Behavior analysis video key frame extraction method and device, medium and computer equipment

Abstract

The invention discloses a behavior analysis video key frame extraction method and device, a medium and computer equipment, and the method comprises the steps: reading a behavior analysis video picture frame by frame, and generating a full-white reference frame with the same size as a first frame; for a non-first frame, the difference between the non-first frame and the reference frame is quantized by using structural similarity indexes (from brightness, contrast and structural three-dimensional measurement), and a bucket index is determined; and judging whether a bucket corresponding to the bucket index has a candidate key frame, if not, storing a non-first-frame picture, and if yes, discarding the non-first-frame picture. And finally, sorting the candidate key frames in all buckets in an ascending order according to bucket indexes of the buckets to obtain a final key frame sequence, distinguishing'different pictures' by simulating human eyes, automatically skipping similar frames from continuous video frames, and screening out frames capable of representing picture core changes to compress data volume and retain key information. Key frame extraction can be carried out by considering behavior semantics and time sequence dynamic characteristics, so that the performance of a behavior analysis task is improved.

Description

Technical Field

[0001] This application relates to the field of video processing technology, and in particular to a method, apparatus, medium, and computer equipment for extracting keyframes from behavioral analysis videos. Background Technology

[0002] With the widespread adoption of applications such as intelligent surveillance, human-computer interaction, and motion analysis, the amount of video data is growing exponentially. Behavior analysis, as a core task in intelligent video understanding, is particularly sensitive to the temporal information and key action changes in video. However, existing keyframe extraction methods often rely on inter-frame pixel differences or low-level features, easily ignoring the semantic information of actions and behaviors, resulting in extracted keyframes that cannot accurately reflect the behavioral process. Furthermore, some methods have high computational complexity, making it difficult to meet real-time requirements. Summary of the Invention

[0003] In view of this, this application provides a method, apparatus, medium, and computer device for extracting key frames from videos for behavior analysis. By simulating the human eye to distinguish "different scenes", it automatically skips similar frames from continuous video frames and selects frames that can represent the core changes of the scene, so as to compress the amount of data and retain key information. It can take into account both behavioral semantics and temporal dynamic features for key frame extraction, thereby improving the performance of behavior analysis tasks.

[0004] According to one aspect of this application, a method for extracting keyframes from behavior analysis videos is provided, the method comprising: Read the video frames of the behavior analysis video frame by frame and generate a full white reference frame video frame with the exact same size as the first video frame read; For any non-first frame video frame read, the structural similarity index is used to quantify the difference between the non-first frame video frame and the all-white reference frame video frame to obtain the quantified difference value. Based on the quantified difference value, the bucket index of the non-first frame video frame is determined. The structural similarity index measures the similarity between the non-first frame video frame and the all-white reference frame video frame as the difference value through three dimensions: brightness, contrast and structure. Determine whether there is a candidate keyframe in the bucket corresponding to the bucket index. If there is no candidate keyframe in the bucket corresponding to the bucket index, the non-first frame video frame is assigned as a candidate keyframe to the bucket. If there is a candidate keyframe in the bucket corresponding to the bucket index, the non-first frame video frame is discarded. The bucket includes multiple buckets, and each bucket corresponds to a bucket index determined based on a preset difference value range. For all candidate keyframes, sort them in ascending order according to the bucket index of the bucket corresponding to each candidate keyframe to obtain the final keyframe sequence.

[0005] According to another aspect of this application, a behavior analysis video keyframe extraction apparatus is provided, the apparatus comprising: The all-white baseline frame generation module is used to read the video frames of the behavior analysis video frame by frame and generate an all-white baseline frame video frame that is exactly the same size as the first video frame read. The difference quantization and bucket index construction module is used to quantify the difference between any non-first frame video frame and the all-white reference frame video frame using a structural similarity index for any non-first frame video frame read, obtain the quantified difference value, and determine the bucket index of the non-first frame video frame based on the quantified difference value. The structural similarity index measures the similarity between the non-first frame video frame and the all-white reference frame video frame as the difference value through three dimensions: brightness, contrast and structure. The candidate keyframe allocation module is used to determine whether there is a candidate keyframe in the bucket corresponding to the bucket index. If there is no candidate keyframe in the bucket corresponding to the bucket index, the non-first frame video frame is allocated as a candidate keyframe to the bucket. If there is a candidate keyframe in the bucket corresponding to the bucket index, the non-first frame video frame is discarded. The bucket includes multiple buckets, and each bucket corresponds to a bucket index determined based on a preset difference value range. The keyframe sequence generation module is used to sort all candidate keyframes in ascending order according to the bucket index of the bucket corresponding to each candidate keyframe, so as to obtain the final keyframe sequence.

[0006] According to another aspect of this application, a medium is provided having a computer program stored thereon, which, when executed by a processor, implements the above-described behavior analysis video keyframe extraction method.

[0007] According to another aspect of this application, a computer device is provided, including a medium, a processor, and a computer program stored on the medium and executable on the processor, wherein the processor executes the program to implement the above-described behavior analysis video keyframe extraction method.

[0008] Using the above technical solution, this application provides a method, apparatus, medium, and computer device for extracting keyframes from behavior analysis videos. The method reads the behavior analysis video frame by frame, generating a full-white reference frame of the same size as the first frame. For non-first frames, the difference between them and the reference frame is quantified using structural similarity indicators (measured from three dimensions: brightness, contrast, and structure), and a bucket index is determined. It is then determined whether there are candidate keyframes within the bucket corresponding to the bucket index; if not, the non-first frame is stored; otherwise, it is discarded. Finally, all candidate keyframes within all buckets are sorted in ascending order by bucket index to obtain the final keyframe sequence. By simulating the human eye's ability to distinguish "different scenes," similar frames are automatically skipped from continuous video frames, and frames representing the core changes in the scene are selected. This compresses data volume while retaining key information, and can extract keyframes while considering both behavioral semantics and temporal dynamic features, thereby improving the performance of behavior analysis tasks.

[0009] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0010] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 A flowchart illustrating a method for extracting keyframes from a behavior analysis video according to an embodiment of this application is shown. Figure 2 A flowchart illustrating another method for extracting keyframes from video for behavior analysis provided in an embodiment of this application is shown. Figure 3 This illustration shows a structural schematic diagram of a behavior analysis video keyframe extraction device provided in an embodiment of this application; Figure 4 A schematic diagram of another behavior analysis video keyframe extraction device provided in an embodiment of this application is shown. Detailed Implementation

[0011] The present application will be described in detail below with reference to the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in the embodiments of the present application can be combined with each other.

[0012] This embodiment provides a method for extracting keyframes from behavior analysis videos, such as... Figure 1 As shown, the method includes: Step 101: Read the video frames of the behavior analysis video frame by frame and generate a full white reference frame video frame with the same size as the first video frame read.

[0013] Currently, assuming data collection and storage are complete, the first step in data processing is to correctly read the data into the analysis environment. In behavioral analysis scenarios such as retail and security, data often exists in the form of surveillance video. Taking a single 1080P camera as an example, at 8Mbps, the amount of data generated per day is around 84GB. Furthermore, due to the continuity of video frames, adjacent frames often have extremely high similarity, meaning there is a large amount of redundant information. In this context, efficient video data processing becomes a prerequisite for accurate analysis, and keyframe extraction, as the foundation of video content analysis, directly affects the efficiency and accuracy of subsequent analysis.

[0014] Currently, the mainstream keyframe extraction methods can be broadly categorized into two types: One method is uniform frame sampling. This method extracts frames from the video at fixed time intervals, such as one frame every 25 frames. It is achieved through simple periodic sampling and does not require complex calculations. However, this method relies entirely on a fixed interval and cannot be dynamically adjusted according to the actual changes in the scene content. If a scene is static for 100 frames, multiple similar frames will be repeatedly extracted, resulting in data redundancy. When the scene changes rapidly, the fixed interval may miss key transition frames, leading to the loss of key information and failing to meet the needs of behavior analysis for capturing key poses.

[0015] The second method is the adjacent frame comparison method. This method uses adjacent frames as a reference and calculates the inter-frame similarity (such as SSIM, hash distance) to determine the changes in the scene. When the similarity is below a threshold, the current frame is retained as the keyframe. However, this mode, which uses the dynamically changing previous frame as a reference, is prone to error propagation. When there are slowly changing scenes in the video, the similarity between adjacent frames is high, but the cumulative change is large, which may lead to the missed detection of keyframes. In addition, the reference frame changes dynamically with the video content, and the similarity threshold cannot be uniform in different scenes, resulting in poor cross-scene applicability.

[0016] In addition, when processing long videos, existing bucketing or clustering methods experience a linear increase in memory used to store reference frames or features as the number of frames increases, which can easily lead to system memory overflow and cannot meet the needs of long-term video processing scenarios such as 24-hour uninterrupted monitoring.

[0017] The embodiments described above in this application are applicable to behavioral analysis scenarios such as retail customer behavior analysis, intelligent security abnormal behavior monitoring, and sports motion technology analysis. They can automatically filter key frames with core behavioral information from continuous video frames, achieving video data compression and key information retention, providing high-quality input data for subsequent tasks such as behavior recognition and motion evaluation. Specifically, the embodiments described above in this application are based on the core logic of "static reference frame + bucket aggregation + sliding window," which effectively avoids problems such as high redundancy, error propagation, and memory overflow in existing methods.

[0018] More specifically, the video data (behavioral analysis video) is first read frame by frame. When reading the first frame, a completely white reference frame with the same size as the first frame is automatically generated and used as the global reference. The features of this completely white frame are completely fixed and do not depend on the dynamic changes of the video content, providing an absolutely static reference system for subsequent difference calculations.

[0019] Furthermore, the video stream to be processed (behavioral analysis video) can be read via RTSP. When initializing the video capture component, the RTSP URL, GStreamer decoder parameters, and maximum number of retries can be configured, and a network video stream connection with an automatic reconnection mechanism can be established. Next, the original resolution and original frame rate of the video (behavioral analysis video) can be obtained through cv::CAP_PROP_FRAME_WIDTH, cv::CAP_PROP_FRAME_HEIGHT, and cv::CAP_PROP_FPS. The time dimension information of the video is recorded by mapping the frame index and timestamp to ensure the correspondence between subsequent keyframes and the time dimension of the original video.

[0020] In particular, the resize_ratio parameter can be configured according to business needs and device computing power, with a range of (0-1.0). cv::resize can be used to scale video frames proportionally, thereby unifying video frames of different resolutions to the same size and avoiding feature calculation deviations caused by different resolutions.

[0021] Step 102: For any non-first frame video image read, the structural similarity index is used to quantify the difference between the non-first frame video image and the all-white reference frame video image to obtain the quantified difference value. Based on the quantified difference value, the bucket index of the non-first frame video image is determined. The structural similarity index measures the similarity between the non-first frame video image and the all-white reference frame video image as the difference value through three dimensions: brightness, contrast and structure.

[0022] Next, the Structural Similarity Index (SSIM) is used to quantify the differences between non-first frames and the all-white baseline frame. This comprehensively measures the visual differences between images from three dimensions: brightness, contrast, and structure, effectively avoiding misjudgments caused by single indicators (such as pixel differences) due to lighting changes and noise interference. This method simulates the characteristics of human visual perception to more accurately capture substantial changes in the image, improving the accuracy of keyframe selection. Simultaneously, mapping the difference values ​​to bucket indices and managing candidate keyframes by region automatically filters redundant similar frames, retaining images within representative difference ranges, simplifying subsequent analysis complexity, ensuring keyframes cover different dynamic behavior stages, and ultimately providing high-information-density and low-redundancy visual data support for behavior analysis. Therefore, quantifying the differences between video frames and the all-white baseline frame using the SSIM index enables an objective assessment of frame content complexity, providing a quantitative basis for subsequent bucketing and aggregation.

[0023] Optionally, in step 102, a structural similarity index is used to quantify the difference between the non-first frame video image and the all-white reference frame video image, obtaining the quantified difference value, including: Step 1021: Based on the structural similarity index calculation formula, quantify the difference between the non-first frame video image and the all-white reference frame video image to obtain the quantified structural similarity index value, wherein the structural similarity index calculation formula is: , , , , This indicates the structural similarity index between a non-first frame video image x and a completely white baseline video image y. The quantified structural similarity index value, This is a brightness comparison function. y represents the average pixel value of the non-first frame video image (x) and the all-white baseline frame video image (y), respectively. , L represents the pixel dynamic range; c represents the contrast comparison function. , Let x represent the pixel standard deviation of the non-first frame video image and y represent the full white baseline frame video image, respectively. , ; s is the structure comparison function, Let x be the pixel covariance between the non-first frame video image and the all-white baseline frame video image y. , , The value range is [0,1]; Step 1022: The structural similarity index values ​​are positively vectorized to obtain the difference values. The formula for positive vectorization is as follows:

[0024] In the above embodiments of this application, the total number of frames in the preset window can be, for example, 10,000. The Structural Similarity Index (SSIM) is used to quantify the difference between each video frame and the all-white reference frame. During calculation, the local window size can be set to 3 to improve the accuracy of local feature comparison. This size can capture local behavioral features within the frame, such as human hand movements and object edges, while avoiding blurring of local details caused by an excessively large window, such as subtle gestures of customers in a retail scene. The calculated SSIM value (range 0-1, the closer the value is to 1, the more similar it is) is converted into a positively vectorized difference value. The larger the difference value, the more significant the difference between the current frame and the all-white reference frame, and the more complex the content of the picture. The calculation formula is: Difference value = (1-SSIM) × 10000.

[0025] Therefore, this method of calculating the difference value with a full white reference frame can effectively suppress error propagation, accurately capture key changes in the image, reduce the loss of key poses, and ensure that the characteristics of the full white reference frame are fixed, making the difference value comparable in different scenarios without the need to adjust parameters for different scenarios, thus improving the versatility of the method.

[0026] Specifically, the final value range of the structural similarity index is [0,1]. The closer the structural similarity index value is to 1, the smaller the difference in structure, brightness, and contrast between the video frame and the all-white reference frame; the closer the structural similarity index value is to 0, the greater the difference.

[0027] The positive vectorization calculation formula maps the SSIM range of [0,1] to [0,10000], which amplifies and visualizes the degree of difference. That is, when two images (video frames) are exactly the same, the structural similarity index value is 1, and the corresponding difference value is 10000; when the image difference is the greatest, the structural similarity index value approaches 0, and the corresponding difference value approaches 0. The smaller the value, the greater the difference between the two images.

[0028] Optionally, in step 102, determining the bucket index of the non-first frame video frame based on the quantized difference value includes: Step 1023: Calculate the ratio of the quantized difference value to the bucket width, and round down the ratio to obtain the bucket index of the non-first frame video frame. The bucket with bucket index n represents a preset difference value range of [n×bucket width, (n+1)×bucket width].

[0029] In the above embodiments of this application, the bucket width can be, for example, 10. Then, the result of "difference value / / 10" can be used as the bucket index to assign the current frame to the corresponding bucket range. This ensures that each bucket only retains the first frame that enters the frame; that is, similar images within the same bucket only store the first frame. This aggregates similar frames and removes redundancy. This mechanism can aggregate similar frames without complex feature matching calculations, significantly reducing data redundancy.

[0030] Specifically, a bucket is a concept for grouping data; it's a series of intervals or categories divided based on the bucket index calculated earlier. Each bucket represents a specific preset range of differences, determined by the preset bucket width. For example, assuming the bucket width is 10, then: A bucket with a bucket index of 0 represents a difference value in the range of 0 (inclusive) to 10 (exclusive); A bucket with a bucket index of 1 represents a difference value in the range of 10 (inclusive) to 20 (exclusive); Similarly, a bucket with index n represents a difference value in the range of n × bucket width (inclusive) to (n+1) × bucket width (exclusive).

[0031] For each video frame, after calculating its bucket index, the corresponding bucket is found based on the bucket index. For example, if a frame has a bucket index of 2 and a bucket width of 10, then this frame is assigned to a bucket representing a difference value in the range of 20 (inclusive) to 30 (exclusive).

[0032] To address this, the quantized difference values ​​are divided by 10 and grouped using bucket indices. For example, difference value 123 corresponds to bucket index 12, and difference value 456 corresponds to bucket index 45. This divides the difference values ​​into different buckets, ensuring that the difference values ​​of frames within the same bucket are ≤10 and have high similarity. At the same time, the increased memory usage caused by an excessive number of buckets can be avoided by periodically clearing the hash table.

[0033] Next, it is necessary to ensure that each bucket only retains the first frame that enters. A flag can be set for each bucket to record whether a frame has already been stored in that bucket.

[0034] When a frame is assigned to a bucket, the system first checks if the bucket already contains a frame. If the bucket is empty (i.e., no frame has been stored yet), the current frame is stored in the bucket, and the bucket is marked as stored. If the bucket already contains a frame (i.e., a similar frame has already been stored), the current frame is discarded, and no storage operation is performed.

[0035] Therefore, behavioral analysis videos may contain a large number of similar video frames. This approach avoids storing a large number of similar frames, saving storage space and improving subsequent processing efficiency. Structural similarity metrics measure the degree of similarity between two frames. By assigning differences to different buckets, frames with similar differences are grouped into the same bucket. Each bucket only retains the first frame it enters, ensuring that similar frames within the same bucket are only stored as their first frame, thus aggregating similar frames and removing redundancy.

[0036] For example, in a surveillance video where the background is relatively fixed and the behavior of the people changes little, the differences between many frames are small. This method can aggregate these similar frames into a single bucket, retaining only one frame, significantly reducing the amount of data that needs to be stored and processed.

[0037] Step 103: Determine whether there is a candidate keyframe in the bucket corresponding to the bucket index. If there is no candidate keyframe in the bucket corresponding to the bucket index, the non-first frame video frame is assigned as a candidate keyframe to the bucket. If there is a candidate keyframe in the bucket corresponding to the bucket index, the non-first frame video frame is discarded. The buckets include multiple buckets, and each bucket corresponds to a bucket index determined based on a preset difference value range.

[0038] Next, by setting up multiple buckets with corresponding bucket indices, the quantified difference values ​​can be reasonably categorized, automatically classifying images with varying degrees of similarity into the appropriate intervals and preventing keyframes from becoming overly concentrated in a single difference range. When there are no candidate keyframes in the target bucket, new images are promptly included to ensure that each difference interval has representative frames; if candidate keyframes already exist, redundant images are discarded, effectively controlling the number of keyframes. This mechanism comprehensively covers different stages of dynamic video changes while preventing similar images from being selected repeatedly, improving the diversity and representativeness of keyframes and providing a more concise and efficient data foundation for subsequent behavioral analysis.

[0039] Optionally, in step 103, it is determined whether there are candidate keyframes in the bucket corresponding to the bucket index. If there are no candidate keyframes in the bucket corresponding to the bucket index, the non-first frame video frame is assigned as a candidate keyframe to the bucket. If there are candidate keyframes in the bucket corresponding to the bucket index, the non-first frame video frame is discarded. This includes: Step 1031: Use the bucket index as the key and the candidate keyframe in the bucket corresponding to the bucket index as the value to construct a key-value pair and store it in a hash table.

[0040] Step 1032: Determine whether there is a candidate keyframe in the bucket corresponding to the bucket index based on the hash table.

[0041] Step 1033: If the value in the key-value pair corresponding to the bucket index in the hash table is empty, it is determined that there is no candidate keyframe in the bucket, and the current non-first frame video frame is stored as a candidate keyframe in the bucket. If the value in the key-value pair corresponding to the bucket index in the hash table is not empty, it is determined that there is a candidate keyframe in the bucket, and the non-first frame video frame is discarded.

[0042] In the above embodiments of this application, a hash table can be used to store key-value pairs, where the key is a bucket index and the value is the frame data corresponding to that bucket (a non-first frame video image that serves as a candidate keyframe). The processing flow is as follows: first, check whether the value corresponding to the bucket index in the hash table is empty. If it is empty, meaning there is no frame in the bucket, store the current frame into the bucket and save it as a candidate keyframe; if the bucket is not empty, meaning there is already a similar frame, skip the current frame directly and do not store it.

[0043] Specifically, an empty hash table (like a dictionary) can be created to store the mapping between bucket indices and candidate keyframes. The key of the hash table is the bucket index (an integer), and the value is the candidate keyframe corresponding to that bucket index. For each non-first frame, calculate its structural similarity difference with the all-white baseline frame and determine the corresponding bucket index (e.g., by dividing the difference by the bucket width and rounding). Use the bucket index of the current frame as the key of the hash table, and add frames only when the bucket is empty, until a key-value pair (bucket index - candidate keyframe) is inserted into the hash table.

[0044] Access the hash table using the bucket index as the key to retrieve the corresponding value.

[0045] If the value is empty (or the key does not exist): it means that there are no candidate keyframes in the bucket corresponding to the bucket index, and the current image needs to be stored in the bucket.

[0046] If the value table is not empty: it means that there are already candidate keyframes in the bucket, and the current frame is discarded because the similarity is too high.

[0047] Therefore, hash tables directly map keys to values, achieving a query time complexity of O(1), which is superior to linear search (such as lists), making them particularly suitable for large-scale video frame processing. By maintaining candidate keyframes for each bucket index in real time using hash tables, it ensures that at most one frame (or a small number of frames) is retained for each difference interval, avoiding redundant storage. Furthermore, the hash table structure supports dynamic addition and deletion of key-value pairs, flexibly adapting to adjustments in bucket width or interval division.

[0048] By mapping bucket indexes to candidate keyframes using hash tables, rapid classification and storage of video frame difference ranges are achieved. During a query, only the existence of the key or whether the value is null needs to be checked to determine whether to retain the current frame, thus optimizing storage and computation efficiency while ensuring keyframe diversity.

[0049] Optionally, the method further includes: Step 105: Determine whether the total number of video frames in the behavior analysis video meets the preset total number of frames for the window.

[0050] Step 106: If the total number of video frames in the behavior analysis video is greater than or equal to the total number of frames in the preset window, then whenever the number of non-first frame video frames allocated or discarded meets the total number of frames in the preset window, save the candidate keyframes in each bucket and then automatically clear all buckets.

[0051] Step 107: If the total number of video frames in the behavior analysis video is less than the preset total number of frames in the window, then after all non-first frame video frames have been processed, all current buckets will be automatically cleared.

[0052] In the above embodiments of this application, to prevent excessive memory consumption due to too many buckets, a sliding window mechanism can be introduced to periodically empty the buckets. A fixed window size of 10,000 frames is set (i.e., ... After processing 10,000 frames, all current buckets are automatically cleared and the index counter is reset, then the process moves to the next window to continue. This mechanism solves the problem of memory usage growing linearly with video length, ensuring that even when processing long videos, such as 24-hour surveillance videos, memory usage remains manageable.

[0053] Specifically, the window size is set to 10,000 frames (period=10,000). Based on common video frame rates: 400 seconds at 25fps and 333 seconds at 30fps, this avoids keyframe loss due to an excessively small window and excessive memory usage due to an excessively large window. When the number of video frames processed (count) reaches the window size, the candidate keyframes stored in the hash table are named by their frame numbers and saved as image files. Then, a window clearing operation is triggered, clearing all bucket data in the hash table and resetting the count. If the total number of video frames is less than one window size, a clearing operation is triggered after video processing is completed. When processing the next window, there is no need to regenerate a full-white baseline frame; the difference value is directly calculated based on the original full-white baseline frame.

[0054] Optionally, in step 101, after generating a full-white reference frame video image with the exact same size as the read first frame video image, the method further includes: Step 108: Store the all-white reference frame video image in a temporary buffer and back it up synchronously to the local disk. When the structural similarity index is used to quantify the difference between the non-first frame video image and the all-white reference frame video image, the all-white reference frame video image is obtained from the temporary buffer.

[0055] Step 109: If the full white reference frame video in the temporary buffer is corrupted when the structural similarity index is used to quantify the difference between the non-first frame video and the full white reference frame video, then the full white reference frame video is reloaded from the local disk.

[0056] In the above embodiments of this application, when the first frame of the video is read, a full white reference frame is automatically generated. The size of the full white reference frame is completely consistent with the first valid frame, and its characteristics are completely fixed and do not change with subsequent video content. The full white reference frame is stored in a temporary buffer area and backed up to local storage for reference recovery when resuming processing. If the reference frame is damaged in subsequent video processing, it can be reloaded from the local backup to ensure reference consistency.

[0057] Step 104: Sort all candidate keyframes in ascending order according to the bucket index of the bucket corresponding to each candidate keyframe to obtain the final keyframe sequence.

[0058] Next, a set of keyframes sorted in ascending order by bucket number is extracted from all windows. Smaller indexes and smaller differences indicate simpler visuals, while larger indexes and larger differences indicate more complex visuals. This forms a keyframe sequence hierarchically based on complexity, which facilitates subsequent layered processing.

[0059] Specifically, the final keyframe set formed after all windows have been processed can be passed through the `vp_frame_meta` structure via the `out_queue` node at the data flow level. This includes core information such as the frame index (`frame_index`) to ensure transmission consistency. Currently, it is sorted in ascending order by bucket number by default. This rule prioritizes the output of high-complexity keyframes, making it suitable for intelligent security scenarios and providing direct input for abnormal behavior recognition models. Furthermore, the sorting logic can be flexibly modified based on actual needs to adapt to different behavior analysis scenarios. For example, in sports motion analysis, to preserve the complete temporal sequence of actions, the naturally increasing `frame_index` in `vp_frame_meta` can be used to output the frames in chronological order.

[0060] In one specific embodiment, such as Figure 2As shown, the system environment is prepared, including loading the video file (behavioral analysis video), initializing parameters (such as bucket width, difference threshold, etc.), and creating data structures (such as hash tables). Next, the video file is read, an initial reference frame is set (i.e., a completely white reference frame video image with the same dimensions as the first read video frame), and a hash table is initialized to store candidate keyframes. The completely white reference frame video image is used for subsequent difference comparisons with other frames. The first completely white frame of the video is selected as the reference frame to ensure consistency in subsequent difference calculations. The video is processed frame by frame to extract candidate keyframes. Starting from the second frame, each frame is read sequentially. The difference between the current frame and the reference frame is quantified to determine whether it is a potential keyframe (candidate keyframe). The structural similarity (SSIM) metric is used to calculate the difference value between the current frame and the reference frame. SSIM measures similarity from three dimensions: brightness, contrast, and structure; a smaller difference value indicates higher similarity. The bucket index to which the current frame belongs is determined based on the difference value (difference value divided by bucket width and rounded down). A hash table is used to efficiently manage candidate keyframes in different difference ranges, avoiding redundant storage. Construct a hash table using bucket indices as keys and candidate keyframes as values. Check if the value corresponding to the current bucket index in the hash table is empty. If empty, store the current frame in that bucket as a candidate keyframe. If not empty, it means there is already a representative frame in that difference interval, so skip the current frame (to avoid repeatedly storing similar frames). Maintain the dynamic nature of the hash table to ensure that keyframes cover different stages of the video. Periodically or based on conditions (such as the number of frames within a bucket reaching a certain threshold). 10000) Empty the buckets to prevent excessive storage consumption in a certain difference range. Update the system state (such as base frame switching) to adapt to changes in video content. Ensure all frames are processed. Check if the last frame of the video has been traversed. If not, continue processing the next frame; if completed, output all candidate keyframes as a keyframe sequence, which is the simplified video frame set required for behavior analysis. Specifically, output the candidate keyframes corresponding to each bucket index in the hash table to form a keyframe sequence.

[0061] Therefore, the key-value mapping of the hash table reduces the time complexity of query and storage operations to O(1), significantly improving processing efficiency. By using difference value bucketing and hash table management, keyframes are ensured to cover different dynamic stages of the video, avoiding redundancy. The bucket width and difference threshold are adjustable to adapt to the needs of different behavior analysis scenarios.

[0062] This method quantifies inter-frame differences using SSIM and dynamically manages candidate keyframes using a hash table, achieving efficient keyframe extraction from behavior analysis videos. Its core lies in utilizing difference value binning and efficient hash table lookups to ensure keyframes are both representative and avoid redundancy, providing a high-quality data foundation for subsequent behavior analysis.

[0063] Existing keyframe extraction methods suffer from high redundancy, easy loss of key information, error propagation, poor cross-scene adaptability, and uncontrollable memory consumption, making it difficult to meet the actual needs of behavioral analysis scenarios such as retail and security. To address these pain points, this paper proposes a video keyframe extraction algorithm based on a static benchmark and a bucketing strategy. Its core lies in simulating the human eye's ability to distinguish "different scenes," automatically skipping similar frames from consecutive video frames, and selecting frames that represent the core changes in the scene, thereby compressing data volume while retaining key information.

[0064] By applying the technical solution of this embodiment, the video footage for behavior analysis is read frame by frame to generate a full-white baseline frame of the same size as the first frame. For non-first frames, the difference between them and the baseline frame is quantified using structural similarity indicators (measured from three dimensions: brightness, contrast, and structure) to determine the bucket index. It is determined whether there are candidate keyframes in the bucket corresponding to the bucket index. If not, the non-first frame is stored; otherwise, it is discarded. Finally, all candidate keyframes in all buckets are sorted in ascending order by bucket index to obtain the final keyframe sequence. By simulating the human eye's ability to distinguish "different scenes," similar frames are automatically skipped from continuous video frames, and frames that can represent the core changes in the scene are selected. This compresses the data volume while retaining key information, and can take into account both behavioral semantics and temporal dynamic features for keyframe extraction, thereby improving the performance of the behavior analysis task.

[0065] Furthermore, as Figure 1 To specifically implement the method, this application provides a device for extracting keyframes from behavioral analysis videos, such as... Figure 3 As shown, the device includes: The all-white baseline frame generation module 201 is used to read the video frames of the behavior analysis video frame by frame and generate an all-white baseline frame video frame that is exactly the same size as the first video frame read. The difference quantization and bucket index construction module 202 is used to quantify the difference between any non-first frame video frame and the all-white reference frame video frame using a structural similarity index for any non-first frame video frame read, to obtain the quantified difference value, and to determine the bucket index of the non-first frame video frame based on the quantified difference value. The structural similarity index measures the similarity between the non-first frame video frame and the all-white reference frame video frame as the difference value through three dimensions: brightness, contrast and structure. The candidate keyframe allocation module 203 is used to determine whether there is a candidate keyframe in the bucket corresponding to the bucket index. If there is no candidate keyframe in the bucket corresponding to the bucket index, the non-first frame video frame is allocated as a candidate keyframe to the bucket. If there is a candidate keyframe in the bucket corresponding to the bucket index, the non-first frame video frame is discarded. The bucket includes multiple buckets, and each bucket corresponds to a bucket index determined based on a preset difference value range. The keyframe sequence generation module 204 is used to sort all candidate keyframes in ascending order according to the bucket index of the bucket corresponding to each candidate keyframe to obtain the final keyframe sequence.

[0066] Optionally, the difference quantization and bucket index construction module 202 is further configured to: Based on the structural similarity index calculation formula, the difference between the non-first frame video image and the all-white reference frame video image is quantified to obtain the quantified structural similarity index value. The structural similarity index calculation formula is as follows: , , , , This indicates the structural similarity index between a non-first frame video image x and a completely white baseline video image y. The quantified structural similarity index value, This is a brightness comparison function. y represents the average pixel value of the non-first frame video image (x) and the all-white baseline frame video image (y), respectively. , L represents the pixel dynamic range; c represents the contrast comparison function. , Let x represent the pixel standard deviation of the non-first frame video image and y represent the full white baseline frame video image, respectively. , ; s is the structure comparison function, Let x be the pixel covariance between the non-first frame video image and the all-white baseline frame video image y. , , The value range is [0,1]; The structural similarity index values ​​are positively vectorized to obtain the difference values. The formula for positive vectorization is as follows:

[0067] Optionally, the difference quantization and bucket index construction module 202 is further configured to: The ratio of the quantized difference value to the bucket width is calculated, and the ratio is rounded down to obtain the bucket index of the non-first frame video frame. The bucket with bucket index n represents a preset difference value range of [n×bucket width, (n+1)×bucket width].

[0068] Optionally, the candidate keyframe allocation module 203 is further configured to: Use the bucket index as the key and the candidate keyframes in the bucket corresponding to the bucket index as the value to build key-value pairs and store them in a hash table; The presence of candidate keyframes in the bucket corresponding to the bucket index is determined based on the hash table.

[0069] Optionally, the candidate keyframe allocation module 203 is further configured to: If the value in the key-value pair corresponding to the bucket index in the hash table is empty, it is determined that there is no candidate keyframe in the bucket, and the current non-first frame video frame is stored as a candidate keyframe in the bucket. If the value in the key-value pair corresponding to the bucket index in the hash table is not empty, it is determined that there is a candidate keyframe in the bucket, and the non-first frame video frame is discarded.

[0070] Furthermore, embodiments of this application provide another device for extracting keyframes from behavior analysis videos, such as... Figure 4 As shown, the device includes: The all-white baseline frame generation module 201 is used to read the video frames of the behavior analysis video frame by frame and generate an all-white baseline frame video frame that is exactly the same size as the first video frame read. The difference quantization and bucket index construction module 202 is used to quantify the difference between any non-first frame video frame and the all-white reference frame video frame using a structural similarity index for any non-first frame video frame read, to obtain the quantified difference value, and to determine the bucket index of the non-first frame video frame based on the quantified difference value. The structural similarity index measures the similarity between the non-first frame video frame and the all-white reference frame video frame as the difference value through three dimensions: brightness, contrast and structure. The candidate keyframe allocation module 203 is used to determine whether there is a candidate keyframe in the bucket corresponding to the bucket index. If there is no candidate keyframe in the bucket corresponding to the bucket index, the non-first frame video frame is allocated as a candidate keyframe to the bucket. If there is a candidate keyframe in the bucket corresponding to the bucket index, the non-first frame video frame is discarded. The bucket includes multiple buckets, and each bucket corresponds to a bucket index determined based on a preset difference value range. The keyframe sequence generation module 204 is used to sort all candidate keyframes in ascending order according to the bucket index of the bucket corresponding to each candidate keyframe to obtain the final keyframe sequence. The frame clearing and restoration module 205 is used to determine whether the total number of video frames in the behavior analysis video meets the preset window total number of frames. If the total number of video frames in the behavior analysis video is greater than or equal to the preset window total number of frames, then whenever the number of non-first frame video frames allocated or discarded meets the preset window total number of frames, the candidate keyframes in each bucket are saved and all buckets are automatically cleared. If the total number of video frames in the behavior analysis video is less than the preset window total number of frames, then all buckets are automatically cleared after all non-first frame video frames have been processed.

[0071] Optionally, the difference quantization and bucket index construction module 202 is further configured to: Based on the structural similarity index calculation formula, the difference between the non-first frame video image and the all-white reference frame video image is quantified to obtain the quantified structural similarity index value. The structural similarity index calculation formula is as follows: , , , , This indicates the structural similarity index between a non-first frame video image x and a completely white baseline video image y. The quantified structural similarity index value, This is a brightness comparison function. y represents the average pixel value of the non-first frame video image (x) and the all-white baseline frame video image (y), respectively. , L represents the pixel dynamic range; c represents the contrast comparison function. , Let x represent the pixel standard deviation of the non-first frame video image and y represent the full white baseline frame video image, respectively. , ; s is the structure comparison function, Let x be the pixel covariance between the non-first frame video image and the all-white baseline frame video image y. , , The value range is [0,1]; The structural similarity index values ​​are positively vectorized to obtain the difference values. The formula for positive vectorization is as follows:

[0072] Optionally, the difference quantization and bucket index construction module 202 is further configured to: The ratio of the quantized difference value to the bucket width is calculated, and the ratio is rounded down to obtain the bucket index of the non-first frame video frame. The bucket with bucket index n represents a preset difference value range of [n×bucket width, (n+1)×bucket width].

[0073] Optionally, the candidate keyframe allocation module 203 is further configured to: Use the bucket index as the key and the candidate keyframes in the bucket corresponding to the bucket index as the value to build key-value pairs and store them in a hash table; The presence of candidate keyframes in the bucket corresponding to the bucket index is determined based on the hash table.

[0074] Optionally, the candidate keyframe allocation module 203 is further configured to: If the value in the key-value pair corresponding to the bucket index in the hash table is empty, it is determined that there is no candidate keyframe in the bucket, and the current non-first frame video frame is stored as a candidate keyframe in the bucket. If the value in the key-value pair corresponding to the bucket index in the hash table is not empty, it is determined that there is a candidate keyframe in the bucket, and the non-first frame video frame is discarded.

[0075] Optionally, the frame clearing and restoration module 205 is further configured to: The all-white reference frame video image is stored in a temporary buffer and synchronously backed up to the local disk. When the structural similarity index is used to quantify the difference between the non-first frame video image and the all-white reference frame video image, the all-white reference frame video image is obtained from the temporary buffer. If the full-white reference frame video in the temporary buffer is corrupted when the structural similarity index is used to quantify the difference between the non-first frame video and the full-white reference frame video, then the full-white reference frame video is reloaded from the local disk.

[0076] It should be noted that other corresponding descriptions of the functional units involved in the behavior analysis video keyframe extraction device provided in this application embodiment can be found in the following references. Figures 1 to 2 The corresponding descriptions in the method will not be repeated here.

[0077] Based on the above, Figures 1 to 2 Accordingly, this application also provides a medium on which a computer program is stored, which, when executed by a processor, implements the above-described method. Figures 1 to 2 The method for extracting keyframes from behavioral analysis videos is shown.

[0078] Based on this understanding, the technical solution of this application can be embodied in the form of a software product, which can be stored in a non-volatile medium (such as CD-ROM, USB flash drive, mobile hard drive, etc.) and includes several instructions to cause a computer device (such as personal computer, server, or network device, etc.) to execute the methods described in various implementation scenarios of this application.

[0079] Based on the above, Figures 1 to 2 The method shown, and Figure 3 , Figure 4 To achieve the above objectives, the present application also provides a computer device, specifically a personal computer, server, network device, etc., as shown in the virtual device embodiment. This computer device includes a medium and a processor; the medium stores a computer program; the processor executes the computer program to achieve the above-described objectives. Figures 1 to 2 The method for extracting keyframes from behavioral analysis videos is shown.

[0080] Optionally, the computer device may also include a user interface, a network interface, a camera, radio frequency (RF) circuitry, sensors, audio circuitry, a Wi-Fi module, etc. The user interface may include a display screen, input units such as a keyboard, etc., and optional user interfaces may also include USB ports, card reader ports, etc. The network interface may optionally include standard wired interfaces, wireless interfaces (such as Bluetooth interfaces, Wi-Fi interfaces), etc.

[0081] Those skilled in the art will understand that the computer device structure provided in this embodiment does not constitute a limitation on the computer device, and may include more or fewer components, or combine certain components, or have different component arrangements.

[0082] The medium may also include an operating system and a network communication module. The operating system is a program that manages and stores the hardware and software resources of a computer device, supporting the operation of information processing programs and other software and / or programs. The network communication module is used to enable communication between the various components within the medium, as well as communication with other hardware and software within the physical device.

[0083] Through the above description of the implementation methods, those skilled in the art can clearly understand that this application can be implemented using software plus necessary general-purpose hardware platforms, or it can be implemented using hardware to read the behavior analysis video frame by frame and generate a full-white reference frame of the same size as the first frame. For non-first frames, the difference between them and the reference frame is quantified using structural similarity indicators (measured from three dimensions: brightness, contrast, and structure) to determine the bucket index. It is determined whether there are candidate keyframes in the bucket corresponding to the bucket index. If not, the non-first frame is stored; otherwise, it is discarded. Finally, all candidate keyframes in the buckets are sorted in ascending order according to the bucket index to obtain the final keyframe sequence. By simulating the human eye's ability to distinguish "different scenes," similar frames are automatically skipped from continuous video frames, and frames that can represent the core changes of the scene are selected. This compresses the data volume while retaining key information, and can take into account both behavioral semantics and temporal dynamic features for keyframe extraction, thereby improving the performance of the behavior analysis task.

[0084] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of a preferred embodiment, and the modules or processes shown in the drawings are not necessarily essential for implementing this application. Those skilled in the art will understand that the modules in the apparatus of the embodiment can be distributed within the apparatus of the embodiment as described, or can be modified to be located in one or more apparatuses different from this embodiment. The modules of the above-described embodiment can be combined into one module, or further divided into multiple sub-modules.

[0085] The serial numbers in this application are for descriptive purposes only and do not represent the superiority or inferiority of any particular implementation scenario. The above disclosures are merely a few specific implementation scenarios of this application; however, this application is not limited thereto, and any modifications that can be made by those skilled in the art should fall within the protection scope of this application.

Claims

1. A method for behavior analysis video key frame extraction, characterized in that, The method comprises: reading video pictures of the behavior analysis video frame by frame, and generating a full-white reference frame video picture which is completely consistent with the size of the first read video picture; for any non-first video picture, quantifying the difference between the non-first video picture and the full-white reference frame video picture by using a structural similarity index, obtaining a quantized difference value, and determining the bucket index of the non-first video picture based on the quantized difference value, wherein the structural similarity index measures the similarity between the non-first video picture and the full-white reference frame video picture as the difference value through three dimensions of brightness, contrast and structure; judging whether there is a candidate key frame in the bucket corresponding to the bucket index, if there is no candidate key frame in the bucket corresponding to the bucket index, assigning the non-first video picture as a candidate key frame to the bucket, if there is a candidate key frame in the bucket corresponding to the bucket index, discarding the non-first video picture, wherein the bucket comprises a plurality of buckets, each bucket corresponds to a bucket index determined based on a preset difference value range interval; for all candidate key frames, sorting the candidate key frames in ascending order according to the bucket index of the bucket corresponding to each candidate key frame, and obtaining a final key frame sequence.

2. The method of claim 1, wherein, The method comprises: quantifying the difference between the non-first video picture and the full-white reference frame video picture based on a structural similarity index calculation formula, and obtaining a quantized structural similarity index value, wherein the structural similarity index calculation formula is: ， ， ， ， denotes the structural similarity index between the non-first frame video picture x and the full white reference frame video picture y quantized structural similarity index value, is the luminance comparison function, are the pixel mean values of the non-first frame video picture x and the full white reference frame video picture y respectively, , L is the pixel dynamic range; c is the contrast comparison function, , are the pixel standard deviations of the non-first frame video picture x and the full white reference frame video picture y respectively, , s is the structure comparison function, is the pixel covariance between the non-first frame video picture x and the full white reference frame video picture y, , , the value range is [0, 1]; performing forward quantization on the structural similarity index value to obtain a difference value, wherein the calculation formula of forward quantization is: The difference value = (1-structural similarity index value) × preset window total frame number.

3. The method of claim 1, wherein, The method comprises: calculating the ratio of the quantized difference value to the bucket width, and taking the ratio down to obtain the bucket index of the non-first video picture, wherein the bucket index n represents a preset difference value range interval of [n×bucket width, (n+1)×bucket width).

4. The method of claim 1, wherein, The method comprises: storing the key-value pair in a hash table by taking the bucket index as the key and the candidate key frame in the bucket corresponding to the bucket index as the value; judging whether there is a candidate key frame in the bucket corresponding to the bucket index based on the hash table.

5. The method of claim 4, wherein, The method comprises: if the value in the key-value pair corresponding to the bucket index in the hash table is empty, it is judged that there is no candidate key frame in the bucket, and the current non-first video picture is stored as a candidate key frame in the bucket, if the value in the key-value pair corresponding to the bucket index in the hash table is not empty, it is judged that there is a candidate key frame in the bucket, and the non-first video picture is discarded.

6. The method according to any one of claims 1 to 5, characterized in that, The method further comprises: judging whether the total frame number of the video pictures of the behavior analysis video meets the preset window total frame number; If the total frame number of the video pictures of the behavior analysis video is greater than or equal to the preset window total frame number, then after the candidate key frames in the current buckets are saved, the current all buckets are automatically emptied when the number of the assigned or discarded non-first frame video pictures meets the preset window total frame number. If the total frame number of the video pictures of the behavior analysis video is less than the preset window total frame number, then after the processing of all the non-first frame video pictures is completed, the current all buckets are automatically emptied.

7. The method of claim 6, wherein, After the all-white reference frame video picture completely consistent with the size of the read first frame video picture is generated, the method further comprises: storing the all-white reference frame video picture to a temporary cache area and synchronously backing up to a local disk, wherein when the difference between the non-first frame video picture and the all-white reference frame video picture is quantified by using the structural similarity index, the all-white reference frame video picture is obtained from the temporary cache area; if the all-white reference frame video picture in the temporary cache area is damaged when the difference between the non-first frame video picture and the all-white reference frame video picture is quantified by using the structural similarity index, the all-white reference frame video picture is reloaded from the local disk.

8. A behavior analysis video key frame extraction apparatus characterized by comprising: The device comprises: an all-white reference frame generation module configured to read the video pictures of the behavior analysis video frame by frame and generate an all-white reference frame video picture completely consistent with the size of the read first frame video picture; a difference quantification and bucket index construction module configured to, for any non-first frame video picture read, quantify the difference between the non-first frame video picture and the all-white reference frame video picture by using the structural similarity index to obtain a quantified difference value, and determine the bucket index of the non-first frame video picture based on the quantified difference value, wherein the structural similarity index measures the similarity between the non-first frame video picture and the all-white reference frame video picture as a difference value through three dimensions of brightness, contrast, and structure; a candidate key frame assignment module configured to determine whether there is a candidate key frame in the bucket corresponding to the bucket index, if there is no candidate key frame in the bucket corresponding to the bucket index, assign the non-first frame video picture as a candidate key frame to the bucket, and if there is a candidate key frame in the bucket corresponding to the bucket index, discard the non-first frame video picture, wherein the buckets comprise a plurality of buckets, and each bucket corresponds to a bucket index determined based on a preset difference value range interval; a key frame sequence generation module configured to, for all the candidate key frames, perform ascending order sorting according to the bucket index of the candidate key frame corresponding to each bucket to obtain a final key frame sequence.

9. A medium having stored thereon a computer program, characterized in that The computer program is executed by the processor to implement the method for extracting key frames of the behavior analysis video according to any one of claims 1 to 7.

10. A computer device comprising a medium, a processor and a computer program stored on the medium and executable on the processor, characterized in that, The processor executes the computer program to implement the method for extracting key frames of the behavior analysis video according to any one of claims 1 to 7.

Images