An object perception and space-time memory-based industrial monitoring video key frame extraction method and system

Abstract

The application discloses an object perception and space-time memory-based industrial monitoring video key frame extraction method and system, and relates to video image processing technology, which comprises the following steps: a video stream preprocessing step: obtaining a current frame in an input video stream and at least one prompt information preset for the current frame; an image coding step: performing feature extraction and feature analysis on the current frame based on the prompt information to obtain a semantic risk score and a time sequence dynamic feature in parallel; a feature fusion step: adaptively weighting and fusing the semantic risk score and the time sequence dynamic feature based on historical feature data in a sliding window to obtain a comprehensive importance score of the current frame; and a discrimination output step: discriminating the comprehensive importance score based on a set dynamic threshold to determine whether the current frame is a key frame. The application realizes continuous perception of a specific object in a monitoring scene and accurate capture of state mutation, and provides reliable technical support for industrial safety monitoring intelligentization.

Description

Technical Field

[0001] This invention relates to video image processing technology, and more specifically, to a method and system for extracting keyframes from industrial surveillance videos based on object perception and spatiotemporal memory. Background Technology

[0002] In the field of industrial safety monitoring, automatically extracting key frames representing abnormal events from massive video streams is crucial for achieving efficient traceability and real-time early warning. Current mainstream methods typically suffer from the following drawbacks:

[0003] 1. Pixel- or optical flow-based methods: These methods detect motion by calculating inter-frame pixel differences or optical flow energy. The fundamental problem is that any minute change in lighting, camera shake, or irrelevant motion (such as swaying leaves or fluttering flags) will be interpreted as a "change," leading to an extremely high false alarm rate. They cannot understand the semantics of the moving subject and cannot distinguish the essential difference between "normal movement of inspection personnel" and "unauthorized intrusion by unauthorized personnel."

[0004] 2. Frame extraction methods based on fixed rules or simple thresholds: such as frame extraction at equal time intervals or abrupt change detection based on the average brightness of the scene. These methods completely ignore the semantic coherence and event logic of the video content, generating a large number of redundant frames in static scenes, while potentially missing key decision moments during complex events, failing to guarantee the completeness of event recording.

[0005] 3. Tracking methods based on traditional object detection: Although they can identify specific objects, they usually require predefined categories and are not sensitive to drastic changes in object shape (such as smoke diffusion, leak spread) or occlusion reproduction. They lack the ability to model the continuous, fine-grained evolution of object states and are difficult to capture slow-changing anomalies such as "equipment parts gradually loosening" or "liquid level slowly rising".

[0006] The reason for this is that existing technologies generally treat videos as a continuous sequence of image frames, employing paradigms of "inter-frame comparison" or "single-frame classification," lacking a fundamental model that can unify the understanding of "what objects are in the video" and "how those objects change over time." This leads to a disconnect between semantic understanding and dynamic analysis, making it difficult to accurately capture security-related state changes at the object level. Summary of the Invention

[0007] The technical problem to be solved by the present invention is to address the shortcomings of the existing technology by providing a method and system for extracting key frames from industrial monitoring videos based on object perception and spatiotemporal memory. By fusing motion optical flow and global semantic features, the method can accurately and efficiently screen key frames in the video stream that represent abnormal events or state changes, aiming to provide high-value, low-redundancy video input for subsequent intelligent analysis.

[0008] The present invention discloses a method for extracting keyframes from industrial surveillance videos based on object perception and spatiotemporal memory. This method includes the following steps: Video stream preprocessing steps: Obtain the current frame in the input video stream and at least one preset prompt message for the current frame; Image encoding steps: Based on the prompt information, perform feature extraction and feature analysis on the current frame to obtain semantic risk scores and temporal dynamic features in parallel; Feature fusion step: Based on historical feature data within the sliding window, the semantic risk score and the temporal dynamic features are adaptively weighted and fused to obtain the comprehensive importance score of the current frame; The output discrimination step is to discriminate the comprehensive importance score based on the set dynamic threshold to determine whether the current frame is a key frame.

[0009] Preferably, the image encoding step specifically includes: The current frame is used to extract features using a visual segmentation model to obtain a feature embedding vector; The feature embedding vector is matched with predefined text semantics for risk analysis, and the semantic risk score is output. By combining the historical memory database of the visual segmentation model, the changes in object morphology and memory association of the current frame within the area corresponding to the prompt information are analyzed to generate the temporal dynamic features.

[0010] Preferably, the feature embedding vector is used to perform risk matching analysis with a predefined text semantic vector, specifically as follows: The feature embedding vector is spatially adjusted using a multilayer perceptron to generate a visual vector aligned with the predefined text semantic vector space. Calculate the cosine similarity between the visual vector and the predefined text semantic vector; The semantic risk score vector is generated based on the cosine similarity, and the maximum value in the semantic risk score vector is extracted as the semantic risk score.

[0011] Preferably, the specific method for generating the temporal dynamic features is as follows: The visual segmentation model generates a segmentation mask based on the prompt information of the current frame, and obtains the historical mask corresponding to the previous frame of the current frame from the historical memory bank; The object morphology change feature value between the current frame and the previous frame is calculated using the segmentation mask and the history mask; Calculate the correlation between the feature embedding vector of the current frame and the historical feature sequence stored in the history memory bank; The temporal dynamic features include the object morphological change feature value and the correlation degree.

[0012] Preferably, the specific calculation method for the object morphological change characteristic value is as follows: The displacement deviation between the current frame and the previous frame is calculated using the following formula. : , In the formula, Mt represents the segmentation mask, and Mt-1 represents the history mask. This indicates a bitwise AND operation. This indicates a bitwise OR operation. Indicates the smoothing factor; The rate of change of area between the current frame and the previous frame is calculated using the following formula. : , In the formula, This indicates the degree of volume expansion of the object corresponding to the segmentation mask in the current frame. This indicates the degree of volume expansion of the object corresponding to the history mask in the previous frame of the current frame; The displacement deviation is expressed by the following formula. With area change rate Weighted fusion is performed to generate the morphological change feature values ​​of the object. : , In the formula, and All of these represent sensitivity weighting coefficients.

[0013] Preferably, the feature fusion step specifically includes: Historical feature data corresponding to the semantic risk score and temporal dynamic features are extracted from the sliding window. Based on the historical feature data, the semantic risk score and temporal dynamic features are normalized respectively to obtain normalized indicators corresponding to the semantic risk score and temporal dynamic features. The dynamic weight of the normalized index in the fusion is calculated based on the degree of disorder in the value distribution of each normalized index within the sliding window. The comprehensive importance score is obtained by weighted summation based on the normalized index and its corresponding dynamic weight.

[0014] Preferably, the specific calculation method for the dynamic weight is as follows: The range of values ​​for each normalized index within the sliding window is divided into K equal intervals; Calculate the frequency with which each normalized index falls within each interval; Calculate the information entropy of each normalized index based on the frequency distribution; Calculate the difference coefficients of each normalized index based on the information entropy; The dynamic weight is obtained by fusing all the difference coefficients.

[0015] Preferably, the discrimination output step specifically includes: Obtain a sliding window sequence containing the scores of M frames, including the current frame; The mean and standard deviation of the scores are calculated based on the sliding window sequence. Based on the mean and standard deviation and combined with a preset sensitivity coefficient, a dynamic threshold that changes over time is set. Determine whether the overall importance score of the current frame is greater than the dynamic threshold, and at the same time determine whether the overall importance score of the current frame is greater than the scores of the frames adjacent to the current frame before and after it. If both of the above conditions are met, then the current frame is determined to be a keyframe.

[0016] A system for implementing the aforementioned method for extracting keyframes from industrial surveillance videos based on object awareness and spatiotemporal memory, the system comprising: A video stream preprocessing module is used to perform the video stream preprocessing steps to obtain the current frame in the input video stream and at least one preset prompt message for the current frame; An image encoder, connected to the video stream preprocessing module, is used to perform the image encoding step to extract and analyze features of the current frame based on the prompt information, so as to obtain semantic risk scores and temporal dynamic features in parallel. A feature fusion unit, connected to the image encoder, is used to perform the feature fusion step to adaptively weight and fuse the semantic risk score and the temporal dynamic features based on historical feature data within a sliding window, so as to obtain a comprehensive importance score for the current frame. A discriminator, connected to the feature fusion unit, is used to execute the discrimination output step to discriminate the comprehensive importance score based on a set dynamic threshold, so as to determine whether the current frame is a key frame. If so, the current frame is determined to be a key frame.

[0017] Preferably, in the discriminator, if the current frame is the last frame, the processing flow of the video stream is directly introduced; in the discriminator, if the current frame is determined to be a key frame, the current frame is saved; if the current frame is determined not to be a key frame, the history memory is updated, the current frame is updated to the next frame, and then the key frame extraction method for industrial monitoring video based on object perception and spatiotemporal memory is executed.

[0018] Beneficial effects The advantages of this invention are: by using a video segmentation model with a spatiotemporal memory mechanism, a dual-stream architecture that combines semantic and motion analysis is constructed, enabling continuous perception of specific objects in the monitoring scene and accurate capture of state changes, thereby extracting key frames with high semantic value and low redundancy, providing reliable technical support for intelligent industrial safety monitoring. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the system architecture of the present invention. Detailed Implementation

[0020] The present invention will be further described below with reference to embodiments, but this does not constitute any limitation on the present invention. Any limited modifications made by any person within the scope of the claims of the present invention are still within the scope of the claims of the present invention.

[0021] This invention discloses a keyframe extraction method for industrial monitoring videos based on object perception and spatiotemporal memory. This method can be used in monitoring systems for high-risk industrial environments such as petrochemicals and energy production. The core idea of ​​this invention is to use a visual segmentation model with a memory bank as a unified perception basis, initialize and focus on monitoring points of interest through prompts, and decouple its output deep features from spatiotemporal memory to form parallel semantic perception streams and temporal motion streams. Finally, the dual-stream signals are fused for decision-making. The method specifically includes the following steps: S1, a video stream preprocessing step, which is used to obtain the current frame in the input video stream and at least one preset prompt message for the current frame.

[0022] Taking a video stream F containing 100 frames as an example, each frame is 224×224×3, that is, H=224, W=224, C=3 (red, green and blue color channels).

[0023] For each frame, the system loads the image area from the two preset static prompt boxes (BoxPrompt) in the image captured by the corresponding camera: Box1 (first area): (x1=80, y1=60, x2=160, y2=140); Box2 (second area): (x1=10, y1=150, x2=100, y2=220).

[0024] It should be noted that in actual industrial deployments, the number of static prompt boxes (BoxPrompt) is determined by the actual needs of the monitoring screen, such as the number of reaction vessels and hazardous valves displayed on the screen. In other words, it can be one or more, such as five.

[0025] Therefore, this step ultimately outputs two prompts: {Prompt1:Box1,Prompt2:Box2}. These prompts are equivalent to telling the image encoder, "Please focus on the objects in these two areas of the image," without needing to analyze the background of the entire image.

[0026] S2, the image encoding step, involves extracting and analyzing features of the current frame based on the prompt information to obtain semantic risk scores and temporal dynamic features in parallel. The semantic risk score represents the semantic risk category matching degree of the current frame, and the temporal dynamic features represent the state change of the current frame relative to historical frames within the region corresponding to the prompt information.

[0027] In this step, the feature encoding module, semantic perception module, and temporal motion module will be called in sequence to acquire the target features.

[0028] S2.1 In the feature encoding module, the input is the current frame and its prompt information, such as inputting frame 100 F. 100 +{Prompt1:Box1,Prompt2:Box2}. The SAM2 Hiera algorithm is used to extract features from the current frame, resulting in a feature embedding vector. It should be noted that the video segmentation model's internal history memory already stores feature information from previous frames after processing.

[0029] In this embodiment, Hiera is used to process a 224×224×3 image into a high-dimensional vector. The working principle is as follows: S2.1.1 Initial Tensor Blocking and Channel Dimensionality Reduction Projection: The purpose of this operation is to divide the image into blocks and project them into a sequence of patch vectors through convolution.

[0030] The input current frame is a three-dimensional tensor T. Where H=224, W=224, and C=3 (red, green, and blue color channels). The system uses a 7×7 convolutional kernel with a stride of 7 to perform a sliding window scan on the three channels of the current frame. Within each 7×7×3 spatial region, the convolutional kernel maps 147 pixel values ​​(7×7×3) into a vector of length D (e.g., 128 dimensions) through weighted summation. Finally, the original pixel array of the current frame is transformed into an initial feature map consisting of 32×32 (224 / 7=32) patch vectors. At this point, the number of channels jumps from 3 (color) to 128 (abstract features), with each vector representing a visual summary of a 7×7 region in the original image.

[0031] S2.1.2 Low-level mapping of the window attention mechanism: This operation aims to transform the image from pixel space to feature space. The Hiera encoder gradually constructs a hierarchical feature representation from local to global through multiple stages and window attention mechanisms.

[0032] ①The multi-stage process is as follows: In the Hiera architecture, Stages 1 through 4 are not simply four steps, but rather four sets of continuous, deeply stacked processing procedures. After each window attention layer performs an operation, its output is a local matrix reconstructed from the features. Each Stage contains multiple layers of such window attention computations. As data flows within a Stage, through continuous window attention mapping, the pixel vectors undergo a transformation from "neighborhood awareness" to "structure awareness."

[0033] Stage 1 (Pixel-level Mapping): The window attention output is a spatial resolution of 32×32×256. At this time, the window attention works on extremely small 7×7 raw pixel blocks, and the output is edge and texture primitives.

[0034] Stage 2 (Geometric Mapping): The system downsamples the output of Stage 1 by 2×2, making the tensor 16×16×512. At this point, the actual area of ​​the original image covered by the window increases. The mapping result evolves from "lines" to "local geometry".

[0035] Stage 3 (Component-level Mapping): The tensor is downsampled again, becoming 8×8×1024. By shifting the window, the boundaries of the fixed window are broken. The mapping result can now identify the "functional components".

[0036] Stage 4 (Semantic Mapping - Final Output): Global average pooling is performed, reducing the tensor to 1×1×2560. Here, the concept of a "window" is eliminated; instead, global attention is calculated. This maps the 1024-dimensional tensor to a 2560-dimensional E. 100 .

[0037] ②The calculation of the window attention mechanism is as follows: The system divides the initial 32×32 feature map into multiple non-overlapping 8×8 local windows. Within each window, self-attention computation is performed. First, for each vector... It will use three different learnable weight matrices , , Transformed into: Query vector (Q): Represents "what features I need to find"; Key vector (K): Represents "what features I can provide"; Value vector (V): Represents "the visual content I contain".

[0038] The system then reconstructs features using a scaled dot product attention mechanism: , Here, d represents the dimension of the key vector. This calculation allows feature vectors to be weighted and interacted based on their semantic relevance, enhancing important features and suppressing noise.

[0039] The output maintains a spatial resolution of 32×32×8, but the internal values ​​have undergone fundamental changes. The output result is a set of visual primitives with local topology awareness. Sensor random noise has been eliminated, and the signal strength of the target object has been enhanced.

[0040] In the feature encoding module, the final output is a high-dimensional feature embedding vector E. 100 It is a 2560-dimensional vector that encapsulates the current frame F. 100 The visual information is then processed. The historical memory bank is also updated at this time, preparing to provide temporal context for the next frame.

[0041] In the feature encoding module, because of the introduction of Prompt1 and Prompt2, the feature embedding vector E 100 In this 2560-dimensional vector, the pixel features falling into the two prompt box regions will be assigned extremely high weights, while the features of the background (such as the sky and walls) will be suppressed.

[0042] S2.2 The semantic awareness module is used to perform risk matching analysis between the feature embedding vector and predefined text semantics, and output the semantic risk score. Its specific execution flow is as follows: S2.2.1 Multilayer Perceptron (MLP) Feature Alignment and Projection: That is, the system first uses a multilayer perceptron to align and project the feature embedding vector E. 100 Spatial dimension adjustments are performed to generate visual vectors aligned with the predefined text semantic vector space. The calculation function for the visual vectors is as follows:

[0043] in: The weight matrix represents the 2560-dimensional feature embedding vector E. 100 It is compressed to 512 dimensions to remove redundant visual background information and retain core semantic features. This represents the bias term, used to shift the characteristic distribution. This indicates that the ReLU activation function is used to enhance the expressive power of features through non-linear mapping, transforming visual features into visual vectors that can be matched with text. .

[0044] S2.2.2 Construction of a predefined industrial risk semantic library (TextEmbedding): The system pre-trains using the CLIP (Contrastive Language-Image Pre-training) model to transform the natural language descriptions of five safety categories ("normal operation", "smoke", "open flame", "personnel violation", and "other anomalies") into 512-dimensional text semantic vectors. ( =1,...,5). These text semantic vectors are stored in the system as "semantic anchors" for identifying various risks.

[0045] S2.2.3 Cosine Similarity Measurement and Risk Score: This involves calculating the cosine similarity between the visual vector and the predefined text semantic vector, which is used to quantify the degree of matching between the current frame and a specific risk category.

[0046] Visual vectors With each text semantic vector The function for calculating the cosine similarity between them is: , in, The L2 norm of a vector is used for length normalization to eliminate the influence of feature intensity fluctuations. Temperature scaling, set to 0.07, is used to "sharpen" the score distribution, making high-similarity risk items score more prominent and low-similarity items score lower.

[0047] S2.2.4 Generate the semantic risk score vector based on the cosine similarity, and extract the maximum value in the semantic risk score vector as the semantic risk score.

[0048] Extract the cosine similarity to the five categories, then perform Softmax normalization to obtain the probability distribution vector, which is the semantic risk score vector.

[0049] Assuming the semantic risk score vector is: ["Normal operation": 0.2, "Smoke": 0.82, "Open flame": 0.1, "Personnel violation": 0.15, "Other abnormalities": 0.05], then the conclusion is: the semantic risk score F sem If (100) = 0.82, then the model considers the current frame to have a high semantic risk of "smoke".

[0050] S2.3 Temporal motion module, which is used to analyze the changes in object shape and memory association of the current frame in the area corresponding to the prompt information by combining the historical memory bank, and generate the temporal dynamic features.

[0051] S2.3.1 Mask Deformation Analysis: SAM2 generates a segmentation mask M based on the prompt information of the current frame. 100 In specific implementation, this segmentation mask M t This covers the "smoke" area. The historical mask M corresponding to the previous frame of the current frame is retrieved from the historical memory database. t-1 Through the segmentation mask M t With the aforementioned history mask M t-1 Calculate the object shape change feature value between the current frame and the previous frame. The calculation of the object shape change feature value includes the following three processes: S2.3.1.1 Displacement between the current frame and the previous frame based on pixel logic operations Calculation: The system evaluates the target's displacement and shape alignment by performing pixel-wise logical operations on two binary matrices. Displacement deviation... The specific calculation function is: , in, : Bitwise AND operation to calculate the total number of pixels covered by the two frames of masks (intersection area) ). : Bitwise OR operation, calculates the total number of pixels covered by the two frames of the mask. ). Smoothing factor (take) This prevents calculation anomalies when the target completely disappears (denominator is 0).

[0052] Regarding displacement deviation Physical meaning: The smaller the value, the more drastic the displacement or shape change of the target relative to the previous frame (such as smoke rapidly spreading in all directions).

[0053] S2.3.1.2 The area change rate between the current frame and the previous frame is calculated using the following formula. : .

[0054] Calculate displacement deviation There is a loss of perception when scaling proportionally, so the system simultaneously calculates the degree of volume expansion of the target, i.e., the rate of change of area. .

[0055] Area change rate The physical meaning of this parameter is that it is specifically designed to capture features such as “liquid leak spread” or “smoke cloud expansion”.

[0056] S2.3.1.3 Characteristic values ​​of object morphological changes Nonlinear synthesis calculation.

[0057] The displacement deviation is expressed by the following formula. With area change rate Weighted fusion is performed to generate the morphological change feature values ​​of the object. : .

[0058] In the characteristic value of object morphological change In the calculation function, This indicates spatial inconsistency in the target. : Represents the scalability of the target. and Sensitivity weighting coefficients (such as 0.6 and 0.4) can be fine-tuned according to actual conditions. Activation function: Compresses the calculation results to the [0,1] interval. Its physical function is that it is insensitive to very small changes, but once the change exceeds the threshold, the score will quickly approach 1, thus marking physical anomalies.

[0059] S2.3.2 Memory Association Analysis: Calculate the feature embedding vector E of the current frame. 100 The degree of correlation between the sequence of historical features stored in the historical memory bank.

[0060] The system uses the feature embedding vector E of the current frame 100 The correlation coefficient is calculated using the query vector, the historical feature sequences stored in the MemoryBank as the key vector, and the value vector. The calculation function is as follows: .

[0061] correlation The physical meaning is: to calculate the probability that the current scene has been "seen" in history.

[0062] In summary, the final output temporal dynamic features in the temporal motion module include: object morphology change feature values. and correlation .

[0063] S3, the feature fusion step, involves adaptively weighting and fusing the semantic risk score vector and the temporal dynamic features based on historical feature data within the sliding window to obtain a comprehensive importance score for the current frame. This step aims to combine three heterogeneous indicators into a single comprehensive importance score through an objective and adaptive weighted fusion strategy. The core of this strategy lies in dynamically allocating weights based on the "uncertainty" of the information provided by each indicator within the current data window.

[0064] S3.1 Indicator Normalization: Eliminating differences in units and numerical ranges among different indicators to make them comparable. Specifically, historical feature data corresponding to the semantic risk score vector and temporal dynamic features are extracted from the sliding window. Based on the historical feature data, the semantic risk score vector and temporal dynamic features are normalized respectively to obtain normalized indicators corresponding to the semantic risk score vector and temporal dynamic features. The calculation function is as follows: .

[0065] Where, index X∈{F sem ,D,A mem {X} represents the historical feature data extracted from the sliding window corresponding to the semantic risk score vector and the temporal dynamic features. min({X}) and max({X}) are the minimum and maximum values ​​within the window, respectively. This represents the normalized value of index X in frame t, which is mapped to [0,1].

[0066] S3.2 Dynamic weight calculation based on information entropy. Specifically, the dynamic weight of the normalized index in the fusion is calculated based on the degree of disorder in the value distribution of each normalized index within the sliding window.

[0067] The calculation process for dynamic weights is as follows: a. Discretization: Discretizing each normalized index The value range [0,1] within the sliding window is divided into K intervals (e.g., K=10).

[0068] b. Statistical frequency: Calculation The frequency p_j of the value falling within each interval j.

[0069] c. Calculate information entropy: Calculate the information entropy of each normalized index based on the frequency distribution. The calculation function is as follows: .

[0070] The information entropy of indicator X measures the uncertainty of its value distribution. The higher the entropy value, the more varied and informational the indicator X is, and the higher its weight should be assigned.

[0071] d. Calculate the difference coefficient: Calculate the difference coefficient of each normalized index based on the information entropy. The calculation function is as follows: .

[0072] Coefficient of variation The smaller the value (the more concentrated the distribution). The larger.

[0073] E: Calculate the dynamic weight: The dynamic weight is obtained by fusing all the difference coefficients, and its calculation function is: .

[0074] The dynamic weights of indicator X satisfy w Fsem +w D +w Amem =1.

[0075] In the formula, These are the "difference coefficients" of three indicators: semantic risk score, deformation features, and memory association degree, which are derived from information entropy.

[0076] S3.3 Comprehensive Importance Score: A unified comprehensive importance score is output by weighted summation based on the normalized index and its corresponding dynamic weight. .

[0077] S4, the output discrimination step, judges the comprehensive importance score based on a set dynamic threshold to determine whether the current frame is a key frame. If so, the current frame is determined to be a key frame. This step aims to intelligently filter out frames that truly represent key events from a continuous score sequence through an adaptive dynamic threshold mechanism, avoiding false detections or missed detections caused by fixed thresholds.

[0078] S4.1 Calculate dynamic statistics: Obtain the sliding window sequence W of the scores for M frames, including the current frame. t ={S(t-M+1),...,S(t)}. Generally, t ≥ M, thus obtaining the complete M-frame sequence of the sliding window as W. t = {S(t-M+1), ..., S(t)}. However, in the early stages of video stream processing (i.e., the warm-up phase), there exists a case where t < M. In this case, the sliding window sequence W... t It should then consist of all available scores from frame 1 to frame t, i.e., W t= {S(1), ..., S(t)}. The mean and standard deviation of the scores are calculated based on the sliding window sequence to describe their "normal" level and fluctuation range, providing a computational basis for the dynamic threshold.

[0079] The function for calculating the mean (central tendency) is: .

[0080] The function for calculating standard deviation (dispersion) is: .

[0081] Where μ represents the average level of recent ratings, and σ represents the magnitude of the fluctuation of the ratings around the mean.

[0082] S4.2 Determine the dynamic threshold: Based on the mean and standard deviation, and in conjunction with a preset sensitivity coefficient, set a dynamic threshold that changes over time. A score exceeding this threshold indicates a significant deviation from the recent "normal" fluctuation range, which may be a signal of an abnormal event.

[0083] The function for calculating the dynamic threshold is: .

[0084] in, : The basic dynamic threshold at frame t. k: Sensitivity coefficient, an adjustable parameter whose value is determined through offline optimization or pre-set according to the scene's security level (e.g., k=1.8 for high-risk areas, k=1.2 for general areas). The larger the k, the higher the threshold and the stricter the screening.

[0085] S4.3 Significance Judgment and Peak Verification. Its purpose is to implement a two-level screening to ensure that the extracted keyframes are both frames with significant anomalies and extreme points (peaks) of local importance changes, thereby accurately locating the critical moment of the event outbreak and avoiding repeated frame extraction within the persistent high importance range.

[0086] Judgment conditions: a. Significance condition: S(t) > T base (t); b. Local peak condition: S(t)>S(t-1) and S(t)>S(t+1).

[0087] Decision logic: The current frame is determined to be a key frame if and only if conditions (a) and (b) are satisfied at the same time.

[0088] It should be noted that the local peak judgment condition requires the information of the comprehensive importance score S(t+1) of the next frame. This requires the system not to make a final judgment immediately when processing the t-th frame, but to wait for the arrival of the next frame. This "causal delay" strategy is a common practice in streaming processing, especially when accuracy requirements are higher than real-time performance. The specific implementation is as follows: The system continuously processes the video stream, calculating the overall importance score S for each frame. After processing the (t+1)th frame and calculating its overall importance score S(t+1), it backtracks to determine whether the t-th frame meets the peak condition. Therefore, the keyframe determination result for the t-th frame is output with a one-frame delay.

[0089] In the output discrimination step, the final output is: (1) Keyframe image: The original image or high-resolution screenshot of the t-th frame that is identified as a keyframe; (2) Structured metadata: A JSON-formatted record associated with this keyframe, containing: json { "frame_index": t, "timestamp": "YYYY-MM-DD HH:MM:SS.fff", "importance_score": S(t), "primary_semantic_label": "e.g., smoke", "trigger_region": "For example: Box1", "dynamic_attributes": { "deformation_score": D(t), "novelty_score": 1-A_mem(t) } }; (3) Event trigger signal: push key frame information and metadata to the monitoring center or alarm system to trigger subsequent alarm, video archiving or manual review process.

[0090] To verify the effectiveness and superiority of this method, a test dataset was constructed in a real-world monitoring scenario within a chemical industrial park, and comparative experiments were conducted. The dataset contains 24-hour monitoring videos over 30 consecutive days, covering key areas such as the reactor area, storage tank area, and loading / unloading area, totaling approximately 720 hours in duration. The resolution is 1920×1080, and the frame rate is 25fps. Three safety experts jointly labeled abnormal events, resulting in 217 valid abnormal events, including smoke (78 events), open flames (23 events), liquid leaks (52 events), and unauthorized personnel entry (64 events). The start and end frame intervals of each event were labeled as the true standard for keyframe extraction.

[0091] Three existing technologies were selected as comparative examples: Comparative Example 1 (Optical Flow Method + Fixed Threshold): Based on the inter-frame optical flow energy change, a global threshold is set, and frames exceeding the threshold are judged as keyframes; Comparative Example 2 (Equal Interval Frame Extraction): 1 frame is extracted at a fixed interval of one second, without considering content changes; Comparative Example 3 (Target Detection Method): YOLOv8 is used to detect predefined targets (such as smoke, flames, and people). When a target is detected, the current frame is extracted.

[0092] The following evaluation metrics are used to measure the performance of each method: Recall: The percentage of successfully detected anomalous events out of all anomalous events; Accuracy: The proportion of keyframes extracted that truly represent anomalous events; F1-score: the harmonic mean of recall and precision; Mean Temporal Offset: The average absolute frame difference between the extracted keyframes and the actual starting frame of the event, measuring the accuracy of time positioning; Event coverage (IoU): The intersection-union ratio of the extracted keyframe intervals to the actual event intervals, reflecting the completeness of the event process.

[0093] The comparison results are shown in Table 1.

[0094]

[0095] As shown in Table 1, the method of the present invention is significantly superior to the prior art in all indicators. Specific analysis is as follows: (1) Significantly improved ability to detect abnormal events: The recall rate of this embodiment reached 91.2%, which is about 30 percentage points higher than the best performing comparative example 3 (61.3%), indicating that the present invention can effectively capture the vast majority of real abnormal events with an extremely low false negative rate. This is due to the synergistic effect of the semantic perception module and the temporal motion module, which can identify both semantically clear abnormalities (such as open flames) and slowly changing abnormalities with morphological changes (such as leakage and diffusion).

[0096] (2) Significantly reduced false alarm rate: The accuracy of this embodiment reaches 94.6%, which is much higher than that of Comparative Example 1 (34.2%) and Comparative Example 3 (45.8%). Comparative Example 1 is easily affected by changes in lighting and camera shake, resulting in a large number of false alarms; although Comparative Example 3 can detect specific targets, it cannot identify undefined anomalies (such as slow liquid leakage) and often misdetects security equipment as intrusion. This invention accurately distinguishes between real anomalies and irrelevant movements through object perception and spatiotemporal memory mechanisms, greatly reducing false alarms.

[0097] (3) Precise keyframe timing: The average time offset in this embodiment is only 6.5 frames (about 0.26 seconds), which is much smaller than the 18.2 frames (0.73 seconds) of Comparative Example 3 and the 23.9 frames (0.96 seconds) of Comparative Example 1. This shows that the present invention can accurately capture the moment when an anomaly occurs, providing an accurate timing for rapid response.

[0098] (4) Complete event process coverage: The event coverage IoU of this embodiment reached 79.8%, which is significantly improved compared with 48.0% of Comparative Example 1. This indicates that the extracted key frame sequence can cover the complete process of the anomaly from occurrence to development to end, which is convenient for subsequent accident cause analysis and tracing.

[0099] (5) Edge real-time processing capability: On the NVIDIA Jetson AGX Orin platform, the average processing time for a single frame is 35ms, and it can process 25fps video streams in real time, meeting the real-time requirements of industrial monitoring.

[0100] In summary, the method of the present invention has made breakthrough progress in terms of the accuracy, completeness and real-time performance of keyframe extraction, providing reliable technical support for intelligent industrial safety monitoring.

[0101] like Figure 1 As shown, the present invention also provides a system for implementing the above-described method for extracting keyframes from industrial surveillance videos based on object perception and spatiotemporal memory. The system includes: Visible light camera: Hikvision DS-2CD7A47EWD-IZS, resolution 1920×1080, frame rate 25fps, deployed directly above the monitoring area to cover the monitoring area; Edge computing device: an industrial control computer with an embedded NVIDIA Jetson AGX Orin 32GB processor, responsible for real-time video stream processing and keyframe extraction; Storage and Alarm Server: Used to receive keyframes and metadata pushed by edge devices, trigger alarms, and archive them.

[0102] The camera is connected to the edge computing device via an industrial Ethernet network, and the video stream is transmitted in real time using the RTSP protocol. The edge computing device runs the Ubuntu 20.04 operating system, is equipped with the PyTorch 1.13 deep learning framework, and is pre-loaded with the SAM2 model (sam2_hiera_base version) and CLIP text encoder.

[0103] In edge computing devices: A video stream preprocessing module is used to perform the video stream preprocessing steps to obtain the current frame in the input video stream and at least one preset prompt message for the current frame.

[0104] Specifically, edge computing devices continuously receive video streams from cameras, denoted as a continuous frame sequence F={F1,F2,…,F…}. t Each frame is 224×224×3 (scaled and aligned). The system loads a predefined static prompt box for this monitoring point. In this embodiment, two regions of interest are set: Box 1: Observation window area of ​​the reactor, coordinates (x1=80, y1=60, x2=160, y2=140); Box2: Pipe valve connection, coordinates (x1=10, y1=150, x2=100, y2=220).

[0105] The video stream preprocessing module will process the current frame F t Combined with the prompt box, it serves as input for the subsequent image encoder.

[0106] An image encoder, connected to the video stream preprocessing module, is used to perform the image encoding steps to extract and analyze features of the current frame based on the prompt information, so as to obtain semantic risk score vectors and temporal dynamic features in parallel.

[0107] Specifically, taking frame 100 as an example... 100 For example, the image encoder performs the following operations: In the feature encoding module: features are extracted using the Hiera backbone network in SAM2. 100 First, the image is divided into 32×32 image blocks using a 7×7 convolution (stride 7), and each image block is mapped to a 256-dimensional vector, resulting in the initial feature map T0∈R. 32×32×256Hierarchical feature extraction then proceeds through four stages (Stages 1-4) of Hiera, with each stage including window attention and downsampling operations: Stage 1: Output size 32×32×256 (local texture features); Stage 2: Downsampling to 16×16×512 (geometric features); Stage 3: Downsampling to 8×8×1024 (component-level features); Stage 4: After global pooling, a 1×1×2560 feature embedding vector E is obtained. 100 .

[0108] In the semantic awareness module: E 100 Input a two-layer MLP (dimensions 2560→1024→512), and obtain visual vectors through ReLU activation. ∈R 512 The system pre-defines five types of industrial risk text descriptions, which are encoded into 512-dimensional text semantic vectors using the CLIP model. (k=1,…,5), corresponding to "normal operation", "smoke", "open flame", "personnel violation", and "other abnormalities" respectively. Calculate With each The cosine similarity, after being scaled by a temperature coefficient τ=0.07, is used to obtain the probability distribution vector through Softmax. For F 100 If the "smoke" category scores the highest, such as 0.82, then the semantic risk score F is... sem (100) = 0.82.

[0109] In the timing motion module: SAM2 adjusts F according to the prompt box. 100 Predictive segmentation mask M 100 (For example, covering a smoke-filled area). Retrieve the previous frame F from the Memory Bank. 99 and its history mask M 99 ,calculate: IoU(M 100 M 99 =0.65; ΔArea=(Area(M 100 ) Area (M) 99 )) / Area(M 99 = +0.4; D(100)=tanh(0.6×(1 0.65)+0.4×∣0.4∣)≈0.35.

[0110] Simultaneously, the correlation degree A between the current frame features and the historical features in the memory is calculated through cross-attention. mem (100). Assume A mem (100) = 0.3, indicating that the correlation between the current frame features and the historical features in the memory bank is low, which in turn indicates that the current frame has a visual feature that has rarely appeared in history, that is, "smoke" is a newly emerging anomaly with low correlation.

[0111] A feature fusion unit, connected to the image encoder, is used to perform the feature fusion step to adaptively weight and fuse the semantic risk score vector and the temporal dynamic features based on historical feature data within a sliding window, to obtain a comprehensive importance score for the current frame.

[0112] Specifically, a sliding window (window size N=100) is constructed to collect the semantic risk score vector F of the most recent 100 frames. sem 1. Object morphological change characteristic value D; 2. Relationship degree A mem Assume the statistical range of these three indicators is: F sem ∈ [0.1,0.9], D ∈ [0.0, 0.8], A mem ∈ [0.2, 1.0]. Perform min-max normalization on each index to obtain F. sem_norm (100) = (0.82 - 0.1) / (0.9 - 0.1) = 0.72 / 0.8 = 0.9,D norm (100) = (0.35- 0.0) / (0.8 - 0.0) = 0.35 / 0.8 ≈ 0.44,A mem_norm (100) = (0.3 - 0.2) / (1.0 -0.2) = 0.1 / 0.8 = 0.125. The normalized values ​​are discretized into 10 intervals, and the information entropy of each indicator is calculated to obtain the dynamic weights: w Fsem =0.5,w D =0.3,w Amem =0.2, then the overall importance score of the current frame is: S(100) = 0.5 × 0.9 + 0.3 × 0.44 + 0.2 × (1 - 0.125) = 0.45 + 0.132 + 0.175 = 0.757 (approximately 0.76).

[0113] A discriminator, connected to the feature fusion unit, is used to execute the discrimination output step to discriminate the comprehensive importance score based on a set dynamic threshold, so as to determine whether the current frame is a key frame. If so, the current frame is determined to be a key frame.

[0114] Specifically, maintain a rating sliding window Wt of length M=150. Since t=100<150 at this point, W... t = {S(1), ..., S(t)}), calculate the mean μ and standard deviation σ, where μ = 0.3 and σ = 0.1 respectively. Take the sensitivity coefficient k = 1.5 to obtain the dynamic threshold T. base (100) = 0.45. Since S(100) = 0.76 > 0.45, the system caches S(100) and S(99) to achieve peak value judgment. After processing the 101st frame, assuming S(101) = 0.60, the final judgment for the 100th frame is then performed: S(100) = 0.76 > T. base Since S(t) = 0.45, S(100) = 0.76 > S(99) = 0.65, and S(100) = 0.76 > S(101) = 0.60, the 100th frame is determined to be a key frame.

[0115] The system saves the original image (or compressed keyframe) of the 100th frame and generates metadata in JSON format. This keyframe and its metadata are pushed to the monitoring center server to trigger an alarm and archive it, while also supporting subsequent event tracing and analysis.

[0116] Furthermore, in the discriminator, if the current frame is the last frame, the processing flow of the video stream is directly introduced; in the discriminator, if the current frame is determined to be a key frame, the current frame is saved; if the current frame is determined not to be a key frame, the history memory is updated, the current frame is updated to the next frame, and then the method for extracting key frames of industrial monitoring videos based on object perception and spatiotemporal memory is executed.

[0117] The above description is only a preferred embodiment of the present invention. It should be noted that those skilled in the art can make several modifications and improvements without departing from the structure of the present invention, and these will not affect the effectiveness of the implementation of the present invention or the practicality of the patent.

Claims

1. A method for extracting keyframes from industrial surveillance videos based on object perception and spatiotemporal memory, characterized in that, The method includes the following steps: Video stream preprocessing steps: Obtain the current frame in the input video stream and at least one preset prompt message for the current frame; Image encoding steps: Based on the prompt information, perform feature extraction and feature analysis on the current frame to obtain semantic risk scores and temporal dynamic features in parallel; Feature fusion step: Based on historical feature data within the sliding window, the semantic risk score and the temporal dynamic features are adaptively weighted and fused to obtain the comprehensive importance score of the current frame; The output discrimination step is to discriminate the comprehensive importance score based on the set dynamic threshold to determine whether the current frame is a key frame.

2. The method for extracting key frames from industrial surveillance videos based on object perception and spatiotemporal memory according to claim 1, characterized in that, The image encoding steps specifically include: The current frame is used to extract features using a visual segmentation model to obtain a feature embedding vector; The feature embedding vector is matched with predefined text semantics for risk analysis, and the semantic risk score is output. By combining the historical memory database of the visual segmentation model, the changes in object morphology and memory association of the current frame within the area corresponding to the prompt information are analyzed to generate the temporal dynamic features.

3. The method for extracting keyframes from industrial surveillance videos based on object perception and spatiotemporal memory according to claim 2, characterized in that, The feature embedding vector is then subjected to risk matching analysis with a predefined text semantic vector, specifically as follows: The feature embedding vector is spatially adjusted using a multilayer perceptron to generate a visual vector aligned with the predefined text semantic vector space. Calculate the cosine similarity between the visual vector and the predefined text semantic vector; The semantic risk score vector is generated based on the cosine similarity, and the maximum value in the semantic risk score vector is extracted as the semantic risk score.

4. The method for extracting key frames from industrial surveillance videos based on object perception and spatiotemporal memory according to claim 2, characterized in that, The specific method for generating the aforementioned temporal dynamic features is as follows: The visual segmentation model generates a segmentation mask based on the prompt information of the current frame, and obtains the historical mask corresponding to the previous frame of the current frame from the historical memory bank; The object morphology change feature value between the current frame and the previous frame is calculated using the segmentation mask and the history mask; Calculate the correlation between the feature embedding vector of the current frame and the historical feature sequence stored in the history memory bank; The temporal dynamic features include the object morphological change feature value and the correlation degree.

5. The method for extracting key frames from industrial monitoring videos based on object perception and spatiotemporal memory according to claim 4, characterized in that, The specific calculation method for the object morphological change characteristic value is as follows: The displacement deviation between the current frame and the previous frame is calculated using the following formula. : , In the formula, Mt represents the segmentation mask, and Mt-1 represents the history mask. This indicates a bitwise AND operation. This indicates a bitwise OR operation. Indicates the smoothing factor; The rate of change of area between the current frame and the previous frame is calculated using the following formula. : , In the formula, This indicates the degree of volume expansion of the object corresponding to the segmentation mask in the current frame. This indicates the degree of volume expansion of the object corresponding to the history mask in the previous frame of the current frame; The displacement deviation is expressed by the following formula. With area change rate Weighted fusion is performed to generate the morphological change feature values ​​of the object. : ， In the formula, and All of these represent sensitivity weighting coefficients.

6. The method for extracting key frames from industrial surveillance videos based on object perception and spatiotemporal memory according to claim 1, characterized in that, The feature fusion step is specifically as follows: Historical feature data corresponding to the semantic risk score and temporal dynamic features are extracted from the sliding window. Based on the historical feature data, the semantic risk score and temporal dynamic features are normalized respectively to obtain normalized indicators corresponding to the semantic risk score and temporal dynamic features. The dynamic weight of the normalized index in the fusion is calculated based on the degree of disorder in the value distribution of each normalized index within the sliding window. The comprehensive importance score is obtained by weighted summation based on the normalized index and its corresponding dynamic weight.

7. The method for extracting key frames from industrial surveillance videos based on object perception and spatiotemporal memory according to claim 1, characterized in that, The specific calculation method for the dynamic weight is as follows: The range of values ​​for each normalized index within the sliding window is divided into K equal intervals; Calculate the frequency with which each normalized index falls within each interval; Calculate the information entropy of each normalized index based on the frequency distribution; Calculate the difference coefficients of each normalized index based on the information entropy; The dynamic weight is obtained by fusing all the difference coefficients.

8. The method for extracting key frames from industrial surveillance videos based on object perception and spatiotemporal memory according to claim 1, characterized in that, The specific steps for determining the output are as follows: Obtain a sliding window sequence containing the scores of M frames, including the current frame; The mean and standard deviation of the scores are calculated based on the sliding window sequence. Based on the mean and standard deviation and combined with a preset sensitivity coefficient, a dynamic threshold that changes over time is set. Determine whether the overall importance score of the current frame is greater than the dynamic threshold, and at the same time determine whether the overall importance score of the current frame is greater than the scores of the frames adjacent to the current frame before and after it. If both of the above conditions are met, then the current frame is determined to be a keyframe.

9. A system for implementing the keyframe extraction method for industrial surveillance video based on object perception and spatiotemporal memory as described in any one of claims 1-8, characterized in that, The system includes: A video stream preprocessing module is used to perform the video stream preprocessing steps to obtain the current frame in the input video stream and at least one preset prompt message for the current frame; An image encoder, connected to the video stream preprocessing module, is used to perform the image encoding step to extract and analyze features of the current frame based on the prompt information, so as to obtain semantic risk scores and temporal dynamic features in parallel. A feature fusion unit, connected to the image encoder, is used to perform the feature fusion step to adaptively weight and fuse the semantic risk score and the temporal dynamic features based on historical feature data within a sliding window, so as to obtain a comprehensive importance score for the current frame. A discriminator, connected to the feature fusion unit, is used to execute the discrimination output step to discriminate the comprehensive importance score based on a set dynamic threshold, so as to determine whether the current frame is a key frame. If so, the current frame is determined to be a key frame.

10. A system according to claim 9, characterized in that, In the discriminator, if the current frame is the last frame, the processing flow of the video stream is directly introduced; in the discriminator, if the current frame is determined to be a key frame, the current frame is saved; if the current frame is determined not to be a key frame, the history memory is updated, the current frame is updated to the next frame, and then the key frame extraction method for industrial monitoring video based on object perception and spatiotemporal memory is executed.

Images