Monitoring video hierarchical storage method and system based on reference background constraint and multi-modal sorting

Abstract

The application discloses a monitoring video hierarchical storage method and system based on reference background constraint and multi-modal sorting, relates to the technical field of computer vision and video monitoring, and comprises the following steps: constructing a local change region corresponding to each sampling frame; determining a determination result of each sampling time according to the local change region corresponding to each sampling time of each candidate segment; wherein the determination result is used for indicating an effective state change of a person and / or an effective state change of an electric bicycle; determining an effective event set according to the determination results of the plurality of sampling times in each candidate segment, constructing a candidate frame set for the effective event set; performing value sorting on each candidate frame based on visual features and semantic features of each candidate frame to determine a central representative frame; determining video data corresponding to the effective event set as storage data, and determining an event index of the storage data based on the central representative frame. The application improves the effectiveness of monitoring video storage and the event retrieval efficiency.

Description

Technical Field

[0001] This application relates to the fields of computer vision and video surveillance technology, and more specifically, to a method and system for hierarchical storage of surveillance videos based on reference background constraints and multimodal sorting. Background Technology

[0002] Surveillance video has become a crucial infrastructure for public safety and social governance. With the expansion of camera deployment, increased video resolution, and longer retention periods, the amount of data generated by continuous recording is growing exponentially, putting continuous pressure on storage system capacity and costs. Tiered storage, by allocating video data of varying value to storage media with different performance levels, can reduce overall storage overhead while ensuring the traceability of critical evidence, and is therefore widely regarded as an effective strategy. However, the effectiveness of tiered storage is highly dependent on the accuracy of the pre-screening process in identifying video content with long-term preservation value; otherwise, invalid data will still occupy higher-level storage resources, diminishing the actual benefits of the tiered strategy.

[0003] Existing surveillance systems typically rely on motion detection technology to trigger recording or generate candidate clips, using this as the primary means of filtering and storing content. However, in real-world scenarios, factors such as changes in lighting, shadow projection, background object disturbances, and even slight camera shake can trigger motion detection. This results in a large number of invalid clips that do not involve changes in the effective state of people or electric bicycles being mistakenly identified as valuable content and entering the storage chain. This crude "any movement is stored" approach leads to a large amount of invalid video occupying storage resources, while events that truly require attention are buried in a sea of ​​recordings.

[0004] Therefore, how to filter motion candidate segments in a fixed monitoring scenario while preserving complete and valid event evidence, and establish a compact and easily accessible event representation accordingly, is a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0005] This application provides a method and system for hierarchical storage of surveillance videos based on reference background constraints and multimodal sorting to solve the above-mentioned technical problems.

[0006] This application provides a hierarchical storage method for surveillance video based on reference background constraints and multimodal sorting, including: determining multiple sampling times for each candidate segment of the surveillance video according to a preset sampling strategy; obtaining sampling frames corresponding to each sampling time; and constructing local change regions corresponding to each sampling frame based on the reference background frame corresponding to each candidate segment and each sampling frame of each candidate segment. The preset sampling strategy includes: for each candidate segment, using the first frame of the candidate segment as the reference background frame of the candidate segment, and determining multiple sampling times within the candidate segment at equal frame intervals based on the reference background frame of the candidate segment; and constructing local change regions corresponding to each sampling frame based on each sampling frame of each candidate segment. The local change region corresponding to each sampling moment is identified, and the judgment result for each sampling moment is determined. The judgment result is used to indicate the effective state changes of personnel and / or electric bicycles. Based on the judgment results of multiple sampling moments within each candidate segment, a set of effective events is determined, and a set of candidate frames is constructed from the set of effective events. Visual and semantic features of each candidate frame in the set of candidate frames are extracted. Based on the visual and semantic features of each candidate frame, the candidate frames are ranked by value to determine the central representative frame. The video data corresponding to the set of effective events is determined as stored data, and the event index of the stored data is determined based on the central representative frame.

[0007] This application provides a hierarchical storage system for surveillance video based on reference background constraints and multimodal sorting, comprising: a construction module, configured to determine multiple sampling times for each candidate segment of the surveillance video according to a preset sampling strategy, acquire sampling frames corresponding to each sampling time, and construct local change regions corresponding to each sampling frame based on the reference background frame corresponding to each candidate segment and each sampling frame of each candidate segment; wherein, the preset sampling strategy includes: for each candidate segment, using the first frame of the candidate segment as the reference background frame of the candidate segment, and determining multiple sampling times within the candidate segment at the same frame interval based on the reference background frame of the candidate segment; and a determination module, configured to determine the local change regions corresponding to each sampling time of each candidate segment. The system comprises: a region for determining the judgment results at each sampling time; wherein the judgment results are used to indicate the effective state changes of personnel and / or electric bicycles; a candidate frame determination module for determining a set of effective events based on the judgment results at multiple sampling times within each candidate segment, and constructing a candidate frame set from the set of effective events; a center representative frame determination module for extracting the visual and semantic features of each candidate frame in the candidate frame set; ranking the candidate frames by value based on the visual and semantic features of each candidate frame to determine the center representative frame; and a hierarchical storage module for determining the video data corresponding to the set of effective events as storage data, and determining the event index of the storage data based on the center representative frame.

[0008] In this application, multiple sampling times are determined for each candidate segment of the surveillance video, and corresponding sampling frames are obtained. Local change regions of each sampling frame are constructed by combining the reference background frame corresponding to each candidate segment. This allows subsequent analysis to focus on local areas in the image that have undergone actual changes relative to a fixed scene, rather than processing the entire surveillance image indiscriminately. This avoids interference from a large area of ​​stable background on the judgment of the target state and improves the accuracy of identifying effective state changes in fixed surveillance scenes. Furthermore, based on the local change regions at each sampling time, it determines whether there are effective state changes for personnel and / or electric bicycles. The selection criteria are limited from the general "pixel motion" to changes in the specific semantic target relative to its reference state before entering the scene. This prevents environmental interference factors that do not involve actual changes in the state of personnel or electric bicycles, such as changes in lighting, shadow projection, background object disturbances, and camera micro-shakes, from being judged. This solves the technical problem in existing technologies where coarse motion detection leads to a large number of invalid segments entering the storage link, weakening the benefits of layered storage.

[0009] The system determines the set of valid events based on the judgment results of multiple sampling moments within the candidate segments and constructs a set of candidate frames, enabling the accurate extraction of complete and valid event segments from continuous video streams. By extracting the visual and semantic features of each candidate frame in the candidate frame set, the system ranks each candidate frame based on the fused features to determine the central representative frame. This eliminates reliance on general visual criteria such as motion intensity and clarity when selecting the central representative frame. Instead, it comprehensively evaluates the value of identification clues such as personnel identification, clothing appearance, electric bicycle model characteristics, and the interaction between people and vehicles in the monitoring scene, ensuring that the selected representative frame has high information content at the monitoring information level. Furthermore, the video data corresponding to the set of valid events is determined as the stored data, and the event index of the stored data is determined based on the central representative frame. This ensures that each valid event is fully stored to retain all evidence information while having a compact index representation corresponding to the moment with the highest information value. During subsequent retrieval, the central representative frame is used for quick browsing and filtering first, eliminating the need to search for the complete event content frame by frame. This significantly reduces the retrieval burden of manual review while preserving the complete evidence chain. Attached Figure Description

[0010] The accompanying drawings, which are included to provide a further understanding of embodiments of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings: Figure 1 This is a flowchart of an optional hierarchical storage method for surveillance video based on reference background constraints and multimodal sorting according to an embodiment of this application; Figure 2This is a flowchart of another optional hierarchical storage method for surveillance video based on reference background constraints and multimodal sorting according to an embodiment of this application; Figure 3 This is a flowchart illustrating an optional center representative frame positioning and the output of a three-frame representative group according to an embodiment of this application. Figure 4 This is a schematic diagram showing the hit rate comparison results of an optional different center representative frame localization method according to an embodiment of this application on a test event.

[0011] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0012] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0013] The key to long-term monitoring and storage is identifying truly valid information before video enters the storage chain. This problem involves two levels. At the segment level, changes in lighting, shadows, reflections, background object disturbances, and even slight camera shake can all trigger motion detection, generating invalid candidate segments that do not involve valid changes in the state of people or electric bicycles, thus allowing invalid video to enter the long-term storage chain. At the event representation level, real events often contain a large number of consecutive frames with similar content. If complete event segments need to be retained, directly retrieving each frame increases the burden of event indexing, rapid browsing, and manual verification. The former requires valid screening of motion candidate segments, while the latter requires building a compact and quickly accessible event representation based on retaining complete event video evidence.

[0014] Fixed cameras provide stable scene conditions to solve the above problems. The camera's perspective, spatial layout, and main background remain unchanged over a long period, allowing footage from before the event to serve as a reference state. Therefore, segment selection can be transformed from "whether there are people or electric bicycles in the current frame" to "whether there have been effective state changes related to people or electric bicycles relative to the initial reference scene." By retaining complete and valid event segments, and then locating the central moment with high monitoring information value within the event and generating a three-frame representative group, a compact index representation of the complete event can be established, reducing the burden of frame-by-frame retrieval during subsequent rapid browsing and manual review.

[0015] The original long video is first used to generate candidate short segments using a lightweight motion trigger based on an improved adaptive Gaussian mixture model (MOG2). However, its foreground mask mainly describes pixel statistical changes, making it difficult to directly distinguish between real targets and environmental disturbances. Depth target detection methods significantly improve the localization capabilities of people and vehicles, and video anomaly detection can also identify events that deviate from normal patterns through temporal features. However, these tasks are not entirely consistent with the effective information screening in long-term storage. Target detectors that independently process the current frame struggle to determine whether a target in the scene was added during an event; anomaly detection is usually based on whether behavior deviates from normal patterns, but changes in target state such as people entering or leaving, or electric bicycles entering or leaving, have clear retention value, while significant changes in illumination and background disturbances do not constitute effective events as defined in this application. Therefore, video screening before long-term storage needs to consider the initial state of the scene, the current target semantics, and evidence of local changes simultaneously.

[0016] Visual language models offer a new technical approach for understanding relative states in fixed monitoring scenarios. However, when directly inputting the complete monitoring footage, state changes of distant, small-scale, or partially occluded targets are easily weakened by a large area of ​​stable background; inputting only the moving area results in the loss of scene reference and spatial context. Furthermore, general visual language models may not be able to directly understand the specific discrimination rule of "whether there are effective state changes related to personnel or electric bicycles relative to the initial reference scene of the event." Therefore, it is still necessary to construct reference conditions for fixed monitoring scenarios and improve the model's ability to recognize scene-specific interference through scene adaptation.

[0017] Beyond segment-level effective event filtering, rapid retrieval of long-term videos also faces significant frame-level redundancy. A short-term event typically contains a large number of images with similar content, but the completeness of the target, occlusion status, interaction relationships, and identification cues such as faces, clothing, or vehicle appearance change over time. Video summarization methods primarily focus on content coverage, diversity, or consistency with human summarization; fixed intermediate frames, motion peak frames, or frames with the highest clarity may not correspond to the event moment with the highest monitoring information value. Individual images may also be affected by factors such as instantaneous motion blur, partial occlusion, or reflections. Therefore, the system not only needs to locate the central representative frame with high information value in effective events, but also needs to retain limited temporal information near that moment while building a compact index representation for the complete event to support subsequent event indexing, rapid browsing, and manual review.

[0018] The analysis falls into two categories: effective video information extraction and intelligent surveillance video analysis. The former focuses on information fidelity under finite representation, while the latter focuses on moving target detection, target recognition, and event understanding. Both provide the technical foundation for candidate generation, semantic filtering, and center representative frame localization, but they do not yet fully cover the continuous processing needs in fixed surveillance scenarios.

[0019] Video summarization aims to summarize the content of an original video using a small number of frames or short segments. Early methods typically relied on clustering or heuristic selection based on shot boundaries, color differences, and motion intensity, but later shifted towards data-driven temporal importance modeling. Methods based on Long Short-Term Memory (LSTM) networks utilize cross-frame dependencies to estimate frame importance, while adversarial reconstruction, reinforcement learning, fully convolutional sequence networks, and cycle consistency constraints reduce reliance on frame-level annotation from the perspectives of reconstruction quality, content diversity, and representativeness. These methods have achieved good results on general summarization datasets, but their goals are usually to cover the main content or match human summaries, making it difficult to directly characterize the completeness of targets, interaction states, and identity identification cues in surveillance footage.

[0020] Visual language pre-training further improves the semantic relevance of key content selection, while methods based on large language models attempt to use event-level semantics to assist video summarization. These works show that high-level semantics can supplement simple visual saliency, but most methods output continuous summaries or independent frame scores, rarely establish explicit partial orders for highly similar candidate frames in the same short-term monitoring event, and do not adequately discuss the model bias caused by fixed sampling positions.

[0021] Content-aware storage and multi-camera collaborative analysis reduce the overhead of video decoding, transmission, and redundant computation at the system level, but they typically treat event-triggered results as reliable input. When upstream motion detection is interfered with by lighting, shadows, or dynamic backgrounds, falsely triggered segments can still enter the long-term storage link; even if the segment is genuine and valid, fixed intermediate frames or motion peak frames may not be suitable as event indexes.

[0022] Background modeling and foreground extraction are common methods for generating candidate segments in surveillance videos. Adaptive Gaussian mixture models (Gaussian mixture models) adapt to background changes by updating pixel distribution online. MOG2 is widely used for motion triggering with fixed cameras due to its low computational cost and mature implementation. However, foreground masks only describe pixel statistical changes, and sudden changes in illumination, shadows, reflections, swaying leaves, and slight camera jitter can all generate false foregrounds; slow-moving or briefly stationary targets may also be gradually absorbed by the background model. Therefore, MOG2 is suitable for generating lightweight motion candidates, but not as the sole final criterion for determining whether a segment has surveillance value.

[0023] Deep object detection significantly improves the localization capabilities of people and vehicles. However, a key issue in fixed scenes is not only the presence of the target, but also whether the target is newly added relative to the initial state of the segment. Detectors that process the current frame independently struggle to exclude vehicles that have been parked for a long time or people that were already present in the first frame; multi-target tracking can supplement trajectory continuity, but it is still affected by occlusion, missed detections, and identity association errors.

[0024] Video anomaly detection identifies events that deviate from normal states based on temporal patterns. Weakly supervised multi-instance learning and robust temporal feature amplitude learning reduce frame-by-frame annotation costs; open-vocabulary anomaly detection and training-free or interpretable methods combined with large language models further enhance category expansion and semantic interpretation capabilities. Research on surveillance video and language has also begun to build dedicated data and models around description generation, question answering, and temporal localization. However, anomalies and business validity are not consistent: changes in target state such as people entering or leaving, electric bicycles entering or leaving, etc., have retention value, while significant environmental disturbances do not constitute valid events as defined in this application.

[0025] In summary, while some methods address motion triggering, target recognition, anomaly understanding, and content summarization, they lack a hierarchical mechanism for fixed cameras: first, generating motion candidate segments from continuous video; then, determining whether valid state changes related to people or electric bicycles have occurred using a fixed reference scene; and finally, locating a representative frame based on the value of the monitoring information and generating a group of three consecutive representative frames. This application combines lightweight motion triggering, reference frame conditional discrimination, and candidate frame sorting within this continuous task framework.

[0026] According to one aspect of the embodiments of this application, such as Figure 1 As shown, this application provides a hierarchical storage method for surveillance videos based on reference background constraints and multimodal sorting, including: Step S1: Determine multiple sampling times for each candidate segment corresponding to the monitoring video, obtain the sampling frames corresponding to each sampling time, and construct the local change region corresponding to each sampling frame based on the reference background frame corresponding to each candidate segment and each sampling frame of each candidate segment. Before determining multiple sampling times for each candidate segment corresponding to the surveillance video, the method further includes: performing motion detection on the acquired surveillance video to generate multiple candidate segments, and determining the first frame of each candidate segment as the reference background frame corresponding to each candidate segment; The method of determining multiple sampling times for each candidate segment corresponding to the surveillance video includes: determining multiple sampling times for each candidate segment corresponding to the surveillance video according to a preset sampling strategy; wherein the preset sampling strategy includes: for each candidate segment, taking the first frame of the candidate segment as the reference background frame of the candidate segment, and determining multiple sampling times within the candidate segment at the same frame interval based on the reference background frame of the candidate segment.

[0027] In one embodiment, multiple sampling times are determined for each candidate segment corresponding to the surveillance video according to a preset sampling strategy, including: Motion detection is performed on the acquired surveillance video to generate multiple candidate segments. Each candidate segment has a clearly defined start frame (i.e., the first frame) and end frame, and the start frame of each candidate segment is determined as the reference background frame corresponding to that candidate segment. For any candidate segment, the preset sampling strategy is as follows: based on the reference background frame of the candidate segment, multiple sampling times are determined within the candidate segment according to the order of the frames and at the same frame interval; the same frame interval means that the number of frames between two adjacent sampling frames within the same candidate segment is the same, preferably 3 frames.

[0028] Specifically, starting from the frame following the reference background frame of the candidate segment, the monitoring video frames are read sequentially in chronological order. During the reading process, whenever the third frame is read consecutively, that frame is identified as a sampling frame, and the acquisition time of that sampling frame in the monitoring video is identified as a sampling time of the candidate segment. After identifying a sampling frame, the counting continues from the next frame in the order of the first, second, and third frames. When the third frame is read consecutively again, the next sampling frame and its corresponding sampling time are identified. This process is repeated until the end frame of the candidate segment is reached.

[0029] The reference background frame corresponding to the candidate segment is only used as the differential reference for each sampling frame, and is not used as a sampling frame, nor is it counted repeatedly in the above-mentioned "continuous reading to the 3rd frame" count; the sampling time of each candidate segment is determined by the above-mentioned rule of reading frame by frame and determining a sampling frame every 3 consecutive frames.

[0030] Optionally, if a candidate segment contains fewer than three frames excluding the reference background frame, and a sampling frame cannot be obtained according to the rule of determining one sampling frame for every three consecutive frames, then the ending frame of the candidate segment is used as the sampling time. If a candidate segment contains only one valid frame, the reference background frame, then no sampling time is determined for that candidate segment. Through this method, multiple sampling times and corresponding sampling frames can be stably determined for each candidate segment generated by motion detection.

[0031] Step S2: Determine the judgment result for each sampling time based on the local change region corresponding to each sampling time of each candidate segment; wherein, the judgment result is used to indicate the effective state change of the person and / or the effective state change of the electric bicycle. Step S3: Based on the judgment results of multiple sampling times within each candidate segment, determine the set of valid events, and construct a set of candidate frames from the set of valid events; Step S4: Extract the visual and semantic features of each candidate frame in the candidate frame set; based on the visual and semantic features of each candidate frame, sort the candidate frames by value to determine the central representative frame. Step S5: Determine the video data corresponding to the valid event set as the storage data, and determine the event index of the storage data based on the central representative frame.

[0032] In one embodiment of the present invention, this application addresses surveillance video continuously captured by a fixed camera, constructing a three-level processing framework consisting of lightweight motion candidate segment generation, effective event filtering with reference scene constraints, and center representative frame localization and continuous three-frame representative group output through visual and semantic fusion. Figure 2 As shown, the front-end module generates time intervals that may contain dynamic targets from long-term videos with low computational overhead; the middle module determines whether candidate segments have undergone effective state changes related to people or electric bicycles relative to the initial reference scene; the back-end module only performs frame-level information compression on the retained events, first locating the central representative frame with the most valuable monitoring information from within the event, and then extracting this frame and its adjacent frames before and after from the original video to form a three-frame representative group.

[0033] This application defines a candidate segment as a valid event if it involves a valid change in the state of a person or electric bicycle relative to the initial reference scene of the candidate segment. Valid changes in the current experiment mainly include events such as people entering or leaving, and electric bicycles entering or leaving; candidate segments caused only by changes in lighting, shadows, reflections, background object disturbances, or slight camera shake, and not involving the aforementioned changes in the state of people or electric bicycles, are defined as invalid events.

[0034] Unlike general target recognition in open environments, fixed cameras maintain a stable viewpoint, spatial layout, and main background structure over long periods, resulting in strong background repetition in video data generated by the same camera. Long-term continuous data acquisition can therefore adequately cover the main operational states and typical interference patterns in the scene, including day-night changes, lighting changes, vehicle lights, shadows, background object disturbances, different target scales, and local occlusion. Based on this characteristic, this application does not directly perform isolated person / vehicle recognition on the current frame. Instead, it uses the scene state before the event as conditional information, transforming "whether there are people or electric bicycles in the current frame" into "whether there have been effective state changes related to people or electric bicycles relative to the initial reference scene." This task transformation reduces the interference of stable elements such as walls, parking spaces, pillars, and long-term stationary vehicles on the model's decision-making and constrains open visual recognition to scene change discrimination under specific background conditions.

[0035] For events deemed valid, the system constructs a candidate frame set of 11 images in chronological order within the event interval, and locates the central representative frame by visual and semantic fusion. Then, using this frame as a reference, the system extracts the preceding frame, the central frame, and the following frame from the original event video, forming a three-frame representative group. The uniform sampling of the 11 frames is only used to unify the candidate size for events of different lengths and is not considered an independent contribution to this application.

[0036] The aforementioned candidate segments can also be referred to as motion candidate segments or candidate short segments generated by the front end; the aforementioned reference background frame can also be referred to as reference scene or reference frame; the aforementioned sampled frame can also be referred to as the current frame; the aforementioned center representative frame can also be referred to as the center moment or key representative moment; the aforementioned continuous representative frame group can also be referred to as a continuous three-frame representative group.

[0037] The high repetition of the background from a fixed camera provides a clear scene prior for candidate segment selection. For the same camera, most of the background structure remains unchanged in long-term video, so the training data does not need to repeatedly learn the complete background, but can be organized around a limited range of scene changes. The total amount of the original continuous surveillance video is approximately 290 gigabytes (GB), and the front end uses a lightweight motion detector based on MOG2 to generate approximately 24GB of candidate segments, reducing the range of video data entering subsequent intelligent analysis by approximately 91.72%.

[0038] Specifically, the system can scale video frames by 0.3 and update the background model every 5 frames. Different foreground detection thresholds and minimum contour areas are set for indoor parking areas and outdoor entrances / exits: the variance threshold (varThreshold) and minimum contour area are 80 and 700 for indoor scenes, respectively, and 30 and 100 for outdoor scenes. After morphological denoising and connected component filtering, the foreground mask triggers a candidate event only when valid motion is detected within three consecutive processing times. The system also retains video content from 5 seconds before the event trigger and 5 seconds after the last valid motion to cover the main processes of target entry, stay, and departure. This stage is only used to narrow down the subsequent processing range with low computational cost; it does not determine whether the moving area contains real people or vehicles. Invalid candidates generated by lighting changes, shadows, reflections, and dynamic backgrounds are further filtered by the subsequent semantic module. For the candidate segment V generated by the front end, its first frame is fixed as the reference scene R (i.e., the aforementioned reference background frame), and the current frame at subsequent sampling times is denoted as... (i.e., the aforementioned sampled frame), the local change region constructed from the difference between the reference frame and the current frame is denoted as... The valid event filtering module, in conjunction with R, and The system determines whether candidate segments should be retained. For events deemed valid, the system extracts candidate frames according to a preset uniform rule within the event's time interval to construct a candidate frame set, and then uses visual and semantic fusion to sort and locate the central representative frame. Subsequently, using this frame as a reference, temporally adjacent frames are selected from the original video data to form a continuous representative frame group.

[0039] In this embodiment, the above-mentioned valid state changes mainly include events such as people entering or leaving, and electric bicycles entering or leaving; candidate segments caused only by changes in lighting, shadows, reflections, background object disturbances, or slight camera shake, and not involving the above-mentioned state changes of people or electric bicycles, are defined as invalid events.

[0040] This embodiment does not directly perform isolated person and vehicle recognition on the current frame. Instead, it uses the scene state before the event as conditional information, transforming "whether there are people or electric bicycles in the current frame" into "whether there have been effective state changes related to people or electric bicycles relative to the initial reference scene." This task transformation can reduce the interference of stable content such as walls, vehicle bodies, pillars, and long-term parked vehicles on the model's decision-making, and constrain open visual recognition to scene change discrimination under specific background conditions.

[0041] For events to be preserved, the system constructs a candidate frame set in chronological order, integrates visual features and structured semantic information to estimate the monitoring information value of each candidate frame, and determines the central representative frame through empirical location calibration. Finally, the central frame and its adjacent frames before and after it are extracted from the original event video to form a continuous representative frame group. The complete event segment is used for long-term video evidence preservation, while the continuous representative frame group is used for event indexing, quick preview, and manual review.

[0042] In this embodiment, the segment-level classification metric is evaluated only for candidate segments generated by MOG2. The recall rate reported in this application reflects the ability of the effective event screening model to retain real and valid events within the candidate set, and is not equal to the end-to-end event detection rate starting from the original continuous video.

[0043] The monitoring data used in this embodiment is stored and processed only in a controlled research environment. Raw videos and images that may contain personally identifiable information are not publicly disclosed. Experimental results are presented only as statistical indicators, flowcharts, and examples without identifiable information. The data processing follows the principle of minimum necessity, extracting only the information required to complete the filtering of valid events and the location of representative frames.

[0044] In the hierarchical storage method for surveillance videos based on reference background constraints and multimodal ranking proposed in this application, the original long video is first used to generate candidate segments through lightweight motion detection. Then, a fixed reference scene is constructed using the first frame of each candidate segment, transforming open target recognition into the discrimination of effective state changes of personnel or electric bicycles under reference background constraints. This application further utilizes the difference between the reference frame and the current frame to generate local change regions, and inputs the reference frame, the complete current frame, and the local change regions into a multimodal visual language model, achieving efficient parameter adaptation for specific surveillance scenes through low-rank adaptation (LoRA). The judgment results from multiple sampling moments are aggregated with segment-level evidence to form stable and effective event decisions. For retained events, the system constructs an 11-frame candidate set in chronological order, fuses visual features from an 18-layer residual network (ResNet18) with structured semantic features to estimate the surveillance information value of each candidate frame, and determines the central representative frame through posterior position calibration outside the network. Finally, the central frame and its adjacent frames before and after it are extracted from the original event video to form a three-frame representative group, serving as an event index and fast preview representation in the long-term video library.

[0045] This application uses an underground electric bicycle parking lot in a residential community as the current verification scenario. The data covers the concentrated parking area and entrances / exits, and includes typical issues such as long-term stationary vehicles, dense occlusion, small targets at a distance, human-vehicle interaction, scale changes, and lighting disturbances. This scenario is used to verify the effective event filtering and compact representation capabilities of the proposed method under fixed camera conditions. The formal experimental data covers 8 fixed cameras and four typical monitoring scenarios. Training, verification, and testing are isolated at the levels of raw video, acquisition time intervals, and independent events. The main contributions of this application include: This application proposes a hierarchical effective information extraction framework for long-term retention of fixed surveillance videos. It organizes lightweight motion candidate segment generation, effective event filtering, central representative frame location, and three consecutive representative frame output into a unified processing link. Among them, effective event filtering is used to reduce invalid candidate segments from entering long-term storage, and the central representative frame and three consecutive representative frame output are used to establish event indexes and improve the efficiency of fast browsing and manual review.

[0046] This application proposes an effective event filtering method based on reference scene conditions. It utilizes the characteristics of long-term stability and repetitive state of a fixed camera background to transform the recognition of absolute targets in the current image into the discrimination of effective state changes of people or electric bicycles relative to the initial reference scene of the event. Furthermore, it enhances information on small-scale, occluded, and edge targets by using local change regions.

[0047] This application proposes a center representative frame localization method that integrates visual and semantic fusion. It sorts candidate frames within an event by comprehensively considering target appearance, subject integrity, event interaction, and identity recognition cues, and alleviates the bias of fixed candidate positions through independent posterior position calibration. Based on this, it generates a group of three consecutive representative frames from the original video using deterministic rules, which serve as the system output for event indexing and fast preview.

[0048] The original surveillance resources covered approximately 34 fixed cameras in the underground electric bicycle parking area of ​​a residential community. Due to differences among the cameras in terms of video integrity, effective monitoring duration, field-of-view redundancy, and available annotation scale, this application screened the cameras based on data integrity, scene representativeness, and sample availability before model training and evaluation began, ultimately selecting 8 fixed cameras for the formal experiment. The selected cameras covered four typical scenarios: parking areas with relatively simple backgrounds, parking areas with complex backgrounds, entrances and exits within the community, and entrances and exits facing outdoors. The camera selection process did not refer to model prediction results, avoiding data selection based on test performance.

[0049] Among them, relatively simple background parking areas include scenes with relatively open passages, few stationary vehicles, or low degree of obstruction; complex background parking areas include densely parked vehicles and obvious obstruction; the main entrances and exits of the venue are people and electric bicycles entering and exiting, changes in target scale, and obstruction by railings; the entrances and exits facing the outdoors are also affected by dynamic background interference such as changes in natural light, swaying leaves, and insects.

[0050] The total volume of the original continuous surveillance video is approximately 290GB. The front-end uses lightweight motion detection to generate approximately 24GB of candidate segments to narrow down the scope of subsequent intelligent analysis. This module generates motion candidates solely based on pixel statistical changes and lacks the ability to semantically recognize targets such as people and vehicles, or to judge the value of events. Therefore, its output still needs to undergo subsequent effective event screening. Since this application did not perform complete manual annotation of the video intervals not triggered by MOG2, the segment-level classification metric is only evaluated based on the candidate segments generated by MOG2. Therefore, the Recall reported in this application reflects the ability of the effective event screening model to retain real and effective events within the candidate set, and is not equivalent to the end-to-end event detection rate starting from the original continuous video.

[0051] This application defines candidate video segments with independent start and end times as an independent event. For candidate segments originating from the same camera and the same original video, the overlap length of their time intervals and the intersection-union ratio (IU / R) of the bounding rectangles of their main motion regions are calculated. When the time interval overlap of two candidate segments is not less than 0.5 seconds and the IU / R of the bounding rectangles of their main motion regions is greater than 0.7, they are determined to be from the same source event. The reference frame, current frame, local change regions, and other derived inputs in the same source event are all assigned to the same data subset and are not split between the training set, validation set, and test set. Within the training set, duplicate samples are further filtered according to the above IU / R rules. Only one representative sample is retained for duplicate segments belonging to the same continuous target process to reduce sample redundancy and lower the risk of information leakage caused by the same source event.

[0052] The test events covered multiple fixed viewpoints across the eight official experimental cameras mentioned above, and were kept isolated from the training and internal validation data at the levels of raw video, acquisition time intervals, event boundaries, and derived images. Therefore, the 713 segment-level results reported in this application were obtained under conditions where the events, video, and temporal relationships were explicitly controlled. Since the current experiment did not establish a separate testing protocol for completely unseen cameras, this application does not interpret these results as broad generalization performance across all approximately 34 cameras or other unseen scenes.

[0053] The monitoring data used in this embodiment is stored and processed only in a controlled research environment. The original videos and images that may contain personally identifiable information are not publicly disclosed. Experimental results are presented only as statistical indicators, flowcharts, and examples without identifiable information. The data processing follows the principle of minimum necessity, extracting only the information required to complete the filtering of valid events and the localization of representative frames. The composition of the experimental data is shown in Table 1.

[0054] Table 1 Data phase Data scale Main uses Original continuous video Approximately 290GB Long-term monitoring video input Motion candidate fragments Approximately 24GB Narrowing the scope of subsequent intelligent analysis Validity judgment training and validation data 6513 image-level samples Scene adaptation Manually labeled fragment test set 713 independent candidate events Keep / Drop Fragment-Level Evaluation Representative frame dataset 786 events, 8646 images Center representative frame positioning

[0055] The sample distribution of the four types of verification scenarios is illustrated in Table 2. The test set mainly consists of relatively simple background parking areas and entrances / exits, while complex background parking areas are fewer in number but include challenging situations such as dense vehicle occlusion and partial target visibility. The camera-scene mappings mentioned above were determined before model evaluation and remained consistent throughout all experiments with the base model and the low-rank adaptation (LoRA) scene adaptation model. The scene distribution of manually annotated fragment-level test events is shown in Table 2.

[0056] Table 2 Verification Scenario number of events Keep Discard (drop) Relatively simple background parking area 247 154 93 Complex background parking area 20 10 10 Entrances and exits inside the venue 393 306 87 Outdoor entrances and exits 53 25 28 total 713 495 218

[0057] The validity determination uses a multimodal visual language model as the basic model. The first frame of each candidate segment is fixed as the reference frame, and subsequent frames are sampled at intervals of 3 frames. The local change region is generated by the reference frame and the current frame. The reference frame, the complete current frame, and the local change region together constitute the model input.

[0058] To evaluate the effectiveness of fixed monitoring scene adaptation, this application compares a multimodal visual language model with a low-rank adaptation scene adaptation model. Both models use the same reference frame, current frame, and local change region inputs, sampling strategies, and fragment-level aggregation rules; the only difference is whether a scene adapter is loaded. The representative frame dataset consists of 786 events, each containing 11 candidate images constructed in chronological order, totaling 8646 images. Manual annotation focuses on evaluating the relative monitoring information value among candidate frames within the same event. If a unique best frame can be reliably determined, the list-level ranking supervision for that event is retained; if medium-value frames and worst-value frames can also be distinguished, a further candidate partial order is constructed. The "invalid" (unrankable) status in the representative frame annotations is not business invalidity, but rather an annotation state resulting from insufficient overall discriminative power among the 11 candidate frames for the same event, making it impossible for manual determination of a unique best. Such events are not included in the training, validation, and rigorous test metric calculation of the ranking model.

[0059] After manual review, 464 events had usable ranking supervision information. Among them, 225 events could be clearly identified as Best, Medium, and Worst partial orders, and were denoted as Rankable events; 239 events could only be reliably identified as Best, and were denoted as Low-separation events. After excluding one event that overlapped with the frozen test video group, the supervision data was divided into 359 training events, 47 validation events, and 57 test events according to the source videos. The remaining 322 events were not included in the formal supervision division, of which 38 were marked as unrankable, and the remaining 284 events did not form formal manual ranking labels. These events did not participate in model training, validation, test metric calculation, or position prior estimation, but were still used for system-level inference and generation of representative groups of three consecutive frames.

[0060] The representative frame ranking model uses a 224×224 letterbox scale input, extracts visual features from a frozen ResNet18 dataset, and fuses structured semantic features generated by a multimodal visual language model. The model locates the center representative frame among 11 candidate images. The system then extracts adjacent frames based on the center representative frame's temporal position in the original video, forming a three-frame representative group. The model learns and evaluates based on the center representative frame's position, while the three-frame representative group serves as the deterministic system output for event indexing, quick browsing, and manual review.

[0061] The maximum number of training epochs for the frame fusion ranking model was set to 30, and an early stopping strategy with a patience value of 7 was adopted. Training logs were recorded up to the 18th training epoch, and the final test used the best.pt file saved during the validation phase, corresponding to the 11th epoch.

[0062] Furthermore, based on the reference background frame corresponding to each candidate segment and each sampled frame of each candidate segment, a local change region corresponding to each sampled frame is constructed, including: The reference background frame corresponding to each candidate segment and the sampled frame of each candidate segment are subjected to size unification and brightness normalization respectively to obtain the normalized reference background frame and the normalized sampled frame. The normalized reference background frame and the normalized sampled frame are converted into background grayscale image and sampled grayscale image, respectively. The background grayscale image and sampled grayscale image are then smoothed using a Gaussian kernel of the first preset size to obtain the smoothed reference background frame and the smoothed sampled frame. Calculate the absolute difference between the smoothed reference background frame and the smoothed sampled frame, and amplify the absolute difference with a preset enhancement coefficient to obtain the difference-enhanced image; The differential enhancement image is binarized using a preset segmentation threshold to generate a binary transformation mask; The binary transformation mask is subjected to morphological opening and morphological closing operations sequentially using a structuring element of the second preset size to obtain the processed mask. Multiple connected regions are extracted from the processed mask, and connected regions with an area smaller than a preset area threshold are filtered out to obtain candidate connected regions. From the candidate connected regions, retain a first preset number of valid regions in descending order of area, and merge the bounding rectangles of the retained valid regions to obtain the merged bounding rectangle; Using the center of the merged bounding rectangle as a reference, the width of the merged bounding rectangle is expanded to a first preset expansion multiple, and the height of the merged bounding rectangle is expanded to a second preset expansion multiple, thus obtaining an expanded bounding rectangle; The bounding rectangle is restricted to a preset effective range of the image to obtain the cropped bounding rectangle; The image region enclosed by the cropping rectangle is scaled to a first preset resolution and used as the local variation region.

[0063] To improve the comparability between the reference frame and the current frame, the two images are first sized and luminance normalized to obtain the normalized reference background frame and the normalized sampled frame.

[0064] The normalized reference background frame and the normalized sampled frame are converted into background grayscale images and sampled grayscale images, respectively. A 5×5 Gaussian kernel (i.e., a Gaussian kernel of the first preset size) is used to smooth the grayscale images to obtain the smoothed reference background frame and the smoothed sampled frame.

[0065] The absolute difference between the smoothed reference background frame and the smoothed sampled frame is calculated, and the differential response is amplified with an enhancement coefficient of 1.8 (i.e., the preset enhancement coefficient) to obtain the differential enhanced image.

[0066] The differential enhancement image is binarized using a threshold of 25 (i.e., a preset segmentation threshold) to generate a binary transformation mask.

[0067] The binary transformation mask is subjected to morphological opening and closing operations sequentially using a 3×3 structuring element (i.e., a structuring element of the second preset size) to obtain the processed mask, which removes discrete noise and connects local broken regions.

[0068] Multiple connected regions are extracted from the processed mask. Regions with an area less than 25 pixels (i.e., the aforementioned preset area threshold) are filtered out, and the top 5 regions with the largest area (i.e., the first preset number) are retained to obtain candidate connected regions.

[0069] When multiple valid regions exist, their bounding rectangles are merged to obtain a merged bounding rectangle.

[0070] Using the center of the merged outer rectangle as a reference, the width and height of the rectangle are expanded to 2.5 times the original size (i.e., the first preset expansion multiple and the second preset expansion multiple, which are the same in this embodiment, both being 2.5), to obtain the expanded outer rectangle.

[0071] The bounding rectangle is restricted to the effective range of the image, and the part that exceeds the image boundary is truncated to the corresponding image edge to obtain the cropped bounding rectangle.

[0072] Finally, the image area enclosed by the cropped bounding rectangle is scaled to a resolution of 224×224 (i.e., the first preset resolution) as the local variation area.

[0073] Furthermore, based on the local change regions corresponding to each sampling time of each candidate segment, the judgment result for each sampling time is determined, including: The reference background frame corresponding to each sampling time and the sampling frame corresponding to each sampling time are uniformly scaled to the second preset resolution, and the local change area corresponding to each sampling time is scaled to the third preset resolution to construct multi-image multimodal input data. Here, the sampling time corresponds to a certain candidate segment, and the reference background frame corresponding to each sampling time is the reference background frame corresponding to this candidate segment.

[0074] Multi-image, multi-modal input data and text instructions are input into the multimodal visual language model. The text instructions are used to instruct the multimodal visual language model to use the reference background frame corresponding to each sampling time as the initial state of the event, and to determine whether the sampling frame corresponding to each sampling time has undergone effective state changes of the person and / or the electric bicycle relative to the reference background frame. A multimodal visual language model is used to perform inference based on multi-image, multimodal input data and text commands, and the generated text is output. A deterministic decoding method is used to extract the last retained or discarded tag in the generated text, and the judgment result (i.e., whether to retain or discard) is determined based on the extracted retained or discarded tag at each sampling time. If no retained or discarded tags are extracted from the generated text, the decision at that sampling moment will be reverted to discard.

[0075] The effective sampling time is used to construct multi-image, multi-modal input data in the order of reference frame, complete current frame, and local change region.

[0076] Specifically, the reference background frame corresponding to each sampling time and the sampling frame corresponding to each sampling time are uniformly scaled to a resolution of 640×640 (i.e., the second preset resolution mentioned above), and the local change area corresponding to each sampling time is scaled to a resolution of 224×224 (i.e., the third preset resolution mentioned above) to construct multi-image multimodal input data.

[0077] Multi-image, multimodal input data and text instructions are fed into a multimodal visual language model. The text instructions instruct the model to use the reference background frame at each sampling time as the initial state of the event, determine whether there are valid state changes for the person and / or the electric bicycle relative to the reference background frame at each sampling time, and use local change regions as dynamic information to aid in the judgment. The generated result is limited to "keep" or "drop". The aforementioned "keep" and "drop" labels correspond to the "keep" and "drop" labels, respectively.

[0078] A multimodal visual language model is used to perform inference based on multi-image, multimodal input data and text commands, and output generated text.

[0079] A deterministic decoding method is used to extract the last "keep" or "drop" tag from the generated text, and the judgment result at each sampling time is determined based on the extracted tag.

[0080] If the output does not contain a valid, parseable label, the judgment result at that sampling time will be uniformly rolled back to drop.

[0081] Furthermore, the multimodal visual language model is adapted to the scene through a preset adapter. The rank of the preset adapter is set to a preset rank parameter, the scaling parameter of the preset adapter is set to a preset scaling parameter, and the dropout rate of the preset adapter is set to a preset dropout rate parameter. The target modules of the preset adapter include the attention projection layer and the linear layer in the feedforward network.

[0082] The aforementioned multimodal visual language model is adapted to the scene via a preset adapter. In this embodiment, the preset adapter is a low-rank adaptation (LoRA) adapter.

[0083] The LoRA adapter's rank r is set to 64 (i.e., the preset rank parameter mentioned above), the scaling parameter is set to 128 (i.e., the preset scaling parameter mentioned above), and the dropout rate is set to 0.05 (i.e., the preset dropout rate parameter mentioned above).

[0084] The target modules of the preset adapter include the attention projection layer and the linear layer (also known as the main linear layer) in the feedforward network.

[0085] The two subtasks employ independent model structures and training processes. The effective event selection model is based on a multimodal visual language model and a LoRA scene adapter, while the representative frame localization model uses frozen ResNet18 (18-layer residual network) visual features, structured semantics from the multimodal visual language model, and a lightweight ranking head. The test results for the former are based on checkpoint-1466 (globalstep=1466) saved at the end of the first epoch, where checkpoint is the check point and globalstep is the global step count. The latter uses best.pt (optimal weight file) saved during the validation phase. A summary of the training and testing configurations for both models is shown in Table 3. "Not Applicable" indicates that this parameter is only used in the representative frame localization phase and is not part of the effective event selection model.

[0086] Table 3 project Effective event filtering model Representative frame localization model Model Composition Multimodal visual language model + LoRA Freeze ResNet18+ multimodal visual language model semantics + ranking head Final test model checkpoint-1466 best.pt (Epoch 11) Maximum number of training rounds 3 epochs 30 epochs Actual training status Epoch 1, Step 1466 Log recording up to Epoch 18 Batch size 1. Gradient accumulation 4 2 Optimizer Adaptive Moment Estimation Optimizer Variant (AdamW) AdamW Initial learning rate 5×10^-5 1×10^-4 Weight decay 0 1×10^-4 Early stop strategy Unrecorded Patience value = 7 Ranking loss interval and weight not applicable m=0.2，β=0.2 Posterior position calibration coefficient not applicable λ=-0.04

[0087] In Table 3, m is the sorting interval, β is the pairwise sorting loss weight, and λ is the posterior position calibration coefficient; the meanings and determination methods of these three are detailed in Formulas (2) and (3) of the instruction manual. The validity discrimination model is pre-trained for 3 epochs, and the model performance is monitored using an internal validation set during training. At the end of the first epoch, the training loss and validation loss of the model are 0.000158 and 0.000053, respectively; the overall accuracy of the validation set sampling evaluation reaches 99.00%, of which the accuracy of the keep class and drop class are 98.68% and 100.00%, respectively. In the subsequent training observation phase, the model validation performance did not show a significant improvement. Considering the validation loss, sampling evaluation results, and model training stability, checkpoint-1466 (globalstep=1466) saved at the end of the first epoch was finally selected as the inference weight for the testing phase. All validity discrimination test results are obtained based on this weight, and the test set does not participate in model selection.

[0088] Furthermore, based on the judgment results of multiple sampling times within each candidate segment, a set of valid events is determined, including: Within each candidate segment, multiple samples are taken at a preset sampling interval. The number of sampling moments that are determined to be retained is counted. When the number reaches a preset retention threshold, the candidate segment is determined to be retained and added to the set of valid events.

[0089] The high repetition of the background in a fixed camera provides a clear scene prior for candidate segment selection. For the same camera, most background structures remain unchanged over long-term video, and training data does not need to repeatedly learn the complete background but can be organized around a limited set of scene changes. This application fixes the first frame of each candidate segment as the reference scene and keeps the reference frame unchanged during segment processing. In this way, the model's discrimination criterion is no longer the absolute presence of the target in the current frame, but whether the person or electric bicycle has undergone a valid state change relative to the initial reference scene of the event, thereby avoiding the misidentification of long-term stationary vehicles, fixed facilities, or other stable background content as event targets.

[0090] To improve the comparability between the reference frame and the current frame, the two images are first subjected to size unification and luminance normalization, and a 5×5 Gaussian kernel is used to smooth the grayscale images. Then the absolute difference between the reference frame and the current frame is calculated, the difference response is amplified with an enhancement coefficient of 1.8, and a binary change mask is generated using a threshold of 25. The change mask is sequentially subjected to morphological opening operation and closing operation using a 3×3 structuring element to remove discrete noise and connect locally broken regions. When screening connected regions, regions with an area smaller than 25 pixels are filtered out, and the top 5 valid regions with the largest areas are retained; when there are multiple valid regions, their bounding rectangles are merged. To preserve the spatial context around the target, based on the center of the merged bounding rectangle, the width and height of the rectangle are respectively expanded to 2.5 times the original. The expanded boundary coordinates are limited within the valid range of the image, and the part beyond the image boundary is truncated to the corresponding image edge. Finally, the obtained local change region is scaled to 224×224 as the local dynamic input of the model.

[0091] The effective sampling moments are input into the multi-modal vision-language model in the order of the reference frame, the complete current frame, and the local change region. The reference frame represents the background benchmark before the event occurs, the current frame retains the target appearance and the complete scene context, and the local change region highlights the position where the state change of a person or an electric bicycle may occur. The three types of information correspond to background prior, global semantics, and local dynamic evidence respectively, and have a complementary effect, enabling the model to perform reference-conditional discrimination under complex backgrounds. At each sampling moment, the system constructs a multi-image multi-modal input once in the fixed order of the reference frame R, the current frame I_t, and the local change region C_t. The reference frame and the current frame are uniformly scaled to 640×640, the local change region is scaled to 224×224, and the tensor conversion and normalization of the images are completed by the vision processor supporting the multi-modal vision-language model. The text instruction requires the model to take the reference frame as the initial state of the event, judge whether an effective state change related to a person or an electric bicycle has occurred in the current frame relative to the reference scene, take the local change region as dynamic evidence to assist the judgment, and limit the generated result to keep or drop. In the inference stage, deterministic decoding is adopted, and the program extracts the last valid keep or drop label appearing in the generated text; if no parsable valid label is contained in the output, the result is uniformly fallback to drop.

[0092] Considering that the single-frame result may be affected by short-term occlusion, motion blur or incomplete local change regions, the present application performs evidence aggregation on the determination results of multiple sampling moments at the clip level.

[0093] Let the set of sampling moments in the clip V be S (t is the index of the sampling moment), and the corresponding frame-level determination be , then the clip-level result is defined as: (1)

[0094] Where 𝕀(•) is the indicator function, and K is the fragment retention threshold. This application samples subsequent frames in the candidate fragment every 3 frames, and fixes the fragment aggregation threshold to K=3 before formal testing. When at least 3 sampling moments within a fragment are determined to be keep, the fragment is ultimately determined to be retained; otherwise, it is determined to be discarded. The 3 keep results that meet the condition do not need to appear consecutively in time. Therefore, this aggregation method can tolerate a small number of frame-level missed detections caused by short-term occlusion, motion blur, or incomplete local change regions, while avoiding the direct triggering of fragment-level retention by occasional noise or erroneous output at a single sampling moment. Smaller K values ​​are more susceptible to isolated misjudgments, while excessively large K values ​​increase the amount of evidence required to obtain retention determinations for short-lived, distant small targets, and edge targets. Considering both the stability of fragment-level determination and the requirement for real-world event retention, this application uses K=3 as a fixed aggregation parameter.

[0095] For events that pass the valid event screening, the system uniformly extracts 11 candidate frames along the timeline based on the start and end positions of the event, and numbers them according to the original chronological order. The candidate images are scaled to 224×224 using Letterbox, preserving the complete image while maintaining a uniform input size and avoiding cropping of pedestrians or vehicles near the image edges. This process converts events of varying durations into a fixed-size candidate set, from which the subsequent ranking model locates the central representative frame that best represents the event content. The system then extracts the preceding frame, the central frame, and the following frame based on the frame's position in the original video, forming a three-frame representative group. The overall structure is as follows: Figure 3 As shown. It should be noted that the 11-frame uniform sampling is only used to unify the candidate size of events of different lengths and is not considered an independent contribution to this embodiment.

[0096] Specifically, multiple samplings are performed within each candidate segment at a preset sampling interval of 3 frames, and the number of sampling moments that are determined to be retained is counted.

[0097] When the number reaches the preset retention threshold K=3, the candidate fragment is determined to be retained and added to the valid event set; otherwise, it is determined to be discarded.

[0098] Of the 713 independent candidate events, 495 involved valid state changes related to people or electric bicycles, while 218 were triggered solely by environmental changes or background disturbances, resulting in invalid candidate events accounting for 30.58%. This composition indicates that pixel-change-based motion detection can compress the processing range of continuous video, but it lacks the ability to judge target semantics and event value. Its output still needs to be filtered for valid events before entering the long-term storage link.

[0099] Table 4 shows the results of the base model and the LoRA scene adaptation model on the same manually annotated fragment-level test set, i.e., the results of valid event screening.

[0100] Table 4 method Accuracy Precision Recall rate F1 score False positive rate (FPR) Specificity Multimodal visual language model 97.90% 97.81% 99.19% 98.50% 5.05% 94.95% This application method 99.02% 99.80% 98.79% 99.29% 0.46% 99.54%

[0101] The multimodal visual language model yielded 491 true positives (TP), 11 false positives (FP), 4 false negatives (FN), and 207 true negatives (TN). After LoRA scene adaptation, the method in this application yielded 489 TP, 1 FP, 6 FN, and 217 TN. Accuracy improved from 97.90% to 99.02%, Precision from 97.81% to 99.80%, and F1 score from 98.50% to 99.29%; the number of false positives decreased from 11 to 1, and the FPR decreased from 5.05% to 0.46%.

[0102] From the processing results of the long-term storage link, the method in this application retains 490 motion candidate segments from 713, reducing the candidate size by 31.28%; at the same time, it correctly removes 217 out of 218 invalid candidates and retains 489 out of 495 real and valid events. This result shows that the proposed method can improve the effective information density in the long-term video library while maintaining a high retention rate of real events. It should be noted that the 31.28% only represents the screening ratio at the 713 motion candidate segment level and is not equivalent to the final storage compression rate of the 290GB original video; the change from 290GB to approximately 24GB reflects the change in data scale during the front-end candidate segment generation stage.

[0103] The main improvement brought about by scene adaptation is the suppression of repetitive environmental disturbances. All 11 false positives generated by the base model came from changes in illumination, while only 1 false positive from an illumination change was retained after LoRA scene adaptation. This indicates that the model learned the difference between common lighting changes under a fixed camera and changes in the actual target state through scene data.

[0104] The recall rate for this application was 98.79% on 713 MOG2 candidate events. This result indicates that the model correctly retained 489 out of 495 valid events triggered by MOG2 and included in the candidate set; this metric does not include missed events that may exist in video intervals where MOG2 was not triggered. The 6 false negatives included 4 new person events and 2 new person / vehicle events. Combined with manual review, these errors mainly occurred with small targets at a distance, dense vehicle occlusion, targets that were only partially visible, and targets that passed through the edge of the frame for a short time. The results of valid event selection in different validation scenarios are shown in Table 5 (N is the number of test events in each validation scenario): Table 5 Verification Scenario N keep drop Accuracy Recall F1 Relatively simple background parking area 247 154 93 98.79% 98.70% 99.02% Complex background parking area 20 10 10 85.00% 70.00% 82.35% Entrances and exits inside the venue 393 306 87 100.00% 100.00% 100.00% Outdoor entrances and exits 53 25 28 98.11% 96.00% 97.96% overall 713 495 218 99.02% 98.79% 99.29%

[0105] The relatively simple background parking area contained 247 samples, and the model achieved an accuracy of 98.79% and a recall of 98.70%. Although this type of scene has some lighting variations and stationary vehicles, the visible range of the passage is relatively large and the occlusion is relatively limited. The reference frame can provide a relatively stable background benchmark for effective state change discrimination.

[0106] The entrances and exits within the venue contain 393 samples, making it the largest sub-scene in the test set. The method presented in this application achieved 100% accuracy and recall on the current test set, indicating that the reference scene constraints perform remarkably well in handling situations such as personnel and electric bicycles entering and exiting, changes in target scale, and occlusion by railings.

[0107] The outdoor entrance / exit is simultaneously affected by changes in natural light, swaying leaves, insects, and other dynamic backgrounds. The model's accuracy in this scene is 98.11%, and its recall is 96.00%, indicating that the scene adaptation can filter out most environmental disturbances in the outdoor direction. The remaining errors are mainly related to objects passing by quickly or being located at the edge of the frame.

[0108] The complex background parking area sample contains 20 samples, 10 for keep and 10 for drop. The accuracy for this scene is 85.00%, and the recall is 70.00%, significantly lower than other scenes. Densely parked vehicles create extensive occlusion, and people or electric bicycles undergoing state changes are often only visible in localized areas, making it difficult to provide sufficient target information from either the complete current frame or locally changed regions. Due to the small sample size of this subset, the results are more suitable for revealing error concentration trends in difficult scenes than for simple absolute performance comparisons with large-scale scenes.

[0109] Furthermore, a candidate frame set is constructed from the valid event set, and the visual and semantic features of each candidate frame in the candidate frame set are extracted, including: For each valid event in the valid event set, a second preset number of candidate frames are extracted within the time interval of each valid event according to a preset uniform rule, and the second preset number of candidate frames are scaled to a preset input size to construct a candidate frame set. A pre-trained convolutional neural network with frozen parameters is used to extract the visual features of the first preset dimension of each candidate frame; Semantic features are extracted through the following steps: A third preset number of structured semantic facts are extracted by a multimodal visual language model based on the interrelationship information of each candidate frame in the candidate frame set. The structured semantic facts include numerical attributes and category attributes. Numerical attributes are clipped and normalized, and categorical attributes are encoded to form a semantic vector with a second preset dimension; wherein, clipping refers to restricting the numerical attributes to a preset value range; The semantic vectors are mapped through a multilayer perceptron with a predefined layer structure to obtain the semantic features of the third predefined dimension; The structured semantic facts include at least one of the following: number of people, number of vehicles, main target, event type, target interaction relationship, subject integrity, visibility of identity recognition clues, target scale, degree of occlusion, edge truncation, and image quality.

[0110] The target localization of the center representative frame differs from general image quality evaluation. The clearest image in a monitored event may not necessarily contain the most complete event information. Therefore, this embodiment prioritizes the value of event information over simple image quality, giving priority to target appearance, number of people or vehicles, target interaction, entry or exit status, subject integrity, and identifiable information such as faces or license plates. When the event content is similar, the target scale, occlusion, edge truncation, and image sharpness are then combined for ranking.

[0111] Visual feature extraction employs a pre-trained convolutional neural network with frozen parameters. In this embodiment, a ResNet18 with frozen parameters is used to extract 512-dimensional (i.e., the first preset dimension) visual features from each candidate frame, primarily encoding appearance information such as target structure, subject scale, sharpness, occlusion status, and edge truncation. Freezing the visual backbone can reduce the risk of overfitting on small-scale manually labeled data and supports pre-computed visual features to improve training efficiency.

[0112] Among them, the interrelated information is the multi-frame input context of the same valid event; the determination of the multi-frame input context includes: inputting each candidate frame in the candidate frame set into the multimodal visual language model to form the multi-frame input context; In one implementation, for each valid event in the valid event set, a second preset number of candidate frames are extracted within the time interval of the valid event according to a preset uniform rule to construct a candidate frame set. Each candidate frame belonging to the same valid event is used as a multi-frame input sample and input into a multimodal visual language model to form a multi-frame input context for the same valid event; the multi-frame input context refers to the interrelated information. The multi-frame input sample includes multiple candidate frames for the same valid event, and each candidate frame retains its corresponding image content. After receiving the multi-frame input sample, the multimodal visual language model performs association understanding on each candidate frame within the same valid event range, outputs a third preset number of structured semantic facts according to preset output items, and parses each output result to obtain the corresponding numerical attribute or category attribute. The structured semantic facts include numerical attributes and category attributes; the numerical attributes include at least one of the following: number of people, number of vehicles, target scale, degree of occlusion, and image quality; the category attributes include at least one of the following: main target, event type, target interaction relationship, subject integrity, visibility of identity recognition clues, and edge truncation status.

[0113] The parsed numerical attributes are trimmed and normalized; trimming refers to restricting the numerical attributes to a preset value range. The parsed categorical attributes are encoded. The processed numerical attributes and categorical attributes are combined into a second preset dimension semantic vector, and then the semantic vector is mapped through a multilayer perceptron with a preset layer structure to obtain a third preset dimension semantic feature. In a preferred embodiment, the third preset dimension is 18, the second preset dimension is 63, and the third preset dimension is 64; the preset layer structure includes an input layer, a hidden layer, and an output layer. The input dimension of the input layer is 63, the hidden layer includes a first hidden layer with 96 nodes and a second hidden layer with 64 nodes, and the output dimension of the output layer is 64; the categorical attributes are encoded using one-hot encoding.

[0114] Furthermore, based on the visual and semantic features of each candidate frame, the candidate frames are ranked by value to determine the central representative frame, including: Based on the visual and semantic features of each candidate frame, the fused representation is determined; After the fused representation is projected through a fully connected layer of the fourth preset dimension, the original scores of each candidate frame are generated by the shared linear scoring head, and the candidate frames are ranked by value based on the original scores. The training objectives for the shared linear scoring head include: For valid events with both manually best frame annotation and manually multi-level sorting annotation, a list-level cross-entropy loss is used to make the original score of the manually annotated best candidate frame higher than the original scores of other candidate frames. Pairwise sorting loss is applied to candidate frame pairs with different sorting levels to make the difference in the original scores of candidate frame pairs with different sorting levels satisfy the preset boundary constraints. The pairwise sorting loss takes the form of hinge loss. For valid events with manually best frame annotations but no manually multi-level sorted annotations, list-level cross-entropy loss is used. For valid events marked as unsortable, list-level cross-entropy loss and pairwise sorting loss are not used.

[0115] Based on the 512-dimensional visual features and 64-dimensional semantic features of each candidate frame, the 512-dimensional visual features, 64-dimensional semantic embedding, and 1-dimensional semantic availability tag of each candidate frame are concatenated into a 577-dimensional fusion representation, and the fusion representation is determined.

[0116] The fused representation is projected onto a fully connected layer with a fourth preset dimension, and then the original scores for each candidate frame are generated by a shared linear scoring head. The candidate frames are then ranked by value based on their original scores. In this embodiment, the aforementioned fully connected layer with the fourth preset dimension includes a 256-dimensional fully connected layer and a 128-dimensional fully connected layer. The fused representation is first projected onto the 256-dimensional fully connected layer, then further processed by the 128-dimensional fully connected layer, and finally the original scores for each candidate frame are generated by the shared linear scoring head. The network does not accept candidate numbers, temporal location encodings, or manual location priors; therefore, the visual and semantic ranking network itself evaluates only the candidate images and their structured semantic information. The shared scoring function ensures that the 11 candidate positions use a consistent discrimination criterion, and semantic availability tags are used to distinguish between true zero semantics and semantic information missing.

[0117] The training objectives of the aforementioned shared linear scoring head include: for valid events with both manually labeled best and manually ranked (Medium, Worst) frames, employing list-level cross-entropy loss (L_rank) to ensure that the original score of the manually labeled best candidate frame is higher than the original scores of other candidate frames; and applying pairwise ranking loss (L_pairwise) to candidate frame pairs at different ranking levels to ensure that the difference in the original scores of candidate frame pairs at different ranking levels satisfies a preset boundary constraint. In this embodiment, the aforementioned pairwise ranking loss adopts the hinge loss form, with a ranking interval m=0.2 and a pairwise ranking loss weight β=0.2.

[0118] For valid events with manually optimal frame annotations but no manually multi-level sorting annotations (i.e., low-discrimination events), only the above-mentioned list-level cross-entropy loss is used.

[0119] For valid events marked as invalid, the list-level cross-entropy loss and the pairwise sorting loss are not used.

[0120] The ranking model consists of a visual branch and a semantic branch. The visual branch uses a frozen ResNet18 to extract 512-dimensional visual features from each candidate image, mainly encoding appearance information such as target structure, subject scale, sharpness, occlusion, and edge truncation. Freezing the visual backbone can reduce the risk of overfitting on small-scale manually labeled data and supports pre-computed visual features to improve training efficiency.

[0121] The semantic branch invokes the multimodal visual language model within the 11-frame context of the same event to extract 18 structured facts, including the number of people, the number of vehicles, the main target, the event type, target interaction, subject integrity, face or license plate visibility, target scale, occlusion, edge truncation, and image quality. Numerical attributes are cropped and normalized, while categorical attributes are encoded using one-hot encoding, ultimately forming a 63-dimensional semantic vector. This vector is then processed through a 63-96-64 multilayer perceptron to obtain a 64-dimensional semantic embedding. In the manual annotations, Best, Medium, and Worst represent the most preferred, medium-valued, and least-valued candidate frames, respectively. The Best, Medium, or Worst suggestions directly output by the multimodal visual language model are not used as supervision labels; the training signal for the ranking model comes solely from manual confirmation results.

[0122] The 512-dimensional visual features, 64-dimensional semantic embedding, and 1-dimensional semantic availability tag for each candidate frame are concatenated into a 577-dimensional fused representation. This representation is then projected through 256-dimensional and 128-dimensional fully connected layers, and the original score for the candidate frame is generated by a shared linear scoring head. The network does not accept candidate numbers, temporal location encodings, or manual location priors; therefore, the visual and semantic ranking network itself evaluates solely based on the candidate image and its structured semantic evidence. The shared scoring function ensures that the 11 candidate locations use a consistent discrimination criterion, and the semantic availability tag is used to distinguish between true zero semantics and semantic information missing.

[0123] The training objective utilizes both manually assigned "Best" labels and the confirmed relative order of candidates. For events with manually assigned "Best" labels, list-level cross-entropy is used to drive the best candidate to achieve the highest score; when both "Medium" and "Worst" labels are available, further interval ordering constraints are imposed on "Best>Medium", "Best>Worst", and "Medium>Worst". The comprehensive loss is defined as: (2)

[0124] in, A set of events with effective ordering supervision within a batch. Let P be the number of active events; P is the set of ordered candidate pairs determined by manual annotation. The number of candidate pairs for the activity; The label weight for event n. This represents the model output score for the i-th candidate frame in event n. This indicates the manually labeled Best candidate position; for ordered candidate pairs (n, i, j) ∈ P, the manually ranked candidate i is better than candidate j. For weighted list-level cross-entropy loss, The weighted hinge pairwise ranking loss is used; both losses are normalized based on the actual number of activity samples involved in the supervision, and zero-weight samples are not included in the denominator. For events where a unique Best can be reliably identified but Medium and Worst cannot be stably distinguished, only list-level cross-entropy is calculated; for events marked as unrankable, neither list-level nor pairwise ranking loss is calculated. This application sets the ranking interval m=0.2 and the pairwise ranking loss weight β=0.2. In the final formal training, the business effectiveness auxiliary loss weight is set to 0, therefore the ranking model is optimized using only the above list-level cross-entropy and pairwise ranking loss. Manually label the optimal frame position in event n. The model outputs a score at that location. This represents the model output score for the j-th candidate frame in event n.

[0125] The formal ranking model does not include a business validity auxiliary task. Input events have already been filtered for validity by the front end; the model only learns the ranking of monitoring information value among 11 candidate frames. Unrankable states in the annotations are only used to identify events for which a unique best supervision cannot be reliably established, and are not included in the model's business classification output.

[0126] The maximum number of training epochs for the frame fusion ranking model was set to 30, and an early stopping strategy with a patience of 7 was adopted. Training logs were recorded up to the 18th epoch, and the final test used the optimal weight file (best.pt) saved during the validation phase, corresponding to the 11th epoch.

[0127] The comparison of different methods across 57 freeze test events is illustrated in Table 6. Figure 4 As shown, Table 6 presents the center representative frame localization results on the frozen test set. Figure 4The diagram shows a comparison of Hit@1 (hit rate @1) and Hit@3 (hit rate @3) for different center representative frame localization methods on the frozen test set. The vertical axis represents Hit@1 (hit rate @1), indicating the proportion of manually annotated best frames that are ranked first by the model, and Hit@3 (hit rate @3), indicating the proportion of manually annotated best frames that are ranked in the top three by the model. On the horizontal axis, the fixed position baseline (Pos-7) represents the method that always determines the 7th candidate position as the center representative frame, the semantic-only model represents the model that only uses structured semantic features for ranking, the visual-only model represents the model that only uses visual features for ranking, the visual and semantic fusion model represents the model that uses visual and semantic fusion features for ranking, and the final model represents the model after adding posterior position calibration on the basis of visual and semantic fusion.

[0128] Table 6 method Hit@1 Hit@3 Average last ranking Normalized loss cumulative gain Sorting affects accuracy Fixed position baseline 15.79% — — — — semantics only 19.30% 40.35% 0.3714 0.5156 22.52% Visual only 33.33% 61.40% 0.5277 0.6404 80.18% Visual and semantic fusion 33.33% 64.91% 0.5317 0.6435 81.08% Final Model 36.84% 66.67% 0.5543 0.6607 81.08%

[0129] The Hit@1 of the fixed-location baseline is 15.79%, indicating that the human-made Best matches exhibit a certain distribution pattern in terms of time and location. However, relying solely on fixed candidate locations cannot reliably pinpoint the moment with the highest value of monitored information within an event. The Hit@1 of the semantic model alone is 19.30%, with a ranking accuracy of 22.52%, indicating that high-level semantics such as event type, number of people, and number of vehicles are insufficient to distinguish approximate candidate frames within the same short-term event.

[0130] The visual model alone improved Hit@1 to 33.33% and achieved a ranking accuracy of 80.18%, indicating that target scale, subject integrity, occlusion level, edge truncation, and sharpness are the main factors for center representative frame localization. After incorporating structured semantics from the multimodal visual language model, Hit@1 remained unchanged, but Hit@3 improved from 61.40% to 64.91%, with improvements in mean reciprocal ranking (MRR), normalized depreciation cumulative gain (NDCG@11), and ranking accuracy. This suggests that semantic information is more suitable as a supplement to visual appearance, used to express the number of people and vehicles, target interactions, and identity cues, rather than replacing fine-grained visual differences.

[0131] After adding posterior position calibration following the visual and semantic fusion scores, Hit@1 improved to 36.84%, Hit@3 to 66.67%, and MRR and NDCG@11 reached 0.5543 and 0.6607, respectively. Calibration improved Hit@1 by 3.51 percentage points compared to the original fusion model without changing the ranking pair accuracy, which is consistent with its functional boundary: this step corrects the overall positional bias formed by fixed candidate sampling without relearning the visual or semantic relationships between candidate frames.

[0132] Unsortable events, lacking a reliable and unique Best label, are not included in the calculation of center representative frame positioning metrics such as Hit@1, Hit@3, MRR, and NDCG@11. The ranking metrics reported in this application are all based on freeze test events whose Best positions can be reliably confirmed manually.

[0133] Furthermore, after ranking the candidate frames by value based on their original scores, the method also includes: The effective events with manually labeled best frames in the effective event set are divided into training events and validation events; the set of candidate frames in each training event is determined as the training set, and the set of candidate frames in each validation event is determined as the validation set. Specifically, the partitioning is performed based on the source video: events originating from the same source video are grouped together, and then each group is assigned to either training or validation events, ensuring that the same event and all its derived images do not cross between the training and validation sets. Assigning each group to either training or validation events involves: obtaining the start time of the capture time period corresponding to each group's source video; sorting the groups according to the order of their start times; determining the partitioning position in the sorting results; assigning groups before the partitioning position to training events; and assigning groups after the partitioning position to validation events. Events not included in the training and validation sets do not participate in prior position estimation but are still used for system-level inference and generation of representative groups for three consecutive frames.

[0134] In other embodiments, before the division, the manual annotations of each event are reviewed according to their sortability. The review results can be divided into three categories: events that can clearly give a three-level ranking of the best candidate frame, the medium candidate frame and the lowest candidate frame are recorded as sortable events; events that can accurately identify the best candidate frame but cannot stably distinguish the ranking among the other candidate frames are recorded as low-discrimination events; and events that have insufficient overall discrimination of candidate frames and cannot accurately determine the unique best candidate frame are recorded as unsortable events.

[0135] For example, after manual review, 464 events had usable ranking supervision signals. Among them, 225 events could clearly provide a three-level ranking of the best candidate frame, intermediate candidate frames, and lowest candidate frames, and were denoted as ranking events; 239 events could only accurately identify the best candidate frame and were denoted as low-discrimination events. After excluding one event that overlapped with the frozen test video, the remaining 463 events were divided into 359 training events, 47 validation events, and 57 test events according to the source video. The remaining 322 events did not enter the formal supervision division, of which 38 events were in an unrankable state, and the remaining 284 events did not form formal manual ranking labels. These events did not participate in model training, validation, test metric calculation, or position prior estimation, but were still used for system-level inference and generation of three consecutive frame representative groups. The number of times each candidate frame in the training set was manually labeled as the best candidate frame was counted to determine the position prior. On the validation set, the calibration coefficients are determined through a grid search with a preset search range and a preset search step size; The original scores of each candidate frame are weighted and summed with the prior position of the corresponding candidate frame position according to the calibration coefficient to obtain the calibration score; the center representative frame is determined based on the calibration score, specifically, the candidate frame with the highest calibration score is selected as the center representative frame.

[0136] After completing the visual and semantic fusion ranking, the system performs lightweight posterior position calibration on the 11 raw scores outside the network. Fixed-time sampling may result in a non-uniform distribution of manually assigned best labels (also known as manually selected best frame annotations) across candidate positions. Therefore, this application only uses manually assigned labels from the training set to estimate a smooth position prior and determines the calibration coefficients on the validation set. The test set is not involved in position prior estimation, coefficient selection, or other model adjustments. Given the network raw score of the i-th candidate frame... and the standardized logarithmic position prior obtained from the artificial best position distribution of the training set. The calibration score and the position of the center representative frame are as follows: (3)

[0137] Where i is the candidate frame number. The network's original score for the i-th candidate frame. Let λ be the normalized logarithmic position prior for the i-th candidate frame position, and λ be the calibration coefficient determined by the validation set. Let be the calibration score of the i-th candidate frame, and let i* be the calibration score among all candidate frames. The position of the largest candidate frame, i.e. the position of the center representative frame.

[0138] The location prior is estimated solely from the manually best distribution of the training set: First, the number of times each of the 11 candidate locations is manually best is counted, and the location probabilities are obtained by additive smoothing with a factor of 0.5. Then, the logarithm is taken and standardized over the 11 locations to obtain the probability. . The calibration coefficient is determined on the validation set through a grid search with the interval [-1.0, 1.0] and a step size of 0.01, and finally λ = -0.04 is selected. The test set does not participate in position prior estimation or calibration coefficient selection. This step is an independent post-processing applied to the original ranking scores, which does not update network weights, nor does it change the visual encoding, semantic encoding or feature fusion process.

[0139] In the system deployment phase, the central representative frame is first determined based on the calibrated candidate scores. Considering that a single image may be affected by factors such as instantaneous occlusion, motion blur and local reflection, the system further extracts this frame and its adjacent preceding and following frames from the original event video to form a representative group of three consecutive frames. Assume that a valid event contains N frames, and the position of the central representative frame is the k-th frame. When N ≤ 3, all available frames in the event are directly output; when N > 3 and 1 < k < N, the (k-1)-th frame, the k-th frame and the (k+1)-th frame are output; when the central representative frame is the first frame of the event, the entire output window is shifted backward, and the 1st to 3rd frames are output; when the central representative frame is the last frame of the event, the entire output window is shifted forward, and the (N-2)-th to N-th frames are output. Through the above boundary adjustment, when the total number of frames of the event is no less than 3, the system always outputs 3 temporally consecutive images located within the range of the valid event, avoiding out-of-bounds indices or invalid empty frames. As a result, the complete method forms a processing link consisting of candidate segment generation, reference scene constrained valid event filtering, visual and semantic fusion scoring, posterior position calibration, central representative frame positioning, and output of the representative group of three consecutive frames. The complete event segment is used for long-term evidence preservation, and the representative group of three consecutive frames is used for event indexing and quick preview.

[0140] The central representative frame is determined according to the calibration score. This calibration step is an independent post-processing applied to the original ranking scores, which does not update network weights, nor does it change the visual encoding, semantic encoding or feature fusion process.

[0141] Surveillance videos have strong temporal redundancy. For the same short-term event, the background structure usually remains unchanged, and the appearance, position and behavior state of persons or vehicles at adjacent moments may only undergo slight changes. Therefore, among the 11 candidate images, there are often multiple candidate frames with highly similar content and close surveillance information value. For events where the Best can still be determined, strict Hit@1 not only requires the model to find high-value candidates, but also requires its top-ranked position to be completely consistent with the unique position selected manually, thus having strong selectivity.

[0142] It is necessary to distinguish between low-discrimination events and unrankable states. Although the candidate frames for low-discrimination events have small differences, they can still be reliably identified as the best by humans; therefore, their list-level supervision is retained and they are included in the ranking evaluation. Unrankable states indicate that there are no differences among the 11 candidate frames sufficient to support a reliable and unique best label; therefore, they are not included in ranking supervision and rigorous metric calculation. Table 7 shows the final model performance data under different event discrimination levels. The 37 Rankable events and 20 Low-separation events in Table 7 together constitute the 57 freeze test events.

[0143] Table 7 test subset number of events Hit@1 Hit@3 Rankable 37 48.65% 72.97% Low-separation 20 15.00% 55.00%

[0144] For rankable events where candidate frames have clear differences, the final model achieves Hit@1 and Hit@3 scores of 48.65% and 72.97%, respectively. For low-separation events where candidate frames are highly similar but the Best can still be reliably identified, Hit@1 drops to 15.00%, but Hit@3 still reaches 55.00%. The results indicate that when candidate frames have discernible differences in subject integrity, occlusion state, or event stage, the model can stably locate the center representative frame. When multiple candidates have only slight pose or positional changes, the model's first-ranked position is more likely to deviate from the manually selected Best.

[0145] Overall, Hit@3 is significantly higher than Hit@1, indicating that the model can rank manually verified high-value images among the top candidates in most events. However, determining the unique highest position among multiple approximate frames remains quite challenging. This result is consistent with the prevalent phenomenon of numerous repeated frames and semantically similar adjacent frames in short-term surveillance videos. The frozen test set contains 57 events, and the final Wilson 95% confidence interval for Hit@1 is 25.52%–49.82%. Therefore, this application interprets the changes before and after calibration as a performance improvement trend, rather than drawing statistical significance conclusions.

[0146] Based on the high temporal redundancy of surveillance videos, the system does not directly output a single center frame as a complete event. Instead, it further extracts its preceding and following frames from the original video, forming a group of three consecutive representative frames. The center representative frame is used to locate key moments in the event with high monitoring information value, while adjacent frames are used to supplement changes in the target state near the center moment. This design mitigates the potential loss of single-frame information caused by momentary occlusion, motion blur, or local reflections. For events marked as unsortable, the system can still determine a center time position based on the sorting score and generate a group of three consecutive representative frames; however, due to the lack of a reliable and unique best human label, such results are not included in the calculation of strict single-frame positioning indicators.

[0147] Further, determining the event index of the stored data based on the central representative frame includes: Taking the central representative frame as a reference, temporally adjacent frames are selected from the video data corresponding to the valid event set, and the temporally adjacent frames and the central representative frame form a continuous representative frame group, including: When the total number of frames of the valid event is not greater than a fourth preset number, all frames of the valid event are selected as the continuous representative frame group; When the total number of frames of the valid event is greater than the fourth preset number and the central representative frame is located at the middle position of the event, the central representative frame and adjacent frames of a fifth preset number before and after the central representative frame are selected to form the continuous representative frame group; When the central representative frame is located at the first frame of the event, consecutive frames of a sixth preset number at the start end of the event are selected to form the continuous representative frame group; When the central representative frame is located at the last frame of the event, consecutive frames of a seventh preset number at the tail end of the event are selected to form the continuous representative frame group; The continuous representative frame group is used as the event index and preview data of the stored data.

[0148] Taking the central representative frame as a reference, temporally adjacent frames are selected from the video data corresponding to the valid event set, and the temporally adjacent frames and the central representative frame form a continuous representative frame group.

[0149] Assuming that the valid event includes N frames, and the position of the central representative frame is the k-th frame: When the total number of frames of the valid event is not greater than 3 (that is, the fourth preset number is 3), all frames of the valid event are directly selected as the continuous representative frame group; When the total number of frames of the valid event is greater than 3 and the central representative frame is located at the middle position of the event (that is, 1<k<N), the central representative frame and 1 adjacent frame before and after the central representative frame (that is, the fifth preset number is 1) are selected to form the continuous representative frame group, and the (k-1)-th frame, the k-th frame and the (k+1)-th frame are output; When the central representative frame is located at the first frame of the event (k=1), the entire output window is moved backward, consecutive 3 frames at the start end of the event (that is, the sixth preset number is 3) are selected to form the continuous representative frame group, and the 1st frame to the 3rd frame are output; When the central representative frame is located at the last frame of the event (k=N), the entire output window is moved forward, consecutive 3 frames at the tail end of the event (that is, the seventh preset number is 3) are selected to form the continuous representative frame group, and the (N-2)-th frame to the N-th frame are output.

[0150] Through the above boundary adjustment, when the total number of event frames is not less than 3, the system always outputs 3 temporally consecutive images located within the range of the valid event, so as to avoid out-of-bounds indexes or invalid empty frames.

[0151] The aforementioned consecutive representative frame groups serve as event indexes and preview data for stored data, used for event indexing, quick browsing, and manual review; while complete event segments are preserved as long-term video evidence.

[0152] In one embodiment of the present invention, the above method can be implemented through the following system deployment: the system receives long-term monitoring video captured by a fixed camera, and sequentially completes the generation of lightweight motion candidate segments, the screening of effective events constrained by the reference scene, the localization of the center representative frame through visual and semantic fusion, the generation of consecutive representative frame groups, and the indexing of event results. MOG2 only undertakes candidate segment generation and does not participate in target semantic classification; the LoRA scene adaptation model is responsible for determining whether the candidate segment has undergone effective state changes related to people or electric bicycles relative to the reference scene; the ranking model is responsible for locating the center representative frame.

[0153] The system-level inference process handled 786 events and 8646 candidate images, generating a center temporal location result and a group of consecutive representative frames for each event. This processing scale is used to verify the executability of the complete pipeline and does not constitute a claim to real-time performance or provide untested throughput and latency. The group of consecutive representative frames does not alter the ranking evaluation of the center representative frame, nor does it replace the complete video evidence provided by the retained event segments.

[0154] The training and testing of the validity discrimination model were both performed on a single NVIDIA RTX PRO 6000 Blackwell GPU with approximately 96 gigabytes (GB) of video memory. The software environment consisted of Python 3.10.20, PyTorch 2.11.0 (CUDA 12.8), Transformers 5.2.0, and PEFT 0.19.1, with NVIDIA driver version 580.119.02. During the online and batch inference stages, the multimodal visual language model's base model and LoRA (low-rank adaptation) scene adapter were loaded through a separate vLLM inference service framework. vLLM primarily handled inference services related to the multimodal visual language model; feature extraction and ranking of the 18-layer ResNet18 residual network were performed by a separate Python ranking pipeline. This deployment environment does not participate in model training, validation, or the calculation of experimental metrics in this application.

[0155] Regarding data construction, this embodiment defines candidate video segments with independent start and end times as an independent event. For candidate segments originating from the same camera and the same original video, the overlap length of their time intervals and the intersection-union ratio (IU / R) of the bounding rectangles of their main motion regions are calculated. When the time interval overlap of two candidate segments is not less than 0.5 seconds and the IU / R of the bounding rectangles of their main motion regions is greater than 0.7, they are determined to be events from the same source. The reference frame, current frame, local change regions, and other derived inputs in the same source event are all assigned to the same data subset and are not split between the training set, validation set, and test set. Within the training set, duplicate samples are further filtered according to the above IU / R rules. Only one representative sample is retained for multiple segments belonging to the same continuous target process to reduce sample redundancy and lower the risk of information leakage caused by events from the same source.

[0156] Data partitioning was first completed at the levels of original video, acquisition time period, and independent events. Then, image-level inputs were constructed from each data subset according to a unified sampling rule. The training and internal validation data were derived from multiple independent candidate events, resulting in 6513 image-level samples, including 4945 keep samples and 1568 drop samples. By scene, the parking area contained 4297 samples, including 3356 keep samples and 941 drop samples; the entrance / exit contained 2216 samples, including 1589 keep samples and 627 drop samples. After grouping by source video, 5861 training samples and 652 internal validation samples were formed; the same event and all its derived images did not cross between the training and internal validation sets. After grouping at the source video and independent event levels, the sample order within each subset was randomized using a fixed random seed of 42. The internal validation set was used for monitoring the training process.

[0157] The manually labeled list contained 716 candidate segments, of which 3 segments were marked as uncertain and excluded due to the target being too small or insufficient image information. The final segment-level test set consisted of 713 independent candidate events, of which 495 events involved valid state changes related to people or electric bicycles relative to the initial reference scene and were marked as keep; the remaining 218 events did not involve the aforementioned target state changes, or were only triggered by changes in lighting, shadows, background objects, and other environments, and were marked as drop. The test events covered multiple fixed viewpoints from the 8 official experimental cameras and were kept isolated from the training and internal validation data at the levels of raw video, acquisition time intervals, event boundaries, and derived images. Therefore, the 713 segment-level results reported in this application were obtained under conditions where the relationships between events, video, and time were clearly controlled.

[0158] It should be noted that, due to differences in video integrity, effective monitoring duration, field-of-view redundancy, and available annotation scale among different cameras, this embodiment selected cameras based on data integrity, scene representativeness, and sample availability before model training and evaluation began. Ultimately, 8 cameras were selected from approximately 34 fixed cameras for the formal experiment. The selected cameras covered four typical scenarios: parking areas with relatively simple backgrounds, parking areas with complex backgrounds, parking lot entrances / exits, and entrances / exits facing outdoors. The camera selection process did not refer to model prediction results, avoiding data selection based on test performance.

[0159] Among them, the relatively simple background parking area includes scenes with relatively open passages, fewer prohibited parking vehicles, or less obstruction; the complex background parking area includes densely parked vehicles and obvious obstruction; the entrances and exits mainly involve people and electric bicycles entering and exiting, changes in target scale, and temporary obstruction; the entrances and exits facing the outdoors are also affected by dynamic background interference such as changes in natural light, swaying tree shadows, and insects.

[0160] This application focuses on fixed surveillance video applications with storage times exceeding three months, conducting two experiments: effective event screening and rapid event retrieval. First, the candidate segments generated by front-end motion detection are evaluated for validity, selecting events showing valid state changes related to personnel or electric bicycles relative to the initial reference scene. Then, a central representative frame with high monitoring information value is located from the retained events, and a three-frame representative group is generated based on this frame. For these two tasks, corresponding datasets and evaluation methods are constructed to assess the performance of segment-level validity screening and the performance of locating representative moments within events.

[0161] For segment-level valid event screening, this application sets `keep` as the positive class and uses Accuracy, Precision, Recall, F1 score, False Positive Rate (FPR), and Specificity to evaluate model performance. Recall measures the ability to retain true valid state change events within the MOG2 candidate set, while FPR and Specificity measure the model's ability to filter invalid candidates such as illumination changes and background disturbances. Since the video intervals not triggered by MOG2 were not fully manually annotated, the Recall reported in this application does not represent the end-to-end event detection rate on the original continuous video. To analyze the role of fixed scene adaptation, experiments compare the basic multimodal visual language model with the LoRA scene adaptation model. Both models use the same input format, sampling strategy, and segment-level aggregation rules; the only difference is whether a LoRA adapter trained on fixed monitoring scene data is loaded. Front-end motion detection is only used to generate candidate segments and is not included in the performance comparison as a classification model.

[0162] It should be noted that Hit@3 and the continuous representative frame group output by the system correspond to different evaluation objects and business objectives. Hit@3 is only used to measure whether the manually annotated Best candidate frame is among the top three in the ranking of 11 candidate images. It evaluates the single-frame localization capability of the central representative frame, corresponding to the selection of a limited number of optimal frames within the candidate set. The continuous representative frame group, on the other hand, is formed by extracting the central representative frame and its adjacent frames from the original event video after the central representative frame is determined. It is not equivalent to the top three images in the candidate set, nor is it limited to the top three (Top 3) results of the 11-frame candidate set. This output is used to supplement local temporal information near the central moment, supporting event indexing, quick browsing, and manual review in the long-term video library. The two do not use the same accuracy metric.

[0163] It should be noted that the results of this application are based on 8 fixed cameras pre-selected from approximately 34 initial monitoring resources. They mainly reflect the effective event filtering and representative frame localization capabilities in the selected fixed scenarios and are not directly extrapolated to the generalization performance of all approximately 34 cameras or other unseen scenarios.

[0164] The current experiment has not yet established an end-to-end event miss evaluation on the original continuous video. Further evaluation of the miss risk in the front-end candidate generation stage is needed by sampling or fully annotating the video intervals that were not triggered by MOG2.

[0165] The system receives long-term surveillance video from fixed cameras and sequentially performs the following steps: generating lightweight motion candidate segments, filtering effective events constrained by the reference scene, locating the center representative frame through visual and semantic fusion, generating a three-frame representative group, and indexing the event results. MOG2 only generates candidate segments and does not participate in target semantic classification; the LoRA scene adaptation model determines whether the candidate segments have undergone effective state changes related to people or electric bicycles relative to the reference scene; the ranking model locates the center representative frame. The three-frame representative group is mainly used for indexing and quick previewing.

[0166] The total volume of the original continuous monitoring video used in the experiment was approximately 290GB. After being processed by MOG2, approximately 24GB of candidate segments were generated, reducing the range of video data entering subsequent intelligent analysis by approximately 91.72%. This result indicates that lightweight motion candidate generation can significantly reduce the range of video data requiring large model analysis; however, this application does not directly interpret the file size ratio as a proportional reduction in actual GPU computing power or inference time.

[0167] The system input includes the original surveillance video, candidate segments generated by the front end, and reference frames and sampled frames constructed from the candidate segments. The system output includes keep / drop segment-level determination results, the position of the center representative frame, a representative group of three consecutive frames, and structured records that can be used for event indexing.

[0168] This application addresses the issues of false triggering of motion candidates in long-term storage of fixed surveillance videos and frame-level redundancy in rapid retrieval of valid events. It constructs a hierarchical processing framework encompassing lightweight motion candidate segment generation, LoRA scene adaptation model for valid event filtering with reference scene constraints, visual and semantic fusion for central representative frame localization, and output of a three-frame representative group. Complete valid event segments are used for long-term evidence preservation, while the central representative frame and the three-frame representative group are used for event indexing, rapid previewing, and manual verification.

[0169] On 713 candidate events generated by MOG2 and manually annotated, the LoRA scene adaptation model achieved an accuracy of 99.02%, an F1 score of 99.29%, and an FPR of 0.46%. Its main benefit was reducing false positives caused by repetitive lighting variations under a fixed camera. On 57 frozen test events, the final representative frame model achieved a Hit@1 accuracy of 36.84% and a Hit@3 accuracy of 66.67%. Experiments also showed that visual features such as target scale, subject integrity, and occlusion are the main basis for fine-grained inter-frame ranking, while structured semantics provides supplementary information such as event type, target quantity, and interaction information. Posterior position calibration further alleviated the bias of fixed candidate positions.

[0170] The main challenges currently lie in the occlusion caused by densely parked vehicles, small targets at long distances and edge targets, and the similar information value of multiple candidate frames due to high temporal redundancy. A group of three consecutive representative frames is used to supplement local temporal information near the central moment to support event indexing, quick browsing, and manual verification in the long-term video library, but it does not replace the video evidence provided by complete event segments. It should be noted that the results of this application are based on eight fixed cameras pre-selected from approximately 34 initial monitoring resources, mainly reflecting the effective event screening and representative frame localization capabilities in the selected fixed scenes, and are not directly extrapolated to broad generalization performance for all monitoring perspectives or unseen scenes. The current experiment has not yet established an end-to-end event recall evaluation on the original continuous video. Further evaluation of the missed detection risk in the front-end candidate generation stage is needed by sampling or fully annotating the MOG2-untriggered video intervals.

[0171] According to another aspect of the embodiments of this application, a hierarchical storage system for surveillance video based on reference background constraints and multimodal sorting is also provided. The system includes: a construction module, configured to determine multiple sampling times for each candidate segment corresponding to the surveillance video according to a preset sampling strategy, acquire sampling frames corresponding to each sampling time, and construct local change regions corresponding to each sampling frame based on the reference background frame corresponding to each candidate segment and each sampling frame of each candidate segment; wherein the preset sampling strategy includes: for each candidate segment, using the first frame of the candidate segment as the reference background frame of the candidate segment, and determining multiple sampling times within the candidate segment at the same frame interval based on the reference background frame of the candidate segment; and a determination module, configured to determine the sampling times based on each candidate segment... The system is divided into several modules: a local change region corresponding to each sampling time of a segment, and a judgment result for each sampling time. The judgment result is used to indicate the effective state changes of personnel and / or electric bicycles. A candidate frame determination module is used to determine the effective event set based on the judgment results of multiple sampling times within each candidate segment, and to construct a candidate frame set from the effective event set. A center representative frame determination module is used to extract the visual and semantic features of each candidate frame in the candidate frame set. Based on the visual and semantic features of each candidate frame, the candidate frames are ranked by value to determine the center representative frame. A hierarchical storage module is used to determine the video data corresponding to the effective event set as the storage data, and to determine the event index of the storage data based on the center representative frame.

[0172] It should be noted that the embodiments implemented in the hierarchical storage system for surveillance videos based on reference background constraints and multimodal sorting in this application can be referenced to the embodiments implemented in the hierarchical storage method for surveillance videos based on reference background constraints and multimodal sorting, and will not be described in detail here.

[0173] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A hierarchical storage method for surveillance video based on reference background constraints and multimodal sorting, characterized in that, include: According to a preset sampling strategy, multiple sampling times are determined for each candidate segment of the surveillance video, and sampling frames corresponding to each sampling time are obtained. Based on the reference background frame corresponding to each candidate segment and each sampling frame of each candidate segment, a local change region corresponding to each sampling frame is constructed. The preset sampling strategy includes: for each candidate segment, the first frame of the candidate segment is used as the reference background frame of the candidate segment, and multiple sampling times are determined within the candidate segment at the same frame interval based on the reference background frame of the candidate segment. Based on the local change region corresponding to each sampling time of each candidate segment, the judgment result for each sampling time is determined; wherein, the judgment result is used to indicate the effective state change of the person and / or the effective state change of the electric bicycle; Based on the determination results of multiple sampling times within each candidate segment, a set of valid events is determined, and a set of candidate frames is constructed from the set of valid events. Extract the visual and semantic features of each candidate frame in the candidate frame set; based on the visual and semantic features of each candidate frame, rank the candidate frames by value to determine the central representative frame. The video data corresponding to the valid event set is determined as the stored data, and the event index of the stored data is determined based on the central representative frame.

2. The hierarchical storage method for surveillance video based on reference background constraints and multimodal sorting according to claim 1, characterized in that, The step of constructing a local change region corresponding to each sample frame based on the reference background frame corresponding to each candidate segment and each sample frame of each candidate segment includes: The reference background frame corresponding to each candidate segment and the sampled frame of each candidate segment are subjected to size unification and brightness normalization respectively to obtain the normalized reference background frame and the normalized sampled frame. The normalized reference background frame and the normalized sampled frame are converted into a background grayscale image and a sampled grayscale image, respectively. The background grayscale image and the sampled grayscale image are then smoothed using a Gaussian kernel of a first preset size to obtain a smoothed reference background frame and a smoothed sampled frame. Calculate the absolute difference between the smoothed reference background frame and the smoothed sampled frame, and amplify the absolute difference with a preset enhancement coefficient to obtain a difference-enhanced image; The differential enhancement image is binarized using a preset segmentation threshold to generate a binary transformation mask; The binary transformation mask is subjected to morphological opening and morphological closing operations sequentially using a structuring element of a second preset size to obtain the processed mask. Multiple connected regions are extracted from the processed mask, and connected regions with an area smaller than a preset area threshold are filtered out to obtain candidate connected regions. From the candidate connected regions, retain a first preset number of valid regions in descending order of area, and merge the bounding rectangles of the retained valid regions to obtain the merged bounding rectangle; Using the center of the merged outer rectangle as a reference, the width of the merged outer rectangle is expanded to a first preset expansion multiple, and the height of the merged outer rectangle is expanded to a second preset expansion multiple, to obtain an expanded outer rectangle; The boundary coordinates of the extended bounding rectangle are restricted to a preset effective range of the image to obtain the cropped bounding rectangle; The image region enclosed by the cropped bounding rectangle is scaled to a first preset resolution and used as the local variation region.

3. The hierarchical storage method for surveillance video based on reference background constraints and multimodal sorting according to claim 1, characterized in that, The determination of the judgment result for each sampling time based on the local change region corresponding to each sampling time of each candidate segment includes: The reference background frame corresponding to each sampling time and the sampling frame corresponding to each sampling time are uniformly scaled to the second preset resolution, and the local change area corresponding to each sampling time is scaled to the third preset resolution to construct multi-image multimodal input data. The multi-image multimodal input data and text instructions are input into the multimodal visual language model; wherein, the text instructions are used to instruct the multimodal visual language model to use the reference background frame corresponding to each sampling time as the initial state of the event, and to determine whether the sampling frame corresponding to each sampling time has undergone an effective state change of the person and / or an effective state change of the electric bicycle relative to the reference background frame. The multimodal visual language model is used to perform inference based on the multi-image multimodal input data and the text instructions, and output generated text. A deterministic decoding method is used to extract the last retained or discarded tags in the generated text, and the judgment result at each sampling time is determined based on the extracted retained or discarded tags; If no retained or discarded tags are extracted from the generated text, the determination result at that sampling moment will be reverted to discard.

4. The hierarchical storage method for surveillance video based on reference background constraints and multimodal sorting according to claim 3, characterized in that, The multimodal visual language model is adapted to a scene via a preset adapter. The rank of the preset adapter is set to a preset rank parameter, the scaling parameter of the preset adapter is set to a preset scaling parameter, and the dropout rate of the preset adapter is set to a preset dropout rate parameter. The target module of the preset adapter includes an attention projection layer and a linear layer in the feedforward network.

5. The hierarchical storage method for surveillance video based on reference background constraints and multimodal sorting according to claim 1, characterized in that, The determination of the set of valid events based on the judgment results of multiple sampling times within each candidate segment includes: Within each candidate segment, multiple samples are taken at a preset sampling interval. The number of sampling moments that are determined to be retained is counted. When the number reaches a preset retention threshold, the candidate segment is determined to be retained and added to the set of valid events.

6. The hierarchical storage method for surveillance video based on reference background constraints and multimodal sorting according to claim 3, characterized in that, Constructing a candidate frame set from the set of valid events, and extracting the visual and semantic features of each candidate frame from the candidate frame set, including: For each valid event in the set of valid events, a second preset number of candidate frames are extracted according to a preset uniform rule within the time interval of each valid event, and the second preset number of candidate frames are scaled to a preset input size to construct the set of candidate frames; A pre-trained convolutional neural network with frozen parameters is used to extract the visual features of the first preset dimension of each candidate frame; The semantic features are extracted through the following steps: The multimodal visual language model extracts a third preset number of structured semantic facts based on the interrelationship information of each candidate frame in the candidate frame set. The structured semantic facts include numerical attributes and category attributes. The interrelated information is a multi-frame input context for the same valid event; the determination of the multi-frame input context includes: inputting each candidate frame in the candidate frame set into the multimodal visual language model to form the multi-frame input context; The numerical attributes are clipped and normalized, and the categorical attributes are encoded to form a semantic vector with a second preset dimension; wherein, the clipping refers to restricting the numerical attributes to a preset value range; The semantic vector is mapped through a multilayer perceptron with a preset layer structure to obtain semantic features in a third preset dimension; The structured semantic facts include at least one of the following: number of people, number of vehicles, main target, event type, target interaction relationship, subject integrity, visibility of identity recognition clues, target scale, degree of occlusion, edge truncation, and image quality.

7. The hierarchical storage method for surveillance video based on reference background constraints and multimodal sorting according to claim 6, characterized in that, The process of ranking candidate frames based on their visual and semantic features to determine the central representative frame includes: Based on the visual and semantic features of each candidate frame, the fused representation is determined; After the fused representation is projected through a fully connected layer of the fourth preset dimension, the original scores of each candidate frame are generated by a shared linear scoring head, and the candidate frames are ranked by value according to the original scores. The training objectives of the shared linear scoring head include: For valid events with both manually best frame annotation and manually multi-level sorting annotation, a list-level cross-entropy loss is used to make the original score of the manually annotated best candidate frame higher than the original scores of other candidate frames. Pairwise sorting loss is applied to candidate frame pairs with different sorting levels so that the difference in the original scores of candidate frame pairs with different sorting levels satisfies the preset boundary constraints. The pairwise sorting loss adopts the hinge loss form. For valid events with manually optimal frame annotations but no manually multi-level sorting annotations, the list-level cross-entropy loss is used. For valid events marked as unsortable, the list-level cross-entropy loss and the pairwise sorting loss are not used.

8. The hierarchical storage method for surveillance video based on reference background constraints and multimodal sorting according to claim 7, characterized in that, After ranking the candidate frames by value based on the original scores, the method further includes: The effective events with manually labeled best frames in the effective event set are divided into training events and validation events; the set of candidate frames in each training event is determined as the training set, and the set of candidate frames in each validation event is determined as the validation set; the number of times each candidate frame in the training set is manually labeled as the best candidate frame is counted to determine the position prior; on the validation set, the calibration coefficient is determined by grid search with a preset search interval and a preset search step size. The original scores of each candidate frame are weighted and summed with the prior position of the corresponding candidate frame position according to the calibration coefficient to obtain the calibration score; the center representative frame is determined based on the calibration score.

9. The hierarchical storage method for surveillance video based on reference background constraints and multimodal sorting according to claim 1, characterized in that, Determining the event index of the stored data based on the central representative frame includes: Using the central representative frame as a reference, temporally adjacent frames are selected from the video data corresponding to the effective event set, and the temporally adjacent frames and the central representative frame are combined to form a continuous representative frame group, including: When the total number of valid event frames is not greater than the fourth preset number, all frames of valid events are selected as the continuous representative frame group. When the total number of valid event frames is greater than the fourth preset number and the center representative frame is located in the middle of the event, the center representative frame and its five preset number of adjacent frames before and after it are selected to form the continuous representative frame group. When the central representative frame is located at the first frame of the event, the sixth consecutive preset number of frames at the beginning of the event are selected to form the consecutive representative frame group. When the central representative frame is located at the end of the event, the seventh consecutive preset number of frames at the end of the event are selected to form the consecutive representative frame group. The consecutive representative frame groups serve as event indexes and preview data for the stored data.

10. A hierarchical storage system for surveillance video based on reference background constraints and multimodal sorting, characterized in that, include: The construction module is used to determine multiple sampling times for each candidate segment of the surveillance video according to a preset sampling strategy, obtain sampling frames corresponding to each sampling time, and construct local change regions corresponding to each sampling frame based on the reference background frame corresponding to each candidate segment and each sampling frame of each candidate segment; wherein, the preset sampling strategy includes: for each candidate segment, taking the first frame of the candidate segment as the reference background frame of the candidate segment, and determining multiple sampling times within the candidate segment at the same frame interval based on the reference background frame of the candidate segment; The determination module is used to determine the determination result for each sampling time based on the local change region corresponding to each sampling time of each candidate segment; wherein, the determination result is used to indicate the effective state change of the person and / or the effective state change of the electric bicycle; The candidate frame determination module is used to determine the set of valid events based on the judgment results of multiple sampling times within each candidate segment, and to construct a set of candidate frames from the set of valid events. The center representative frame determination module is used to extract the visual and semantic features of each candidate frame in the candidate frame set; and to rank the candidate frames by value based on the visual and semantic features of each candidate frame to determine the center representative frame. The hierarchical storage module is used to determine the video data corresponding to the valid event set as the storage data, and to determine the event index of the storage data based on the central representative frame.

Images