A video anomaly detection method based on cloud-edge collaborative semantic sparse coding

Abstract

The application discloses a video anomaly detection method based on cloud-edge collaborative semantic sparse coding, comprising the following steps: S1, deploying a multi-modal large model comprising a semantic coding module, an adaptive network and a semantic decoding module on the cloud, storing a semantic prototype matrix on the edge side, and deploying a trained lightweight semantic coding model; S2, acquiring an image sequence, inputting the lightweight semantic coding model, and obtaining a semantic sparse feature vector corresponding to a time window; S3, calculating a semantic change measure of adjacent time windows, and when the semantic change measure is higher than or equal to a threshold value, generating a sparse feature data packet and uploading the sparse feature data packet to the cloud; S4, processing the sparse feature data packet through the adaptive network and the semantic decoding module, obtaining an anomaly detection result, and outputting abnormal information or an alarm. The application replaces full-volume video data with semantic sparse features, reduces the demand for edge computing power and the communication burden, and improves the practicability and scalability of video anomaly detection in complex environments.

Description

Technical Field

[0001] This invention relates to the field of visual anomaly detection, and more particularly to a video anomaly detection method based on cloud-edge collaborative semantic sparse coding. Background Technology

[0002] Video anomaly detection is a fundamental problem in industrial vision, requiring systems to automatically identify events deviating from normal operating conditions, such as equipment defects, environmental hazards, and personnel intrusions, within continuous video streams. It has significant application value in ensuring production safety and reducing the cost of manual inspections. Compared to traditional static tasks such as image classification and object detection, video anomaly detection needs to consider both dynamic changes in the temporal dimension and complex structures in the spatial dimension. Anomalies often exhibit characteristics such as uncertainty, rarity, and strong scene dependence, leading to higher modeling challenges.

[0003] In outdoor infrastructure inspection scenarios such as railway lines, power transmission lines, and petrochemical pipeline corridors, fixed-position cameras or drones have been widely used for video acquisition of line conditions to monitor equipment integrity, changes in the surrounding environment, and operational safety. However, in remote areas such as mountains and wilderness, front-end equipment often relies on solar energy or batteries for power, resulting in limited computing and storage resources. Video data is typically uploaded via IoT cards, making link bandwidth and communication costs highly sensitive. Furthermore, the limited size of available server racks or edge computing boxes on-site makes deploying large-scale GPUs or entire large-model inference systems at monitoring points impractical in engineering.

[0004] Existing engineering solutions mainly fall into two categories:

[0005] One type of method directly transmits the original video or high-bitrate compressed video back to the central server or cloud for unified analysis, and performs anomaly detection through traditional convolutional networks or temporal convolutional networks. This type of method performs well in environments with sufficient computing power, but when a large number of front-end nodes work simultaneously, it will cause link congestion and cloud computing power pressure, making it difficult to adapt to the cost and energy consumption constraints of large-scale deployment.

[0006] Another approach deploys lightweight detection models at the edge, uploading only the detected key frames or cropped regions to identify predefined targets such as falling rocks, people, and vehicles. However, due to the high uncertainty of anomaly categories in outdoor industrial scenarios and the lack of large-scale, comprehensive labeled samples, these fixed-category lightweight model solutions are significantly insufficient in identifying rare anomalies, combined scenarios, and new hazards, often requiring frequent re-labeling and retraining.

[0007] In recent years, visual models and multimodal large models based on large-scale data pre-training have made significant progress in tasks such as open-world recognition, video understanding, and text description. They can meticulously characterize semantic relationships in complex scenes and are theoretically very suitable for industrial video anomaly detection. However, these large models have a massive parameter scale, and their requirements for computing power, memory, and power consumption are far higher than traditional models, usually only deployable in centralized cloud or data centers. However, uploading massive amounts of front-end video frames frame by frame or at high frequency to the cloud for full inference will not only cause serious bandwidth consumption and transmission latency under weak network conditions, but also significantly increase the overall inference cost, which is not conducive to the implementation of applications on a wide range of lines and numerous monitoring points.

[0008] Meanwhile, much of the footage in outdoor inspection videos is in a stable and normal state. For example, the mountains and vegetation around railway lines change slowly with sunlight and weather, but their semantic categories remain stable over a long period. Meanwhile, the frequency of truly abnormal events (such as slope collapses, blocked drainage ditches with water accumulation, construction sites encroaching on the safety zone of the railway line, flammable buildings like corrugated steel houses near the line, and personnel or large machinery encroaching on the line) is extremely low. Without semantic-level filtering, relying solely on pixel-level changes or threshold triggers to determine whether to upload footage easily leads to a high false alarm rate and a low proportion of valid data, further increasing the burden on cloud-based analysis.

[0009] Therefore, existing technologies cannot simultaneously meet engineering requirements in the following aspects: they must leverage the strong expressive capabilities of large cloud models in understanding complex scenes and identifying open-category anomalies, while also strictly controlling the consumption of computing resources at the edge and the data transmission burden in the cloud, and avoiding the indiscriminate uploading of a large number of semantically normal video frames. Summary of the Invention

[0010] To address the problems existing in the prior art, this invention provides a video anomaly detection method based on cloud-edge collaborative semantic sparse coding, which can significantly reduce data transmission and computation costs while maintaining the adaptability of multimodal large-scale open scenes and detection accuracy.

[0011] Therefore, the present invention adopts the following technical solution:

[0012] A video anomaly detection method based on cloud-edge collaborative semantic sparse coding includes the following steps:

[0013] S1, Deploy a multimodal large model in the cloud, store a semantic prototype matrix on the edge of the target area to be inspected, and deploy a trained lightweight semantic encoding model; the multimodal large model includes a semantic encoding module, an adaptation network, and a semantic decoding module.

[0014] S2, on the edge side, the video acquisition device acquires the monitoring scene video of the target area and obtains the extracted image sequence. The image sequence is input into the lightweight semantic coding model deployed on the edge side to obtain the semantic sparse feature sequence. The semantic sparse feature sequence contains semantic sparse feature vectors that correspond one-to-one with each time window.

[0015] S3, on the edge side, a semantic change metric is calculated based on the semantic sparse feature vectors of adjacent time windows. When the semantic change metric is lower than a preset semantic change threshold, data upload is not triggered. When the semantic change metric is higher than or equal to the semantic change threshold, the semantic sparse feature vectors of the corresponding time window and their time and location information are packaged into a sparse feature data packet and uploaded to the cloud through a communication link with limited bandwidth.

[0016] S4. On the cloud side, the sparse feature data packets uploaded from the edge side are received. The sparse feature data packets are processed by the adaptation network and semantic decoding module to obtain anomaly detection results containing anomaly categories and occurrence locations. Anomaly information or alarms are output based on the anomaly detection results.

[0017] In step S1 above:

[0018] The multimodal large model is a multimodal large model with visual encoding, language text encoding, and classification decoding capabilities;

[0019] The lightweight semantic coding model is obtained by training the semantic coding module of the cloud-based multimodal large model through feature distillation, as follows:

[0020] Representative video samples used for training are input into the semantic coding module and the lightweight semantic coding model to be trained, respectively, to obtain cloud-based semantic coding features and edge semantic coding features. Based on the difference between the cloud-based semantic coding features and the edge semantic coding features, a feature distillation loss function is constructed to optimize the parameters of the lightweight semantic coding model so that the lightweight semantic coding model can approximate the semantic coding effect of the cloud-based multimodal large model under given computational resource constraints, thus completing the training.

[0021] During the feature distillation training phase, the training dataset used consists of representative video data collected in similar target scenarios, including a large number of video clips in normal operation and collected hazard samples. For hazard categories with scarce samples, data is supplemented through scenario simulation and historical event reconstruction.

[0022] In step S1 above:

[0023] Building and The semantic prototype matrix corresponding to the key security hazard concepts:

[0024] , ,

[0025] Each semantic prototype vector This corresponds to a key security hazard concept.

[0026] In step S1 above:

[0027] By mapping text prompts describing specific key hidden danger concepts to a vector aligned with the visual semantic space through the text encoding function of a multimodal large model, and then combining a small number of corresponding image samples through clustering or fine-tuning, the semantic prototype vector is obtained.

[0028] In step S2 above:

[0029] Image frames of the railway line and its surrounding environment are acquired from the image sequence at preset time intervals, and the continuous... Frames of images form a time window, denoted as the . Time window , ;

[0030] Let the lightweight semantic encoding model on the edge side be denoted as , for the Each frame within a time window is encoded to obtain a frame-level basic semantic vector:

[0031] ,

[0032] in, Indicates the first Within the first time window The basic semantic vector of a frame image, dimension For output dimensions;

[0033] Temporal aggregation of frame-level semantic basis vectors within the same time window yields window-level basic semantic representations. :

[0034] ,

[0035] in, For time aggregation operators, it can be any of simple average, weighted average, or attention aggregation;

[0036] Based on the aforementioned basic semantic representation and semantic prototype matrix Calculate the time window Semantic reinforcement vectors of key security hazard concepts in China :

[0037] ,

[0038] in, It is a monotonic nonlinear function;

[0039] The basic semantic representation With semantic reinforcement vector By concatenating the features, we obtain a semantically sparse feature vector. :

[0040] ,

[0041] in, This represents a concatenation operation along the feature dimension;

[0042] The entire image sequence is processed according to the above procedure to obtain a semantic sparse feature sequence in which semantic sparse feature vectors are sorted by time. ,in, This represents the total number of semantically sparse feature vectors.

[0043] The video acquisition equipment includes fixed cameras, pan / tilt / zoom cameras, or drones.

[0044] In step S3 above: the semantic change metric is one of cosine similarity, Euclidean distance, or a weighted combination thereof; the semantic change threshold is determined based on historical inspection data statistics or empirical values.

[0045] In step S3 above:

[0046] For adjacent time windows and Their corresponding semantic sparse feature vectors are respectively and Euclidean distance is used to calculate semantic change measure The formula is:

[0047] ,

[0048] in, and These are non-negative weighting coefficients.

[0049] In step S4 above:

[0050] By adapting the network The semantic sparse feature vector Mapping to the feature space that the semantic encoding module can receive, we obtain the aligned semantic representation. :

[0051] ,

[0052] The adaptation network includes several fully connected layers, normalization layers, or lightweight Transformer layers;

[0053] The aligned semantic representation Input semantic decoding module Obtain the anomaly detection output vector :

[0054] ,

[0055] in, This includes the probability distribution of various anomalies, the overall risk score, and parameters related to the route location.

[0056] In step S4 above: the cloud platform is based on Generate structured anomaly detection records and, as needed, generate alarm information in natural language format, which is then pushed to railway line maintenance personnel or monitoring platforms.

[0057] Compared with the prior art, the present invention has the following beneficial effects:

[0058] 1. This invention decomposes the encoding and decoding functions of a large multimodal model in the cloud and performs feature distillation on the visual encoding part. A lightweight multimodal semantic encoder is deployed on the edge side, and the semantic encoding process is performed only on the edge side, while the semantic alignment and decoding process is performed on the cloud side. This reduces the computing power and storage requirements of edge devices while ensuring the semantic representation capability of complex scenes.

[0059] 2. This invention uses semantically sparse features instead of complete video data for cloud-edge transmission at the edge. By constructing a semantic change metric, it enables uploading to be triggered only when there are significant changes in scene semantics that may affect security. This filters out a large number of semantically stable normal inspection images, significantly reducing uplink bandwidth usage and cloud inference load.

[0060] 3. This invention relies on the strong semantic understanding capabilities of cloud-based multimodal large models for open scenes and rare hidden dangers. It can comprehensively analyze the surrounding environmental structure, construction site, temporary structures, and the status of roadbed and drainage facilities without acquiring all video pixels, thereby achieving unified detection of environmental and equipment anomalies and improving the intelligence level and risk identification capabilities of UAV railway inspection.

[0061] 4. This invention achieves good scalability for different industrial outdoor inspection scenarios through semantic prototype construction and feature distillation mechanism. When new types of hidden dangers or new infrastructure environments appear in the business, the cloud-edge collaborative anomaly detection capability can be iteratively upgraded through cloud model and distillation process updates without replacing edge hardware. Attached Figure Description

[0062] Figure 1 This is a flowchart of a video anomaly detection method based on cloud-edge collaborative semantic sparse coding, according to an embodiment of the present invention. Detailed Implementation

[0063] The technical solution of the invention will be clearly and completely described below with reference to the accompanying drawings and embodiments. Obviously, the following embodiments are only some embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0064] Example

[0065] Taking railway line drone inspection as a typical application scenario, this embodiment provides a video anomaly detection method based on cloud-edge collaborative semantic sparse coding, such as... Figure 1 As shown, it includes the following steps:

[0066] S1, Training and Deployment: Deploy a multimodal large model in the cloud, store the semantic prototype matrix at the edge of the target region to be inspected, and deploy the trained lightweight semantic encoding model, as detailed below:

[0067] In this embodiment, the multimodal large model Qwen3-VL (Tongyi Qianwen Multimodal Model) is deployed in the cloud. The multimodal large model can also be other multimodal large models with visual encoding, language text encoding, and classification decoding capabilities.

[0068] The multimodal large model includes a semantic encoding module, an adaptation network, and a semantic decoding module.

[0069] The lightweight semantic coding model adopts a multimodal coding architecture consistent with or compatible with the semantic encoder of the large multimodal model. The lightweight semantic coding model is obtained by training the semantic coding module of the cloud-based large multimodal model through feature distillation, specifically including:

[0070] Representative video samples used for training are input into the semantic coding module of the cloud-based multimodal large model and the lightweight semantic coding model to be trained, respectively, to obtain cloud-based semantic coding features and edge semantic coding features. Based on the difference between the cloud-based semantic coding features and the edge semantic coding features, a feature distillation loss function is constructed to optimize the parameters of the lightweight semantic coding model so that the lightweight semantic coding model can approximate the semantic coding effect of the cloud-based multimodal large model under given computational resource constraints, thus completing the training.

[0071] In this embodiment, during the feature distillation training phase, the training dataset used is representative video data collected from UAV inspections of railway lines or similar target scenarios. The training dataset includes a large number of video clips showing normal operation, as well as samples collected to include potential hazards in the surrounding environment, construction areas, roadbed anomalies, and water accumulation in drainage ditches. For hazard categories with scarce samples, data can be supplemented through scenario simulation and historical event reconstruction. In this embodiment, the trained and optimized lightweight semantic coding model is deployed in edge computing boxes at various UAV airports.

[0072] To further highlight abstract semantic information related to railway safety, a framework was constructed... The semantic prototype matrix corresponding to the key security hazard concepts:

[0073] , ,

[0074] Each semantic prototype vector This corresponds to a key security hazard concept.

[0075] In this embodiment, text prompts describing specific key safety hazards (such as "personnel intruding into the power line," "prefabricated steel-framed houses near the power line," "construction site," "water accumulation in drainage ditches," "roadbed collapse," etc.) can be mapped to a vector aligned with the visual semantic space using the text encoding function of the multimodal large model Qwen3-VL. This vector is then combined with a small number of corresponding image samples and obtained through clustering or fine-tuning to obtain the semantic prototype vector. The specific prototype generation method is not limited to any particular method; any semantic prototype vector associated with the safety hazard concept is acceptable.

[0076] S2, lightweight multimodal semantic coding and feature extraction for video capture and edge computing:

[0077] On the edge side, video acquisition devices (fixed cameras, PTZ cameras, drones) acquire monitoring scene videos of the target area, resulting in a sequence of extracted frames. This image sequence is then input into the lightweight semantic coding model deployed on the edge side to obtain a semantic sparse feature sequence representing the main target, spatial layout, and environmental elements of the scene. This semantic sparse feature sequence contains multiple semantic sparse feature vectors, each describing the scene's semantic information within a corresponding time window, including the main target, spatial layout, and environmental elements. Pixel details irrelevant to anomaly detection are omitted compared to the original pixel data, retaining only high-level semantic features related to the target state and environmental changes.

[0078] In this embodiment, image frames of the railway line and its surrounding environment are acquired from the image sequence at preset time intervals. (The text repeats itself here.) Frames of images form a time window, denoted as the . Time window , .

[0079] Let the lightweight semantic encoding model on the edge side be denoted as , for the Each frame within a time window is encoded to obtain a frame-level basic semantic vector:

[0080] ,

[0081] in, Indicates the first Within the first time window The basic semantic vector of a frame image, dimension For output dimensions.

[0082] Temporal aggregation of frame-level semantic basis vectors within the same time window yields window-level basic semantic representations. :

[0083] ,

[0084] in, The time aggregation operator can be a simple average, a weighted average, or an attention aggregation, etc. This embodiment does not limit the specific aggregation method.

[0085] Based on the aforementioned basic semantic representation and semantic prototype matrix Calculate the time window Semantic reinforcement vectors of key security hazard concepts in China :

[0086] ,

[0087] in, It can be an element-wise sigmoid function, softmax function, or other monotonic nonlinear function, used to constrain the concept response value within a preset range.

[0088] The basic semantic representation With semantic reinforcement vector By concatenating the features, we obtain a semantically sparse feature vector. :

[0089] ,

[0090] in, This represents a concatenation operation along the feature dimension. While preserving the semantics of the global scene, the concept of potential hazards related to railway line safety has been strengthened and expressed, and abstract semantic information can be reflected in a more compact vector form.

[0091] By processing the entire image sequence according to the above process, a semantic sparse feature sequence in temporal order of semantic sparse feature vectors can be obtained. This is used for subsequent semantic change measurement and upload decisions, where, This represents the total number of semantically sparse feature vectors.

[0092] S3, Semantic Change Measurement Calculation and Upload Decision:

[0093] On the edge side, for consecutive time windows, a semantic change metric representing the degree of semantic change in the scene is calculated based on the semantic sparse feature vectors of adjacent time windows. When the semantic change metric is lower than a preset semantic change threshold, it is determined that the scene semantics within the time window remain stable, and data upload is not triggered. When the semantic change metric is higher than or equal to the semantic change threshold, the semantic sparse feature vectors of the corresponding time window and their time and location information are packaged into a sparse feature data packet and uploaded to the cloud through a bandwidth-constrained communication link. The semantic change metric includes:

[0094] The similarity or distance index calculated for semantic sparse feature vector sequences corresponding to adjacent time windows, wherein the similarity or distance index includes one of cosine similarity, Euclidean distance or a weighted combination thereof;

[0095] The similarity or distance metric is compared with the preset semantic change threshold to determine whether to trigger the upload of the sparse feature data packet.

[0096] In this embodiment, changes in semantic sparse features over time are used to determine whether safety-related semantic changes have occurred in the railway line scenario, thereby deciding whether data for the corresponding time window needs to be uploaded to the cloud. Specifically:

[0097] For adjacent time windows and Their corresponding semantic sparse feature vectors are respectively and Semantic change measurement The calculation formula is:

[0098] ,

[0099] in, and These are non-negative weighting coefficients.

[0100] In this embodiment, the changes in the basic semantic space and the potential danger concept space are measured simultaneously by Euclidean distance. This approach considers both the changes in the overall scene structure and focuses on the activation changes of concepts related to railway safety.

[0101] In obtaining semantic change measurement Then, based on the preset semantic change threshold Make the upload decision:

[0102] when At that time, it was believed that the scenario had not changed significantly in terms of basic semantics and the concept of hidden dangers, and the corresponding time window mainly reflected the normal operation status of the line;

[0103] when When a semantic change is considered to potentially affect safety, such as the appearance of new structures (e.g., corrugated steel houses, temporary sheds) near the line, the presence of construction machinery or a construction area, or significant water accumulation or debris buildup in drainage ditches, the [unclear text - possibly related to safety] will be considered. The semantic sparse feature vector corresponding to each time window The time and location information are packaged into sparse feature data packets and uploaded to the cloud via a communication network.

[0104] The semantic change threshold Based on historical inspection data or empirical values, the upload can be determined so that under normal weather and operating conditions, only a small number of windows will trigger the upload. However, when abnormal changes occur in the environment or equipment facilities, the data can be detected in a timely manner and uploaded to the cloud for further analysis.

[0105] S4, cloud-based anomaly detection based on semantic sparse features:

[0106] On the cloud side, sparse feature data packets uploaded from the edge side are received. The received sparse feature data packets are processed through the adaptation network and semantic decoding module. Anomaly analysis is performed on the semantic sparse features without reconstructing the original video pixels, resulting in anomaly detection results containing anomaly categories and locations. Anomaly information or alarms are then output based on the anomaly detection results.

[0107] The decoding process of cloud-based multimodal large models includes two stages: semantic alignment and anomaly reasoning. Specifically:

[0108] In the semantic alignment stage, through network adaptation The semantic sparse feature vectors are mapped to the feature space that the semantic encoding module can receive, resulting in the aligned semantic representation. :

[0109] ,

[0110] The adaptation network may include several fully connected layers, normalization layers, or lightweight Transformer layers to compensate for the differences in structure and resolution between the lightweight semantic coding model and the semantic coding module.

[0111] During the anomaly reasoning phase, the aligned semantic representation is... Input to the semantic decoding module of a cloud-based multimodal large model Obtain the anomaly detection output vector :

[0112] ,

[0113] in, This can include probability distributions of various anomalies, overall risk scores, and parameters related to the location of the line.

[0114] The anomalies include, but are not limited to: potential hazards in the surrounding environment of the railway line during railway line inspection applications, such as corrugated steel houses, temporary construction sites, and piles of lightweight debris that are easily blown away by the wind; as well as potential hazards in the line equipment and facilities, such as abnormal roadbed conditions, water accumulation caused by blocked drainage ditches, and debris encroaching on the road shoulders.

[0115] For example, in railway line inspection applications, the cloud-based decoding module can output anomaly categories such as "suspected construction site exists around the line", "large area of ​​water accumulation exists in the drainage ditch on the right side of the line", and "a large area of ​​corrugated steel tile roofs near the line may pose a risk of falling off", and provide the corresponding spatial location information.

[0116] cloud platform based on The system generates structured anomaly detection records and, as needed, generates alarm information in natural language format, which is then pushed to railway line maintenance personnel or monitoring platforms to support on-site inspections, risk assessments, and decision-making.

[0117] The method of the present invention achieves functional decomposition of semantic encoding and decoding between the cloud and the edge by performing only the semantic encoding process of the lightweight semantic encoding model on the edge side and only the decoding process of the multimodal large model on the cloud side.

[0118] In summary, the method of this invention relies solely on the semantic sparse feature sequence uploaded from the edge side for the entire process of cloud anomaly detection and interpretation, without the need to receive and store complete video pixel data, thereby significantly reducing cloud storage pressure and computational consumption, while fully utilizing the semantic understanding capabilities of multimodal large models for complex and rare anomalies.

Claims

1. A video anomaly detection method based on cloud-edge collaborative semantic sparse coding, characterized in that, Includes the following steps: S1, Deploy a multimodal large model in the cloud, store a semantic prototype matrix on the edge of the target area to be inspected, and deploy a trained lightweight semantic encoding model; the multimodal large model includes a semantic encoding module, an adaptation network, and a semantic decoding module. S2, on the edge side, the video acquisition device acquires the monitoring scene video of the target area and obtains the extracted image sequence. The image sequence is then input into the lightweight semantic coding model deployed on the edge side to obtain the semantic sparse feature sequence. The semantic sparse feature sequence contains semantic sparse feature vectors that correspond one-to-one with each time window; S3, on the edge side, a semantic change metric is calculated based on the semantic sparse feature vectors of adjacent time windows. When the semantic change metric is lower than a preset semantic change threshold, data upload is not triggered. When the semantic change metric is higher than or equal to the semantic change threshold, the semantic sparse feature vectors of the corresponding time window and their time and location information are packaged into a sparse feature data packet and uploaded to the cloud through a communication link with limited bandwidth. S4. On the cloud side, the sparse feature data packets uploaded from the edge side are received. The sparse feature data packets are processed by the adaptation network and semantic decoding module to obtain anomaly detection results containing anomaly categories and occurrence locations. Anomaly information or alarms are output based on the anomaly detection results.

2. The video anomaly detection method according to claim 1, characterized in that, In S1: The multimodal large model is a multimodal large model with visual encoding, language text encoding, and classification decoding capabilities; The lightweight semantic coding model is obtained by training the semantic coding module of the cloud-based multimodal large model through feature distillation, as follows: Representative video samples used for training are input into the semantic coding module and the lightweight semantic coding model to be trained, respectively, to obtain cloud-based semantic coding features and edge semantic coding features; Based on the difference between the cloud-based semantic coding features and the edge semantic coding features, a feature distillation loss function is constructed to optimize the parameters of the lightweight semantic coding model so that the lightweight semantic coding model can approximate the semantic coding effect of the cloud-based multimodal large model under given computing resource constraints, thus completing the training. During the feature distillation training phase, the training dataset used consists of representative video data collected in similar target scenarios, including a large number of video clips in normal operation and collected hazard samples. For hazard categories with scarce samples, data is supplemented through scenario simulation and historical event reconstruction.

3. The video anomaly detection method according to claim 2, characterized in that, In S1: Building and The semantic prototype matrix corresponding to the key security hazard concepts: ， ， Each semantic prototype vector This corresponds to a key security hazard concept.

4. The video anomaly detection method according to claim 3, characterized in that, In S1: By mapping text prompts describing specific key hidden danger concepts to a vector aligned with the visual semantic space through the text encoding function of a multimodal large model, and then combining a small number of corresponding image samples through clustering or fine-tuning, the semantic prototype vector is obtained.

5. The video anomaly detection method according to claim 3, characterized in that, In S2: Image frames of the railway line and its surrounding environment are acquired from the image sequence at preset time intervals, and the continuous... Frames of images form a time window, denoted as the . Time window , ; Let the lightweight semantic encoding model on the edge side be denoted as , for the Each frame within a time window is encoded to obtain a frame-level basic semantic vector: ， in, Indicates the first Within the first time window The basic semantic vector of a frame image, dimension For output dimensions; Temporal aggregation of frame-level semantic basis vectors within the same time window yields window-level basic semantic representations. : ， in, For time aggregation operators, it can be any of simple average, weighted average, or attention aggregation; Based on the aforementioned basic semantic representation and semantic prototype matrix Calculate the time window Semantic reinforcement vectors of key security hazard concepts in China : ， in, It is a monotonic nonlinear function; The basic semantic representation With semantic reinforcement vector By concatenating the features, we obtain a semantically sparse feature vector. : ， in, This represents a concatenation operation along the feature dimension; The entire image sequence is processed according to the above procedure to obtain a semantic sparse feature sequence in which semantic sparse feature vectors are sorted by time. ,in, This represents the total number of semantically sparse feature vectors.

6. The video anomaly detection method according to claim 5, characterized in that: The video acquisition equipment includes fixed cameras, pan / tilt / zoom cameras, or drones.

7. The video anomaly detection method according to claim 5, characterized in that, In S3: the semantic change metric is one of cosine similarity, Euclidean distance, or a weighted combination thereof; the semantic change threshold is determined based on historical inspection data statistics or empirical values.

8. The video anomaly detection method according to claim 7, characterized in that, In S3: For adjacent time windows and Their corresponding semantic sparse feature vectors are respectively and Euclidean distance is used to calculate semantic change measure The formula is: ， in, and These are non-negative weighting coefficients.

9. The video anomaly detection method according to claim 7, characterized in that, In S4: By adapting the network The semantic sparse feature vector Mapping to the feature space that the semantic encoding module can receive, we obtain the aligned semantic representation. : ， The adaptation network includes several fully connected layers, normalization layers, or lightweight Transformer layers; The aligned semantic representation Input semantic decoding module Obtain the anomaly detection output vector : ， in, This includes the probability distribution of various anomalies, the overall risk score, and parameters related to the location of the line.

10. The video anomaly detection method according to claim 9, characterized in that, In S4: The cloud platform is based on Generate structured anomaly detection records and, as needed, generate alarm information in natural language format, which is then pushed to railway line maintenance personnel or monitoring platforms.

Images