Video monitoring scene intelligent analysis and event classification system based on semantic segmentation

Abstract

The invention discloses a video monitoring scene intelligent analysis and event classification system based on semantic segmentation. The system comprises a data preprocessing module, a preliminary classification module, a weight generation module and a correction and detection module. The data preprocessing module is used for acquiring and preprocessing original video data of a video monitoring scene, and extracting scene semantic features of the preprocessed video data through a semantic segmentation model; and the preliminary classification module is used for inputting the scene semantic features into a classifier for performing preliminary event classification to obtain an initial classification result. The invention relates to the technical field of video monitoring and intelligent analysis. According to the video monitoring scene intelligent analysis and event classification system based on semantic segmentation, the coupling degree between modules of the system is low, the semantic category number, the classifier structure and the updating strategy can be freely configured according to different monitoring tasks, and the system is suitable for various application scenes such as traffic flow analysis, personnel gathering monitoring and industrial operation anomaly detection.

Description

Technical Field

[0001] This invention relates to the field of video surveillance and intelligent analysis, and more specifically, to a video surveillance scene intelligent analysis and event classification system based on semantic segmentation. Background Technology

[0002] Traditional video surveillance systems primarily rely on manual observation and analysis of surveillance footage to detect anomalies or classify events. This method is not only labor-intensive and time-consuming, but also prone to missed detections and misjudgments due to the limitations of human attention. With the development of computer vision and deep learning technologies, automated video surveillance analysis systems are gradually emerging.

[0003] However, existing deep learning-based video surveillance analysis systems still have some shortcomings. On the one hand, in the data preprocessing stage, fixed processing methods are often used for video data from different scenarios, without fully considering the dynamic characteristics of the scenes. For example, using the same frame extraction rate for high-speed motion scenes and static scenes may lead to the loss of key information in high-speed motion scenes and the generation of a large amount of redundant data in static scenes. On the other hand, in the event classification process, semantic segmentation models and classifiers usually work independently, without fully considering the consistency between them, which affects the accuracy and reliability of the classification results. For example, the semantic segmentation model may have biases in its semantic understanding of certain regions, and the classifier may not effectively utilize this bias information to correct it, resulting in incorrect classification results. Furthermore, existing systems typically use fixed thresholds for anomaly detection, lacking adaptive calibration and dynamic update mechanisms, making it difficult to adapt to different scenarios and complex and changing realities, and prone to false positives or false negatives. Summary of the Invention

[0004] The purpose of this invention is to provide a video surveillance scene intelligent analysis and event classification system based on semantic segmentation. This system addresses the shortcomings of existing deep learning-based video surveillance analysis systems. In the data preprocessing stage, these systems often employ fixed processing methods for video data from different scenarios, failing to fully consider the dynamic characteristics of the scenes. Furthermore, in the event classification process, the semantic segmentation model and classifier typically operate independently, without adequately considering their consistency, thus affecting the accuracy and reliability of the classification results. In anomaly detection, fixed thresholds are usually used, lacking adaptive calibration and dynamic update mechanisms, making it difficult to adapt to different scenarios and complex, ever-changing realities, easily leading to false alarms or missed alarms, and ultimately failing to meet usage requirements.

[0005] This invention achieves the above objectives through the following technical solution: a video surveillance scene intelligent analysis and event classification system based on semantic segmentation, the system comprising: Data preprocessing module, preliminary classification module, weight generation module, correction and detection module; The data preprocessing module is used to acquire and preprocess the raw video data of the video surveillance scene, and extract the scene semantic features of the preprocessed video data through a semantic segmentation model. The preliminary classification module is used to input scene semantic features into the classifier to perform preliminary event classification and obtain initial classification results; The weight generation module is used to automatically generate model consistency weights by comparing and learning the differences between the feature output of the semantic segmentation model and the initial classification results of the classifier. The correction and detection module is used to dynamically adjust the decision contributions of the semantic segmentation model and classifier based on the model consistency weight, adaptively correct the initial classification results, and complete accurate event classification and anomaly detection.

[0006] Furthermore, when the data preprocessing module preprocesses the raw video data, it includes the following steps: Frames are extracted from the raw video data to obtain a continuous sequence of video frames. The frame extraction rate is dynamically adjusted according to the monitoring scene. Each video frame is normalized in size, and the resolution of the video frames is uniformly adjusted to the preset standard resolution. Perform grayscale correction on the normalized video frames and adjust the brightness distribution; Noise filtering is performed using a filtering algorithm; Motion regions in a video frame sequence are extracted using the inter-frame difference method to generate motion masks.

[0007] Furthermore, the adjustment rules for the frame extraction frame rate are as follows: High-speed motion scenes use a first preset frame rate, while static or low-speed scenes use a second preset frame rate. The first preset frame rate is higher than the second preset frame rate. Size normalization is achieved through a preset normalization formula, which normalizes the pixel values ​​of the video frame based on the minimum and maximum pixel values ​​of the video frame. Grayscale correction uses a gamma correction algorithm, and the gamma coefficient is determined based on the average brightness of the video frame.

[0008] Furthermore, Gaussian filtering is used for noise filtering, and the filter kernel is set to a preset size; During the motion mask generation process, the binarization threshold is first calculated by statistically distributing the inter-frame difference pixel values ​​of the previous preset number of frames, and the preset quantile of this distribution is taken as the initial threshold. The threshold is dynamically updated every preset number of frames. The updated threshold is calculated based on the historical threshold and the corresponding quantile of the current preset number of frames.

[0009] Furthermore, the semantic segmentation model adopts an encoder-decoder architecture, with the encoder using a deep convolutional neural network and the decoder using a pyramid pooling module; The semantic segmentation model takes the fused data of standardized video frames and motion masks as input, and the fusion method is element-wise multiplication; The encoder extracts multi-scale spatial features of video frames through multiple convolutional stages. The decoder upsamples the deepest feature map multiple times and concatenates it with the feature map of the corresponding stage of the encoder until it is restored to the original video frame size, and outputs pixel-level semantic segmentation results.

[0010] Furthermore, the value of each pixel in the semantic segmentation result corresponds to a semantic category label; The range of values ​​for semantic category labels is determined based on the preset number of semantic categories, which is configured according to the monitoring scenario. When extracting high-dimensional semantic feature vectors of target regions from semantic segmentation results, the output feature map of the last convolutional layer in the semantic segmentation model is obtained, a feature map mask is generated based on motion mask, region filtering is performed on the feature map, and the spatial dimension is compressed through global average pooling operation to obtain semantic feature vectors.

[0011] Furthermore, the classifier is a Transformer classifier based on a multi-head self-attention mechanism, including: Embedding layer, multiple encoder layers, and classification layer; The input to the classifier is a sequence of semantic feature vectors for a preset number of consecutive frames. After the temporal position information is added by the position encoding module, the input sequence is processed sequentially through each encoder layer. The encoder layer contains a multi-head self-attention sub-layer and a feedforward neural network sub-layer. After processing by the encoder layer, the output features of the last position of the sequence are taken and input into the classification layer. The features are then mapped to the event category space through the fully connected layer, outputting the initial classification probability distribution. The initial classification result is determined based on this probability distribution.

[0012] Furthermore, the position encoding module adds temporal position information to the feature vector sequence through a preset position encoding formula, which is constructed based on the position index and feature dimension index in the sequence; The multi-head self-attention sublayer calculates the attention output through multiple attention heads, and obtains the multi-head self-attention result through splicing and projection; The feedforward neural network sublayer processes the feature vector through two layers of linear transformation and activation function; The initial classification probability distribution output by the classification layer is implemented using the Softmax function, and the initial classification result is the category label corresponding to the maximum probability in the initial classification probability distribution.

[0013] Furthermore, the contrastive learning correction network includes a feature mapping layer, a similarity calculation layer, and a weight generation layer, with semantic feature vectors and initial classification probability distributions as dual inputs; The feature mapping layer maps semantic feature vectors to a feature distribution with the same dimension as the classifier output space through a fully connected network; The similarity calculation layer calculates the cosine similarity between the mapped feature distribution and the initial classification probability distribution, which serves as a measure of model output consistency. The weight generation layer generates model consistency weights based on cosine similarity. The model consistency weights include semantic segmentation model weights and classifier weights, and the sum of the two is 1.

[0014] Furthermore, the correction and detection module uses a weighted summation method to fuse the semantic feature distribution and the initial classification probability distribution to obtain the corrected classification probability distribution, with the weighting coefficients being the model consistency weights. The corrected classification probability distribution is normalized to ensure that the sum of probabilities is 1, and the final event classification result is determined based on the normalized probability distribution. Anomaly detection is achieved by setting a preset anomaly detection threshold, which is obtained through adaptive calibration using a validation set and dynamically updated at a preset period during subsequent online operation. When the maximum probability value of the corrected probability distribution is lower than the anomaly detection threshold, it is determined to be an abnormal event, and an anomaly alarm signal is output. When a preset number of consecutive abnormal frames are all determined to be abnormal events, a level 2 alarm is triggered, and the motion region mask and semantic segmentation results related to the abnormal event are output.

[0015] The beneficial effects of this invention are as follows: 1. The system adopts a semantic segmentation model with an encoder-decoder structure to perform pixel-level semantic segmentation of video frames. It can extract semantic features of people, objects and regions in the scene at multiple scales, realize a structured understanding of complex video environments, and effectively enhance the saliency of motion regions and improve the accuracy of feature representation by fusing input with motion masks.

[0016] 2. The preliminary classification module adopts a Transformer structure based on a multi-head self-attention mechanism, which can fully model the temporal correlation and contextual dependency between video frames, and achieve accurate discrimination of continuous behavioral events. Compared with the traditional RNN structure, it has stronger global modeling capabilities and parallel performance.

[0017] 3. The weight generation module calculates the cosine similarity between semantic segmentation features and classification results through comparative learning, and automatically generates model consistency weights to achieve a dynamic balance between the semantic segmentation model and the classifier. This enables the system to adaptively adjust feature contributions in different scenarios, significantly improving classification stability.

[0018] 4. The correction and detection module performs weighted fusion of semantic features and classification results based on consistency weights, obtains the final classification probability distribution through normalization, and introduces an anomaly detection strategy with adaptive thresholds to achieve accurate identification of abnormal events and a multi-level alarm mechanism, reducing false alarms and missed alarms.

[0019] 5. The system has low coupling between its modules and can freely configure the number of semantic categories, classifier structure and update strategy according to different monitoring tasks. It is suitable for various application scenarios such as traffic flow analysis, personnel gathering monitoring and industrial operation anomaly detection. Attached Figure Description

[0020] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a system block diagram of the present invention; Figure 2 This is a flowchart of the data preprocessing module of the present invention; Figure 3 This is a flowchart of the correction and detection module of the present invention. Detailed Implementation

[0021] The present application will now be described in further detail with reference to the accompanying drawings. It should be noted that the following specific embodiments are only used to further illustrate the present application and should not be construed as limiting the scope of protection of the present application. Those skilled in the art can make some non-essential improvements and adjustments to the present application based on the above application content.

[0022] Example 1: Please see Figure 1-3 This invention provides a technical solution: a video surveillance scene intelligent analysis and event classification system based on semantic segmentation, the system comprising: Data preprocessing module, preliminary classification module, weight generation module, correction and detection module; The data preprocessing module is used to acquire and preprocess the raw video data of the video surveillance scene; and to extract the scene semantic features of the preprocessed video data through a semantic segmentation model. Raw video data refers to unprocessed video information directly collected from video surveillance scenes, including original materials such as video footage and audio, which serves as the foundational data source for subsequent analysis. Preprocessing involves a series of operations on the raw video data to improve video data quality, remove noise, and standardize formats, enabling subsequent modules to process the data more effectively. Common preprocessing operations may include video frame extraction, video noise reduction, color correction, and size normalization. Semantic segmentation models are deep learning models whose task is to classify each pixel in an image or video, dividing the image or video into regions with different semantic meanings. For example, in video surveillance scenes, different objects in the image, such as people, vehicles, and buildings, can be segmented, and each pixel can be assigned a corresponding semantic label to extract scene semantic features. The preliminary classification module is used to input scene semantic features into the Transformer classifier to perform preliminary event classification and obtain initial classification results. Scene semantic features are extracted from preprocessed video data using a semantic segmentation model. These features reflect the semantic content represented by different objects and regions in the video scene. For example, one region might be labeled as "person," while another might be labeled as "vehicle." This semantic information helps classify events in the video. The Transformer classifier is a classification model based on the Transformer architecture. The Transformer architecture initially achieved great success in the field of natural language processing and has since been widely applied in fields such as computer vision. It uses a self-attention mechanism to capture long-distance dependencies in the input data, effectively analyzing the input scene semantic features and outputting preliminary event classification results. The initial classification result is a preliminary judgment of the event category in the video obtained after the scene semantic features are input into the Transformer classifier and processed by the model. This result may only be a preliminary and relatively coarse classification, and further correction and improvement are needed. The weight generation module is used to automatically generate model consistency weights by comparing and learning the differences between the feature output of the semantic segmentation model and the initial classification results of the Transformer classifier. Among them, the contrastive learning correction network is a special neural network structure. Its core idea is to learn effective feature representations by comparing the similarities and differences between different data samples. In this system, it is used to analyze the difference between the feature output of the semantic segmentation model and the initial classification result of the Transformer classifier. Through this contrastive learning method, model consistency weights are automatically generated. The model consistency weights are weight values ​​generated by the contrastive learning correction network to measure the degree of consistency between the semantic segmentation model and the Transformer classifier in the event classification process. These weights reflect the contribution of the two models to the classification result, and the decision contribution of the two models will be dynamically adjusted according to these weights. The correction and detection module is used to dynamically adjust the decision contributions of the semantic segmentation model and the Transformer classifier based on the model consistency weight, adaptively correct the initial classification results, and complete accurate event classification and anomaly detection. Dynamic adjustment involves dynamically and flexibly altering the influence of the semantic segmentation model and the Transformer classifier in the final decision-making process based on model consistency weights. For example, if a model has a higher weight, indicating better performance in the current classification task, it will be given greater decision weight when correcting the initial classification results. Adaptive correction involves targeted modification and improvement of the initial classification results based on the decision contributions of the dynamically adjusted semantic segmentation model and the Transformer classifier. This correction is not fixed but automatically adjusts the correction strategy according to different situations to improve the accuracy of event classification. Precise event classification, after processing by the correction and detection modules, yields a more accurate and detailed classification result for events in the video. Compared to the initial classification result, the precise classification result can more accurately identify the specific type and characteristics of the event. Anomaly detection simultaneously detects whether there are any anomalies in the video during the event classification process. By learning and analyzing the classification features of normal and abnormal events, the system can identify events that do not conform to the normal pattern and issue alarms or take other corresponding actions in a timely manner.

[0023] It should be noted that during use, the data preprocessing module acquires and processes the raw video, extracting scene semantic features to lay the foundation for subsequent analysis, thereby improving data quality and usability. The preliminary classification module uses the Transformer classifier to perform preliminary classification of semantic features, quickly obtaining initial results and improving processing efficiency. The weight generation module generates consistent weights through comparative learning to correct differences in network analysis, which can quantify the performance of the two models and provide a scientific basis for subsequent adjustments. The correction and detection module dynamically adjusts the decision contributions of the two models based on the weights, adaptively correcting the initial results, fully leveraging the advantages of the two models, improving the accuracy of event classification, and simultaneously completing anomaly detection, enhancing the system's practicality and reliability, and effectively addressing the needs of complex and ever-changing video surveillance scenarios.

[0024] In one embodiment, acquiring raw video data from a video surveillance scene and performing preprocessing includes: Frame extraction is performed on the original video data to obtain a continuous sequence of video frames:

[0025] in, This indicates the number of video frames, and the frame extraction frame rate is set to... The frame rate is dynamically adjusted according to the monitoring scene: for high-speed motion scenes, such as traffic intersections, a higher frame rate is used. In static or low-speed scenarios, such as office areas, adopt ; For each video frame Size normalization was performed to uniformly adjust the video frame resolution to [value missing]. The normalization formula is as follows: , in, Represents video frames The minimum pixel value, Represents video frames The maximum pixel value; Grayscale correction is performed on the normalized video frames, and a gamma correction algorithm is used to adjust the brightness distribution. The gamma coefficients are... The rules for determining it are as follows: Calculate the average brightness of video frames ; when At that time, take , when At that time, take , when At that time, take , The correction formula is as follows:

[0026] Noise filtering is performed using a Gaussian filter, with the filter kernel size set to [value missing]. The calculation formula is as follows:

[0027] in, For Gaussian kernel function, Let (x,y) be the standard deviation, and (x,y) represent the pixel coordinates of the video frame. Motion regions in a video frame sequence are extracted using the inter-frame difference method to generate motion masks. Binarization threshold The rules for determining it are as follows: Calculate the statistical distribution of pixel values ​​for the inter-frame differences of the first 100 frames, and take the distribution... The quantile is used as the initial threshold, and it is updated every 500 frames thereafter, using the following formula:

[0028] The motion mask is calculated using the 75th percentile of the current 500 frames, as follows:

[0029]

[0030] in, The range of values ​​is , This represents the normalized video frame of frame t-1. This represents the pixel value of the motion mask at the (x,y) coordinates.

[0031] This design allows for flexible adjustment of the frame extraction frame rate to suit different monitoring scenarios. High frame rates are used to capture details in high-speed scenes, while low frame rates are used in static scenes to reduce data volume. Size normalization unifies the resolution for easier subsequent processing. Grayscale correction adjusts the gamma coefficient based on average brightness to improve the brightness distribution of the image. Gaussian filtering effectively filters noise and improves image quality. The inter-frame difference method extracts moving regions to generate motion masks and can dynamically update thresholds. This design fully considers the differences between different scenarios, optimizes the data processing flow, reduces data redundancy, improves data quality, and accurately extracts moving regions. This provides a more accurate foundation for subsequent semantic segmentation and event classification, helping the system to analyze video content more efficiently and accurately, and enhancing its adaptability to various monitoring scenarios.

[0032] In one embodiment, scene semantic features of preprocessed video data are extracted using a semantic segmentation model, including: A semantic segmentation model based on an encoder-decoder architecture is constructed. The encoder uses a ResNet-50 network, and the decoder uses a pyramid pooling module (PPM) to normalize video frames. With motion mask Fusion data:

[0033] As input, This indicates element-wise multiplication; The encoder module extracts multi-scale spatial features from video frames through five convolutional stages, with the output feature map sizes for each stage being as follows: , , , , ; The decoder module upsamples the deepest feature map by a factor of 4 and concatenates it with the feature map from the corresponding stage of the encoder. This process is repeated until the original video frame size is restored, and the pixel-level semantic segmentation result is output. semantic segmentation results Each pixel's value corresponds to a semantic category label, with a value range of [value range missing]. , The preset number of semantic categories is configured according to the monitoring scenario: Traffic scenarios Includes vehicles, pedestrians, traffic lights, etc. Office Scene This includes personnel, tables, chairs, equipment, etc. From semantic segmentation results Extracting high-dimensional semantic feature vectors of the target region eigenvectors ,in The feature dimension is represented, and the extraction process is as follows: Obtain the output feature map of the last convolutional layer in the semantic segmentation model. , This refers to the number of channels in the feature map. Based on motion masking Generate feature map mask Region filtering is performed on the feature map:

[0034] in, Channel-space coordinates representing the feature map; The semantic feature vector is obtained by compressing the spatial dimension through global average pooling: , in, Indicates the number of valid feature pixels. This represents the value of the c-th dimension of the eigenvector.

[0035] This design employs an encoder-decoder architecture. The encoder uses ResNet-50 to extract multi-scale spatial features, while the decoder uses a pyramid pooling module to restore the dimensions and output semantic segmentation results. High-dimensional semantic feature vectors of the target region are extracted from the results. Valid feature regions are selected through specific operations and then pooled for compression. This encoder-decoder architecture can fully extract the spatial features of video frames and accurately restore their dimensions, resulting in refined semantic segmentation results. The extracted high-dimensional semantic feature vectors can better represent the features of the target region, providing rich and accurate feature information for subsequent event classification and improving the accuracy and reliability of classification.

[0036] In one embodiment, scene semantic features are input into a Transformer classifier to perform preliminary event classification, obtaining an initial classification result, including: A Transformer classifier based on a multi-head self-attention mechanism is constructed, consisting of one embedding layer, six encoder layers, and one classification layer. The input is a sequence of semantic feature vectors from K consecutive frames. ; Temporal position information is added to the feature vector sequence through a position encoding module. The position encoding formula is as follows:

[0037]

[0038] in, Represents the position index in the sequence , Indicates feature dimension index The generated position encoding matrix The sequence after adding position encoding is ; The encoder layer consists of a multi-head self-attention sublayer and a feedforward neural network sublayer. The multi-head self-attention is calculated as follows:

[0039]

[0040]

[0041] in, For the number of attention heads, For each dimension of attention head, Let be the projection matrix. To output the projection matrix; The sublayer calculations of the feedforward neural network are as follows:

[0042] in, , This is the weight matrix. , For bias terms; After processing by six encoder layers, the output feature from the last position of the sequence is input into the classification layer, and then mapped to the event category space through a fully connected layer. The number of event categories is [number missing]. Matching with the number of semantic categories, Output the initial classification probability distribution The calculation formula is as follows:

[0043] in, The end features of the sequence output by the encoder layer. , For classification layer parameters, The function satisfies , The logits value output by the fully connected layer; Based on the initial classification probability distribution Determine the initial classification results ,Right now:

[0044] in, This represents the category label corresponding to the highest probability.

[0045] This design constructs a Transformer classifier with a multi-head self-attention mechanism, adding positional encoding to preserve temporal information. The encoder layer processes features through multi-head self-attention and a feedforward neural network. Finally, the features at the end of the sequence are taken and passed to the classification layer to output the initial classification probability distribution, determining the initial classification result. The multi-head self-attention mechanism can capture the complex relationships between features, and the positional encoding preserves temporal information, which helps to understand the development of events. The encoder layer structure can fully explore the potential of features, and the classification layer outputs the probability distribution and determines the initial result, providing a basis for subsequent corrections and improving the initial accuracy of event classification.

[0046] In one embodiment, model consistency weights are automatically generated by analyzing the difference between the feature output of the semantic segmentation model and the initial classification result of the Transformer classifier through a contrastive learning correction network, including: Construct a contrastive learning correction network, including a feature mapping layer, a similarity calculation layer, and a weight generation layer, using semantic feature vectors. and initial classification probability distribution As a dual input; The feature mapping layer uses a fully connected network to map semantic feature vectors. Mapped to a feature distribution with the same dimension as the classifier output space The mapping process is as follows:

[0047] in, , For mapping layer parameters; Similarity calculation layer calculation and Cosine similarity between As a measure of model output consistency, it is expressed as follows:

[0048] in, Represents the vector dot product. Represents the L2 norm of a vector; The weight generation layer is based on cosine similarity. Generative Model Consistency Weights Weight Includes semantic segmentation model weights and Transformer classifier weights ,satisfy The weighting rules are as follows: The similarity was normalized.

[0049] in, and These represent the minimum and maximum similarity values ​​recorded during training, obtained through statistics from the training set. , ; The weight distribution ratio and gain coefficient are adjusted using the Sigmoid function. and offset coefficient The determination rule is: dynamically configured based on the complexity of the monitoring scenario, where the scenario complexity is... By semantic category number and moving target density Comprehensive calculation , when When (simple scenario), take , , when When (medium complexity scenario), take , , when When (complex scenarios), take , , The weight calculation formula is as follows:

[0050]

[0051] when Force setting , The models are highly consistent, primarily based on semantic segmentation features; when Force setting , The models show significant divergence, with the classifier results taking precedence.

[0052] This design constructs a network that includes feature mapping, similarity calculation, and weight generation layers. It takes semantic feature vectors and initial classification probability distributions as inputs, maps them, calculates cosine similarity, and then dynamically generates model consistency weights based on the similarity normalization value and scene complexity. By comparing and analyzing the differences in the outputs of the two models, it automatically generates weights, which can reasonably allocate the decision contributions of the two models. The weights are dynamically adjusted according to the scene complexity, enabling the system to effectively integrate the advantages of the two models in different scenarios and improve the accuracy and stability of classification results.

[0053] In one embodiment, the decision contributions of the semantic segmentation model and the Transformer classifier are dynamically adjusted based on model consistency weights to adaptively correct the initial classification results, thereby achieving accurate event classification and anomaly detection. This includes: Based on model consistency weights The semantic feature distribution is fused using a weighted summation method. Compared with the initial classification probability distribution The corrected classification probability distribution is obtained. The calculation formula is as follows:

[0054] in, ; The fused probability distribution is normalized to ensure that the sum of the probabilities is 1:

[0055] Based on the corrected classification probability distribution Determine the final event classification results ,Right now

[0056] Anomaly detection threshold The determination rule is as follows: Adaptive calibration is used with a validation set. The maximum probability distribution of all normal event classification results is calculated in the validation set, and the distribution is taken as... quantiles as the initial threshold The update will be dynamically performed every 1000 frames during subsequent online runtime, using the following formula:

[0057] This is the 95th percentile of the highest probability of normal events in the current 1000 frames. The range of values ​​is limited to When the mean of the maximum probability of normal events in the validation set At that time, take , when At that time, take , when At that time, take ; Based on threshold Determine abnormal events: Calculate the maximum probability value of the corrected probability distribution.

[0058] when If an event is identified as abnormal, an abnormal alarm signal is output, and the timestamp of the abnormality is recorded. and video frames The specific determination formula is as follows: , in, This indicates normal events and their corresponding category tags. Indicates an exception event marker; Level 2 alarm trigger threshold The determination rule is as follows: Configure according to the security level of the monitoring scenario. For high-security scenarios, such as financial data centers, take... Frame, medium security level scenario, such as shopping mall, take Frames, in low-security scenarios, such as parks, are taken. Frames, when consecutive When the average frame rate is determined to be an abnormal event, a level 2 alarm is triggered, and the motion region mask of the abnormal event is output. and semantic segmentation results It is used to assist in manual review.

[0059] This design employs a weighted summation fusion of semantic feature distribution and initial classification probability distribution, normalizes the result to determine the final classification, and uses a validation set for adaptive calibration to determine the anomaly detection threshold. Based on the threshold, abnormal events are identified, and a secondary alarm triggering threshold is configured according to the scene's security level. The weighted fusion fully leverages the advantages of both models to improve classification accuracy, the adaptive calibration of the anomaly detection threshold can adapt to different scenarios and accurately identify anomalies, and the secondary alarm triggering mechanism is set according to the scene's security level, enabling timely and effective alarm triggering in different scenarios, providing reliable protection for security monitoring.

[0060] Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0061] The above embodiments provide a detailed description of the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A video surveillance scene intelligent analysis and event classification system based on semantic segmentation, characterized in that, The system includes: Data preprocessing module, preliminary classification module, weight generation module, correction and detection module; The data preprocessing module is used to acquire and preprocess the raw video data of the video surveillance scene, and extract the scene semantic features of the preprocessed video data through a semantic segmentation model. The preliminary classification module is used to input scene semantic features into the classifier to perform preliminary event classification and obtain initial classification results; The weight generation module is used to automatically generate model consistency weights by comparing and learning the differences between the feature output of the semantic segmentation model and the initial classification results of the classifier. The correction and detection module is used to dynamically adjust the decision contributions of the semantic segmentation model and classifier based on the model consistency weight, adaptively correct the initial classification results, and complete accurate event classification and anomaly detection.

2. The intelligent video surveillance scene analysis and event classification system based on semantic segmentation according to claim 1, characterized in that, When the data preprocessing module preprocesses the raw video data, it includes the following steps: Frames are extracted from the raw video data to obtain a continuous sequence of video frames. The frame extraction rate is dynamically adjusted according to the monitoring scene. Each video frame is normalized in size, and the resolution of the video frames is uniformly adjusted to the preset standard resolution. Perform grayscale correction on the normalized video frames and adjust the brightness distribution; Noise filtering is performed using a filtering algorithm; Motion regions in a video frame sequence are extracted using the inter-frame difference method to generate motion masks.

3. The intelligent video surveillance scene analysis and event classification system based on semantic segmentation according to claim 2, characterized in that, The frame extraction frame rate adjustment rule is as follows: High-speed motion scenes use a first preset frame rate, while static or low-speed scenes use a second preset frame rate. The first preset frame rate is higher than the second preset frame rate. Size normalization is achieved through a preset normalization formula, which normalizes the pixel values ​​of the video frame based on the minimum and maximum pixel values ​​of the video frame. Grayscale correction uses a gamma correction algorithm, and the gamma coefficient is determined based on the average brightness of the video frame.

4. The intelligent video surveillance scene analysis and event classification system based on semantic segmentation according to claim 2, characterized in that: Noise filtering uses Gaussian filtering, and the filter kernel is set to a preset size; During the motion mask generation process, the binarization threshold is first calculated by statistically distributing the inter-frame difference pixel values ​​of the previous preset number of frames, and the preset quantile of this distribution is taken as the initial threshold. The threshold is dynamically updated every preset number of frames. The updated threshold is calculated based on the historical threshold and the corresponding quantile of the current preset number of frames.

5. The intelligent video surveillance scene analysis and event classification system based on semantic segmentation according to claim 1, characterized in that: The semantic segmentation model adopts an encoder-decoder architecture, with the encoder using a deep convolutional neural network and the decoder using a pyramid pooling module; The semantic segmentation model takes the fused data of standardized video frames and motion masks as input, and the fusion method is element-wise multiplication; The encoder extracts multi-scale spatial features of video frames through multiple convolutional stages, and the decoder upsamples the deepest feature map multiple times and concatenates it with the feature map of the corresponding stage of the encoder until it is restored to the original video frame size, and outputs pixel-level semantic segmentation results.

6. The intelligent video surveillance scene parsing and event classification system based on semantic segmentation according to claim 5, characterized in that: In the semantic segmentation result, the value of each pixel corresponds to a semantic category label; The range of values ​​for semantic category labels is determined based on the preset number of semantic categories, which is configured according to the monitoring scenario. When extracting high-dimensional semantic feature vectors of target regions from semantic segmentation results, the output feature map of the last convolutional layer in the semantic segmentation model is obtained, a feature map mask is generated based on motion mask, region filtering is performed on the feature map, and the spatial dimension is compressed through global average pooling operation to obtain semantic feature vectors.

7. The intelligent video surveillance scene analysis and event classification system based on semantic segmentation according to claim 1, characterized in that, The classifier is a Transformer classifier based on a multi-head self-attention mechanism, including: Embedding layer, multiple encoder layers, and classification layer; The input to the classifier is a sequence of semantic feature vectors for a preset number of consecutive frames. After the temporal position information is added by the position encoding module, the input sequence is processed sequentially through each encoder layer. The encoder layer contains a multi-head self-attention sub-layer and a feedforward neural network sub-layer. After processing by the encoder layer, the output feature of the last position of the sequence is taken and input into the classification layer. It is then mapped to the event category space through a fully connected layer, outputting an initial classification probability distribution. The initial classification result is determined based on this probability distribution.

8. The intelligent video surveillance scene analysis and event classification system based on semantic segmentation according to claim 7, characterized in that: The position encoding module adds temporal position information to the feature vector sequence through a preset position encoding formula, which is constructed based on the position index and feature dimension index in the sequence. The multi-head self-attention sublayer calculates the attention output through multiple attention heads, and obtains the multi-head self-attention result through splicing and projection; The feedforward neural network sublayer processes the feature vector through two layers of linear transformation and activation function; The initial classification probability distribution output by the classification layer is implemented using the Softmax function, and the initial classification result is the category label corresponding to the maximum probability in the initial classification probability distribution.

9. The intelligent video surveillance scene analysis and event classification system based on semantic segmentation according to claim 1, characterized in that: The contrastive learning correction network consists of a feature mapping layer, a similarity calculation layer, and a weight generation layer, with semantic feature vectors and an initial classification probability distribution as two inputs; The feature mapping layer maps semantic feature vectors to a feature distribution with the same dimension as the classifier output space through a fully connected network; The similarity calculation layer calculates the cosine similarity between the mapped feature distribution and the initial classification probability distribution, which serves as a measure of model output consistency. The weight generation layer generates model consistency weights based on cosine similarity. The model consistency weights include semantic segmentation model weights and classifier weights, and the sum of the two is 1.

10. The intelligent video surveillance scene analysis and event classification system based on semantic segmentation according to claim 1, characterized in that: The correction and detection module uses a weighted summation method to fuse the semantic feature distribution and the initial classification probability distribution to obtain the corrected classification probability distribution. The weighting coefficients are the model consistency weights. The corrected classification probability distribution is normalized to ensure that the sum of probabilities is 1, and the final event classification result is determined based on the normalized probability distribution. Anomaly detection is achieved by setting a preset anomaly detection threshold, which is obtained through adaptive calibration using a validation set and dynamically updated at a preset period during subsequent online operation. When the maximum probability value of the corrected probability distribution is lower than the anomaly detection threshold, it is determined to be an abnormal event, and an anomaly alarm signal is output. When a preset number of consecutive abnormal frames are all determined to be abnormal events, a level 2 alarm is triggered, and the motion region mask and semantic segmentation results related to the abnormal event are output.

Images