Abnormal video detection method and device based on differential semantic communication

Abstract

The invention relates to the technical field of computer vision, and provides an abnormal video detection method and device based on differential semantic communication. According to the method, the differential semantic communication strategy is introduced, dynamic evaluation is performed on the video data, different transmission strategies are adopted for the video data with scene mutation and the video data with stable scene according to the evaluation result, and bandwidth use can be optimized on the premise that the detection precision is not sacrificed. A multi-stage spatio-temporal feature extraction network composed of a feature extraction network, a three-dimensional convolutional network and a multi-scale Transform module is used to capture a long-term and short-term dependency relationship in a video, and robust feature extraction is ensured so as to realize accurate abnormal score prediction. And memory enhanced feature vectors are generated through the double-memory vector library, so that feature discriminative separation is realized, and the false alarm rate is reduced. According to the invention, through integrated collaborative optimization of transmission, perception and discrimination, the balance between the detection precision and the calculation efficiency is realized.

Description

Technical Field

[0001] This application relates to the fields of intelligent video surveillance and computer vision technology, and in particular to an abnormal video detection method and apparatus based on differentiated semantic communication. Background Technology

[0002] Video anomaly detection (VAD) is a technology that automatically identifies events or segments in a video stream that deviate from normal behavioral patterns under unsupervised or weakly supervised conditions. This technology has been widely applied in several key areas of smart cities, such as public safety and crime prevention, and urban traffic flow monitoring.

[0003] Traditional video surveillance systems primarily rely on manual monitoring or alarm mechanisms based on simple rules (such as motion detection), making it difficult to handle complex, dynamic, and rare anomalies. While existing automatic detection methods have incorporated deep learning models, they still face three major bottlenecks: First, the widespread use of full-frame transmission and processing incurs significant computational and communication overhead on edge devices due to the massive amounts of video data, making it difficult to meet low-latency requirements. Second, they struggle to model the complex spatiotemporal dependencies in real-world video scenes, resulting in poor scalability and generalization capabilities, leading to insufficient ability to distinguish between slowly evolving or sudden anomalies. Third, anomaly video clip samples are scarce and annotation costs are extremely high, forcing most systems to adopt weakly supervised or unsupervised paradigms, lacking the ability to accurately distinguish between normal and abnormal features, and prone to false alarms. These problems hinder the practical application of video anomaly detection technology in real-world scenarios. Summary of the Invention

[0004] Therefore, it is necessary to provide an abnormal video detection method and device based on differentiated semantic communication to address the above-mentioned technical problems.

[0005] An anomaly video detection method based on differential semantic communication includes the following steps:

[0006] An abnormal video detection model is constructed, which includes a feature extraction network, a three-dimensional convolutional network, a multi-scale Transformer module, a dual-memory vector library, and a prediction module. The dual-memory vector library includes a normal vector library and an abnormal vector library. Real-time acquisition of video data and dynamic evaluation of the video data; transmission of complete video frames for video data with sudden scene changes; and transmission of key area frames for video data without sudden scene changes. The feature extraction network receives the transmitted video data and extracts spatial features to obtain a spatial feature sequence, which is then input into a 3D convolutional network for temporal modeling to obtain a spatiotemporal feature sequence. The multi-scale Transformer module performs long-term and short-term modeling on the spatiotemporal feature sequence, and performs weighted fusion on the modeled feature vectors to generate a fused feature vector containing multi-scale semantics. The fused feature vector interacts with the dual-memory vector library through query and read operations to generate memory-enhanced feature vectors. The memory-enhanced feature vectors include normal video memory-enhanced feature vectors and abnormal video memory-enhanced feature vectors, which are stored in the normal vector library and the abnormal vector library, respectively. The prediction module performs anomaly detection on video data based on fused feature vectors and memory-enhanced feature vectors.

[0007] In one embodiment, video data is acquired in real time, and the video data is dynamically evaluated, including: Real-time acquisition of video data, using optical flow graph algorithms to calculate the spatial cumulative amplitude between adjacent frames in the video:

[0008] in, Cumulative amplitude in space; for t A video frame at a given moment; for t Video frame at time -1; H For height; W Width; For optical flow graph functions; Extract the histograms of each channel between adjacent frames in the video, and calculate the variance of the differences between the histograms of the same channel between adjacent frames:

[0009] in, The variance of the histogram differences between adjacent frames in the same channel; c For channel indexing; This is the variance calculation function; For the first c Channel histogram extraction function; Determine whether the variance between the cumulative spatial amplitude and the difference between the histograms of adjacent frames in the same channel exceeds a preset threshold. If it does, it is determined that a scene change has occurred.

[0010] In one embodiment, the spatial feature sequence is extracted according to the following formula:

[0011]

[0012] in, for t Spatial characteristics at any given time; for t A video frame at a given moment; For learnable parameters Defined ShuffleNetV2 backbone network; and for t The video frame at any given moment is along the two input branches of the channel dimension; For channel splicing; Recalculation for cross-channel groups; and These are the convolution parameters.

[0013] In one embodiment, timing modeling is performed according to the following steps: Time t Centered on, with a length of k +1 spatial feature sequence Stacked along the time dimension, forming a spacetime volume. ; spacetime Inputting a 3D convolutional network for temporal modeling:

[0014] in, for t The spatiotemporal characteristics of a moment; It is a 3D convolutional network encoder.

[0015] In one embodiment, the multi-scale Transformer module performs long-term and short-term modeling on the spatiotemporal feature sequences, respectively, and weights and fuses the resulting feature vectors to generate a fused feature vector containing multi-scale semantics, including: Introducing learnable location embedding vectors to record spatiotemporal feature sequences Timing sequence information:

[0016] in, To add spatiotemporal features after location embedding; for t The learnable position embedding vector at each time step; Map the spatiotemporal features after adding location embedding to h Each attention head has a parallel subspace, and the query vector, key vector, and value vector for each subspace are calculated. The output features of each attention head are then calculated. The output features of each attention head are concatenated and fused through a linear projection layer. The fused features are then normalized with the spatiotemporal features after adding position embeddings. A nonlinear transformation is then applied to the normalized features to obtain long-term temporal features.

[0017] in, for t Long-term temporal characteristics of time; For layer normalization operation; It is a feedforward neural network used to perform nonlinear transformations on features; These are the features after layer normalization; Local time window The spatiotemporal features within are added to the corresponding location embedding vectors:

[0018]

[0019] in, The characteristics are those after addition; for Spatiotemporal characteristics within; for The corresponding position embedding vector; m As a scale; This is the basic time granularity; The summed features are decomposed using a multi-head attention mechanism corresponding to the local time window, and the local attention weights are normalized within the local time window. Multi-scale feature output is then obtained through a Transformer layer. The multi-scale features are then fused into short-time features through splicing and linear projection.

[0020] in, for t Short-term temporal characteristics of time; This is the fusion projection matrix for multi-scale features; For feature splicing; The long-term and short-term features obtained from modeling are weighted and fused to generate a fused feature vector containing multi-scale semantics:

[0021] in, for t The fused feature vector at each time step; This is a learnable fusion parameter matrix.

[0022] In one embodiment, the fused feature vector interacts with a dual-memory vector library through query and read operations to generate a memory-enhanced feature vector, including: The query operation calculates the similarity between the fused feature vector and the vectors in the dual-memory vector library:

[0023] in, The prototype matching score for the normal vector library; For the Sigmoid function; This is a normal vector library; For feature dimensions; The prototype matching score for the anomaly vector library; An anomaly vector library; The read operation, based on the query result, generates a memory-enhanced feature vector through a weighted summation of vectors:

[0024] in, Enhance the feature vector for normal video memory; Enhance the feature vector for abnormal video memory.

[0025] In one embodiment, the fused feature vector interacts with a dual-memory vector library through query and read operations to generate a memory-enhanced feature vector. This memory-enhanced feature vector includes normal video memory-enhanced feature vectors and abnormal video memory-enhanced feature vectors, which are stored in a normal vector library and an abnormal vector library, respectively. The embodiment also includes: Construct clear class boundaries using a two-vector separation loss:

[0026] in, For two-vector separation loss; These are the preset hyperparameters.

[0027] In one embodiment, the prediction module performs anomaly detection on video data based on fused feature vectors and memory-enhanced feature vectors, including: The prediction module performs anomaly detection on video data based on fused feature vectors and memory-enhanced feature vectors:

[0028]

[0029] in, These are the input features for the classifier; For activation functions; The feature mapping weight matrix; For classification probability; For activation functions; It is a three-layer fully connected network.

[0030] In one embodiment, the loss function used by the abnormal video detection model is:

[0031] in, The loss function; Cross-entropy; and These are custom hyperparameters.

[0032] An anomaly video detection device based on differential semantic communication includes: The model building module is used to build an abnormal video detection model. The model includes a feature extraction network, a three-dimensional convolutional network, a multi-scale Transformer module, a dual vector memory library, and a prediction module. The dual vector memory library includes a normal vector library and an abnormal vector library. The data transmission module is used to acquire video data in real time, perform dynamic evaluation of the video data, transmit complete video frames for video data with sudden scene changes, and transmit key area frames for video data without sudden scene changes. The feature extraction module is used to extract spatial features from the video data received and transmitted by the feature extraction network, obtain a spatial feature sequence, and input it into a 3D convolutional network for temporal modeling to obtain a spatiotemporal feature sequence. The multi-scale Transformer module performs long-term and short-term modeling on the spatiotemporal feature sequence respectively, and performs weighted fusion on the feature vectors obtained from the modeling to generate a fused feature vector containing multi-scale semantics. The memory enhancement module is used to fuse feature vectors and interact with the dual memory vector library through query and read operations to generate memory-enhanced feature vectors. The memory-enhanced feature vectors include normal video memory-enhanced feature vectors and abnormal video memory-enhanced feature vectors, which are stored in the normal vector library and the abnormal vector library respectively. The anomaly detection module is used by the prediction module to detect anomalies in video data based on fused feature vectors and memory-enhanced feature vectors.

[0033] The aforementioned anomaly video detection method and apparatus based on differentiated semantic communication introduces a differentiated semantic communication strategy. By dynamically evaluating video data, different transmission strategies are adopted for video data with sudden scene changes and video data with stable scenes, optimizing bandwidth usage without sacrificing detection accuracy. A multi-stage spatiotemporal feature extraction network, composed of a feature extraction network, a 3D convolutional network, and a multi-scale Transformer module, captures long-term and short-term dependencies in the video, ensuring robust feature extraction for accurate anomaly score prediction. Memory-enhanced feature vectors are generated using a dual-memory vector library to achieve discriminative feature separation and reduce the false alarm rate.

[0034] This invention achieves a balance between detection accuracy and computational efficiency through integrated and coordinated optimization of transmission, sensing, and discrimination, effectively alleviating the problem of balancing high accuracy with low resource consumption. Attached Figure Description

[0035] Figure 1 This is a flowchart illustrating an anomaly video detection method based on differential semantic communication in one embodiment. Figure 2 This is a schematic diagram of the architecture of an anomaly video detection method based on differential semantic communication in one embodiment; Figure 3 This is a schematic diagram of the structure of a multi-scale Transformer module in one embodiment; Figure 4 This is a schematic diagram of the structure of a dual-memory vector library in one embodiment; Figure 5 This is a structural block diagram of an anomaly video detection device based on differential semantic communication in one embodiment. Detailed Implementation

[0036] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0037] In one embodiment, such as Figure 1 , Figure 2 As shown, an anomaly video detection method based on differential semantic communication is provided, including the following steps: Step 201: Construct an abnormal video detection model. The model includes a feature extraction network, a three-dimensional convolutional network, a multi-scale Transformer module, a dual-memory vector library, and a prediction module. The dual-memory vector library includes a normal vector library and an abnormal vector library.

[0038] Step 202: Real-time acquisition of video data and dynamic evaluation of the video data; transmission of complete video frames for video data with sudden scene changes; and transmission of key area frames for video data without sudden scene changes.

[0039] It should be noted that the video data is collected in real time by intelligent sensing terminals deployed at the edge. When transmitting key area frames of video data that has not undergone sudden scene changes, the key areas are located using YOLOv8.

[0040] Step 203: The feature extraction network receives the transmitted video data and performs spatial feature extraction to obtain a spatial feature sequence, which is then input into a 3D convolutional network for temporal modeling to obtain a spatiotemporal feature sequence. The multi-scale Transformer module performs long-term and short-term modeling on the spatiotemporal feature sequence, and performs weighted fusion on the modeled feature vectors to generate a fused feature vector containing multi-scale semantics.

[0041] Step 204: The fused feature vector interacts with the dual-memory vector library through query and read operations to generate memory-enhanced feature vectors. The memory-enhanced feature vectors include normal video memory-enhanced feature vectors and abnormal video memory-enhanced feature vectors, which are stored in the normal vector library and the abnormal vector library, respectively.

[0042] Step 205: The prediction module performs anomaly detection on the video data based on the fused feature vector and the memory-enhanced feature vector.

[0043] The aforementioned anomaly video detection method based on differentiated semantic communication introduces a differentiated semantic communication strategy. By dynamically evaluating video data, different transmission strategies are adopted for video data with sudden scene changes and video data with stable scenes, optimizing bandwidth usage without sacrificing detection accuracy. A multi-stage spatiotemporal feature extraction network, composed of a feature extraction network, a 3D convolutional network, and a multi-scale Transformer module, captures long-term and short-term dependencies in the video, ensuring robust feature extraction for accurate anomaly score prediction. Memory-enhanced feature vectors are generated using a dual-memory vector library to achieve discriminative feature separation and reduce the false alarm rate. Through integrated collaborative optimization of transmission, perception, and discrimination, a trade-off between detection accuracy and computational efficiency is achieved, effectively alleviating the challenge of balancing high accuracy with low resource consumption.

[0044] In one embodiment, step 202 involves real-time acquisition of video data and dynamic evaluation of the video data, including: Real-time acquisition of video data, using optical flow graph algorithms to calculate the spatial cumulative amplitude between adjacent frames in the video:

[0045] in, The spatial cumulative amplitude reflects the entire image in time step. t Total exercise intensity; for t Video frames at any given moment ; for t Video frame at time -1; H For height; W Width; For optical flow graph functions; Extract the histograms of each channel between adjacent frames in the video, and calculate the variance of the differences between the histograms of the same channel between adjacent frames:

[0046] in, It is the variance of the histogram difference between adjacent frames in the same channel, and it is highly sensitive to visual information such as sudden changes in lighting, explosions, and smoke diffusion. c For channel indexing; This is the variance calculation function; For the first c Channel histogram extraction function; Determine whether the variance between the cumulative spatial amplitude and the difference between the histograms of adjacent frames in the same channel exceeds a preset threshold. If it does, it is determined that a scene change has occurred.

[0047] In this embodiment, dynamic evaluation is performed using optical flow maps and histograms. Based on the evaluation results, a differentiated semantic communication strategy is adaptively implemented. For video data experiencing scene changes, complete video frames are transmitted to preserve high-semantic-value information contained in the current frame. For video data without scene changes, only key region frames are transmitted. By significantly compressing redundant pixel data and transmitting semantic elements related to higher-level tasks, bandwidth usage is optimized. By dynamically allocating communication resources in a task-driven and content-aware manner, limited bandwidth is prioritized for semantic information that is discriminative for anomaly detection. This significantly reduces uplink communication overhead while ensuring the input quality of downstream recognition models and the real-time response of the overall system.

[0048] In one embodiment, in step 203, the spatial feature sequence is extracted according to the following formula:

[0049]

[0050] in, for t Spatial characteristics at any given time; for t A video frame at a given moment; For learnable parameters Defined ShuffleNetV2 backbone network; and for t The video frame at any given moment is along the two input branches of the channel dimension; For channel splicing; Recalculation for cross-channel groups; and These are the convolution parameters, which need to be obtained through training.

[0051] It should be noted that, Pass the build mapping path directly. After calculation using depthwise convolution kernels and pointwise convolution kernels.

[0052] In this embodiment, ShuffleNetV2 (a lightweight convolutional neural network based on efficient network design principles) is used as the feature extraction network. Through mechanisms such as channel segmentation, depthwise separable convolution, and channel rearrangement, it maintains excellent feature representation capabilities while significantly reducing memory access costs and floating-point operations, thereby achieving optimized use of computing power and bandwidth.

[0053] In one embodiment, in step 203, timing modeling is performed according to the following steps: Time t Centered on, with a length of k +1 spatial feature sequence Stacked along the time dimension, forming a spacetime volume. , , T In terms of time dimension, C For channel dimensions; spacetime Inputting a 3D convolutional network for temporal modeling:

[0054] in, for t The spatiotemporal characteristics of a moment; It is a 3D convolutional network encoder.

[0055] It should be noted that, Composed of 3D convolutions pre-trained on large-scale datasets, it possesses good generalization ability. 3D convolution operations target the convolution kernel function. The definition is as follows:

[0056] in, d Size is determined by the time dimension; h and wThese represent the spatial dimensions, specifically the height and width of the convolutional kernel within a single frame; C in Input the number of channels; C out This represents the number of output channels. The 3D convolution operation slides synchronously across the spatial and temporal dimensions, fusing visual and motion information from the local domain to achieve a deep abstraction of the spatiotemporal structure.

[0057] In this embodiment, temporal modeling is performed using a three-dimensional convolutional network to achieve joint modeling of motion dynamics and contextual relationships in video clips, significantly improving the ability to perceive complex behavioral patterns.

[0058] In one embodiment, reference Figure 3 The multi-scale Transformer module includes long-term modeling and short-term modeling. In step 203, the multi-scale Transformer module performs long-term and short-term modeling on the spatiotemporal feature sequences respectively, and performs weighted fusion on the feature vectors obtained from the modeling to generate a fused feature vector containing multi-scale semantics, including: Introducing learnable location embedding vectors to record spatiotemporal feature sequences Timing sequence information:

[0059] in, To add spatiotemporal features after location embedding; for t The learnable position embedding vector at each time step; Map the spatiotemporal features after adding location embedding to h Each attention head has a parallel subspace, and the query vector, key vector, and value vector for each subspace are calculated. The output features of each attention head are then calculated. Specifically, the first i The query vector of the attention head Key vector Sum value vector Calculate using the following formula: , ,

[0060] in, , , These are the weight matrices corresponding to the query vector, key vector, and value vector, respectively. The output characteristics of each attention head are calculated according to the following formula:

[0061] in, For the first i The output features of the attention head.

[0062] The output features of each attention head are concatenated and fused through a linear projection layer. The fused features are then normalized with the spatiotemporal features after adding position embeddings. A nonlinear transformation is then applied to the normalized features to obtain long-term temporal features.

[0063]

[0064]

[0065] in, for t Long-term temporal characteristics of time; For layer normalization operation; It is a feedforward neural network used to perform nonlinear transformations on features; These are the features after layer normalization; Features after fusion; To output the projection matrix, .

[0066] For feedforward neural networks, the features are nonlinearly transformed according to the following formula:

[0067] in, x These are the input features for FFN; This is the weight matrix of the first fully connected layer; This is the bias term for the first layer fully connected layer; This is the weight matrix for the second fully connected layer; For the bias term of the second fully connected layer; This is the activation function.

[0068] Long-term modeling employs a Transformer encoding layer consisting of a multi-head attention mechanism and a feedforward neural network to model long-term dependencies.

[0069] Local time window The spatiotemporal features within are added to the corresponding location embedding vectors:

[0070]

[0071] in, The characteristics are those after addition; for Spatiotemporal characteristics within; for The corresponding position embedding vector; m As a scale; This is the basic time granularity; The summed features are decomposed using a multi-head attention mechanism corresponding to the local time window, and the local attention weights are normalized within the local time window. Multi-scale feature output is then obtained through a Transformer layer. The multi-scale features are then fused into short-time features through splicing and linear projection.

[0072] in, for t Short-term temporal characteristics of time; The fusion projection matrix represents the multi-scale features. ; For feature splicing; Specifically, the features obtained by decomposing and summing the features through the multi-head attention mechanism corresponding to the local time window are as follows: .

[0073] The long-term and short-term features obtained from modeling are weighted and fused to generate a fused feature vector containing multi-scale semantics:

[0074] in, for t The fused feature vector at each time step; For learnable fusion parameter matrix, .

[0075] In one embodiment, reference Figure 4 This is a schematic diagram of the structure of a dual-memory vector library. The dual-memory vector library stores the latent representations of normal vectors and abnormal vectors through two independent memory units. Each vector library contains... M One vector slot is used for storage. D 3D feature vector. In step 204, the fused feature vector interacts with the dual-memory vector library through query and read operations to generate a memory-enhanced feature vector, including: The query operation calculates the similarity between the fused feature vector and the vectors in the dual-memory vector library:

[0076] in, The prototype matching score for the normal vector library; For the Sigmoid function; This is a normal vector library; For feature dimensions; The prototype matching score for the anomaly vector library; An anomaly vector library; The read operation, based on the query result, generates a memory-enhanced feature vector through a weighted summation of vectors:

[0077] in, Enhance the feature vector for normal video memory; Enhance the feature vector for abnormal video memory.

[0078] In one embodiment, step 204 involves fusing the feature vectors through query and read operations with a dual-memory vector library to generate memory-enhanced feature vectors. These memory-enhanced feature vectors include normal video memory-enhanced feature vectors and abnormal video memory-enhanced feature vectors, which are stored in a normal vector library and an abnormal vector library, respectively. The step also includes: Construct clear class boundaries using a two-vector separation loss:

[0079] in, For two-vector separation loss; These are the preset hyperparameters.

[0080] In this embodiment, a clear category boundary is constructed by using a two-vector separation loss, thereby enhancing the distinguishability of normal and abnormal features in the vector space.

[0081] In one embodiment, step 205, the prediction module performs anomaly detection on the video data based on the fused feature vector and the memory-enhanced feature vector, including: The prediction module performs anomaly detection on video data based on fused feature vectors and memory-enhanced feature vectors:

[0082]

[0083] in, These are the input features for the classifier; For activation functions; This is the feature mapping weight matrix, used to map features to the input dimensions of the classifier. ; For classification probability; For activation functions; It is a three-layer fully connected network.

[0084] In this embodiment, the prediction module integrates fused feature vectors and memory-enhanced feature vectors, and uses a classifier to achieve frame-level classification to distinguish between normal and abnormal frames. In this embodiment, a three-layer fully connected network is used as the classifier.

[0085] In one embodiment, the loss function used by the abnormal video detection model is:

[0086] in, The loss function; Cross-entropy; and These are custom hyperparameters, set based on the training results.

[0087] In this embodiment, a loss function is constructed using cross-entropy and bi-vector separation loss. Cross-entropy loss supervises the video-level anomaly scores output by the model, enabling the network to gradually learn to identify the most discriminative anomaly clues from a large number of segments during training. Bi-vector separation loss enhances the separation of the two types of features in the embedding space, enabling the model to more reliably distinguish between "normal events" and "abnormal events" during the inference phase.

[0088] It should be understood that, although Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order in which these steps are executed, and they can be performed in other orders. Furthermore, Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0089] In one embodiment, such as Figure 5 As shown, an anomaly video detection device based on differential semantic communication is provided, comprising: The model building module 901 is used to build an abnormal video detection model. The model includes a feature extraction network, a three-dimensional convolutional network, a multi-scale Transformer module, a dual-memory vector library, and a prediction module. The dual-memory vector library includes a normal vector library and an abnormal vector library.

[0090] The data transmission module 902 is used to acquire video data in real time, perform dynamic evaluation of the video data, transmit complete video frames for video data with sudden scene changes, and transmit key area frames for video data without sudden scene changes.

[0091] The feature extraction module 903 is used to receive and transmit video data from the feature extraction network and extract spatial features to obtain a spatial feature sequence. This sequence is then input into a three-dimensional convolutional network for temporal modeling to obtain a spatiotemporal feature sequence. The multi-scale Transformer module performs long-term and short-term modeling on the spatiotemporal feature sequence and performs weighted fusion on the modeled feature vectors to generate a fused feature vector containing multi-scale semantics.

[0092] The memory enhancement module 904 is used to fuse feature vectors and interact with the dual memory vector library through query and read operations to generate memory-enhanced feature vectors. The memory-enhanced feature vectors include normal video memory-enhanced feature vectors and abnormal video memory-enhanced feature vectors, which are stored in the normal vector library and the abnormal vector library, respectively.

[0093] The anomaly detection module 905 is used by the prediction module to perform anomaly detection on video data based on the fused feature vector and the memory-enhanced feature vector.

[0094] Specific limitations regarding the anomaly video detection device based on differentiated semantic communication can be found in the limitations of the anomaly video detection method based on differentiated semantic communication described above, and will not be repeated here. Each module in the aforementioned anomaly video detection device based on differentiated semantic communication can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0095] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0096] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. An abnormal video detection method based on differentiated semantic communication, characterized in that, Includes the following steps: An abnormal video detection model is constructed, which includes a feature extraction network, a three-dimensional convolutional network, a multi-scale Transformer module, a dual-memory vector library, and a prediction module. The dual-memory vector library includes a normal vector library and an abnormal vector library. Real-time acquisition of video data and dynamic evaluation of the video data; transmission of complete video frames for video data with sudden scene changes; and transmission of key area frames for video data without sudden scene changes. The feature extraction network receives the transmitted video data and extracts spatial features to obtain a spatial feature sequence, which is then input into a 3D convolutional network for temporal modeling to obtain a spatiotemporal feature sequence. The multi-scale Transformer module performs long-term and short-term modeling on the spatiotemporal feature sequence, and performs weighted fusion on the modeled feature vectors to generate a fused feature vector containing multi-scale semantics. The fused feature vector interacts with the dual-memory vector library through query and read operations to generate memory-enhanced feature vectors. The memory-enhanced feature vectors include normal video memory-enhanced feature vectors and abnormal video memory-enhanced feature vectors, which are stored in the normal vector library and the abnormal vector library, respectively. The prediction module performs anomaly detection on video data based on fused feature vectors and memory-enhanced feature vectors.

2. The abnormal video detection method based on differentiated semantic communication according to claim 1, characterized in that, Real-time acquisition of video data and dynamic evaluation of the video data, including: Real-time acquisition of video data, using optical flow graph algorithms to calculate the spatial cumulative amplitude between adjacent frames in the video: in, Cumulative amplitude in space; for t A video frame at a given moment; for t Video frame at time -1; H For height; W Width; For optical flow graph functions; Extract the histograms of each channel between adjacent frames in the video, and calculate the variance of the differences between the histograms of the same channel between adjacent frames: in, The variance of the histogram differences between adjacent frames in the same channel; c For channel indexing; This is the variance calculation function; For the first c Channel histogram extraction function; Determine whether the variance between the cumulative spatial amplitude and the difference between the histograms of adjacent frames in the same channel exceeds a preset threshold. If it does, it is determined that a scene change has occurred.

3. The abnormal video detection method based on differentiated semantic communication according to claim 1, characterized in that, The spatial feature sequence is extracted according to the following formula: in, for t Spatial characteristics at any given time; for t A video frame at a given moment; For learnable parameters Defined ShuffleNetV2 backbone network; and for t The video frame at any given moment is input along the two input branches of the channel dimension; For channel splicing; Recalculation for cross-channel groups; and These are the convolution parameters.

4. The abnormal video detection method based on differentiated semantic communication according to claim 3, characterized in that, Temporal modeling is performed according to the following steps: Time t Centered on, with a length of k +1 spatial feature sequence Stacked along the time dimension, forming a spacetime volume. ; spacetime Inputting a 3D convolutional network for temporal modeling: in, for t The spatiotemporal characteristics of a moment; It is a 3D convolutional network encoder.

5. The abnormal video detection method based on differentiated semantic communication according to claim 4, characterized in that, The multi-scale Transformer module performs long-term and short-term modeling on the spatiotemporal feature sequences, respectively, and then weights and fuses the resulting feature vectors to generate a fused feature vector containing multi-scale semantics, including: Introducing learnable location embedding vectors to record spatiotemporal feature sequences Timing sequence information: in, To add spatiotemporal features after location embedding; for t The learnable position embedding vector at each time step; Map the spatiotemporal features after adding location embedding to h Each attention head has a parallel subspace, and the query vector, key vector, and value vector for each subspace are calculated. The output features of each attention head are then calculated. The output features of each attention head are concatenated and fused through a linear projection layer. The fused features are then normalized with the spatiotemporal features after adding position embeddings. A nonlinear transformation is then applied to the normalized features to obtain long-term temporal features. in, for t Long-term temporal characteristics of time; For layer normalization operation; It is a feedforward neural network used to perform nonlinear transformations on features; These are the features after layer normalization; Local time window The spatiotemporal features within are added to the corresponding location embedding vectors: in, The characteristics are those after addition; for Spatiotemporal characteristics within; for The corresponding position embedding vector; m As a scale; This is the basic time granularity; The summed features are decomposed using a multi-head attention mechanism corresponding to the local time window, and the local attention weights are normalized within the local time window. Multi-scale feature output is then obtained through a Transformer layer. The multi-scale features are then fused into short-time features through splicing and linear projection. in, for t Short-term temporal characteristics of time; This is the fusion projection matrix for multi-scale features; For feature splicing; The long-term and short-term features obtained from modeling are weighted and fused to generate a fused feature vector containing multi-scale semantics: in, for t The fused feature vector at each time step; This is a learnable fusion parameter matrix.

6. The abnormal video detection method based on differentiated semantic communication according to claim 5, characterized in that, The fused feature vectors interact with a dual-memory vector library through query and read operations to generate memory-enhanced feature vectors, including: The query operation calculates the similarity between the fused feature vector and the vectors in the dual-memory vector library: in, The prototype matching score for the normal vector library; For the Sigmoid function; This is a normal vector library; For feature dimensions; The prototype matching score for the anomaly vector library; An anomaly vector library; The read operation, based on the query result, generates a memory-enhanced feature vector through a weighted summation of vectors: in, Enhance the feature vector for normal video memory; Enhance the feature vector for abnormal video memory.

7. The abnormal video detection method based on differentiated semantic communication according to claim 6, characterized in that, The fused feature vector interacts with the dual-memory vector library through query and read operations to generate memory-enhanced feature vectors. These memory-enhanced feature vectors include normal video memory-enhanced feature vectors and abnormal video memory-enhanced feature vectors, which are stored in the normal vector library and the abnormal vector library, respectively. The system also includes: Construct clear class boundaries using a two-vector separation loss: in, For two-vector separation loss; These are the preset hyperparameters.

8. The abnormal video detection method based on differentiated semantic communication according to claim 7, characterized in that, The prediction module performs anomaly detection on video data based on fused feature vectors and memory-enhanced feature vectors, including: The prediction module performs anomaly detection on video data based on fused feature vectors and memory-enhanced feature vectors: in, The input features for the classifier; For activation functions; The feature mapping weight matrix; For classification probability; For activation functions; It is a three-layer fully connected network.

9. The abnormal video detection method based on differentiated semantic communication according to claim 7, characterized in that, The loss function used in the abnormal video detection model is: in, The loss function; Cross-entropy; and These are custom hyperparameters.

10. An anomaly video detection device based on differentiated semantic communication, characterized in that, include: The model building module is used to build an abnormal video detection model. The model includes a feature extraction network, a three-dimensional convolutional network, a multi-scale Transformer module, a dual vector memory library, and a prediction module. The dual vector memory library includes a normal vector library and an abnormal vector library. The data transmission module is used to acquire video data in real time, perform dynamic evaluation of the video data, transmit complete video frames for video data with sudden scene changes, and transmit key area frames for video data without sudden scene changes. The feature extraction module is used to extract spatial features from the video data received and transmitted by the feature extraction network, obtain a spatial feature sequence, and input it into a 3D convolutional network for temporal modeling to obtain a spatiotemporal feature sequence. The multi-scale Transformer module performs long-term and short-term modeling on the spatiotemporal feature sequence respectively, and performs weighted fusion on the feature vectors obtained from the modeling to generate a fused feature vector containing multi-scale semantics. The memory enhancement module is used to fuse feature vectors and interact with the dual memory vector library through query and read operations to generate memory-enhanced feature vectors. The memory-enhanced feature vectors include normal video memory-enhanced feature vectors and abnormal video memory-enhanced feature vectors, which are stored in the normal vector library and the abnormal vector library respectively. The anomaly detection module is used by the prediction module to detect anomalies in video data based on fused feature vectors and memory-enhanced feature vectors.

Images