Video anomaly detection method based on dynamic collaborative attention and VA-Mama decoder

Abstract

The invention provides a video anomaly detection method and device based on dynamic collaborative attention and a VA-Mama decoder, and relates to the technical field of computer vision. The method comprises the steps of performing feature extraction according to a real-time video based on a CLIP model and an MFCC algorithm; on the basis of a multi-head attention mechanism, feature weighted fusion is carried out according to the key visual features and the key frequency spectrum features, and fusion features are obtained; the fusion features are input into a VA-Mamba decoder for data decoding, and video-audio dual-channel high-dimensional features are obtained; performing video anomaly detection on the real-time video according to the video-audio dual-channel high-dimensional features based on a multi-classification anomaly detection model to obtain an anomaly detection result; based on a preset Prompt template, according to the anomaly detection result, performing anomaly analysis by using the fine-tuned Qwen2.5 model to obtain an anomaly analysis result; and storing the anomaly detection result and the anomaly analysis result, and giving an alarm. The video anomaly detection method is an efficient and accurate video anomaly detection method combining the video stream and the audio stream.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and particularly to a video anomaly detection method and device based on dynamic collaborative attention and a VA-Mamba decoder. Background Art

[0002] In the industrial production process, video clip detection is an important link to ensure production stability and product quality. Traditional video clip detection methods mainly rely on video data, and usually use image processing technology or rule-based determination methods to identify abnormal clips. These methods often cannot provide high enough detection accuracy when dealing with complex and variable production environments. Especially when an anomaly occurs, the visual information in the video stream often fails to fully reveal all the abnormal features.

[0003] The combination of video stream and audio stream can make up for this deficiency. Audio data can capture sound features that cannot be reflected by the video stream, such as noise changes in equipment failures, environmental interference sounds, etc. Through the collaborative analysis of video and audio, abnormal clips can be more comprehensively identified, and the detection accuracy and robustness can be improved.

[0004] With the development of industrial Internet and artificial intelligence technologies, video clip detection methods based on multi-modal data fusion have gradually become a trend. By combining video data and audio data, not only can the limitations of a single data source be overcome, but also the detection accuracy and robustness can be improved. The combination of deep learning and inference models can conduct a more in-depth analysis of video and audio data, thereby providing more accurate decision-making support for anomaly handling in industrial production.

[0005] In the prior art, there is a lack of an efficient and accurate video anomaly detection method that combines video stream and audio stream. Summary of the Invention

[0006] To solve the technical problem that the prior art cannot provide high enough detection accuracy in complex and variable production environments, embodiments of the present invention provide a video anomaly detection method and device based on dynamic collaborative attention and a VA-Mamba decoder. The technical solution is as follows:

[0007] On the one hand, a video anomaly detection method based on dynamic collaborative attention and a VA-Mamba decoder is provided. This method is implemented by a video anomaly detection device, and the method includes:

[0008] Collect data through the real-time monitoring system of the industrial site to obtain real-time video;

[0009] Based on the CLIP model and the MFCC algorithm, extract features from the real-time video to obtain key visual features and key spectral features;

[0010] Based on the multi-head attention mechanism, feature weighted fusion is performed according to the key visual features and key spectral features to obtain fused features;

[0011] The fused features are input into the VA-Mamba decoder for data decoding to obtain the visual-audio dual-channel high-dimensional features;

[0012] Based on the multi-class anomaly detection model, video anomaly detection is performed on the real-time video according to the visual-audio dual-channel high-dimensional features to obtain the anomaly detection result;

[0013] Based on the preset Prompt template, anomaly parsing is performed using the fine-tuned Qwen2.5 model according to the anomaly detection result to obtain the anomaly parsing result;

[0014] The anomaly detection result and the anomaly parsing result are saved and an alarm is issued.

[0015] On the other hand, a video anomaly detection device based on dynamic collaborative attention and the VA-Mamba decoder is provided. This device is applied to the video anomaly detection method based on dynamic collaborative attention and the VA-Mamba decoder. The device includes:

[0016] A video acquisition module for collecting data through the real-time monitoring system in the industrial site to obtain real-time video;

[0017] A feature extraction module for extracting features from the real-time video based on the CLIP model and the MFCC algorithm to obtain key visual features and key spectral features;

[0018] A feature fusion module for performing feature weighted fusion according to the key visual features and key spectral features based on the multi-head attention mechanism to obtain fused features;

[0019] A VA-Mamba decoding module for inputting the fused features into the VA-Mamba decoder for data decoding to obtain the visual-audio dual-channel high-dimensional features;

[0020] A first anomaly detection module for performing video anomaly detection on the real-time video based on the multi-class anomaly detection model according to the visual-audio dual-channel high-dimensional features to obtain the anomaly detection result;

[0021] A second anomaly detection module for performing anomaly parsing using the fine-tuned Qwen2.5 model according to the anomaly detection result based on the preset Prompt template to obtain the anomaly parsing result;

[0022] An anomaly alarm module for saving the anomaly detection result and the anomaly parsing result and issuing an alarm.

[0023] On the other hand, a video anomaly detection device is provided, which includes: a processor; a memory storing computer-readable instructions, and when the computer-readable instructions are executed by the processor, any one of the video anomaly detection methods based on dynamic collaborative attention and VA-Mamba decoder as described above is implemented.

[0024] On the other hand, a computer-readable storage medium is provided, in which at least one instruction is stored, and the at least one instruction is loaded and executed by a processor to implement any one of the video anomaly detection methods based on dynamic collaborative attention and VA-Mamba decoder as described above.

[0025] The beneficial effects brought by the technical solutions provided in the embodiments of the present invention at least include:

[0026] The present invention proposes a video anomaly detection method based on dynamic collaborative attention and VA-Mamba decoder. By fusing video stream and audio stream data, a unified feature representation is generated using the dynamic collaborative attention mechanism, and efficient anomaly detection is achieved in combination with the VA-Mamba decoder. The present invention effectively improves the detection accuracy, especially the anomaly recognition ability in complex environments. By combining the Qwen model for in-depth reasoning of anomaly segments, the cause of the anomaly can be traced, potential risks can be evaluated, and a detailed analysis report can be generated. Compared with traditional methods, the present invention has significant advantages in detection accuracy, real-time responsiveness, and environmental adaptability, and is applicable to safety monitoring, equipment maintenance, and process optimization in industrial production, with broad application prospects and important industrial value. The present invention is an efficient and accurate video anomaly detection method that combines video stream and audio stream. Description of the Drawings

[0027] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present invention, and those of ordinary skill in the art can obtain other drawings without creative efforts based on these drawings.

[0028] Figure 1 is a flowchart of a video anomaly detection method based on dynamic collaborative attention and VA-Mamba decoder provided by an embodiment of the present invention;

[0029] Figure 2 is a block diagram of a video anomaly detection device based on dynamic collaborative attention and VA-Mamba decoder provided by an embodiment of the present invention;

[0030] Figure 3It is a schematic structural diagram of a video anomaly detection device provided by an embodiment of the present invention. Detailed implementation manners

[0031] The following describes the technical solutions in the present invention in conjunction with the accompanying drawings.

[0032] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to represent examples, illustrations or explanations. Any embodiment or design solution described as an "example" in the present invention should not be construed as being more preferred or having more advantages than other embodiments or design solutions. Exactly speaking, the use of the word "example" is intended to present concepts in a specific way. In addition, in the embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one of the two.

[0033] In the embodiments of the present invention, "image" and "picture" can sometimes be used interchangeably. It should be noted that when the difference is not emphasized, the meanings they express are the same. "(of)", "corresponding", and "corresponding" can sometimes be used interchangeably. It should be noted that when the difference is not emphasized, the meanings they express are the same.

[0034] In the embodiments of the present invention, sometimes subscripts such as W1 may be written in a non-subscript form such as W1. When the difference is not emphasized, the meanings they express are the same.

[0035] To make the technical problems, technical solutions and advantages to be solved by the present invention clearer, the following will be described in detail in conjunction with the accompanying drawings and specific embodiments.

[0036] The embodiments of the present invention provide a video anomaly detection method based on dynamic collaborative attention and a VA-Mamba decoder. This method can be implemented by a video anomaly detection device, and this video anomaly detection device can be a terminal or a server. As Figure 1 shown in the flowchart of the video anomaly detection method based on dynamic collaborative attention and a VA-Mamba decoder, the processing flow of this method can include the following steps:

[0037] S1. Collect data through the real-time monitoring system of the industrial site to obtain real-time video.

[0038] In a feasible implementation manner, multi-view monitoring video data is collected through the real-time monitoring system of the industrial site to ensure that key equipment and areas in the production scene are comprehensively covered from different shooting angles; the collected monitoring video data is time-stamped and marked with corresponding times to ensure the consistency of different modality data in the unified time dimension, providing an accurate data basis for subsequent feature extraction and fusion.

[0039] S2. Based on the CLIP model and the MFCC algorithm, extract features from the real-time video to obtain key visual features and key spectral features.

[0040] Optionally, based on the CLIP model and the MFCC algorithm, extract features from the real-time video to obtain key visual features and key spectral features, including:

[0041] Perform data modality decomposition on the real-time video to obtain a picture stream and an audio stream;

[0042] According to the picture stream, extract features through the CLIP model to obtain key visual features;

[0043] According to the audio stream, use the MFCC algorithm to extract features to obtain key spectral features.

[0044] In a feasible implementation, extract picture stream information from the video, and use the image encoder in the Contrastive Language-Image Pre-training (CLIP) model to extract features from each frame of the picture to obtain the semantic features of the picture stream. The extraction process is as follows in Equation (1):

[0045] (1);

[0046] Where, I represents the sampled picture stream, represents the extracted semantic features of the picture stream.

[0047] Extract the audio stream separately from the video. After processing, input the audio signal into the Mel-Frequency Cepstral Coefficients (MFCC) algorithm for feature extraction to generate Mel-Frequency Cepstral Coefficient features, which are used to capture the key spectral information of the audio signal. The audio signal needs to be , divided into multiple short-time windows, and perform Fourier transform on each window to obtain the spectrum of each time window. The calculation formula is as follows in Equation (2):

[0048] (2);

[0049] Where, X[k] is the amplitude of the k-th frequency component; x[n] is the time-domain signal of the n-th window; w[n] is the window function; N is the length of the window; is a complex exponential function; j represents the imaginary unit, indicating the rotation components of different frequencies. By setting the window size, make its eigenvalue sequence aligned with the picture stream features.

[0050] C(m) is the generated MFCC eigenvalue, and the process formula is as follows in Equation (3):

[0051] (3);

[0052] where X(n) is the Fourier transform result of the audio signal, and D is the number of filters.

[0053] S3. Based on the multi-head attention mechanism, perform feature weighted fusion according to the key visual features and key spectral features to obtain the fused features.

[0054] Optionally, based on the multi-head attention mechanism, perform feature weighted fusion according to the key visual features and key spectral features to obtain the fused features, including:

[0055] Align the key visual features and key spectral features in the time series to obtain the time-aligned visual features and time-aligned spectral features;

[0056] Align the time-aligned visual features and time-aligned spectral features in the feature dimension to obtain the aligned visual features and aligned spectral features.

[0057] According to the aligned visual features and aligned spectral features, perform feature weighted fusion through the dynamic collaborative attention mechanism module to obtain the fused features.

[0058] In a feasible implementation manner, since the feature extraction process may cause the picture stream features and audio stream features to not be completely corresponding in the time series, it is necessary to align the two in the time series. If the sequence lengths of the picture stream features and audio stream features are inconsistent, select the one with the shorter time series as the benchmark for alignment, and the process formula is as follows in Equation (4):

[0059] (4);

[0060] where is the unified sequence length after alignment, respectively represent the sequence lengths of the picture stream and audio stream features.

[0061] The aligned picture stream features and audio stream features may have different feature dimensions. For the convenience of subsequent fusion, it is necessary to perform dimension mapping and normalization processing on them. Assuming the target feature dimension is d, the mapping formula is as follows in Equation (5):

[0062] (5);

[0063] Among them, V and A respectively represent the picture stream and audio stream features before alignment, V′ and A′ respectively represent the picture stream and audio stream features after dimension mapping, d is the unified feature dimension (such as set to 512); Map is the mapping function.

[0064] The aligned picture stream feature V′ and audio stream feature A′ are input into the cross-attention fusion module, and the multi-head attention mechanism is used to perform weighted fusion on different modality features.

[0065] Generate query vectors for the picture stream feature V′ and audio stream feature A′ respectively 、key vectors and value vectors , and this process is as shown in the following formulas (6), (7):

[0066] (6);

[0067] (7);

[0068] Among them, is the weight matrix for the picture stream feature, is the weight matrix for the audio stream feature.

[0069] Calculate the cross-attention scores between the picture stream and the audio stream, and this process is as shown in the following formulas (8), (9):

[0070] (8);

[0071] (9);

[0072] Perform weighted summation on the results after co-attention fusion, and this process is as shown in the following formula (10):

[0073] (10);

[0074] After being processed by the co-attention mechanism, a unified multi-modal fusion feature representation is generated for the decoding and anomaly detection analysis of the subsequent VA-Mamba decoder. are the adaptive weight coefficients of the picture stream cross-attention score and the audio stream cross-attention score respectively.

[0075] S4. Input the fusion feature into the VA-Mamba decoder for data decoding to obtain the visual-audio dual-channel high-dimensional feature.

[0076] Optionally, input the fusion feature into the VA-Mamba decoder for data decoding to obtain the visual-audio dual-channel high-dimensional feature, including:

[0077] Normalize the fused features to obtain the processed fused features;

[0078] Perform bidirectional hidden layer space mapping based on the processed fused features to obtain bidirectional hidden layer states;

[0079] Based on a preset output projection matrix, perform feature calculation according to the bidirectional hidden layer states to obtain global features;

[0080] Based on the residual connection method, according to the fused features and the global features, perform feature decoding through a multi-layer perceptron to obtain the visual-audio dual-channel high-dimensional features.

[0081] In a feasible implementation, the fused features are input into the VA-Mamba decoder module. To improve the stability and adaptability of the features, a normalization operation (Root Mean Square LayerNormalization, RMS) is performed on them to reduce the influence of outliers in the feature distribution on subsequent processing and avoid numerical instability caused by feature dimension differences. The normalization process is as shown in Equation (11) below:

[0082] (11);

[0083] Where, is the normalized feature for subsequent calculation, is the input feature, d is the dimension of the input feature to ensure that each dimension of the feature is normalized, and ϵ is a very small positive number used to prevent division by zero in the denominator.

[0084] The normalization process adjusts the overall amplitude of the input features, making the subsequent VA-Mamba module more robust during processing and effectively reducing the sensitivity of the model to the scale of the input features.

[0085] The core of the VA-Mamba decoder lies in its bidirectional feature processing ability, comprehensively capturing the dynamic information of the input features from both the video and audio dimensions, and optimizing the feature representation through residual connections and a multi-layer perceptron (MLP).

[0086] Calculation of the forward and backward bidirectional hidden layer states. This calculation process is as shown in Equations (12) and (13) below:

[0087] (12);

[0088] (13);

[0089] Where, and [[ID=4S]] They are the hidden layer states of the forward propagation and the backward propagation respectively, which are used to represent the dependencies of the feature sequence in the time dimension; and They are the forward and backward state transition matrices respectively, which define how the time series features are transmitted as the moment t changes; and are the input projection matrices, which are used to project the input features and into the hidden layer space, so that the original features and the context information are fully integrated.

[0090] The forward and backward bidirectional hidden layer states are fused in the time dimension to integrate the global context information of the sequence and generate the global feature output at the current moment. The calculation process is as follows in Equation (14);

[0091] (14);

[0092] where, and are the output projection matrices of the forward and backward hidden layer states, which are used to convert the hidden layer information into the features in the output space; is the feature output after fusion at the current moment, which combines the essence of the front and back information in the time series. This fusion process can significantly improve the globality of feature expression, enabling subsequent modules to perform abnormal segment detection based on more comprehensive features.

[0093] To further improve the robustness and expression ability of the output features, the residual connection and the multi-layer perceptron mechanism are used to optimize the features. The optimization formula is as follows (15):

[0094] (15);

[0095] where, y′ is the high-dimensional visual-audio dual-channel feature of the final output of the VA-Mamba decoder, represents the input of the residual connection, which can alleviate the problem of gradient disappearance and retain the characteristics of the original input; RMS is the normalization operation, which smooths the feature distribution again; MLP is the multi-layer perceptron structure, which further extracts high-level features through non-linear activation functions.

[0096] This optimization process combines the original information and the fusion information of the input features, and introduces non-linear characteristics with the help of MLP, making the output features have stronger expression ability.

[0097] S5. Based on the multi-class anomaly detection model, video anomaly detection is performed on the real-time video according to the high-dimensional visual-audio dual-channel feature to obtain the anomaly detection result.

[0098] Among them, the training dataset of the multi-class anomaly detection model includes historical production data and labeled anomaly categories; the anomaly categories include normal, equipment failure, environmental interference, and process anomaly.

[0099] In a feasible implementation, a multi-class anomaly detection model based on historical data and annotation information is constructed to achieve time period anomaly detection of multi-modal features, data preparation, and feature standardization. Using historical production data, combined with manually labeled anomaly categories (such as normal, equipment failure, environmental interference, process anomaly, etc.), a training set of multi-modal features and class labels is constructed.

[0100] Standardize the input features to eliminate the influence of dimensions and improve the convergence speed of the classification model. The standardization formula is as follows in Equation (16):

[0101] (16);

[0102] Where, and are the mean and standard deviation of the features respectively, represents the standardized feature, which is used for subsequent classification tasks.

[0103] Use the softmax function to map the standardized features to the probability distributions of each category, and the calculation is as follows in Equation (17):

[0104] (17);

[0105] Where, is the activation value of the feature in the category , C is the total number of classification categories, represents the probability that the input feature belongs to the category . According to the calculation results, select the category corresponding to the maximum probability as the prediction result, and the formula is as follows in Equation (18):

[0106] (18);

[0107] Adopt the cross-entropy loss function as the optimization objective to train the model to improve the classification accuracy. The loss function is as follows in Equation (19):

[0108] (19);

[0109] Where, is the one-hot encoding of the true class label, is the probability distribution predicted by the model. Calculate the gradient through the backpropagation algorithm and use the Adam optimizer to iteratively update the model parameters to make the model gradually reach a higher classification accuracy during the training stage.

[0110] Use the trained classification model to perform anomaly detection on real-time production data. For each time period, record the key information: the anomaly category (such as equipment failure, environmental interference, etc.), the time interval when the anomaly occurred, and the key features or feature combinations that caused the anomaly.

[0111] S6. Based on the preset Prompt template, according to the anomaly detection results, use the fine-tuned Qwen2.5 model to perform anomaly parsing and obtain the anomaly parsing results.

[0112] Optionally, based on the preset Prompt template, according to the anomaly detection results, use the fine-tuned Qwen2.5 model to perform anomaly parsing and obtain the anomaly parsing results, including:

[0113] Perform data processing on the anomaly detection results to obtain the processed detection results;

[0114] Based on the preset Prompt template and the industrial knowledge base of the fine-tuned Qwen2.5 model, perform anomaly cause reasoning according to the processed detection results to obtain the anomaly cause;

[0115] According to the anomaly cause, use the fine-tuned Qwen2.5 model to perform impact assessment and obtain the assessment results;

[0116] Generate an anomaly analysis report based on the anomaly detection results, anomaly causes, and assessment results.

[0117] In a feasible implementation, organize the anomaly detection results (such as anomaly category, time, feature values, etc.) into a structured data format, such as JSON or table form. Filter redundant data and eliminate irrelevant anomaly records to ensure the efficiency and accuracy of the input information.

[0118] Design a scientific Prompt template and a regularized output format, and input the anomaly data into the Qwen model for reasoning and cause mining. The Qwen model combines the built-in industrial knowledge base, performs pattern matching on the input anomaly information, determines whether there are similar anomaly cases in history, and uses the reasoning ability of the model to analyze the causal chain of the anomaly phenomenon based on the time series relationship of the anomaly data and infer the root cause.

[0119] Use the Qwen model to conduct potential causal analysis of abnormal events and comprehensively evaluate the scope and severity of the impact: Based on the time series relationship of multimodal data, the Qwen model infers the possible root causes of anomalies (such as equipment failures, process design defects, external environmental interferences, etc.). Evaluate the potential impact of the anomaly on the production process, such as decreased output, equipment damage, increased energy consumption, etc.; generate an evaluation report and classify it (low risk, medium risk, high risk). Predict the possible development direction of the abnormal problem, such as the problem continuing to worsen, self-healing in the short term, or sporadic events.

[0120] Generate an evaluation report and decision-making suggestions for anomaly detection based on the anomaly analysis results obtained from the Qwen model. The evaluation report includes anomaly description, cause analysis, risk assessment, and solutions.

[0121] S7. Save the anomaly detection results and the anomaly analysis results, and issue an alarm.

[0122] In a feasible implementation, save the anomaly detection results and the anomaly analysis results to the abnormal event database for subsequent retrieval and learning. While analyzing and making decisions, trigger a real-time alarm through the alarm module to achieve a rapid response to industrial production anomalies.

[0123] The present invention proposes a video anomaly detection method based on dynamic collaborative attention and a VA-Mamba decoder. By fusing video stream and audio stream data, using the dynamic collaborative attention mechanism to generate a unified feature representation, and combining with the VA-Mamba decoder to achieve efficient anomaly detection. The present invention effectively improves the detection accuracy, especially the anomaly recognition ability in complex environments. By combining the Qwen model to conduct in-depth reasoning on abnormal segments, it is possible to trace the causes of anomalies, evaluate potential risks, and generate a detailed analysis report. Compared with traditional methods, the present invention has significant advantages in detection accuracy, real-time responsiveness, and environmental adaptability, and is applicable to safety monitoring, equipment maintenance, and process optimization in industrial production, with broad application prospects and important industrial value. The present invention is an efficient and accurate video anomaly detection method that combines video stream and audio stream.

[0124] Figure 2 It is a block diagram of a video anomaly detection device shown according to an exemplary embodiment. This device is used for the video anomaly detection method based on dynamic collaborative attention and a VA-Mamba decoder. Refer to Figure 2 As shown, this device includes a video acquisition module 210, a feature extraction module 220, a feature fusion module 230, a VA-Mamba decoding module 240, a first anomaly detection module 250, a second anomaly detection module 260, and an anomaly alarm module 270. Among them:

[0125] The video acquisition module 210 is used to collect data through the real-time monitoring system of the industrial site to obtain real-time videos;

[0126] The feature extraction module 220 is used to extract features based on the CLIP model and the MFCC algorithm according to the real-time video to obtain key visual features and key spectral features;

[0127] The feature fusion module 230 is used to perform feature weighted fusion based on the multi-head attention mechanism according to the key visual features and key spectral features to obtain fused features;

[0128] The VA-Mamba decoding module 240 is used to input the fused features into the VA-Mamba decoder for data decoding to obtain visual-audio dual-channel high-dimensional features;

[0129] The first anomaly detection module 250 is used to perform video anomaly detection on the real-time video based on the multi-class anomaly detection model according to the visual-audio dual-channel high-dimensional features to obtain anomaly detection results;

[0130] The second anomaly detection module 260 is used to perform anomaly parsing based on the preset Prompt template according to the anomaly detection results using the fine-tuned Qwen 2.5 model to obtain anomaly parsing results;

[0131] The anomaly alarm module 270 is used to save the anomaly detection results and anomaly parsing results and issue an alarm.

[0132] Optionally, the feature extraction module 220 is further used for:

[0133] Perform data modal decomposition on the real-time video to obtain a picture stream and an audio stream;

[0134] Extract features through the CLIP model according to the picture stream to obtain key visual features;

[0135] Extract features using the MFCC algorithm according to the audio stream to obtain key spectral features.

[0136] Optionally, the feature fusion module 230 is further used for:

[0137] Perform time series alignment on the key visual features and key spectral features to obtain time-aligned visual features and time-aligned spectral features;

[0138] Perform feature dimension alignment on the time-aligned visual features and time-aligned spectral features to obtain aligned visual features and aligned spectral features.

[0139] Perform feature weighted fusion on the aligned visual features and aligned spectral features through the dynamic collaborative attention mechanism module to obtain fused features.

[0140] Optionally, the VA-Mamba decoding module 240 is further configured to:

[0141] Normalize the fused features to obtain the processed fused features;

[0142] Perform bidirectional hidden layer space mapping based on the processed fused features to obtain bidirectional hidden layer states;

[0143] Based on a preset output projection matrix, perform feature calculation according to the bidirectional hidden layer states to obtain global features;

[0144] Based on the residual connection method, perform feature decoding through a multi-layer perceptron according to the fused features and the global features to obtain the visual-audio dual-channel high-dimensional features.

[0145] Among them, the training dataset of the multi-class anomaly detection model includes historical production data and labeled anomaly categories; the anomaly categories include normal, equipment failure, environmental interference, and process anomaly.

[0146] Optionally, the second anomaly detection module 260 is further configured to:

[0147] Process the anomaly detection results to obtain the processed detection results;

[0148] Based on a preset Prompt template and the industrial knowledge base of the fine-tuned Qwen2.5 model, perform anomaly cause reasoning according to the processed detection results to obtain the anomaly cause;

[0149] According to the anomaly cause, use the fine-tuned Qwen2.5 model to perform impact assessment to obtain the assessment results;

[0150] Generate an anomaly analysis report according to the anomaly detection results, the anomaly cause, and the assessment results.

[0151] The present invention proposes a video anomaly detection method based on dynamic collaborative attention and a VA-Mamba decoder. By fusing video stream and audio stream data, using the dynamic collaborative attention mechanism to generate a unified feature representation, and combining with the VA-Mamba decoder to achieve efficient anomaly detection. The present invention effectively improves the detection accuracy, especially the anomaly recognition ability in complex environments. By combining the Qwen model to perform in-depth reasoning on anomaly segments, it can trace the anomaly cause, evaluate potential risks, and generate a detailed analysis report. Compared with traditional methods, the present invention has significant advantages in detection accuracy, real-time responsiveness, and environmental adaptability, and is applicable to safety monitoring, equipment maintenance, and process optimization in industrial production, with broad application prospects and important industrial value. The present invention is an efficient and accurate video anomaly detection method that combines video stream and audio stream.

[0152] Figure 3 This is a schematic structural diagram of a video anomaly detection device provided by an embodiment of the present invention. As shown in Figure 3 the figure, the video anomaly detection device may include the above-mentioned Figure 2 video anomaly detection device based on dynamic collaborative attention and VA-Mamba decoder as shown. Optionally, the video anomaly detection device 310 may include a first processor 2001.

[0153] Optionally, the video anomaly detection device 310 may further include a memory 2002 and a transceiver 2003.

[0154] Among them, the first processor 2001, the memory 2002, and the transceiver 2003 may be connected through a communication bus.

[0155] Next, the various components of the video anomaly detection device 310 will be specifically introduced in conjunction with Figure 3 the figure:

[0156] Among them, the first processor 2001 is the control center of the video anomaly detection device 310, which may be a single processor or a collective term for multiple processing elements. For example, the first processor 2001 is one or more central processing units (CPUs), or may be an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention, such as: one or more digital signal processors (DSPs), or one or more field programmable gate arrays (FPGAs).

[0157] Optionally, the first processor 2001 may execute various functions of the video anomaly detection device 310 by running or executing software programs stored in the memory 2002 and calling data stored in the memory 2002.

[0158] In a specific implementation, as an embodiment, the first processor 2001 may include one or more CPUs, such as Figure 3 CPU0 and CPU1 shown in

[0159] In a specific implementation, as an embodiment, the video anomaly detection device 310 may also include multiple processors, such as Figure 3The first processor 2001 and the second processor 2004 shown in [the figure]. Each of these processors can be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor here can refer to one or more devices, circuits, and / or processing cores for processing data (such as computer program instructions).

[0160] Among them, the memory 2002 is used to store the software program for implementing the solution of the present invention and is controlled by the first processor 2001 for execution. The specific implementation manner can refer to the above method embodiments and will not be elaborated here.

[0161] Optionally, the memory 2002 can be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or can also be an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM), or other optical disc storage, optical disc storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), magnetic disk storage media, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 2002 can be integrated with the first processor 2001 or exist independently and is coupled to the first processor 2001 through the interface circuit of the video anomaly detection device 310 ( Figure 3 not shown in [the figure]). The embodiments of the present invention do not make specific limitations on this.

[0162] The transceiver 2003 is used to communicate with a network device or with a terminal device.

[0163] Optionally, the transceiver 2003 can include a receiver and a transmitter ( Figure 3 not separately shown in [the figure]). Among them, the receiver is used to implement the receiving function, and the transmitter is used to implement the sending function.

[0164] Optionally, the transceiver 2003 can be integrated with the first processor 2001 or exist independently and is coupled to the first processor 2001 through the interface circuit of the video anomaly detection device 310 ( Figure 3 not shown in [the figure]). The embodiments of the present invention do not make specific limitations on this.

[0165] It should be noted that Figure 3 The structure of the video anomaly detection device 310 shown in Figure 3 does not limit the router. The actual knowledge structure recognition device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0166] In addition, the technical effects of the video anomaly detection device 310 can refer to the technical effects of the video anomaly detection method based on dynamic collaborative attention and VA-Mamba decoder described in the above method embodiments, which will not be elaborated here.

[0167] It should be understood that the first processor 2001 in the embodiments of the present invention may be a central processing unit (CPU), and this processor may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), off-the-shelf programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or this processor may also be any conventional processor, etc.

[0168] It should also be understood that the memory in the embodiments of the present invention can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of random access memory (RAM) are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchlink dynamic random access memory (SLDRAM), and direct rambus random access memory (DR RAM).

[0169] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware, or any other combination. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the processes or functions described in the embodiments of the present invention are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by wired (such as infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access, or a data storage device such as a server or data center that contains one or more collections of available media. The available medium can be a magnetic medium (such as a floppy disk, hard disk, or magnetic tape), an optical medium (such as a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state drive.

[0170] It should be understood that the term "and / or" in this document is merely a description of the association relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone. Here, A and B can be singular or plural. In addition, the character " / " in this document generally represents an "or" relationship between the associated objects before and after, but it may also represent an "and / or" relationship, which can be specifically understood with reference to the context before and after.

[0171] In the present invention, "at least one" means one or more, and "a plurality" means two or more. "At least one of the following" or its similar expressions refer to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can represent: a, b, c, a - b, a - c, b - c, or a - b - c, where a, b, and c can be single or multiple.

[0172] It should be understood that in various embodiments of the present invention, the magnitudes of the sequence numbers of the above processes do not mean the order of execution. The order of execution of each process should be determined by its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present invention.

[0173] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professionals can use different methods for each specific application to implement the described functions, but such implementation should not be considered to exceed the scope of the present invention.

[0174] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the above-described devices, apparatuses, and units can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated herein.

[0175] In several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division, and there can be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces, and the indirect couplings or communication connections of the devices or units can be in electrical, mechanical, or other forms.

[0176] The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0177] In addition, the functional units in each embodiment of the present invention can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit.

[0178] If the above-mentioned functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical discs.

[0179] The above is only the specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention can easily think of changes or substitutions, which should all be covered by the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.

Claims

1. A video anomaly detection method based on dynamic collaborative attention and VA-Mamba decoder, characterized in that The method includes: Collecting data through the real-time monitoring system of the industrial site to obtain real-time videos; Based on the CLIP model and the MFCC algorithm, extracting features from the real-time videos to obtain key visual features and key spectral features; Based on the multi-head attention mechanism, performing feature weighted fusion according to the key visual features and key spectral features to obtain fused features; Inputting the fused features into the VA-Mamba decoder for data decoding to obtain visual-audio dual-channel high-dimensional features; Based on the multi-class anomaly detection model, performing video anomaly detection on the real-time videos according to the visual-audio dual-channel high-dimensional features to obtain anomaly detection results; Based on the preset Prompt template, using the fine-tuned Qwen2.5 model to perform anomaly parsing according to the anomaly detection results to obtain anomaly parsing results; Saving the anomaly detection results and anomaly parsing results and sending an alarm.

2. The video anomaly detection method based on dynamic collaborative attention and VA-Mamba decoder according to claim 1, wherein The extracting features from the real-time videos based on the CLIP model and the MFCC algorithm to obtain key visual features and key spectral features includes: Performing data modal decomposition on the real-time videos to obtain a picture stream and an audio stream; According to the picture stream, extracting features through the CLIP model to obtain key visual features; According to the audio stream, using the MFCC algorithm to extract features to obtain key spectral features.

3. The video anomaly detection method based on dynamic collaborative attention and VA-Mamba decoder according to claim 1, characterized in that The performing feature weighted fusion according to the key visual features and key spectral features based on the multi-head attention mechanism to obtain fused features includes: Aligning the key visual features and key spectral features in time series to obtain time-aligned visual features and time-aligned spectral features; Aligning the feature dimensions of the time-aligned visual features and time-aligned spectral features to obtain aligned visual features and aligned spectral features; According to the aligned visual features and aligned spectral features, performing feature weighted fusion through the dynamic collaborative attention mechanism module to obtain fused features.

4. The video anomaly detection method based on dynamic collaborative attention and VA-Mamba decoder according to claim 1, characterized in that, The inputting the fused features into the VA-Mamba decoder for data decoding to obtain visual-audio dual-channel high-dimensional features includes: Performing normalization processing on the fused features to obtain processed fused features; Performing bidirectional hidden layer space mapping according to the processed fused features to obtain bidirectional hidden layer states; Based on the preset output projection matrix, performing feature calculation according to the bidirectional hidden layer states to obtain global features; Based on the residual connection method, performing feature decoding through a multi-layer perceptron according to the fused features and global features to obtain visual-audio dual-channel high-dimensional features.

5. The video anomaly detection method based on dynamic collaborative attention and VA-Mamba decoder according to claim 1, characterized in that The training data set of the multi-class anomaly detection model includes historical production data and labeled anomaly categories; the anomaly categories include normal, equipment failure, environmental interference, and process anomaly.

6. The video anomaly detection method based on dynamic collaborative attention and VA-Mamba decoder according to claim 1, characterized in that The performing anomaly parsing using the fine-tuned Qwen2.5 model according to the anomaly detection results based on the preset Prompt template to obtain anomaly parsing results includes: Performing data processing on the anomaly detection results to obtain processed detection results; Based on the preset Prompt template and the industrial knowledge base of the fine-tuned Qwen2.5 model, inferring the anomaly causes according to the processed detection results to obtain anomaly causes; Based on the abnormal cause, use the fine-tuned Qwen2.5 model to conduct impact assessment and obtain the assessment result; Generate an abnormal analysis report according to the abnormal detection result, the abnormal cause, and the assessment result.

7. A video anomaly detection device based on dynamic collaborative attention and a VA-Mamba decoder, the video anomaly detection device based on dynamic collaborative attention and a VA-Mamba decoder is used to implement the video anomaly detection method based on dynamic collaborative attention and a VA-Mamba decoder according to any one of claims 1-6, characterized in that, The device includes: A video acquisition module, configured to collect data through the real-time monitoring system of the industrial site to obtain real-time video; A feature extraction module, configured to extract features based on the CLIP model and the MFCC algorithm according to the real-time video to obtain key visual features and key spectral features; A feature fusion module, configured to perform feature weighted fusion based on the multi-head attention mechanism according to the key visual features and the key spectral features to obtain fused features; A VA-Mamba decoding module, configured to input the fused features into a VA-Mamba decoder for data decoding to obtain video-audio dual-channel high-dimensional features; A first abnormal detection module, configured to perform video abnormal detection on the real-time video based on a multi-class abnormal detection model according to the video-audio dual-channel high-dimensional features to obtain an abnormal detection result; A second abnormal detection module, configured to perform abnormal parsing using the fine-tuned Qwen2.5 model according to the abnormal detection result based on a preset Prompt template to obtain an abnormal parsing result; An abnormal alarm module, configured to save the abnormal detection result and the abnormal parsing result and issue an alarm.

8. The video anomaly detection device based on dynamic collaborative attention and VA-Mamba decoder according to claim 1, wherein The feature extraction module is further configured to: Perform data modality decomposition on the real-time video to obtain a picture stream and an audio stream; Extract features through the CLIP model according to the picture stream to obtain key visual features; Extract features using the MFCC algorithm according to the audio stream to obtain key spectral features.

9. A video anomaly detection device, characterized in that, The video abnormal detection device includes: A processor; A memory, on which computer-readable instructions are stored, and when the computer-readable instructions are executed by the processor, the method described in any one of claims 1 to 6 is implemented.

10. A computer-readable storage medium, characterized in that, Program code is stored in the computer-readable storage medium, and the program code can be called by the processor to execute the method described in any one of claims 1 to 6.

Images