Cloud edge multi-mode abnormal event detection system based on dynamic decision-making mechanism

Abstract

The invention is suitable for the technical field of anomaly detection, and particularly relates to a cloud edge multi-mode abnormal event detection system based on a dynamic decision-making mechanism, and the system comprises a data collection module which is used for collecting videos and audios, and obtaining multi-mode data; the data preprocessing module is used for preprocessing to obtain preprocessed data; the key feature extraction module is used for carrying out key frame feature extraction and key frequency band feature extraction; the multi-mode anomaly detection module is used for carrying out anomaly judgment; and the cloud large-model intelligent processing module is used for analyzing the detected abnormal fragments and outputting a generative text result. According to the cloud edge multi-mode abnormal event detection system based on the dynamic decision-making mechanism provided by the invention, more comprehensive event understanding and anomaly detection are provided through combination of video and audio data. And deep features are extracted at the edge end by using the convolutional neural network, so that the detection accuracy is improved, and the false alarm rate is effectively reduced.

Description

Technical Field

[0001] The present invention belongs to the technical field of anomaly detection, and in particular to a cloud-edge multimodal abnormal event detection system based on a dynamic decision-making mechanism. Background Art

[0002] Existing abnormal event detection systems usually rely on single-modality data (such as video or audio only) and use traditional feature extraction and anomaly detection algorithms (e.g., threshold-based detection, simple statistical methods, etc.).

[0003] In many security monitoring and event detection systems, traditional abnormal event detection usually relies on single-modality data, such as using only video surveillance or audio monitoring. For example, some surveillance cameras only capture video images and determine whether there is suspicious activity by analyzing the movement in the image. Such systems often use simple feature extraction methods, such as threshold-based detection: when the movement detected in a certain area exceeds the preset threshold, the system will determine that an abnormality may have occurred in the area, so it is easy to be disturbed in a dynamic environment, resulting in false alarms or missed alarms. Summary of the invention

[0004] The purpose of the present invention is to provide a cloud-edge multimodal abnormal event detection system based on a dynamic decision-making mechanism, aiming to solve the problem of insufficient accuracy of abnormality identification in the prior art and prone to false alarms or missed alarms.

[0005] The present invention is implemented as follows: a cloud-edge multimodal abnormal event detection system based on a dynamic decision-making mechanism, the system comprising:

[0006] A data acquisition module is used to collect video and audio to obtain multimodal data;

[0007] A data preprocessing module, used for preprocessing multimodal data to obtain preprocessed data;

[0008] A key feature extraction module is used to extract key frame features and key frequency band features from preprocessed data;

[0009] Multimodal anomaly detection module, used to make anomaly judgments based on the extracted key frame features and key frequency band features;

[0010] The cloud-based large-model intelligent processing module is used to analyze the detected abnormal fragments and output generative text results.

[0011] Preferably, the data acquisition module includes a monitoring camera and an audio sensor, and collects audio data through the audio sensor, and collects video data through the monitoring camera.

[0012] Preferably, when the data preprocessing module processes the video, it uses the optical flow method to establish a background model and updates it, establishes a conversion model between foreground objects and background objects, uses a Gaussian mixture model to separate the foreground and background, monitors moving targets by establishing a static background model, extracts the foreground and background, analyzes the motion characteristics of the foreground, and determines whether there are any picture abnormalities; when the data preprocessing module processes the audio, it extracts features from the collected audio data, converts the audio signal from the time domain to the frequency domain, identifies the frequency spectrum information of the audio data based on the Mel-frequency cepstral coefficients, and determines whether there are any sound abnormalities.

[0013] Preferably, the key feature extraction module is implemented by a convolutional neural network, which includes a downsampling module for extracting key features, a global pooling module for integrating multimodal information, an upsampling decoder for fine classification, and a multi-layer connection and fusion module for fusing multimodal information.

[0014] Preferably, in the feature extraction process, the downsampling module retains important information by reducing the image resolution layer by layer, and the global pooling module outputs the feature map by calculating the mean of each channel to reduce the number of feature map channels;

[0015] The global pooling module merges the fusion feature layers and performs predictions through down-convolution operations respectively:

[0016] P = GlobalAvgPool (M)

[0017]

[0018] Among them, P is the feature vector after global pooling, is the prediction vector for the kth category, W k is the weight matrix corresponding to the kth category, B k is the bias term and M is the characteristic scale.

[0019] Preferably, the upsampling decoder performs prediction via transposed convolution and convolution layers, as expressed by:

[0020] M'=TransposeConv(M,W)

[0021] F1=Conv(M′,W1)+B1

[0022] F2=Conv(M′,W2)+B2

[0023] Concatenate the convolution output prediction vector:

[0024]

[0025] The classification results are expressed as:

[0026]

[0027] Where: W is the weight matrix in the transposed convolution operation, M is the feature scale, M′ is the feature map after the transposed convolution, F1 and F2 are the outputs after two convolutional layers, with dimensions of c×h×w and c′×h×w respectively, W1 and W2 are the weight matrices of the two convolutional layers, B1 and B2 are the bias terms of the two convolutional layers, is the final classification result;

[0028] In the detection result analysis, the upsampling decoder restores the feature map through transposed convolution to classify different anomaly types.

[0029] Preferably, the multimodal anomaly detection module compares and analyzes the detection results by establishing a decision analysis model, and the specific steps include:

[0030] Obtain audio data and video data, calculate the time domain features at the initialization moment, and obtain the initial anomaly score;

[0031] Through forward transformation, the audio time domain features and video time domain features are calculated;

[0032] According to the acquired time domain information, audio frequency domain features and video frequency domain features are obtained through backward transformation;

[0033] The obtained audio frequency domain features and video frequency domain features are input into the convolutional neural network to obtain the abnormality score E;

[0034] A system alertness threshold WR is set, WR = |E-E0|. When WR exceeds the threshold, an alarm is sent to remind the user, and the maximum value of the key frame segment and the CNN softmax classification abnormal result is intercepted and sent to the cloud large model processing module.

[0035] Preferably, the cloud-based large model intelligent processing module includes a generative large language model based on a pre-trained Transfomer, receives key video frame fragments, audio fragments and keywords sent by the edge, performs abnormal event analysis, generates a generative language text that can be consulted by users, and transmits the abnormal event content to the user.

[0036] Preferably, the initial anomaly score is expressed as:

[0037]

[0038] Wherein, L is the time window width, max(N) is the maximum noise value of the audio or video signal in the time window, μ(N) is the average noise value of the audio or video signal in the time window, and σ is the proportional constant.

[0039] Preferably, the decision analysis model includes several classifiers, and the abnormal problem types are divided into several categories. Different classifiers are set for different types of abnormalities. Each classifier determines whether the abnormal event of this type occurs according to its characteristics and adjusts the alarm level. The total number of categories is represented as k, and each classifier is used to detect whether there is an abnormality in the category and classify whether there is an abnormality. Each classifier corresponds to a multi-class cross entropy loss LC. If an abnormality is detected, the loss function corresponding to the classifier is:

[0040]

[0041] Where N is the total number of pixels, y i is the true label of the ith pixel. If the ith pixel is labeled as abnormal, then y i =1; if not, then y i =0, p i is the probability that the i-th pixel is labeled as an anomaly as predicted by the model.

[0042] The cloud-edge multimodal abnormal event detection system based on a dynamic decision-making mechanism provided by the present invention provides a more comprehensive event understanding and anomaly detection through the combination of video and audio data. A convolutional neural network is used at the edge to extract deep features, improve detection accuracy, and effectively reduce the false alarm rate. The reasonable allocation of complex tasks and simple tasks improves the efficiency of computing and storage, and ensures the real-time nature of abnormal monitoring. The user feedback mechanism enables the system to dynamically adjust parameters, improve response speed and adaptability. At the same time, it effectively reduces the bandwidth occupancy and operating costs of the system, and improves the reliability of the system and its ability to respond to home security threats. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 An architectural diagram of a cloud-edge multimodal abnormal event detection system based on a dynamic decision-making mechanism provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0045] It is understood that the terms "first", "second", etc. used in this application may be used herein to describe various elements, but unless otherwise specified, these elements are not limited by these terms. These terms are only used to distinguish a first element from another element. For example, without departing from the scope of this application, a first xx script may be referred to as a second xx script, and similarly, a second xx script may be referred to as a first xx script.

[0046] like Figure 1 As shown, it is an architecture diagram of a cloud-edge multimodal abnormal event detection system based on a dynamic decision-making mechanism provided by an embodiment of the present invention, and the system includes:

[0047] The data acquisition module is used to collect video and audio to obtain multimodal data.

[0048] In this system, the data acquisition module uses a combination of surveillance cameras and audio sensors, which work together to capture video and audio information of the surrounding environment. For example, when you walk on the street, the surveillance camera will record the movements of passers-by, while the audio sensor will capture their conversations. The remaining modules are all processed using devices with computing power such as computers. The data acquisition module is responsible for receiving these multimodal signals, and the data preprocessing module denoises and enhances the video and filters the audio to ensure the accuracy of subsequent processing.

[0049] The data preprocessing module is used to preprocess the multimodal data to obtain preprocessed data.

[0050] In this system, the data preprocessing module uses the background modeling method to obtain the moving target for subsequent multimodal anomaly detection. When preprocessing the video, in the detection video initialization stage, the optical flow method is used to establish and update the background model in real time. When the background model is initialized, the background frame B at time t is obtained. t , a model is constructed to represent the color mean and variance of each frame. The model of color mean is expressed as:

[0051]

[0052] Where N is the background frame B t Bt(i) is the total number of pixels in , and Bt(i) is the grayscale value of the i-th pixel.

[0053] The model for color variance is:

[0054]

[0055] in, Represents background frame B t The mean gray value of each pixel, Represents background frame B t The variance of the gray value of each pixel;

[0056] At the same time, a transformation model between foreground objects and background objects is established, which is expressed as follows:

[0057] Y=XT+Z

[0058] The variables Y, X, T, and Z represent input observations, latent variables, random driving matrices, and random errors, respectively.

[0059] In video surveillance, foreground objects usually refer to dynamic or interesting objects in the scene, such as pedestrians, vehicles or some moving objects; while background objects refer to static and unchanging parts of the scene, such as buildings, roads or trees. The core idea of ​​the foreground and background conversion model is to identify and separate the foreground and background in the video, and update the background in real time so that environmental changes or background interference will not have too much impact on foreground detection. This model usually uses some image processing and machine learning methods, such as Gaussian Mixture Model (GMM), optical flow method or convolutional neural network (CNN).

[0060] The foreground-background conversion model can improve the accuracy of abnormal event detection: the separation of foreground and background enables the system to focus only on dynamic foreground objects. For example, in a scene monitoring a station, passengers walking and standing in the background are identified as foreground, while static parts such as the ground, walls, and billboards are classified as background. Through this conversion model, the system can more accurately detect specific abnormal behaviors, such as pedestrians suddenly falling to the ground, objects left behind, etc., without false alarms due to changes in the background (such as light changes or leaves blowing in the breeze).

[0061] The foreground and background conversion model can dynamically adapt to environmental changes: The foreground and background conversion model allows the background to be automatically updated under certain conditions, which means that the system can continue to adapt even if the background objects in the environment change slightly. For example, changes in light often affect the brightness of the image, causing misidentification. The conversion model can ignore this natural change by dynamically adjusting the background, ensuring that the model always focuses on abnormal foreground activities. This dynamic adaptability is ideal for outdoor environment monitoring because it can reduce the false detection rate in situations such as day and night transitions and weather changes.

[0062] The foreground and background conversion model can reduce the demand for computing resources: Traditional video analysis systems often need to analyze the entire picture frame by frame, which consumes a lot of computing resources. By using the foreground and background conversion model, the system can only process the changing foreground part most of the time, rather than the entire image, reducing unnecessary computing resource consumption. For example, for a long video surveillance clip, the system can intelligently identify the static background area, reduce the frequency of background processing, and allocate more computing power to the foreground object, thereby optimizing resource allocation.

[0063] For example, suppose we want to detect whether there are any abnormal behaviors of passengers in the surveillance video of a station. First, the system will build a background model, classify static objects (such as the ground, walls, and benches) as background, and mark dynamic objects (such as pedestrians and moving vehicles) as foreground. At a certain moment, if a pedestrian suddenly falls to the ground, this change will be identified as a significant event in the foreground. The system will extract abnormal features in this frame and mark it as an "abnormal frame."

[0064] Specifically, the Gaussian mixture model (GMM) can be used to separate the foreground from the background. By fitting multiple Gaussian distributions, the GMM can treat frequently appearing pixels as background and briefly appearing pixels as foreground based on the change in pixel values. In addition, the optical flow method can also be used to detect the movement direction and speed changes in the foreground, thereby further distinguishing abnormal activities.

[0065] Set the background frame B t Perform mean and variance splitting, where The background frame B is represented in sequence t The mean and variance of the background frame at time t+1 are expressed as:

[0066]

[0067]

[0068] Among them, the variable λ represents the learning rate, F t represents the current frame, the input image at time t, and Represents the current frame F t The mean and variance of the pixel values.

[0069] When the data preprocessing module preprocesses the audio, it uses short-time Fourier transform to convert the audio data into frequency domain signals:

[0070]

[0071] Among them, X[m,n]: represents the short-time Fourier transform result at frequency number m and time point n, g[t] represents the function used to fill the data with zeros, t represents the sampling time, n is the sampling point number, N represents the number of sampling points, m is the frequency number, ω is the circular frequency, Ts represents the unit sampling time, and quantization noise is eliminated by adding Gaussian white noise.

[0072] In a shopping mall, the data preprocessing module builds a static background model in the initial stage, and then monitors moving targets in real time. For example, the background model can identify the normal flow of people in the mall, and when a certain area suddenly moves violently (such as someone running), the system will mark it as abnormal. In another example, in a campus environment, the audio monitoring system can analyze the surrounding noise through short-time Fourier transform. When the system detects an abnormal frequency band (such as a scream), it will be marked for timely processing.

[0073] For action features such as running, the system can rely on video background modeling and motion trajectory analysis for recognition. The background model usually uses Gaussian mixture model (GMM) or background subtraction method, which can learn the static parts of the background in the video (such as the ground, shopping mall counters, etc.). These algorithms will regard frequently appearing static areas as background and suddenly changing areas as foreground.

[0074] Once the system detects a distinct, fast-moving object in the foreground, it analyzes the object's motion characteristics, such as speed and trajectory. The characteristics of running usually include the following aspects:

[0075] Speed: Normal walking speed is low, while running speed is significantly higher than normal walking speed. The system estimates the speed by calculating the displacement change of the moving object. If the speed exceeds the preset threshold, the system will preliminarily determine that this is running behavior.

[0076] Directional consistency and acceleration: There are differences in the movement trajectories of walking and running. Runners usually maintain a consistent movement direction and a large acceleration for a short period of time, while normal walkers have slower speeds, smaller accelerations, and more random changes in direction. By analyzing the trajectory of the moving object, the system can further confirm whether it is running behavior.

[0077] In terms of specific implementation, the optical flow method can be used to extract the direction and speed of movement. The optical flow method can analyze the pixel changes in two consecutive frames and calculate the motion vector. These vectors can help the system estimate the speed and direction of movement more accurately. For crowded areas such as shopping malls, the advantage of the optical flow method is that it can capture the movement characteristics of each pedestrian in detail, so as to more accurately distinguish between walking and running.

[0078] The recognition of screams mainly depends on the audio signal processing method. Short-time Fourier transform (STFT) is one of the commonly used technologies, which can convert audio signals from the time domain to the frequency domain in order to analyze the frequency components. Through STFT, the system can decompose the audio signal into a set of frequency components within a specific time window, and each frequency corresponds to an amplitude value. The characteristic of screams is that their frequency and energy distribution are different from normal conversations and background noise.

[0079] Frequency characteristics: Screams usually have higher frequency components. Compared with normal conversations, the frequency distribution of screams is more concentrated in the mid-high frequency band (approximately in the range of 1000-3000 Hz). By analyzing the peaks in the spectrum graph, the system can identify whether there is a frequency anomaly in this frequency band.

[0080] Instantaneous energy: Screams are often accompanied by large instantaneous energy changes. In audio signals, this energy change is manifested as a sharp increase in volume. The system can detect screams by calculating the short-term energy of the audio signal. The short-term energy formula is as follows: E = ∑ N-1 x(n) 2 .

[0081] Among them, x(n) is the amplitude of the audio signal, and N is the window size. By calculating the energy of the audio signal in real time, if it is found that the energy rises sharply in a short period of time and is concentrated in the frequency range of the scream, the system can identify that this is a scream.

[0082] Spectral texture analysis: In addition to direct frequency and energy features, the spectral texture of screams also tends to have unique changes. To better identify this change, the system can further analyze the spectral features using Mel-frequency cepstral coefficients (MFCC). MFCC can compress the spectral features of the audio signal into a low-dimensional feature vector, which helps to distinguish screams from other high-frequency noises (such as the sound of objects falling).

[0083] In actual application, the system will follow the steps below to identify running and screaming sounds:

[0084] Foreground object detection and motion analysis: When a fast-moving foreground object is detected in the video surveillance, the system will start the motion analysis module to calculate the speed and acceleration of the object to determine whether it meets the characteristics of running.

[0085] Real-time audio analysis: The collected audio signal will be divided into short time windows (e.g. 25 milliseconds), and the system will perform STFT analysis in each window to calculate the spectrum characteristics and short-time energy. If the energy in a window surges and the frequency falls within the screaming feature range, it will be marked as a possible screaming event.

[0086] Abnormal event confirmation: Once a running or screaming event is detected, the system will pass this information to the dynamic decision module to further determine whether it meets the criteria of an abnormal event, such as whether it occurs in an inactive area or an abnormal time period.

[0087] The key feature extraction module is used to extract key frame features and key frequency band features from the preprocessed data.

[0088] In this system, the key frame feature extraction module is implemented using a convolutional neural network, which includes a downsampling module for extracting key features, a global pooling module for integrating multimodal information, an upsampling decoder for fine classification, and a multi-layer connection and fusion module for fusing multimodal information.

[0089] In surveillance videos, the key frame feature extraction module extracts the features of each moving target, such as pedestrians and vehicles, from the video stream through the ResNet network to identify their movement trajectories and behavioral characteristics. When processing audio data, the system can use CNN to analyze and extract specific frequency band features in the audio signal to identify different types of sound events, such as human voices and car horns.

[0090] The downsampling module includes several convolutional layers as encoders. After each convolutional layer, the image resolution and number of channels are halved, but the feature scale remains unchanged.

[0091] During the downsampling process, audio and video are fused with multimodal information through feature scale fusion; the feature scale fusion is achieved through concatenated convolution operations:

[0092] M c =αM a +(1-α)M v

[0093] Among them, M c represents the fused features, M a represents the first feature map, M v denotes the second characteristic map, and α denotes a proportionality constant.

[0094] During the feature extraction process, the downsampling module retains important information by reducing the image resolution layer by layer, thereby improving the system's ability to recognize moving targets, such as effectively distinguishing between moving pedestrians and stationary objects.

[0095] The global pooling module outputs the feature map by calculating the mean of each channel, reducing the number of feature map channels.

[0096] The global pooling module merges the fusion feature layers and performs predictions through down-convolution operations respectively:

[0097] P = GlobalAvgPool (M)

[0098]

[0099] Among them, P is the feature vector after global pooling, is the prediction vector for the kth category, W k is the weight matrix corresponding to the kth category, B kis the bias term and M is the characteristic scale.

[0100] When the system processes multimodal features, the global pooling module can integrate information by calculating the mean of each channel. For example, it can combine video and audio features in surveillance scenarios to provide a more comprehensive understanding of events.

[0101] Upsampling decoder,The upsampling decoder makes predictions through transposed convolution and convolution layers, and the formula can be expressed as:

[0102] M'=TransposeConv(M,W)

[0103] F1=Conv(M′,W1)+B1

[0104] F2=Conv(M′,W2)+B2

[0105] Concatenate the convolution output prediction vector:

[0106]

[0107] The classification results are expressed as:

[0108]

[0109] Where: W is the weight matrix in the transposed convolution operation, M is the feature scale, and M ′ is the feature map after transposed convolution, F1 and F2 are the outputs after two convolutional layers, with dimensions of c×h×w and c′×h×w respectively, W1 and W2 are the weight matrices of the two convolutional layers, B1 and B2 are the bias terms of the two convolutional layers, is the final classification result.

[0110] In the analysis of detection results, the upsampling decoder restores the feature map through transposed convolution, ensuring that the system can accurately classify different types of anomalies, such as distinguishing emergency events from ordinary events.

[0111] The multimodal anomaly detection module is used to make anomaly judgments based on the extracted key frame features and key frequency band features.

[0112] In this system, the multimodal anomaly detection module compares and analyzes the detection results by establishing a decision analysis model; at the same time, a user feedback mechanism is introduced to send the detection results to the user, and the user adjusts the system parameters based on the feedback results. This module can analyze the data in the video and audio, and determine whether an abnormal event has occurred by comparing the characteristics of the two. For example, if someone moves quickly in the video and is accompanied by a high-decibel sound, the system will mark it as abnormal.

[0113] The anomaly detection method of the multimodal anomaly detection module includes:

[0114] S1. Obtain audio and video information, calculate the time domain features at the initialization moment, and obtain the initial anomaly score. When the system starts monitoring, it first analyzes the video and audio data over the past period of time, calculates the time domain features, and gives an initial anomaly score. For example, if the activity frequency in a certain area is higher than the normal level, the system will record this score for subsequent reference. The initial anomaly score is expressed as:

[0115]

[0116] Wherein, L is the time window width, max(N) is the maximum noise value of the audio or video signal in the time window, μ(N) is the average noise value of the audio or video signal in the time window, and σ is the proportional constant.

[0117] Through this formula, the initial anomaly score E0 can be obtained, which is used to measure the anomaly at the initialization moment.

[0118] S2. Obtain audio and video data, and calculate the audio time domain feature A(t) and video time domain feature V(t) through the forward transformation obtained in S1.

[0119] S3. According to the time domain information obtained in S2, the audio frequency domain feature A'(t') and the video frequency domain feature V'(t') are obtained through backward transformation.

[0120] S4. Input the audio frequency domain features A'(t') and video frequency domain features V'(t') obtained in S3 into the convolutional neural network to obtain the anomaly score E.

[0121] S5. The user sets a system alertness threshold WR, WR = |E-E0|. When WR exceeds the threshold, the system will send an alarm to remind the user, and capture the key frame fragments and the maximum value of the CNN softmax classification abnormal result as a prompt to be transmitted to the cloud multimodal large model for further processing. The user can adjust this parameter at any time to improve the system adaptability and speed.

[0122] The decision analysis model contains several classifiers. According to system requirements, users divide abnormal problem types into several categories. The system can set different classifiers for different types of abnormalities (such as fighting, accidents, fires, etc.). Each classifier determines whether the abnormal event of this type occurs based on its characteristics, and then adjusts the alarm level. The total number of categories is represented as k. Each classifier is used to detect whether there is an abnormality in the category and classify whether there is an abnormality. Each classifier corresponds to a multi-class cross entropy loss LC. If an abnormality is detected, the loss function corresponding to the classifier is:

[0123]

[0124] Where N is the total number of pixels, y i is the true label of the ith pixel. If the ith pixel is labeled as abnormal, then y i =1; if not, then y i =0, p i is the probability that the i-th pixel is labeled as an anomaly as predicted by the model.

[0125] If multiple categories are considered (a total of k), the loss function can be expanded to:

[0126]

[0127] where y ij is the true label of the i-th pixel in the j-th class, p ij is the probability that the model predicts that the i-th pixel is in the j-th class.

[0128] In this case, if an anomaly is detected (i.e., j=1), the loss for that class is calculated; if no anomaly is detected (i.e., j=0), the loss is zero. In this way, the loss can be calculated and optimized separately for each classifier.

[0129] The cloud-based large-model intelligent processing module is used to analyze the detected abnormal fragments and output generative text results.

[0130] In this system, a generative large language model based on the pre-trained Transfomer is built. The key video frame fragments, audio fragments and keyword prompts sent by the edge are received, and further abnormal event analysis is performed to finally generate a generative language text that can be consulted by users, and finally the abnormal event content is transmitted to the user.

[0131] It should be understood that, although each step in the flow chart of each embodiment of the present invention is shown in sequence according to the indication of the arrow, these steps are not necessarily performed in sequence according to the order indicated by the arrow. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be performed in other orders. Moreover, at least a portion of the steps in each embodiment may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these sub-steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with at least a portion of other steps or sub-steps or stages of other steps.

[0132] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0133] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0134] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

[0135] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A cloud-edge multimodal abnormal event detection system based on a dynamic decision-making mechanism, characterized in that: The system comprises: A data acquisition module is used to collect video and audio to obtain multimodal data; A data preprocessing module, used for preprocessing multimodal data to obtain preprocessed data; A key feature extraction module is used to extract key frame features and key frequency band features from preprocessed data; Multimodal anomaly detection module, used to make anomaly judgments based on the extracted key frame features and key frequency band features; The cloud-based large-model intelligent processing module is used to analyze the detected abnormal fragments and output generative text results.

2. The cloud-edge multimodal abnormal event detection system based on dynamic decision-making mechanism according to claim 1 is characterized in that: The data acquisition module includes a monitoring camera and an audio sensor, and collects audio data through the audio sensor and collects video data through the monitoring camera.

3. The cloud-edge multimodal abnormal event detection system based on dynamic decision-making mechanism according to claim 1 is characterized in that: When the data preprocessing module processes the video, it uses the optical flow method to establish a background model and update it, establishes a conversion model between foreground objects and background objects, uses a Gaussian mixture model to separate the foreground and background, monitors moving targets by establishing a static background model, extracts the foreground and background, analyzes the motion characteristics of the foreground, and determines whether there is an abnormality in the picture; When the data preprocessing module processes the audio, it extracts features from the collected audio data, converts the audio signal from the time domain to the frequency domain, identifies the spectral information of the audio data based on the Mel-frequency cepstral coefficients, and determines whether there is any sound abnormality.

4. The cloud-edge multimodal abnormal event detection system based on dynamic decision-making mechanism according to claim 1 is characterized in that: The key feature extraction module is implemented by a convolutional neural network, which includes a downsampling module for extracting key features, a global pooling module for integrating multimodal information, an upsampling decoder for fine classification, and a multi-layer connection and fusion module for fusing multimodal information.

5. The cloud-edge multimodal abnormal event detection system based on dynamic decision-making mechanism according to claim 4 is characterized in that: In the feature extraction process, the downsampling module retains important information by reducing the image resolution layer by layer, and the global pooling module outputs the feature map by calculating the mean of each channel to reduce the number of feature map channels; The global pooling module merges the fusion feature layers and performs predictions through down-convolution operations respectively: P = GlobalAvgPool (M) Among them, P is the feature vector after global pooling, is the prediction vector for the kth category, W k is the weight matrix corresponding to the kth category, B k is the bias term and M is the characteristic scale.

6. The cloud-edge multimodal abnormal event detection system based on dynamic decision-making mechanism according to claim 4 is characterized in that: The upsampling decoder makes predictions through transposed convolution and convolution layers, and the formula is expressed as: M′=TransposeConv(M,M) F1=Conv(M′,W1)+B1 F2=Conv(M′,W2)+B2 Concatenate the convolution output prediction vector: The classification results are expressed as: Where: W is the weight matrix in the transposed convolution operation, M is the feature scale, M′ is the feature map after the transposed convolution, F1 and F2 are the outputs after two convolutional layers, with dimensions of c×h×w and c′×h×w respectively, W1 and W2 are the weight matrices of the two convolutional layers, B1 and B2 are the bias terms of the two convolutional layers, is the final classification result; In the detection result analysis, the upsampling decoder restores the feature map through transposed convolution to classify different anomaly types.

7. The cloud-edge multimodal abnormal event detection system based on dynamic decision-making mechanism according to claim 1 is characterized in that: The multimodal anomaly detection module compares and analyzes the detection results by establishing a decision analysis model. The specific steps include: Obtain audio data and video data, calculate the time domain features at the initialization moment, and obtain the initial anomaly score; Through forward transformation, the audio time domain features and video time domain features are calculated; According to the acquired time domain information, audio frequency domain features and video frequency domain features are obtained through backward transformation; The obtained audio frequency domain features and video frequency domain features are input into the convolutional neural network to obtain the abnormality score E; A system alertness threshold WR is set, WR = |E-E0|. When WR exceeds the threshold, an alarm is sent to remind the user, and the maximum value of the key frame segment and the CNN softmax classification abnormal result is intercepted and sent to the cloud large model processing module.

8. The cloud-edge multimodal abnormal event detection system based on dynamic decision-making mechanism according to claim 1 is characterized in that: The cloud-based large model intelligent processing module includes a generative large language model based on a pre-trained Transfomer, which receives key video frame fragments, audio fragments and keywords sent by the edge end, performs abnormal event analysis, generates user-readable generative language text, and transmits the abnormal event content to the user.

9. The cloud-edge multimodal abnormal event detection system based on dynamic decision-making mechanism according to claim 7 is characterized in that: The initial anomaly score is expressed as: Wherein, L is the time window width, max(N) is the maximum noise value of the audio or video signal in the time window, μ(N) is the average noise value of the audio or video signal in the time window, and σ is the proportional constant.

10. The cloud-edge multimodal abnormal event detection system based on dynamic decision-making mechanism according to claim 7, characterized in that: The decision analysis model contains several classifiers, which divide the abnormal problem types into several categories. Different classifiers are set for different types of abnormalities. Each classifier determines whether the abnormal event of this type occurs according to its characteristics and adjusts the alarm level. The total number of categories is represented as k. Each classifier is used to detect whether there is an abnormality in the category and classify whether there is an abnormality. Each classifier corresponds to a multi-class cross entropy loss LC. If an abnormality is detected, the loss function corresponding to the classifier is: Where N is the total number of pixels, y i is the true label of the ith pixel. If the ith pixel is labeled as abnormal, then y i =1; if not, then y i =0, p i is the probability that the i-th pixel is labeled as an anomaly as predicted by the model.

Images