Video anomaly detection method and device based on diffusion model, medium and equipment

Abstract

The invention discloses a video anomaly detection method and device based on a diffusion model, a medium and equipment, and the method comprises the steps: carrying out the analysis of obtained video stream data, obtaining single-frame image data, and generating a human body posture sequence of each frame of image data; setting an attitude sequence masking strategy, and generating an observation frame sequence and a to-be-predicted first masking motion sequence; and performing anomaly detection processing on the human body posture sequence of each frame of image data based on a video anomaly detection model to obtain a video anomaly detection result. According to the method, the motion sequence is generated from the random noise based on the diffusion model by setting the attitude sequence masking strategy, and the motion completion is performed in the detection process, so that the abnormal frame of the video can be judged based on the error between the masking motion and the generated motion; the optimized video anomaly detection model obtained through training is subjected to anomaly detection processing, detection of the abnormal frames in the video stream data is achieved, and the anomaly detection precision is good.

Description

Technical Field

[0001] The present invention relates to the technical field of video anomaly detection, and in particular to a video anomaly detection method, device, medium and equipment based on a diffusion model. Background Art

[0002] At present, video surveillance is widely deployed in various public places and industrial production scenarios. Detecting abnormal behaviors in videos is crucial to maintaining public safety and regulating production activities. However, existing video anomaly detection technologies mainly rely on manual analysis of video frames, which is time-consuming and laborious. In addition, existing video anomaly detection methods are ineffective in dealing with diverse abnormal behaviors in complex scenarios. For example, traditional anomaly detection methods usually use an autoencoder (AE) architecture to extract features from video frames and reconstruct or predict them to detect anomalies. However, this method has problems such as low computational efficiency and susceptibility to background noise.

[0003] With the development of posture estimation technology, anomaly detection methods based on human skeleton data have begun to attract attention. Skeleton data is a low-dimensional and structured feature with the advantage of being free from background interference. However, existing methods based on skeleton data mainly use encoder-decoder architecture, which cannot fully cover the diversity of various motion patterns, resulting in poor detection results. Therefore, there is an urgent need for a new video anomaly detection method that can better capture the diversity of motion patterns. Summary of the invention

[0004] In view of the above problems, the present invention is proposed to provide a method, device, medium and equipment for video anomaly detection based on a diffusion model that overcomes the above problems or at least partially solves the above problems.

[0005] Other features and advantages of the present invention will become apparent from the following detailed description, or may be learned in part by practice of the present invention.

[0006] According to a first aspect of an embodiment of the present invention, a method for detecting video anomalies based on a diffusion model is provided. The method for detecting video anomalies based on a diffusion model includes:

[0007] Parsing the acquired video stream data to obtain single-frame image data, performing human posture detection processing on each frame of image data in the video stream data through a posture estimation algorithm, detecting human key points in each frame of image data, and generating a human posture sequence for each frame of image data;

[0008] Setting a posture sequence masking strategy, generating an observation frame sequence and a first masked motion sequence to be predicted based on the posture sequence masking strategy for each frame of the human body posture sequence;

[0009] The first masked motion sequence is converted into a noisy motion sequence by gradually adding noise data using a diffusion model, and the noisy motion sequence is denoised by a denoising network model to restore the second masked motion sequence;

[0010] Determine a sampling loss expectation parameter based on the added noise data and a noise estimation result of the denoising network model, and determine a prediction loss parameter based on all the first masked motion sequences and the corresponding second masked motion sequences;

[0011] The parameters of the diffusion model are adjusted based on the loss expectation parameter and the predicted loss parameter to obtain a video anomaly detection model, and anomaly detection processing is performed on the human posture sequence of each frame of the image data based on the video anomaly detection model to obtain a video anomaly detection result.

[0012] In some embodiments of the present invention, after generating the human body posture sequence of each frame of the image data, the method further includes converting the human body posture sequence of each frame of the image data into a topological graph, including:

[0013] Convert the human posture sequence of the image data of the i-th frame into a topological graph G i : G i =(J i ,A i ), where J i is a node set, the node set includes a global node set and local node set The local graph nodes in the local node set represent the joints of the human body, and the global graph nodes in the global node set represent the center of gravity of the human body; i is the adjacency matrix, which represents the connection relationship between nodes.

[0014] In some embodiments of the present invention, generating an observation frame sequence and a first masked motion sequence to be predicted comprises:

[0015] Masking is performed based on a temporal relationship, and a plurality of frames of the human body posture sequence are selected at uniform intervals in the time dimension to alternately serve as mask frames and observation frames, so as to obtain the observation frame sequence and the masked motion sequence to be predicted.

[0016] In some embodiments of the present invention, the step of converting the first masked motion sequence into a noisy motion sequence using a diffusion model, and denoising the noisy motion sequence using a denoising network model to restore the second masked motion sequence comprises:

[0017] Performing motion completion processing on the first masked motion sequence using a diffusion model, and converting the first masked motion sequence into a noisy motion sequence through a forward process of gradually adding noise data;

[0018] The conditional control information is extracted according to the observed frame sequence through a denoising network model, and the noisy motion sequence is denoised through the conditional control information to restore the second masked motion sequence.

[0019] In some embodiments of the present invention, performing anomaly detection processing on the human posture sequence of each frame of the image data based on the video anomaly detection model to obtain the video anomaly detection result includes:

[0020] Performing motion completion processing on the human posture sequence based on the video anomaly detection model to generate multiple restored masked motion sequences corresponding to the same human posture sequence;

[0021] Performing comprehensive completion according to the multiple restored masked motion sequences to obtain a final completed motion sequence;

[0022] Anomaly detection processing is performed on the completed motion sequence to obtain a video anomaly detection result.

[0023] In some embodiments of the present invention, performing anomaly detection processing on the completed motion sequence to obtain a video anomaly detection result includes:

[0024] The error parameters between the plurality of restored masked motion sequences and the human body posture sequence are calculated respectively;

[0025] Determining a minimum value of the plurality of error parameters, and taking the minimum value as an abnormality score;

[0026] The anomaly score is compared with the preset specified threshold to obtain the video anomaly detection result.

[0027] In some embodiments of the present invention, after obtaining the video anomaly detection result, the method further includes: when the detected video anomaly detection result is that the image data is an abnormal frame, marking the abnormal frame in the video stream data and generating alarm information.

[0028] According to a second aspect of an embodiment of the present invention, a diffusion model-based video anomaly detection device is provided, and the diffusion model-based video anomaly detection device includes:

[0029] A posture detection module is used to parse the acquired video stream data to obtain single-frame image data, perform human posture detection processing on each frame of image data in the video stream data through a posture estimation algorithm, detect human key points in each frame of image data, and generate a human posture sequence for each frame of image data;

[0030] A mask processing module, used for setting a posture sequence masking strategy, generating an observation frame sequence and a first masked motion sequence to be predicted based on the posture sequence masking strategy for each frame of the human body posture sequence;

[0031] A motion sequence restoration module, configured to convert the first masked motion sequence into a noisy motion sequence by gradually adding noise data using a diffusion model, and denoise the noisy motion sequence using a denoising network model to restore the second masked motion sequence;

[0032] A loss calculation module, configured to determine a sampling loss expectation parameter based on the added noise data and a noise estimation result of the denoising network model, and to determine a prediction loss parameter based on all the first masked motion sequences and the corresponding second masked motion sequences;

[0033] The anomaly detection module is used to adjust the parameters of the diffusion model based on the loss expectation parameter and the predicted loss parameter to obtain a video anomaly detection model, and perform anomaly detection processing on the human posture sequence of each frame of the image data based on the video anomaly detection model to obtain a video anomaly detection result.

[0034] According to a third aspect of an embodiment of the present invention, a computer-readable storage medium is provided, in which computer program instructions are stored. The computer program instructions are loaded and executed by a processor to implement the operations performed by any of the methods described above.

[0035] According to a fourth aspect of an embodiment of the present invention, there is provided an electronic device, comprising a processor and a memory, wherein the memory stores computer program instructions that can be executed by the processor, and when the processor executes the computer program instructions, it implements instructions of any of the methods described above.

[0036] The technical solution provided in the embodiments of the present invention has at least the following technical effects or advantages:

[0037] The embodiment of the present invention provides a method, device, medium and equipment for video anomaly detection based on a diffusion model. The method of the embodiment of the present invention sets a posture sequence masking strategy, alternately divides each frame of the human posture sequence obtained into an observation frame sequence and a first masked motion sequence to be predicted, generates a motion sequence from random noise based on the diffusion model, and performs motion completion during the detection process, so that the abnormal frame of the video can be judged based on the error between the masked motion and the generated motion. At the same time, through the masked sequences generated multiple times, the diversity of normal or abnormal behavior patterns is covered, and a variety of possible motion completion results are provided, thereby improving the accuracy and applicability of motion prediction; the diffusion model is parameterized by adjusting the loss expectation parameters and the predicted loss parameters obtained by the posture sequence masking strategy, and the optimized video anomaly detection model is trained, and the human posture sequence of each frame of the image data is subjected to anomaly detection processing based on the video anomaly detection model, thereby realizing the detection of abnormal frames in the video stream data, and having good anomaly detection accuracy.

[0038] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0040] Figure 1 A schematic diagram of a flow chart of a method for detecting video anomalies based on a diffusion model provided by an embodiment of the present invention;

[0041] Figure 2 A schematic diagram of a flow chart of executing a gesture sequence masking strategy according to an embodiment of the present invention;

[0042] Figure 3 A schematic diagram of the process of anomaly detection based on a video anomaly detection model;

[0043] Figure 4 A schematic diagram of a video anomaly detection model performing motion completion on the human posture sequence;

[0044] Figure 5 This is a scoring result diagram for abnormal behavior detection in sidewalk scene videos;

[0045] Figure 6 This is a scoring result diagram for abnormal behavior detection in lane scene videos;

[0046] Figure 7 Schematic diagram of the effect of the length of the input human posture sequence on the AUC score;

[0047] Figure 8 Schematic diagram showing the effect of the number of masked sequence generation on the AUC score for the diffusion model;

[0048] Fig. 9 A schematic diagram of the principle structure of a video anomaly detection device based on a diffusion model provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0049] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings.

[0050] Various structural schematic diagrams according to embodiments of the present disclosure are shown in the accompanying drawings. These figures are not drawn to scale, and some details are magnified and some details may be omitted for the purpose of clear expression. The shapes of various regions and layers shown in the figures and the relative sizes and positional relationships therebetween are only exemplary, and may deviate in practice due to manufacturing tolerances or technical limitations, and those skilled in the art may further design regions / layers with different shapes, sizes, and relative positions according to actual needs.

[0051] In the context of the present disclosure, when a layer / element is referred to as being "on" another layer / element, the layer / element may be directly on the other layer / element, or there may be an intervening layer / element therebetween. Additionally, if a layer / element is "on" another layer / element in one orientation, the layer / element may be "below" the other layer / element when the orientation is reversed. In the context of the present disclosure, similar or identical components may be denoted by the same or similar reference numerals.

[0052] In order to better understand the above-mentioned technical scheme, the above-mentioned technical scheme will be described in detail below in combination with specific implementation methods. It should be understood that the embodiments of the present disclosure and the specific features in the embodiments are detailed descriptions of the technical scheme of the present invention, rather than limitations on the technical scheme of the present invention. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments can be combined with each other.

[0053] Figure 1 FIG. 1 is a flow chart of a method for detecting video anomalies based on a diffusion model provided by an embodiment of the present invention. Figure 1 As shown, the video anomaly detection method based on the diffusion model includes the following steps:

[0054] S1, parsing the acquired video stream data to obtain single-frame image data, performing human posture detection processing on each frame of image data in the video stream data through a posture estimation algorithm, detecting human key points in each frame of image data, and generating a human posture sequence for each frame of image data;

[0055] Combination Figure 2 As shown, it is a flow chart of the execution of the posture sequence masking strategy of an embodiment of the present invention. The embodiment of the present invention obtains video stream data through a video acquisition device, and the video acquisition device can be a camera, a monitoring device, etc.; the embodiment of the present invention parses the acquired video stream data, and sequentially divides the frames according to the time sequence relationship to obtain a plurality of continuous single-frame image data, and then performs human posture detection processing on each frame of image data in the video stream data through a posture estimation algorithm, detects the key points of the human body in each frame of the image data, and generates a human posture sequence for each frame of the image data according to a specified window size; the specified window size is determined according to the actual needs of identifying the target (such as a natural person) in the image data.

[0056] After generating the human body posture sequence of each frame of the image data, the embodiment of the present invention further includes converting the human body posture sequence of each frame of the image data into a topological map (eg Figure 3 By using low-dimensional structured human posture features to model normal behavior patterns and abnormal behavior patterns in video stream data, the noise caused by illumination changes and background changes in the original frame data can be avoided. Specifically, the embodiment of the present invention converts the human posture sequence of the image data of the i-th frame into a topological graph G i , G i =(J i ,A i ), i is a positive integer; where J i is a node set, the node set includes a global node set and local node set The local graph nodes in the local node set represent the joints of the human body, and the global graph nodes in the global node set represent the center of gravity of the human body; i It is the adjacency matrix, which represents the connection relationship between nodes, that is, the edge in the topological graph.

[0057] S2, setting a posture sequence masking strategy, generating an observation frame sequence and a first masked motion sequence to be predicted based on the posture sequence masking strategy for each frame of the human body posture sequence;

[0058] The posture sequence masking strategy is used to generate a masked motion sequence that needs to be predicted. For example, an embodiment of the present invention performs masking based on a timing relationship, and selects several frames of the human posture sequence at uniform intervals in the time dimension to alternately serve as masked frames and observation frames to obtain the observation frame sequence and the masked motion sequence to be predicted, thereby increasing the diffusion model's ability to capture timing dependencies.

[0059] S3, using a diffusion model to convert the first masked motion sequence into a noisy motion sequence by gradually adding noise data, and performing denoising on the noisy motion sequence through a denoising network model to restore the second masked motion sequence;

[0060] Specifically, the embodiment of the present invention divides the input human posture sequence into observation frame sequences F O and the mask frame sequence F M In an embodiment of the present invention, the first masked motion sequence is converted into a noisy motion sequence by gradually adding noise data using a diffusion model, and the noisy motion sequence is denoised by a denoising network model to restore the second masked motion sequence, including:

[0061] A diffusion model is used to perform motion completion processing on the first masked motion sequence, and the first masked motion sequence is converted into a noisy motion sequence through a forward process of gradually adding noise data; a denoising network model is used to extract conditional control information according to the observed frame sequence, and the noisy motion sequence is denoised by the conditional control information to restore the second masked motion sequence.

[0062] The embodiment of the present invention performs motion completion through the diffusion model. In the forward process of the diffusion model, the first masked motion sequence of the original input is converted into random noise through the forward process of gradually adding noise, and denoising is learned through the inverse process to restore the second masked motion sequence.

[0063] Specifically, the embodiment of the present invention performs a first masked motion sequence F M Add noise data where the size of the noise data increases with the diffusion time step until a motion sequence completely corrupted by noise is obtained:

[0064]

[0065] In the formula, ∈ represents the noise data randomly sampled from the normal distribution, α t represents the controller for the amount of noise data increase in the t-th diffusion step, Represents the noisy motion sequence obtained in the tth diffusion step; this process aims to transform the distribution of the observed data (ie, the first masked motion sequence) into a simple prior distribution.

[0066] The denoising network model is, for example, a denoising U-Net network based on a spatiotemporal graph convolutional network (ST-GCN). The denoising network model is used to obtain the denoising information of the observed motion sequence F O The extracted conditional control information e is gradually de-noised to restore the second masked motion sequence:

[0067]

[0068] In the formula, ∈ θ is the noise estimation network, The inverse denoising process aims to restore the second masked motion sequence from the noisy motion sequence contaminated by noise.

[0069] S4, determining a sampling loss expectation parameter based on the added noise data and a noise estimation result of the denoising network model, and determining a prediction loss parameter based on all the first masked motion sequences and the corresponding second masked motion sequences;

[0070] In the embodiment of the present invention, the noise data added in each diffusion step t of the forward diffusion and the noise estimation result of each step of the denoising network are input into the following noise estimation loss function to calculate the expected parameter of the sampling loss:

[0071]

[0072] In the embodiment of the present invention, the diffusion model finally predicts the first masked motion sequence to obtain the second masked motion sequence. And the real motion sequence of the mask frame, that is, the first masked motion sequence F M , input into the following motion prediction loss function to calculate the prediction loss parameters:

[0073]

[0074] The optimized parameters are obtained by combining noise estimation loss and motion prediction loss. This optimizes the objective function of the diffusion model and improves the prediction accuracy of the diffusion model for motion posture:

[0075]

[0076] where λ is a hyperparameter that balances the prediction loss; specifically, the noise estimation loss Used to ensure that the model can accurately estimate the added noise data, motion prediction loss Used to ensure that the model can accurately restore masked motion sequences.

[0077] S5. Adjust the parameters of the diffusion model based on the loss expectation parameter and the predicted loss parameter to obtain a video anomaly detection model, and perform anomaly detection processing on the human posture sequence of each frame of the image data based on the video anomaly detection model to obtain a video anomaly detection result.

[0078] See also Figure 3 As shown, Figure 3 The flowchart of performing anomaly detection processing based on a video anomaly detection model is as follows. After obtaining the optimized and adjusted video anomaly detection model, the embodiment of the present invention performs anomaly detection processing on the human posture sequence of each frame of the image data based on the video anomaly detection model to obtain a video anomaly detection result, including:

[0079] S51, performing motion completion processing on the human posture sequence based on the video anomaly detection model to generate multiple restored masked motion sequences corresponding to the same human posture sequence;

[0080] See also Figure 4 As shown, in the process of performing anomaly detection processing on each frame of image data in the video stream data, the present invention uses the observed human posture sequence as conditional information and generates D diversified masked motion prediction results through the video anomaly detection model That is, the restored masked motion sequence, D is a positive integer.

[0081] S52, performing comprehensive completion according to the multiple restored masked motion sequences to obtain a final completed motion sequence;

[0082] The embodiment of the present invention performs smoothing post-processing on the predicted restored masked motion sequence to eliminate noise and discontinuity, thereby ensuring that the sequence complies with physical rationality and continuity.

[0083] The embodiment of the present invention integrates multiple generation results of the same human posture sequence, for example, taking the average value of each generated posture in the human posture sequence to obtain the final completed motion sequence:

[0084]

[0085] The embodiments of the present invention can cover the diversity of normal or abnormal behavior patterns through multiple generation, provide multiple possible motion completion results, and improve the robustness and accuracy of abnormality detection.

[0086] S53: Perform anomaly detection processing on the completed motion sequence to obtain a video anomaly detection result.

[0087] The embodiment of the present invention performs anomaly detection processing on the completed motion sequence, and obtains a video anomaly detection result including: respectively calculating the error parameters between the multiple restored masked motion sequences and the human body posture sequence; confirming the minimum value of the multiple error parameters, and taking the minimum value as the anomaly score; and comparing and judging the anomaly score with a preset specified threshold to obtain a video anomaly detection result.

[0088] For example, the embodiment of the present invention calculates anomaly scores based on the prediction results generated multiple times, calculates the error parameter between each generated result and the human posture sequence, and selects the smallest error parameter as the final anomaly score:

[0089]

[0090] By selecting the minimum error value as the final anomaly score, the impact of random factors on the anomaly score can be reduced and the robustness of anomaly detection can be improved.

[0091] It should be noted that, in the same frame of image data, if there are multiple target individuals, the anomaly score of each individual can be calculated separately, and the average value can be taken as the anomaly score of the frame:

[0092]

[0093] Where N is the number of individuals in the frame, is the anomaly score of the i-th individual; combining the anomaly scores of multiple individuals can provide a more comprehensive frame-level anomaly detection result.

[0094] As shown in Table 1, in order to compare the embodiment of the present invention with the commonly used methods in the video anomaly detection task, the AUC scores were tested on three video anomaly detection datasets, HR-ShanghaiTech, HR-Avenue, and HR-Ubnormal, to conduct comparative experimental verification.

[0095] Table 1. Comparison of anomaly detection performance with existing algorithms

[0096]

[0097] Note: The methods marked with * in the table are those that use the diffusion model

[0098] The Conv-AE, Frame-Pred, MPED-RNN, GEPC, ​​and BiPOCO methods in Table 1 all use autoencoders to predict or reconstruct pose sequences, and thus make decisions on abnormal frames based on prediction or reconstruction errors; the methods marked with * in Table 1 use the generation capability of the diffusion model to detect abnormal behaviors. It can be seen that the video anomaly detection results obtained by the diffusion model-based methods on the three data sets are better than those obtained by the autoencoder-based methods. Among them, Ours is the result of the comparison between the experimental results of the present invention and other mainstream methods. It can be seen from the table that the present invention has achieved the best results on the three mainstream data sets.

[0099] Figure 5 This is a scoring result diagram of abnormal behavior detection in a sidewalk scene video. The area under the ROC curve of anomaly detection can be used as the final anomaly score. A true value label of 1 indicates that there is abnormal behavior in the video frame, and a true value label of 0 indicates that there is no abnormality. When the true value label is abnormal, the anomaly score obtained by the embodiment of the present invention is higher. When the true value label shows no abnormality, the anomaly score obtained by the embodiment of the present invention remains at a low level, indicating that the algorithm of the present invention has a strong ability to distinguish abnormal behavior.

[0100] Figure 6 This is a scoring result diagram for abnormal behavior detection in lane scene videos. When people in the scene suddenly have abnormal behavior (such as fighting, etc.), the embodiment of the present invention can provide an accurate and timely abnormal score response to the frame where the abnormal behavior occurs, thereby realizing real-time video anomaly detection.

[0101] Figure 7 This is a schematic diagram of the effect of the length of the human posture sequence input in the present invention on the AUC score (the area enclosed by the coordinate axis under the ROC curve, i.e., the anomaly score), indicating that appropriate parameter settings can significantly improve the model performance. Figure 8 This is a schematic diagram of the effect of the number of masked sequence generation of the diffusion model in the present invention on the AUC score, indicating that the diffusion model (or video anomaly detection model) in the present invention can cover the diversity of normal or abnormal behavior patterns and provide a variety of possible motion completion results.

[0102] After obtaining the video anomaly detection result, the embodiment of the present invention further includes: when the detected video anomaly detection result is that the image data is an abnormal frame, marking the abnormal frame in the video stream data and generating alarm information. By marking the abnormal frame and generating alarm information, the embodiment of the present invention can intuitively prompt the user, thereby improving convenience.

[0103] Compared with the prior art, the video anomaly detection method based on the diffusion model described in the embodiment of the present invention has the following advantages:

[0104] 1. By setting a posture sequence masking strategy, each frame of the human posture sequence obtained is alternately divided into an observation frame sequence and a first masked motion sequence to be predicted, a motion sequence is generated from random noise based on a diffusion model, and motion completion is performed during the detection process, so that the abnormal frame of the video can be judged based on the error between the masked motion and the generated motion;

[0105] 2. The parameters of the diffusion model are adjusted by the loss expectation parameters and the predicted loss parameters obtained by the posture sequence masking strategy, and the optimized video anomaly detection model is trained. Based on the video anomaly detection model, anomaly detection processing is performed on the human posture sequence of each frame of the image data, thereby realizing the detection of abnormal frames in the video stream data, with good anomaly detection accuracy;

[0106] 3. Convert human posture detection into a topological map, use low-dimensional structured human posture features to model normal and abnormal behavior patterns in the video, and avoid the noise caused by illumination changes and background changes in the original frame data in the traditional way;

[0107] 4. Construct a hierarchical space-time graph to represent video events, and construct a hierarchical space-time graph of human posture to capture the dynamic interactions within and between individuals, thereby improving the accuracy of anomaly detection.

[0108] Based on the above embodiment, as the above Figure 1 The present invention provides an embodiment of a video anomaly detection device based on a diffusion model. Figure 1 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices, see Fig. 9 As shown, the video anomaly detection device based on the diffusion model includes:

[0109] The posture detection module 100 is used to parse the acquired video stream data to obtain single-frame image data, perform human posture detection processing on each frame of image data in the video stream data through a posture estimation algorithm, detect human key points in each frame of image data, and generate a human posture sequence for each frame of image data;

[0110] The mask processing module 200 is used to set a posture sequence masking strategy, and each frame of the human posture sequence generates an observation frame sequence and a first masked motion sequence to be predicted based on the posture sequence masking strategy;

[0111] A motion sequence restoration module 300 is used to convert the first masked motion sequence into a noisy motion sequence by gradually adding noise data using a diffusion model, and to perform denoising on the noisy motion sequence using a denoising network model to restore the second masked motion sequence;

[0112] A loss calculation module 400, configured to determine a sampling loss expectation parameter based on the added noise data and a noise estimation result of the denoising network model, and to determine a prediction loss parameter based on all the first masked motion sequences and the corresponding second masked motion sequences;

[0113] The anomaly detection module 500 is used to adjust the parameters of the diffusion model based on the loss expectation parameter and the predicted loss parameter to obtain a video anomaly detection model, and perform anomaly detection processing on the human posture sequence of each frame of the image data based on the video anomaly detection model to obtain a video anomaly detection result.

[0114] The video anomaly detection device based on the diffusion model described in the embodiment of the present invention can execute the video anomaly detection method based on the diffusion model provided in the above embodiment. The video anomaly detection device based on the diffusion model has the corresponding functional steps and beneficial effects of the video anomaly detection method based on the diffusion model described in the above embodiment. Please refer to the embodiment of the video anomaly detection method based on the diffusion model for details. The embodiment of the present invention will not be repeated here.

[0115] The embodiment of the present invention also provides an electronic device, which may include a processor and a memory, wherein the processor and the memory may be connected via a bus or other means. The processor may be a central processing unit (CPU). The processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components and other chips, or a combination of the above-mentioned various types of chips. The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs, non-transient computer executable programs and modules, such as program instructions / modules corresponding to the video anomaly detection method based on the diffusion model in the embodiment of the present invention. The processor executes various functional applications and data processing of the processor by running the non-transient software programs, instructions and modules stored in the memory, that is, the video anomaly detection method based on the diffusion model in the above method embodiment is realized.

[0116] The memory may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required by at least one function; the data storage area may store data created by the processor, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. The one or more modules are stored in the memory, and when executed by the processor, the video anomaly detection method based on the diffusion model in the above method embodiment is executed. The specific details of the above electronic device can be understood by corresponding to the corresponding descriptions and effects in the above method embodiment, and will not be repeated here. Those skilled in the art can understand that all or part of the processes in the above embodiment method can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and the program, when executed, may include the processes of the embodiments of the above methods. Among them, the storage medium can be a read-only memory (ROM), a random access memory (RAM), a flash memory (Flash Memory), a hard disk drive (HDD) or a solid-state drive (SSD), etc.; the storage medium can also include a combination of the above types of memory.

[0117] In the description provided herein, a large number of specific details are described. However, it is understood that embodiments of the present invention can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0118] Similarly, it should be understood that in order to streamline the present disclosure and aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of the present invention, various features of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, this disclosed method should not be interpreted as reflecting the intention that the claimed invention requires more features than those expressly recited in each claim. Rather, as reflected in the claims, inventive aspects lie in less than all of the features of the individual embodiments previously disclosed. Therefore, the claims that follow the detailed description are hereby expressly incorporated into the detailed description, with each claim itself serving as a separate embodiment of the present invention.

[0119] It should be noted that the above-mentioned embodiments illustrate rather than limit the invention and that those skilled in the art will be able to design alternative embodiments without departing from the scope of the appended claims.

Claims

1. A video anomaly detection method based on a diffusion model, characterized in that: The video anomaly detection method based on the diffusion model includes: Parsing the acquired video stream data to obtain single-frame image data, performing human posture detection processing on each frame of image data in the video stream data through a posture estimation algorithm, detecting human key points in each frame of image data, and generating a human posture sequence for each frame of image data; Setting a posture sequence masking strategy, generating an observation frame sequence and a first masked motion sequence to be predicted based on the posture sequence masking strategy for each frame of the human body posture sequence; The first masked motion sequence is converted into a noisy motion sequence by gradually adding noise data using a diffusion model, and the noisy motion sequence is denoised by a denoising network model to restore the second masked motion sequence; Determine a sampling loss expectation parameter based on the added noise data and a noise estimation result of the denoising network model, and determine a prediction loss parameter based on all the first masked motion sequences and the corresponding second masked motion sequences; The parameters of the diffusion model are adjusted based on the loss expectation parameter and the predicted loss parameter to obtain a video anomaly detection model, and anomaly detection processing is performed on the human posture sequence of each frame of the image data based on the video anomaly detection model to obtain a video anomaly detection result.

2. The video anomaly detection method based on diffusion model according to claim 1, characterized in that: After generating a human body posture sequence of each frame of the image data, the method further includes converting the human body posture sequence of each frame of the image data into a topological graph, including: Convert the human posture sequence of the image data of the i-th frame into a topological graph G i : G i =(J i ,A i ), where J i is a node set, the node set includes a global node set and local node set The local graph nodes in the local node set represent the joints of the human body, and the global graph nodes in the global node set represent the center of gravity of the human body; i is the adjacency matrix, which represents the connection relationship between nodes.

3. The video anomaly detection method based on diffusion model according to claim 1, characterized in that: The generating of the observation frame sequence and the first masked motion sequence to be predicted comprises: Masking is performed based on a temporal relationship, and a plurality of frames of the human body posture sequence are selected at uniform intervals in the time dimension to alternately serve as mask frames and observation frames, so as to obtain the observation frame sequence and the masked motion sequence to be predicted.

4. The video anomaly detection method based on diffusion model according to claim 1, characterized in that: The step of converting the first masked motion sequence into a noisy motion sequence by using a diffusion model, and performing denoising on the noisy motion sequence by using a denoising network model to restore the second masked motion sequence comprises: Performing motion completion processing on the first masked motion sequence using a diffusion model, and converting the first masked motion sequence into a noisy motion sequence through a forward process of gradually adding noise data; The conditional control information is extracted according to the observed frame sequence through a denoising network model, and the noisy motion sequence is denoised through the conditional control information to restore the second masked motion sequence.

5. The video anomaly detection method based on diffusion model according to claim 1, characterized in that: The performing of anomaly detection processing on the human posture sequence of each frame of the image data based on the video anomaly detection model to obtain the video anomaly detection result includes: Performing motion completion processing on the human posture sequence based on the video anomaly detection model to generate multiple restored masked motion sequences corresponding to the same human posture sequence; Performing comprehensive completion according to the multiple restored masked motion sequences to obtain a final completed motion sequence; Anomaly detection processing is performed on the completed motion sequence to obtain a video anomaly detection result.

6. The video anomaly detection method based on diffusion model according to claim 5, characterized in that: The performing anomaly detection processing on the completed motion sequence to obtain a video anomaly detection result includes: The error parameters between the plurality of restored masked motion sequences and the human body posture sequence are calculated respectively; Determining a minimum value of the plurality of error parameters, and taking the minimum value as an abnormality score; The anomaly score is compared with the preset specified threshold to obtain the video anomaly detection result.

7. The video anomaly detection method based on diffusion model according to claim 6 is characterized in that: After obtaining the video anomaly detection result, the method further includes: when the detected video anomaly detection result is that the image data is an abnormal frame, marking the abnormal frame in the video stream data and generating alarm information.

8. A video anomaly detection device based on a diffusion model, applied to the method according to any one of claims 1 to 7, characterized in that: The video anomaly detection device based on the diffusion model comprises: A posture detection module is used to parse the acquired video stream data to obtain single-frame image data, perform human posture detection processing on each frame of image data in the video stream data through a posture estimation algorithm, detect human key points in each frame of image data, and generate a human posture sequence for each frame of image data; A mask processing module, used for setting a posture sequence masking strategy, generating an observation frame sequence and a first masked motion sequence to be predicted based on the posture sequence masking strategy for each frame of the human body posture sequence; A motion sequence restoration module, configured to convert the first masked motion sequence into a noisy motion sequence by gradually adding noise data using a diffusion model, and denoise the noisy motion sequence using a denoising network model to restore the second masked motion sequence; A loss calculation module, configured to determine a sampling loss expectation parameter based on the added noise data and a noise estimation result of the denoising network model, and to determine a prediction loss parameter based on all the first masked motion sequences and the corresponding second masked motion sequences; The anomaly detection module is used to adjust the parameters of the diffusion model based on the loss expectation parameter and the predicted loss parameter to obtain a video anomaly detection model, and perform anomaly detection processing on the human posture sequence of each frame of the image data based on the video anomaly detection model to obtain a video anomaly detection result.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions, which are loaded and executed by a processor to implement the operations performed by the method according to any one of claims 1 to 7.

10. An electronic device comprising a processor and a memory, characterized in that: The memory stores computer program instructions that can be executed by the processor, and when the processor executes the computer program instructions, the processor implements instructions of any one of the methods described in claims 1-7.

Images