Video anomaly detection method and device based on dynamic heterogeneous redundant architecture

Abstract

The invention provides a video anomaly detection method and device based on a dynamic heterogeneous redundant architecture, and the method comprises the steps: obtaining an overall motion sequence of a human skeleton, the overall motion sequence comprising a historical motion sequence and a future motion sequence; encoding the historical motion sequence to obtain a historical motion code; performing historical motion reconstruction based on historical motion coding through a noise prediction network to obtain a reconstructed motion sequence; performing future motion prediction based on the target time step and the historical motion code through a noise prediction network to obtain a predicted motion sequence; comparing the reconstructed motion sequence with a historical motion sequence to obtain a reconstruction error; comparing the predicted motion sequence with a future motion sequence to obtain a prediction error; and determining a video abnormal frame in the overall motion sequence based on the reconstruction error and the prediction error. According to the invention, the detection precision of video anomaly detection can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of video anomaly detection, and in particular to a video anomaly detection method and device based on a dynamic heterogeneous redundant architecture. Background Art

[0002] Video anomaly detection is an important topic in computer vision and security applications. It can quickly identify abnormal events such as human behavior, accidents, and diseases. Anomalies are usually described as unexpected or unusual phenomena.

[0003] Existing video anomaly detection methods mainly include prediction-based methods. Prediction-based methods focus on predicting future frames or events based on historical data. When the future data deviates greatly from the predicted data, anomalies will be displayed. However, when the area occupied by the abnormal event in the video frame is too small, the corresponding prediction error may also be very small, resulting in these abnormal events being difficult to detect accurately, that is, the recall rate is low.

[0004] It can be seen that the video anomaly detection method in the related art has the technical problem of inaccurate detection. Summary of the invention

[0005] The present invention provides a video anomaly detection method and device based on a dynamic heterogeneous redundant architecture, which are used to solve the defect of inaccurate detection in the video anomaly detection method in the prior art and to increase the detection accuracy of video anomaly detection.

[0006] The present invention provides a video anomaly detection method based on a dynamic heterogeneous redundant architecture, comprising the following steps: obtaining an overall motion sequence of a human skeleton, wherein the overall motion sequence is used to represent the movement changes of human joints over time during the movement of the human skeleton, and the overall motion sequence includes a historical motion sequence and a future motion sequence; encoding the historical motion sequence to obtain a historical motion code, wherein the historical motion sequence is a sequence before a target time step in the overall motion sequence; based on a noise prediction network, the historical motion code performs historical motion reconstruction to obtain a reconstructed motion sequence; through the noise prediction network, future motion prediction is performed based on the target time step and the historical motion code to obtain a predicted motion sequence; the reconstructed motion sequence is compared with the historical motion sequence to obtain a reconstruction error; the predicted motion sequence is compared with the future motion sequence to obtain a prediction error; based on the reconstruction error and the prediction error, the video abnormal frame in the overall motion sequence is determined.

[0007] According to a video anomaly detection method based on a dynamic heterogeneous redundant architecture provided by the present invention, the reconstructed motion sequence is compared with the historical motion sequence to obtain a reconstruction error; the predicted motion sequence is compared with the future motion sequence to obtain a prediction error, including: determining multiple target objects corresponding to the overall motion sequence; looping through the overall motion sequence according to a preset target frame number based on a sliding window to determine multiple target frames corresponding to each of the multiple target objects; determining a current motion sequence in which each of the multiple target frames is located based on the target time step; when the current motion sequence is in the historical motion sequence, comparing the reconstructed motion sequence with the historical motion sequence to obtain a reconstruction error; when the current motion sequence is in the future motion sequence, comparing the predicted motion sequence with the future motion sequence to obtain a prediction error.

[0008] According to a video anomaly detection method based on a dynamic heterogeneous redundant architecture provided by the present invention, the method of determining the abnormal video frame in the overall motion sequence based on the reconstruction error and the prediction error includes: taking the reconstruction error as a first abnormality score and taking the prediction error as a second abnormality score; determining an abnormal score set corresponding to the multiple target frames, and taking the average value of the abnormal score set as a local score of the target object, wherein the abnormal score set includes the first abnormal score and the second abnormal score; determining an attribute abnormal score set based on the local score of each target object in the multiple target objects; and determining the abnormal video frame in the overall motion sequence based on the attribute abnormal score set.

[0009] According to a video anomaly detection method based on a dynamic heterogeneous redundant architecture provided by the present invention, the future motion prediction is performed based on the target time step and the historical motion coding to obtain a predicted motion sequence, including: through a diffusion model of the Transformer architecture, in a forward process, the overall motion sequence is reparameterized to obtain a noisy discrete cosine transform spectrum corresponding to the target time step; through the diffusion model of the Transformer architecture, in a reverse process, future motion prediction is performed based on the target time step, the historical motion coding and the noisy discrete cosine transform spectrum to obtain a predicted motion sequence.

[0010] According to a video anomaly detection method based on a dynamic heterogeneous redundant architecture provided by the present invention, the future motion prediction is performed based on the target time step, the historical motion coding and the noisy discrete cosine transform spectrum to obtain a predicted motion sequence, including: using the historical motion coding as a conditional embedding; determining the multi-head self-attention of the noisy discrete cosine transform spectrum and the sinusoidal time step embedding of the target time step; performing feature modulation based on the multi-head self-attention, the sinusoidal time step embedding, and the conditional embedding to obtain a first feature; performing a residual connection between the first feature representation and the noisy discrete cosine transform spectrum to obtain a hidden representation; inputting the hidden representation into a feedforward neural network to obtain an integrated feature output by the feedforward neural network; performing feature modulation based on the integrated feature, the sinusoidal time step embedding, and the conditional embedding to obtain a second feature; performing a residual connection between the second feature representation and the hidden representation to obtain an output representation; performing an inverse discrete cosine transform based on the output representation to obtain a predicted motion sequence.

[0011] According to a video anomaly detection method based on a dynamic heterogeneous redundant architecture provided by the present invention, after the future motion prediction is performed based on the target time step and the historical motion code to obtain a predicted motion sequence, the method also includes: when the length of the historical motion sequence matches that of the overall motion sequence, discrete cosine transform is performed on the historical motion sequence to obtain compact historical information; noise processing is performed on the compact historical information to obtain a noise observation spectrum corresponding to the target time step; inverse discrete cosine transform and masking are performed on the noise observation spectrum and the denoised prediction spectrum to obtain a spectral combination sequence, wherein the denoised prediction spectrum is determined based on the predicted motion sequence; the historical motion code is used as a conditional embedding, and the spectral combination sequence is used as a conditional motion; the conditional embedding and the conditional motion are used as dual conditions of the reverse process in the diffusion model of the Transformer architecture.

[0012] According to a video anomaly detection method based on a dynamic heterogeneous redundant architecture provided by the present invention, the method also includes: determining the joint correlation difference of each frame based on the relative time distance and relative numerical difference of each frame in the overall motion sequence; taking the product of the average value of the joint correlation difference and a preset parameter as the joint correlation difference loss of the overall motion sequence; determining the reconstruction loss function of the noise prediction network based on the reconstruction error; determining the prediction loss function of the noise prediction network based on the prediction error; determining the total loss function of the noise prediction network based on the reconstruction loss function, the prediction loss function and the joint correlation difference loss; and updating the noise prediction network through the total loss function.

[0013] According to a video anomaly detection method based on a dynamic heterogeneous redundant architecture provided by the present invention, the joint correlation difference of each frame is determined based on the relative time distance and relative numerical difference of each frame in the overall motion sequence, including: taking each frame in the overall motion sequence as the current frame and performing the following operations to obtain the joint correlation difference of each frame: determining the relative time distance between the current frame and other frames based on the time axis of the overall motion sequence; determining the time correlation of the current frame based on the relative time distance; determining the relative numerical difference between the current frame and the other frames based on the self-attention value of each frame in the overall motion sequence; determining the global correlation of the current frame based on the relative numerical difference; and determining the joint correlation difference of the current frame based on the time correlation and the global correlation.

[0014] The present invention also provides a video anomaly detection device based on a dynamic heterogeneous redundant architecture, comprising the following modules: an acquisition module, used to acquire an overall motion sequence of a human skeleton, wherein the overall motion sequence is used to represent the movement changes of human joints over time during the movement of the human skeleton, and the overall motion sequence includes a historical motion sequence and a future motion sequence; an encoding module, used to encode the historical motion sequence to obtain a historical motion code, wherein the historical motion sequence is a sequence before a target time step in the overall motion sequence; a reconstruction module, used to reconstruct the historical motion based on the historical motion code through a noise prediction network to obtain a reconstructed motion sequence; a prediction module, used to predict the future motion based on the target time step and the historical motion code through the noise prediction network to obtain a predicted motion sequence; a first comparison module, used to compare the reconstructed motion sequence with the historical motion sequence to obtain a reconstruction error; a second comparison module, used to compare the predicted motion sequence with the future motion sequence to obtain a prediction error; an output module, used to determine the video abnormal frame in the overall motion sequence based on the reconstruction error and the prediction error.

[0015] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the video anomaly detection method based on a dynamic heterogeneous redundant architecture as described above is implemented.

[0016] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the video anomaly detection method based on a dynamic heterogeneous redundant architecture as described in any one of the above is implemented.

[0017] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned video anomaly detection methods based on a dynamic heterogeneous redundant architecture.

[0018] The video anomaly detection method and device based on a dynamic heterogeneous redundant architecture provided by the present invention obtain an overall motion sequence of a human skeleton used to represent the movement changes of human joints over time, encode the historical motion sequence before the target time step of the overall motion sequence, and obtain the historical motion code; reconstruct the historical motion code to obtain a reconstructed motion sequence, and at the same time, predict future motion based on the target time step and the historical motion code to obtain a predicted motion sequence, thereby, by combining the reconstruction-based and prediction-based anomaly detection methods, it is possible to achieve anomaly detection with higher accuracy and robustness in a variety of environments; the joint analysis of the reconstruction error and the prediction error can more accurately identify video anomaly frames that do not conform to the normal motion pattern, which is more accurate than the method that relies solely on historical information or future prediction; thereby solving the technical problem of inaccurate detection in the video anomaly detection method of the human skeleton in the related art. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced one by one below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 It is a flow chart of a video anomaly detection method based on a dynamic heterogeneous redundant architecture provided by the present invention.

[0021] Figure 2 It is a structural schematic diagram of a dual-conditional motion diffusion model of a human skeleton based on a dynamic heterogeneous redundant architecture provided by the present invention.

[0022] Figure 3 It is a visualization diagram of the experimental results of the dual-conditional motion diffusion video anomaly detection method of the human skeleton based on the dynamic heterogeneous redundant architecture provided by the present invention.

[0023] Figure 4 It is the anomaly score of the dual-conditional motion diffusion video anomaly detection method of the human skeleton based on the dynamic heterogeneous redundant architecture provided by the present invention.

[0024] Figure 5 It is a structural schematic diagram of a video anomaly detection device based on a dynamic heterogeneous redundant architecture provided by the present invention.

[0025] Figure 6It is a schematic diagram of the physical structure of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] Video anomaly detection is an important topic in computer vision and security applications. It is able to quickly identify abnormal events such as human behavior, accidents, and diseases. Anomalies are usually described as unexpected or unusual phenomena. In contrast, normality is defined as expected and frequently encountered situations. Video anomaly detection is challenging because abnormal events occur rarely and belong to a variety of different categories. Labeling data is expensive and time-consuming, and it is impractical to collect all possible anomaly samples. Therefore, the anomaly detection problem is usually treated as a first-class classification problem, that is, only normal data is used during model training, and during testing, data that significantly deviates from the normal pattern is identified as anomalies.

[0028] Existing video anomaly detection methods mainly utilize videos containing recognizable facial features and appearance-based features. Using videos with recognizable faces raises privacy concerns.

[0029] Using skeletons to depict human motion in videos is a very effective way to protect privacy and circumvent restrictions based on appearance attributes. In addition to privacy protection, skeleton features are compact, well-structured, and highly describe human motion. In addition, diffusion models have achieved great success in generating high-quality samples. One of the outstanding features of diffusion models is their ability to flexibly handle a variety of input conditions that can contain different types of data, enabling customized generation tasks.

[0030] Existing video anomaly detection methods mainly focus on two technical paths: reconstruction-based methods and prediction-based methods. Reconstruction-based methods first compress the input data into a low-dimensional representation, and then restore the original data from the representation. By comparing the reconstruction errors of the original data and the restored data, anomalies such as irregular patterns or structures in the data can be found. However, reconstruction-based video anomaly detection methods depend largely on the model's ability to represent the data. If the model's representation ability is not sufficient to capture the complexity and potential characteristics of the data, it will be difficult to effectively distinguish between normal and abnormal data. Prediction-based methods focus on predicting future frames or events based on historical data. When future data deviates from the predicted trend, anomalies are revealed. These methods are effective in modeling and revealing the temporal connection between consecutive frames. However, when making long-term predictions, prediction-based video anomaly detection methods may gradually accumulate prediction errors due to various factors (such as data noise, model errors, etc.), resulting in an increasing deviation between the prediction results and the actual data. Due to the inherent multimodality of human motion and the diversity of normal and abnormal patterns, simply applying reconstruction-based or prediction-based video anomaly detection methods has limitations.

[0031] It can be seen that existing video anomaly detection methods focus on learning a single reconstruction or prediction of the input, fail to consider the diversity of normal and abnormal, and only adopt a single proxy model based on reconstruction or prediction.

[0032] Due to the inherent multimodality of human motion and the diversity of normal and abnormal patterns, simply applying reconstruction-based or prediction-based video anomaly detection methods has limitations. Therefore, in an embodiment of the present invention, a video anomaly detection method based on a dynamic heterogeneous redundant architecture seamlessly integrates the advantages of reconstruction-based and prediction-based video anomaly detection methods, uses multiple models with different structures in processing video anomaly detection tasks, and integrates more semantic information, thereby increasing the reliability of video anomaly detection and improving detection accuracy.

[0033] Optionally, the video anomaly detection method based on a dynamic heterogeneous redundant architecture in an embodiment of the present application can be executed by a server, or by a terminal device, or jointly by a server and a terminal device, taking the example of the video anomaly detection method based on a dynamic heterogeneous redundant architecture in this embodiment being executed by a server.

[0034] refer to Figure 1 , Figure 1 is a flow chart of a video anomaly detection method based on a dynamic heterogeneous redundant architecture provided by the present invention, such as Figure 1 As shown, the method includes the following: Step 101, obtaining an overall motion sequence of a human skeleton, wherein the overall motion sequence is used to represent the movement changes of human joints over time during the movement of the human skeleton, and the overall motion sequence includes a historical motion sequence and a future motion sequence.

[0035] The video anomaly detection method in the related art mainly utilizes videos containing recognizable facial features and appearance features, but the use of the above videos may cause privacy issues.

[0036] Therefore, in the embodiment of the present invention, the human body motion process in the video is depicted in the form of a human body skeleton, that is, the overall motion sequence of the human body skeleton is used to represent the movement changes of the human body joints over time during the motion process.

[0037] In an embodiment of the present invention, since the subsequent processing process includes historical motion reconstruction and future motion prediction, the overall motion sequence is divided into a historical motion sequence and a future motion sequence according to the target time step, that is, the sequence before the target time step in the overall motion sequence is the historical motion sequence, and the sequence after the target time step is the future motion sequence.

[0038] Through the embodiments of the present invention, it is possible to utilize skeletons that are compact in features, well-structured, and highly describe human motion to simulate changes in human joints over time, which is a very effective method for protecting privacy and circumventing restrictions based on appearance attributes.

[0039] Step 102, encoding the historical motion sequence to obtain the historical motion code, wherein the historical motion sequence is a sequence before the target time step in the overall motion sequence.

[0040] Step 103, reconstructing the historical motion based on the historical motion coding through the noise prediction network to obtain a reconstructed motion sequence.

[0041] In an embodiment of the present invention, the noise prediction network includes a reconstruction module, and the reconstruction module is constructed using an autoencoder architecture, wherein the reconstruction module is used to reconstruct a historical motion sequence to obtain a reconstructed motion sequence, and the reconstruction module includes an encoder and a decoder.

[0042] Here, the historical motion sequence is encoded based on the encoder to obtain the historical motion coding; the historical motion coding is reconstructed (decoded) based on the decoder to obtain the historical motion coding.

[0043] Step 104, using a noise prediction network, predicting future motion based on the target time step and the historical motion coding, to obtain a predicted motion sequence; In an embodiment of the present invention, the noise prediction network includes a prediction module, and the prediction module is constructed using a diffusion model based on the Transformer architecture, wherein the prediction module is used to predict future motion based on the target time step and historical motion coding to obtain a predicted motion sequence.

[0044] Here, future motion prediction includes a forward process and a reverse process. In the forward process, the overall motion sequence is reparameterized to obtain a noisy discrete cosine transform spectrum corresponding to the target time step; in the reverse process, future motion prediction is performed based on the target time step, historical motion coding and noisy discrete cosine transform spectrum to obtain a predicted motion sequence.

[0045] Step 105: compare the reconstructed motion sequence with the historical motion sequence to obtain a reconstruction error.

[0046] In an embodiment of the present invention, each frame (or each time step) in the reconstructed motion sequence is compared with the frame at the corresponding time point in the historical motion sequence, for example, motion parameters such as position, velocity, acceleration, etc. of each joint are compared.

[0047] The difference of each joint parameter between the reconstructed frame and the historical frame is calculated. For example, the difference can be quantified by various metrics, such as Euclidean distance, Manhattan distance or similarity metric. Then, these differences are determined as a single numerical value (such as average error, mean square error, etc.) as the reconstruction error of the frame.

[0048] Step 106, comparing the predicted motion sequence with the future motion sequence to obtain a prediction error.

[0049] In an embodiment of the present invention, each frame (or each time step) in the predicted motion sequence is compared with the frame at the corresponding time point in the future motion sequence, for example, motion parameters such as position, velocity, acceleration, etc. of each joint are compared.

[0050] The difference of each joint parameter between the predicted frame and the future frame is calculated. For example, the difference can be quantified by various metrics, such as Euclidean distance, Manhattan distance, or similarity measure. These differences are then determined as a single numerical value (such as mean error, mean square error, etc.) as the prediction error of the frame.

[0051] Step 107: determining abnormal video frames in the overall motion sequence based on the reconstruction error and the prediction error.

[0052] In the embodiment of the present invention, the reconstructed motion sequence and the predicted motion sequence are compared with the true value data respectively, so as to obtain the reconstruction error and the prediction error. For example, the Euclidean distance between the reconstructed motion sequence and the reconstructed true value data (i.e., the historical motion sequence) is used as the reconstruction error, and the Euclidean distance between the predicted motion sequence and the predicted true value data (i.e., the future motion sequence) is used as the prediction error.

[0053] According to the reconstruction error and prediction error of each frame in the overall motion sequence, the attribute anomaly score of each frame is determined, and based on the attribute anomaly score of each frame in the overall motion sequence, the abnormal frame of the video is determined.

[0054] Due to the inherent multimodality of human motion and the diversity of normal and abnormal modes, in the related art, simply applying reconstruction-based video anomaly detection methods or prediction-based video anomaly detection methods has limitations.

[0055] Therefore, in an embodiment of the present invention, a video anomaly detection method based on a dynamic heterogeneous redundancy architecture (DHR) is proposed, which integrates the advantages of reconstruction-based and prediction-based video anomaly detection methods, uses multiple models with different structures (i.e., the reconstruction module and prediction module mentioned above) in processing video anomaly detection tasks, and integrates more semantic information, thereby increasing the reliability of video anomaly detection and improving detection accuracy.

[0056] In addition, the dynamic heterogeneous redundant architecture in which each module (reconstruction module and prediction module) can be used independently further enhances the flexibility and stability of the system and ensures the high efficiency of data processing and anomaly detection.

[0057] refer to Figure 2 , Figure 2 It is a structural schematic diagram of a dual-conditional motion diffusion model of a human skeleton based on a dynamic heterogeneous redundant architecture provided by the present invention, which includes an input part, a reconstructed motion, a predicted motion, a reconstructed branch and a predicted branch.

[0058] The input part includes the historical sequence ( , that is, the historical movement sequence) and the future sequence ( , i.e. future motion sequence) and DCT module; reconstructed motion includes reconstructed motion coding ; The reconstruction branch includes the encoder, decoder, and (conditional) embedding; the prediction branch includes conditional motion, DCT module, motion transformer, conditional embedding, and denoising motion after masking; the predicted motion includes the IDCT module and the future sequence.

[0059] refer to Figure 3 , Figure 31 is a visualization diagram of experimental results of the dual-conditional motion diffusion video anomaly detection method of the human skeleton based on the dynamic heterogeneous redundant architecture provided by the present invention, which includes (a) fast running, (b) throwing a school bag, (c) riding a motorcycle, and (d) chasing.

[0060] In order to enhance the intuitive understanding of the results of the proposed method, the present invention simultaneously visualizes the skeleton and anomaly score of each individual. Four anomaly sequences in the dataset scenario are illustrated, where: Figure 3 The abnormal event described in (a) involves fast running, Figure 3 The abnormal event described in (b) involves throwing a schoolbag, Figure 3 The unusual event described in (c) involves riding a motorcycle, Figure 3 The abnormal event described in (d) involves a chase. The proposed video anomaly detection method based on a dynamic heterogeneous redundancy architecture is compared with variants that only use reconstruction and prediction, respectively.

[0061] The results show that the reconstruction model has a higher detection rate for location-related events, and the prediction model has a higher detection rate for motion-related events, and both can work independently in the video anomaly detection task. The video anomaly detection method based on dynamic heterogeneous redundant architecture proposed in the present invention can better utilize the complementarity between the two methods to detect anomalies.

[0062] refer to Figure 4 , Figure 4 It is the anomaly score of the dual-conditional motion diffusion video anomaly detection method of the human skeleton based on the dynamic heterogeneous redundant architecture provided by the present invention.

[0063] In addition, the embodiments of the present invention provide more qualitative results to demonstrate the effectiveness of the method proposed in the present invention in detecting anomalies in the test video. Figure 4 In the embodiment, the method of the present invention (Ours) is compared with the detection results of MoCoDAD. The method of the embodiment of the present invention shows superior performance, and has strong processing capabilities and higher detection accuracy for both short videos and long videos.

[0064] According to a video anomaly detection method based on a dynamic heterogeneous redundant architecture provided by an embodiment of the present invention, a reconstructed motion sequence is compared with a historical motion sequence to obtain a reconstruction error; a predicted motion sequence is compared with a future motion sequence to obtain a prediction error, including: determining a plurality of target objects corresponding to the overall motion sequence; Based on the sliding window, the entire motion sequence is traversed according to a preset number of target frames to determine a plurality of target frames corresponding to each of the plurality of target objects; Based on the target time step, determining a current motion sequence in which each target frame in the plurality of target frames is located; When the current motion sequence is in the historical motion sequence, the reconstructed motion sequence is compared with the historical motion sequence to obtain the reconstruction error; When the current motion sequence is in the future motion sequence, the predicted motion sequence is compared with the future motion sequence to obtain a prediction error.

[0065] In the embodiment of the present invention, a sliding window W is used to analyze the overall motion sequence, wherein the sliding window W includes a target number of continuous video frames, and firstly all participants of all frames in the sliding window W of the overall motion sequence are extracted, wherein the participants are the target objects in each frame, for example, the human objects in each frame. Then, starting from random noise, the Euclidean distance of each joint parameter of the participant between the reconstructed motion sequence and the historical motion sequence, and the predicted motion sequence and the future motion sequence is calculated using a noise prediction network, and the reconstruction error or prediction error of each target frame in multiple target frames of each participant is obtained.

[0066] For example, the overall motion sequence is 30 frames, and the target number of the sliding window W is set to 5 frames. At this time, frames 1-5, 2-6... until 25-30 of the overall motion sequence are extracted sequentially, and all target characters in all frames in the sliding window W of the overall motion sequence are extracted. For each target character, multiple target frames corresponding to the target character are obtained. When the target frame is in the historical motion sequence, the Euclidean distance of each joint parameter of the target character between the reconstructed motion sequence and the historical motion sequence is determined to obtain the reconstruction error of the target frame of the target character; when the target frame is in the future motion sequence, the Euclidean distance of each joint parameter of the target character between the predicted motion sequence and the future motion sequence is determined to obtain the prediction error of the target frame of the target character.

[0067] It should be noted that, according to the target time step, it is determined that the multiple target frames of the target person are in the historical motion sequence or the future motion sequence. For example, the multiple target frames of the target person are the 5th frame, the 6th frame and the 10th frame respectively; When the target time step is the third frame, the frames before the third frame are the historical motion sequence, and the frames after the third frame are the future motion sequence; at this time, multiple target frames of the target person are in the future motion sequence; When the target time step is the 9th frame, the frames before the 9th frame are the historical motion sequence, and the frames after the 9th frame are the future motion sequence; at this time, the target frames of the target person are in the historical motion sequence (the 5th and 6th frames) and the future motion sequence (the 10th frame).

[0068] Through the embodiments of the present invention, it is possible to determine, based on the target time step, whether each target frame of the target object is in a historical motion sequence or a future motion sequence, and then determine the reconstruction error or prediction error corresponding to the target frame, and traverse the entire motion sequence based on a sliding window loop to determine the reconstruction error or prediction error of each target frame in multiple target frames of the target object.

[0069] According to an embodiment of the present invention, a video anomaly detection method based on a dynamic heterogeneous redundant architecture is provided, which determines an abnormal video frame in an overall motion sequence based on a reconstruction error and a prediction error, including: The reconstruction error is taken as the first anomaly score and the prediction error is taken as the second anomaly score; Determine anomaly score sets corresponding to the plurality of target frames, and use an average value of the anomaly score sets as a local score of the target object, wherein the anomaly score set includes a first anomaly score and a second anomaly score; determining a set of attribute anomaly scores based on the local score of each target object in the plurality of target objects; Based on the set of attribute anomaly scores, the abnormal video frames in the overall motion sequence are determined.

[0070] In a cyclic manner, a sliding window is used to traverse all frames in the overall motion sequence to evaluate the video, determine the reconstruction error or prediction error corresponding to each frame, and use the reconstruction error as the first anomaly score and the prediction error as the second anomaly score; then the set of anomaly scores for each participant (including the first anomaly score and the second anomaly score) is averaged into a single value Ha, representing the anomaly score of participant a. This process is then performed for each participant, and these local scores are aggregated into the set Then, we use the set To determine the attribute anomaly score set (AS) of the overall motion sequence, the anomaly score calculation method is as follows: (1)

[0071] in, represents the set of attribute anomaly scores, Indicates the first frame, Indicates frames, of which Represents the total number of frames of the overall motion sequence, The maximum error distribution for all participants (all frames) in each frame is encapsulated, Representing a collection The maximum value in Representing a collection The minimum value in .

[0072] Here, each anomaly score in the attribute anomaly score set corresponds to an abnormal video frame.

[0073] Through the embodiments of the present invention, it is possible to calculate anomaly scores for multiple target frames of each target object in the overall motion sequence. Here, the multiple target frames may be in the historical motion sequence and the future motion sequence respectively. Therefore, a heterogeneous redundant architecture is used to jointly determine the local score of the target object according to the reconstruction error or prediction error corresponding to each frame in the target frame. Subsequently, a set of attribute anomaly scores is determined according to the local score of each target object in the multiple target objects to obtain the video anomaly frame in the overall motion sequence.

[0074] In some embodiments, the historical motion sequence is encoded based on the encoder to obtain the historical motion coding; and the historical motion coding is reconstructed (decoded) based on the decoder to obtain the historical motion coding.

[0075] Considering the temporal and spatial dimensions of the input historical motion sequence, the reconstruction module uses a spatiotemporal separable autoencoder, which relies on a U-Net-like architecture to gradually shrink the skeleton motion network and expand the spatial dimension of the input motion sequence.

[0076] In the embodiment of the present invention, the encoder E of the reconstruction module encodes the historical motion sequence into historical motion coding, which can be specifically expressed by the following formula (2): (2)

[0077] in, represents the historical motion encoding, i.e., conditional embedding, represents the encoder of the reconstruction module, It means starting from the first element to the elements (including elements) historical movement sequence.

[0078] The decoder D of the reconstruction module uses the reconstruction loss to reconstruct the historical motion code into the reconstructed motion code , where the reconstruction loss function can be expressed by the following formula (3): (3)

[0079] in, represents the reconstruction loss, Represents the square of the L2 norm of the difference between the reconstructed motion code and the historical motion sequence.

[0080] Through the embodiments of the present invention, the reconstruction loss of the noise prediction model in the reconstruction branch for reconstructing the historical motion coding can be determined.

[0081] According to a video anomaly detection method based on a dynamic heterogeneous redundant architecture provided by the present invention, future motion prediction is performed based on a target time step and historical motion coding to obtain a predicted motion sequence, including: Through the diffusion model of the Transformer architecture, the overall motion sequence is reparameterized in the forward process to obtain the noisy discrete cosine transform spectrum corresponding to the target time step; Through the diffusion model of the Transformer architecture, in the reverse process, future motion prediction is performed based on the target time step, historical motion coding, and noisy discrete cosine transform spectrum to obtain a predicted motion sequence.

[0082] In the embodiment of the present invention, a prediction module is constructed using a diffusion model based on the Transformer architecture. By using the reparameterization technique, we can calculate the Noise Discrete Cosine Transform (DCT) spectrum at , please refer to the following formula (4) for details: (4)

[0083] in, Representation and time step The corresponding noisy discrete cosine transform spectrum, represents the overall motion sequence, is a time step Varying parameters to control and noise exist The relative contribution of , ; is a predefined variance parameter, represents a random noise from a standard normal distribution.

[0084] Through the embodiments of the present invention, reparameterization allows the model to dynamically adjust its internal parameters during the training process to better adapt to different motion sequences and target time steps, thereby improving the adaptability and expressiveness of the model; by converting the overall motion sequence into a noisy DCT spectrum, the model can learn richer motion features and noise patterns in the forward process.

[0085] According to a video anomaly detection method based on a dynamic heterogeneous redundant architecture provided by the present invention, future motion prediction is performed based on a target time step, historical motion coding, and a noise discrete cosine transform spectrum to obtain a predicted motion sequence, including: embedding historical movement coding as a condition; Determine the multi-head self-attention of the noisy discrete cosine transform spectrum and the sinusoidal timestep embedding of the target timestep; The first feature is obtained by feature modulation based on multi-head self-attention, sinusoidal time step embedding, and conditional embedding; Based on the first feature representation, a residual connection is performed with the noise discrete cosine transform spectrum to obtain a hidden representation; Input the hidden representation into the feedforward neural network to obtain the integrated features of the feedforward neural network output; Based on the integrated features, sinusoidal time step embedding, and conditional embedding, feature modulation is performed to obtain the second feature; Perform residual connection based on the second feature representation and the hidden representation to obtain the output representation; An inverse discrete cosine transform is performed based on the output representation to obtain a predicted motion sequence.

[0086] In the embodiment of the present invention, the prediction of each back diffusion step is completed by using a motion transformer, wherein the motion transformer is characterized in that it allows the potential correlation to be obtained from multiple layers of features, L Transformer blocks are stacked by skip connections, and each block contains two linear-based FiLM modules, which are connected by diffusion time steps. The (target time step) embedding and conditional embedding modulate the FiLM module to establish temporal relationships, where the overall motion sequence of the input is When , the overall equation can be formalized by the following formula (5) to formula (6): (5)

[0087] in, To hide the representation, represents a real matrix, represents the sequence length, Indicates the number of channels, Represents the first feature, Represents the FiLM module represents multi-head self-attention, represents the target time step Sinusoidal time step embedding (TimeEmbedding), represents the conditional embedding, i.e., historical motion encoding, Represents the overall motion sequence.

[0088] (6)

[0089] in, Indicates that the channel is The output indicates that represents the sequence length, Represents the second feature, represents the FiLM module, Indicates hidden representation, represents the integrated features of the feedforward neural network output, represents the target time step The sinusoidal time step embedding of ; Represents conditional embedding.

[0090] Here, the time step embedding It is often used to encode temporal information (such as position in a sequence) into a vector so that the model can process time-dependent data.

[0091] Here, multiple attention It can be expressed by the following formula (7): (7)

[0092] in, , here, represents the sequence length (or spatial dimension), Indicates the number of channels, Indicates the number of heads. represents the multi-head attention mechanism, represents the overall motion sequence, , , They are The query, key and value of each head are obtained by converting the whole motion sequence With three different weight matrices , , Multiplying together, Represents the transposed matrix.

[0093] Finally, the outputs of multiple heads are concatenated through the Motion Transformer to obtain the final result (i.e., the output representation), and the output representation is inversely discrete cosine transformed to obtain the predicted motion sequence.

[0094] Subsequently, the noise prediction loss function is used to optimize the noise parameters of the noise prediction network. , where the noise prediction network includes a reconstruction module based on the autoencoder architecture and a prediction module based on the diffusion model.

[0095] Here, the noise prediction loss function of the noise prediction network is determined based on the noise discrete cosine transform spectrum, the target time step and the historical motion coding, wherein the target time step has a corresponding time relationship with the historical motion coding.

[0096] Based on the noise discrete cosine transform spectrum, target time step and historical motion coding, the noise prediction loss function of the noise prediction network is determined, which can be expressed by the following formula (8): (8)

[0097] in, represents the noise prediction loss function, represents the expectation operator for random variables (noise) and the target time step The joint distribution of takes the expectation (mean), Indicates smooth Loss function, represents the noise parameter, Representation and time step The corresponding noisy discrete cosine transform spectrum, represents historical motion encoding, i.e., conditional embedding.

[0098] In the embodiment of the present invention, the noise prediction network is updated by the noise prediction loss function to obtain an updated noise prediction network, and a predicted motion sequence corresponding to the future motion sequence is output based on the updated noise prediction network.

[0099] In the embodiment of the present invention, it is assumed that the noise prediction network is Parameterized. Let E denote the reconstruction (module) branch encoder. In the proposed reverse process, the recovered human motion can refer to the following formula (9): (9)

[0100] in, Indicates the recovery of human movement, is a time step Varying parameters to control and noise exist Relative contribution of Represents the target time step The corresponding noisy discrete cosine transform spectrum, , ; is a predefined variance parameter, represents noise, represents the noise parameter, Represents the motion coding including the content information of the historical motion sequence, i.e., the historical motion coding, which is embedded as a condition. Represents the target time step The associated standard deviation, To hide the representation.

[0101] Through the embodiments of the present invention, by introducing a linear-based FiLM (Feature-wise Linear Modulation) module and combining the embedding of the target time step and the conditional embedding for modulation, the relationship between the feature and the time step can be established, and the motion converter can capture potential correlations from multiple layers of features and make predictions in combination with time information, so it can generate more accurate and stable prediction results in the back diffusion process.

[0102] According to a video anomaly detection method based on a dynamic heterogeneous redundant architecture provided by the present invention, after future motion prediction is performed based on a target time step and historical motion coding to obtain a predicted motion sequence, the method further includes: When the length of the historical motion sequence matches that of the overall motion sequence, the historical motion sequence is subjected to discrete cosine transform to obtain compact historical information; Perform noise processing on the compact historical information to obtain the noise observation spectrum corresponding to the target time step; Performing inverse discrete cosine transformation and masking on the noise observation spectrum and the denoised predicted spectrum to obtain a spectrum combination sequence, wherein the denoised predicted spectrum is determined based on the predicted motion sequence; The historical motion encoding is used as a conditional embedding, and the spectral combination sequence is used as the conditional motion; Conditional embeddings and conditional motion, as dual conditioning in the backward process in a diffusion model of the Transformer architecture.

[0103] In the embodiment of the present invention, the reconstruction-based and prediction-based methods are integrated to propose a dual-conditional motion diffusion model of the human skeleton based on a dynamic heterogeneous redundant architecture. The architecture of the proposed model is as follows Figure 2 The above model integrates the conditional embedding in the reconstruction module and the conditional motion in the prediction module as dual conditions in the reverse process (reverse process) to gradually predict the future sequence from the noise variable distribution.

[0104] In the embodiment of the present invention, in the inference stage, the last frame of the observation sequence (historical motion sequence) is repeated until the length matches the length of the complete sequence (overall motion sequence); then the historical motion sequence is subjected to DCT transformation to obtain compact historical information, which is recorded as .

[0105] Then, the compact history information Add noise and get the time step (target time step) The noise spectrum of the observation value at is denoted as the noise observation spectrum .

[0106] Aggregate the noisy historical sequence and the denoised predicted sequence together, that is, denoise the historical motion sequence and the predicted motion sequence to obtain the denoised predicted spectrum , to observe the noise observation spectrum and denoised predicted spectrum The approximate distribution is similar.

[0107] Subsequently, the inverse discrete cosine transform (iDCT) operation is used to transform the frequency spectra of the noisy observation spectrum and the denoised prediction spectrum into the time domain, and the two spectra are combined using a mask mechanism to obtain a spectral combination sequence. The entire spectral combination sequence is then sent to the reverse process as a conditional motion.

[0108] Through the embodiments of the present invention, reconstruction and prediction are integrated by using conditional embedding (i.e., historical motion encoding) in the reconstruction module and conditional motion in the prediction module as dual conditions in the inverse process (reverse), and the posture features and potential semantics of the observed motion are utilized to improve the accuracy and robustness of anomaly detection.

[0109] According to a video anomaly detection method based on a dynamic heterogeneous redundant architecture provided by the present invention, the method further includes: Determine the joint correlation difference of each frame based on the relative temporal distance and relative numerical difference of each frame in the overall motion sequence; The product of the average value of the joint correlation difference and the preset parameter is taken as the joint correlation difference loss of the whole motion sequence; Based on the reconstruction error, determine the reconstruction loss function of the noise prediction network; Based on the prediction error, determining the prediction loss function of the noise prediction network; Based on the reconstruction loss function, the prediction loss function and the joint correlation difference loss, the total loss function of the noise prediction network is determined; and the noise prediction network is updated through the total loss function.

[0110] It should be noted that based on the relative time distance and relative numerical difference of each frame in the overall motion sequence, the joint correlation difference of each frame is determined; the product of the average value of the joint correlation difference and the preset parameter is used as the joint correlation difference loss of the overall motion sequence. This can be performed after obtaining the overall motion sequence of the human skeleton, or after obtaining the reconstruction error and the prediction error. That is, the processes of determining the joint correlation difference loss of the overall motion sequence, determining the reconstruction error, and determining the prediction error are independent of each other, and there is no logical order of execution.

[0111] In an embodiment of the present invention, a joint correlation difference regularization is designed. Its purpose is to enhance the distinction between normal frames and abnormal frames. The main components of the joint correlation difference regularization loss include temporal correlation and global correlation.

[0112] In an embodiment of the present invention, a reconstruction loss function is determined based on a reconstruction error, and a noise prediction loss function is determined based on a prediction error; a total loss function is determined based on the reconstruction loss function, the noise prediction loss function and the joint correlation difference loss, and then the noise prediction network is updated by the total loss function to output a predicted motion sequence and a reconstructed motion sequence through the noise prediction network, and the predicted motion sequence and the reconstructed motion sequence are compared with the corresponding data true value to determine the abnormal frame of the video.

[0113] In the embodiment of the present invention, the total loss function of the noise prediction network can refer to the following formula (10): (10)

[0114] in, represents the total loss function, represents the reconstruction loss function (refer to Formula 3 above), represents the noise prediction loss function (refer to the above formula 8), The contribution (weight) of the additional loss is the preset parameter, represents the L1 norm of the joint association difference (UAD) (i.e., joint association difference loss), where Represents time association and Represents global association, Represents the overall motion sequence.

[0115] In the embodiment of the present invention, in order to minimize the total loss. In the case of , the goal is to optimize the additional loss by enhancing UAD. However, directly maximizing UAD is problematic because abnormal frames are rare and can be easily ignored; at the same time, the scale parameter of the Gaussian kernel is significantly reduced, making the temporal association irrelevant. To better control the joint learning, the present invention adopts a min-max strategy, which uses a specially designed stopping gradient mechanism to constrain the temporal association and the global association to obtain a more distinguishable joint association difference.

[0116] Specifically, in the UAD minimization phase, the goal is to make Converges to the learned from the original motion sequence . Stop gradient backpropagation, this convergence process helps Adjust the different time modes. In the UAD Maximization stage, focus on stopping Back propagation of gradients to optimize , to increase UAD, thus forcing More emphasis is placed on non-adjacent frames.

[0117] According to a video anomaly detection method based on a dynamic heterogeneous redundant architecture provided by the present invention, based on the relative time distance and relative numerical difference of each frame in the overall motion sequence, the joint correlation difference of each frame is determined, including: Take each frame in the overall motion sequence as the current frame and perform the following operations to obtain the joint correlation difference of each frame: Based on the time axis of the overall motion sequence, determine the relative time distance between the current frame and other frames; Based on the relative temporal distance, determine the temporal relevance of the current frame; Based on the self-attention value of each frame in the overall motion sequence, determine the relative value difference between the current frame and other frames; Based on the relative numerical difference, the global relevance of the current frame is determined; Based on the temporal correlation and the global correlation, the joint correlation difference of the current frame is determined.

[0118] In the embodiment of the present invention, the temporal association is considered only from the dimension of temporal distance. Frames with closer distances should have stronger associations (greater than the temporal association threshold), while frames with farther distances should have weaker associations (less than the temporal association threshold). The temporal association follows a normal distribution curve, forming a single peak shape.

[0119] In order to establish temporal association, the embodiment of the present invention uses a learnable Gaussian kernel to predetermine the relative temporal distance. In addition, a learnable scale parameter is added to the Gaussian kernel, so that temporal association can be concentrated in adjacent areas and adapted to various time series patterns, including anomalies of different durations. Temporal association can be expressed by the following formula (11): (11)

[0120] in, Indicates time association, Indicates that for The standard deviation of the Gaussian function of the frames is Indicates frames, is the specific form of the Gaussian function, which is used to calculate each frame element in the predicted motion sequence. Indicates the length of the overall motion sequence (total number of frames), The function is used to scale the original data to a specified range, commonly [0,1] and [-1,1].

[0121] Here, Used to indicate The head can learn the scale parameter, where represents the standard deviation of the Gaussian function, represents a real matrix, Indicates the length of the real-question motion sequence (total number of frames), Represents the header by removing the line and using Convert the association weights to a discrete distribution.

[0122] In an embodiment of the present invention, considering the global correlation from the numerical dimension, the frames of abnormal behavior have a stronger (numerical) correlation with frames that are closer in the time dimension, and a weaker (numerical) correlation with frames that are farther away, presenting a unimodal pattern, where the numerical value can be the self-attention value (self-attention strength) of each frame in the overall motion sequence.

[0123] Similarly, frames with normal behavior have strong (numerical) associations with frames that are farther or closer in the temporal dimension. Global association learns associations from the original motion sequence and then adaptively identifies the most effective associations, which can be expressed by the following formula (12): (12)

[0124] in, Represents global association, It is used to convert each element in the matrix into a probability value (i.e., relative numerical difference), so that each row of the matrix is ​​a valid probability distribution. and Represents the matrices used in attention mechanisms (such as self-attention or encoder-decoder attention in the Transformer model), representing the query and key matrices respectively, represents the transposed matrix, is the scaling factor, Usually the key matrix The scaling factor is used to prevent the result from being too large when calculating the dot product, which can lead to gradient vanishing or exploding problems.

[0125] Abnormal frames show more significant similarities between temporal associations and global associations, resulting in smaller relative entropy (KL, Kullback-Leibler Divergence) values. In contrast, normal frames show lower similarities between these associations, resulting in larger KL values. Therefore, in an embodiment of the present invention, the joint association difference is defined as the symmetric KL divergence between temporal associations and global associations. The union association difference (UAD) is calculated by averaging multiple layers, thereby integrating the associations of a series of feature layers into a more informative and robust metric. For details, please refer to the following formula (13).

[0126] (13)

[0127] in, represents the joint association difference, Indicates time association, Represents global association, represents the overall motion sequence, represents the number of feature layers, Represents the index of the feature layer, represents the total number of feature layers, represents KL divergence (Kullback-Leibler Divergence), Indicates the time association at time step A slice (a row) of Indicates the global correlation at time step A slice (a row), ) represents the total length of the time step (frame).

[0128] Here, the KL divergence is calculated for each row (time association) and Calculated between two discrete distributions of (global association), the UAD (Joint Association Difference) of the anomalous frames is smaller than that of the normal frames, making it an intrinsically distinguishable factor.

[0129] Through the embodiments of the present invention, a joint correlation difference regularization is designed, which utilizes the temporal correlation based on Gaussian kernel and the global correlation based on self-attention, and effectively enhances the model's ability to capture temporal and spatial correlation.

[0130] It should be noted that general video anomaly detection often has limitations when dealing with complex and dynamic scenes because it only uses a single proxy model based on reconstruction or prediction. However, the present invention proposes a unified framework that seamlessly integrates the advantages of reconstruction-based and prediction-based skeleton-based video anomaly detection methods, thereby being able to utilize the posture features and potential semantics of the observed motion to improve the accuracy and robustness of anomaly detection. At the same time, in addition to the losses inherent in reconstruction and prediction themselves, the embodiment of the present invention also designs a joint correlation difference regularization, which utilizes the temporal correlation based on Gaussian kernel and the global correlation based on self-attention, effectively enhancing the model's ability to capture temporal and spatial correlations.

[0131] The video anomaly detection method based on dynamic heterogeneous redundant architecture proposed by the present invention can achieve anomaly detection with higher accuracy and robustness in a variety of environments by combining reconstruction-based and prediction-based anomaly detection methods. In addition, the dynamic heterogeneous redundant architecture in which each module can be applied separately further improves the flexibility and stability of the system and ensures the high efficiency of data processing and anomaly detection.

[0132] The video anomaly detection device based on a dynamic heterogeneous redundant architecture provided by the present invention is described below. The video anomaly detection device based on a dynamic heterogeneous redundant architecture described below and the video anomaly detection method based on a dynamic heterogeneous redundant architecture described above can be referenced to each other.

[0133] refer to Figure 5 , Figure 5 It is a structural schematic diagram of a video anomaly detection device based on a dynamic heterogeneous redundant architecture provided by the present invention, which includes an acquisition module 501, an encoding module 502, a reconstruction module 503, a prediction module 504, a first comparison module 505, a second comparison module 506 and an output module 507.

[0134] An acquisition module 501 is used to acquire an overall motion sequence of a human skeleton, wherein the overall motion sequence is used to represent the motion changes of human joints over time during the motion of the human skeleton, and the overall motion sequence includes a historical motion sequence and a future motion sequence; An encoding module 502 is used to encode the historical motion sequence to obtain a historical motion code, wherein the historical motion sequence is a sequence before a target time step in the overall motion sequence; A reconstruction module 503 is used to reconstruct the historical motion based on the historical motion coding through a noise prediction network to obtain a reconstructed motion sequence; A prediction module 504 is used to perform future motion prediction based on the target time step and the historical motion coding through the noise prediction network to obtain a predicted motion sequence; A first comparison module 505, used to compare the reconstructed motion sequence with the historical motion sequence to obtain a reconstruction error; A second comparison module 506 is used to compare the predicted motion sequence with the future motion sequence to obtain a prediction error; The output module 507 is used to determine the abnormal video frame in the overall motion sequence based on the reconstruction error and the prediction error.

[0135] Specifically, the video anomaly detection device based on the dynamic heterogeneous redundant architecture provided by the present invention can implement all the method steps implemented by the above-mentioned video anomaly detection method embodiment based on the dynamic heterogeneous redundant architecture, and can achieve the same technical effect. The parts and beneficial effects that are the same as the method embodiment in this embodiment will not be described in detail here.

[0136] Figure 6 is a schematic diagram of the physical structure of the electronic device provided by the present invention, such as Figure 6 As shown, the electronic device may include: a processor (processor) 610 , a communication interface (Communications Interface) 620 , a memory (memory) 630 and a communication bus 640 , wherein the processor 610 , the communication interface 620 , and the memory 630 communicate with each other through the communication bus 640 . The processor 610 can call the logic instructions in the memory 630 to execute a video anomaly detection method based on a dynamic heterogeneous redundant architecture, the method comprising: obtaining an overall motion sequence of a human skeleton, wherein the overall motion sequence is used to represent the movement changes of human joints over time during the movement of the human skeleton, and the overall motion sequence comprises a historical motion sequence and a future motion sequence; encoding the historical motion sequence to obtain a historical motion code, wherein the historical motion sequence is a sequence before a target time step in the overall motion sequence; reconstructing the historical motion based on the historical motion code through a noise prediction network to obtain a reconstructed motion sequence; predicting the future motion based on the target time step and the historical motion code through a noise prediction network to obtain a predicted motion sequence; comparing the reconstructed motion sequence with the historical motion sequence to obtain a reconstruction error; comparing the predicted motion sequence with the future motion sequence to obtain a prediction error; and determining abnormal video frames in the overall motion sequence based on the reconstruction error and the prediction error.

[0137] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0138] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the video anomaly detection method based on a dynamic heterogeneous redundant architecture provided by the above methods, and the method includes: obtaining an overall motion sequence of a human skeleton, wherein the overall motion sequence is used to represent the movement changes of human joints over time during the movement of the human skeleton, and the overall motion sequence includes a historical motion sequence and a future motion sequence; encoding the historical motion sequence to obtain a historical motion code, wherein the historical motion sequence is a sequence before a target time step in the overall motion sequence; reconstructing the historical motion based on the historical motion code through a noise prediction network to obtain a reconstructed motion sequence; predicting the future motion based on the target time step and the historical motion code through a noise prediction network to obtain a predicted motion sequence; comparing the reconstructed motion sequence with the historical motion sequence to obtain a reconstruction error; comparing the predicted motion sequence with the future motion sequence to obtain a prediction error; and determining the video abnormal frame in the overall motion sequence based on the reconstruction error and the prediction error.

[0139] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the processor executes the video anomaly detection method based on a dynamic heterogeneous redundant architecture provided by the above-mentioned methods, the method comprising: obtaining an overall motion sequence of a human skeleton, wherein the overall motion sequence is used to represent the movement changes of human joints over time during the movement of the human skeleton, and the overall motion sequence comprises a historical motion sequence and a future motion sequence; encoding the historical motion sequence to obtain a historical motion code, wherein the historical motion sequence is a sequence before a target time step in the overall motion sequence; reconstructing the historical motion based on the historical motion code through a noise prediction network to obtain a reconstructed motion sequence; predicting the future motion based on the target time step and the historical motion code through a noise prediction network to obtain a predicted motion sequence; comparing the reconstructed motion sequence with the historical motion sequence to obtain a reconstruction error; comparing the predicted motion sequence with the future motion sequence to obtain a prediction error; and determining abnormal video frames in the overall motion sequence based on the reconstruction error and the prediction error.

[0140] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0141] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0142] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A video anomaly detection method based on a dynamic heterogeneous redundant architecture, characterized in that: include: Acquire an overall motion sequence of a human skeleton, wherein the overall motion sequence is used to represent the motion changes of human joints over time during the motion of the human skeleton, and the overall motion sequence includes a historical motion sequence and a future motion sequence; Encoding the historical motion sequence to obtain a historical motion code, wherein the historical motion sequence is a sequence before a target time step in the overall motion sequence; Reconstructing the historical motion based on the historical motion coding through a noise prediction network to obtain a reconstructed motion sequence; By using the noise prediction network, future motion prediction is performed based on the target time step and the historical motion coding to obtain a predicted motion sequence; Comparing the reconstructed motion sequence with the historical motion sequence to obtain a reconstruction error; Comparing the predicted motion sequence with the future motion sequence to obtain a prediction error; Based on the reconstruction error and the prediction error, abnormal video frames in the overall motion sequence are determined.

2. The video anomaly detection method based on dynamic heterogeneous redundant architecture according to claim 1 is characterized in that: The comparing the reconstructed motion sequence with the historical motion sequence to obtain a reconstruction error; and comparing the predicted motion sequence with the future motion sequence to obtain a prediction error, comprises: determining a plurality of target objects corresponding to the overall motion sequence; Based on the sliding window, the overall motion sequence is traversed in a loop according to a preset number of target frames to determine a plurality of target frames corresponding to each of the plurality of target objects; Based on the target time step, determining a current motion sequence in which each target frame in the plurality of target frames is located; When the current motion sequence is in the historical motion sequence, comparing the reconstructed motion sequence with the historical motion sequence to obtain a reconstruction error; When the current motion sequence is in the future motion sequence, the predicted motion sequence is compared with the future motion sequence to obtain a prediction error.

3. The video anomaly detection method based on dynamic heterogeneous redundant architecture according to claim 2 is characterized in that: The determining of abnormal video frames in the overall motion sequence based on the reconstruction error and the prediction error includes: Using the reconstruction error as a first anomaly score and the prediction error as a second anomaly score; Determine anomaly score sets corresponding to the multiple target frames, and use an average value of the anomaly score sets as the local score of the target object, wherein the anomaly score set includes the first anomaly score and the second anomaly score; determining a set of attribute anomaly scores based on the local score of each target object in the plurality of target objects; Based on the attribute anomaly score set, abnormal video frames in the overall motion sequence are determined.

4. The video anomaly detection method based on dynamic heterogeneous redundant architecture according to claim 1 is characterized in that: The performing future motion prediction based on the target time step and the historical motion coding to obtain a predicted motion sequence includes: Through the diffusion model of the Transformer architecture, in the forward process, the overall motion sequence is reparameterized to obtain a noisy discrete cosine transform spectrum corresponding to the target time step; Through the diffusion model of the Transformer architecture, in the reverse process, future motion prediction is performed based on the target time step, the historical motion coding and the noise discrete cosine transform spectrum to obtain a predicted motion sequence.

5. The video anomaly detection method based on dynamic heterogeneous redundant architecture according to claim 4 is characterized in that: The future motion prediction based on the target time step, the historical motion coding and the noise discrete cosine transform spectrum to obtain a predicted motion sequence includes: embedding the historical motion code as a condition; Determining a multi-head self-attention of the noisy discrete cosine transform spectrum and a sinusoidal time step embedding of the target time step; Performing feature modulation based on the multi-head self-attention, the sinusoidal time step embedding, and the conditional embedding to obtain a first feature; Performing a residual connection between the first feature representation and the noise discrete cosine transform spectrum to obtain a hidden representation; Inputting the hidden representation into a feedforward neural network to obtain an integrated feature output by the feedforward neural network; Perform feature modulation based on the integrated feature, the sinusoidal time step embedding, and the conditional embedding to obtain a second feature; Performing a residual connection between the second feature representation and the hidden representation to obtain an output representation; An inverse discrete cosine transform is performed based on the output representation to obtain a predicted motion sequence.

6. The video anomaly detection method based on dynamic heterogeneous redundant architecture according to claim 4 is characterized in that: After performing future motion prediction based on the target time step and the historical motion coding to obtain a predicted motion sequence, the method further includes: When the length of the historical motion sequence matches the length of the overall motion sequence, performing discrete cosine transform on the historical motion sequence to obtain compact historical information; Performing noise processing on the compact history information to obtain a noise observation spectrum corresponding to the target time step; Performing inverse discrete cosine transformation and masking on the noise observation spectrum and the denoised predicted spectrum to obtain a spectrum combination sequence, wherein the denoised predicted spectrum is determined based on the predicted motion sequence; embedding the historical motion encoding as a condition and using the spectral combination sequence as a conditional motion; The conditional embedding is combined with the conditional motion as a dual condition of the reverse process in the diffusion model of the Transformer architecture.

7. The video anomaly detection method based on dynamic heterogeneous redundant architecture according to claim 1 is characterized in that: The method further comprises: Determining a joint correlation difference of each frame based on a relative temporal distance and a relative numerical difference of each frame in the overall motion sequence; The product of the average value of the joint association difference and a preset parameter is used as the joint association difference loss of the overall motion sequence; Based on the reconstruction error, determining a reconstruction loss function of the noise prediction network; Based on the prediction error, determining a prediction loss function of the noise prediction network; Based on the reconstruction loss function, the prediction loss function and the joint correlation difference loss, a total loss function of the noise prediction network is determined; and the noise prediction network is updated by the total loss function.

8. The video anomaly detection method based on dynamic heterogeneous redundant architecture according to claim 7 is characterized in that: The determining of the joint correlation difference of each frame based on the relative time distance and the relative numerical difference of each frame in the overall motion sequence comprises: Take each frame in the overall motion sequence as the current frame and perform the following operations to obtain the joint correlation difference of each frame: Based on the time axis of the overall motion sequence, determining the relative time distance between the current frame and other frames; Determining a temporal association of the current frame based on the relative temporal distance; Determining a relative value difference between the current frame and the other frames based on the self-attention value of each frame in the overall motion sequence; Determining a global relevance of the current frame based on the relative numerical difference; Based on the temporal correlation and the global correlation, a joint correlation difference of the current frame is determined.

9. A video anomaly detection device based on a dynamic heterogeneous redundant architecture, characterized in that: include: An acquisition module, used for acquiring an overall motion sequence of a human skeleton, wherein the overall motion sequence is used for representing the motion changes of human joints over time during the motion of the human skeleton, and the overall motion sequence includes a historical motion sequence and a future motion sequence; An encoding module, used for encoding the historical motion sequence to obtain a historical motion code, wherein the historical motion sequence is a sequence before a target time step in the overall motion sequence; A reconstruction module, used for reconstructing historical motion based on the historical motion coding through a noise prediction network to obtain a reconstructed motion sequence; A prediction module, configured to perform future motion prediction based on the target time step and the historical motion coding through the noise prediction network to obtain a predicted motion sequence; A first comparison module, used for comparing the reconstructed motion sequence with the historical motion sequence to obtain a reconstruction error; A second comparison module is used to compare the predicted motion sequence with the future motion sequence to obtain a prediction error; The output module is used to determine the abnormal video frame in the overall motion sequence based on the reconstruction error and the prediction error.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the video anomaly detection method based on a dynamic heterogeneous redundant architecture as described in any one of claims 1 to 8 is implemented.

11. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the video anomaly detection method based on a dynamic heterogeneous redundant architecture as described in any one of claims 1 to 8 is implemented.

12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the video anomaly detection method based on a dynamic heterogeneous redundant architecture as described in any one of claims 1 to 8 is implemented.

Images