Video anomaly detection model training method, video anomaly detection method, and device

Abstract

The application provides a video anomaly detection model training method, a video anomaly detection method and equipment, which can be applied to the technical field of computer vision and machine learning. The training method comprises the following steps: obtaining a video source domain and a video target domain; using normal video slices to train an initial video anomaly detection model to obtain a prediction network; inputting randomly selected samples from a frame-level labeled source domain into an adversarial generation network to output first intermediate domain samples; using the first intermediate domain samples and the frame-level labeled source domain to train the initial video anomaly detection model to obtain a teacher network, and constructing a student network based on the teacher network; inputting second intermediate domain samples corresponding to the video target domain and the video target domain into the student network, inputting the first intermediate domain samples and the frame-level labeled source domain into the teacher network, adjusting network parameters of the student network based on output results of the teacher network and the student network, and obtaining a video anomaly detection model.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and machine learning, and specifically to a training method for a video anomaly detection model, a video anomaly detection method, and a device. Background Technology

[0002] Video anomaly detection aims to automatically identify events that deviate from normal patterns in video streams, and is widely used in traffic monitoring, public safety monitoring, and industrial production inspection. Currently, most video anomaly detection relies on video anomaly detection models, whose performance depends on the anomaly samples in the dataset. However, anomaly behaviors are often infrequent and unpredictable, resulting in extremely sparse anomaly samples in the dataset, which limits the detection accuracy of video anomaly detection models. Further complicating matters, when video anomaly detection models are applied to different environmental conditions (such as different weather, lighting, or camera quality), the data distribution changes significantly. This makes it difficult for a video anomaly detection model trained in the source domain (e.g., sunny weather) to accurately detect anomalies in the target domain (e.g., rainy weather, foggy weather), thus reducing the video detection accuracy. Summary of the Invention

[0003] In view of the above problems, the present invention provides a training method for a video anomaly detection model, a video anomaly detection method and device for improving the accuracy of video detection.

[0004] This invention provides a training method for a video anomaly detection model. The method includes: acquiring a video source domain and a video target domain, wherein the video source domain includes normal video slices with normal labels, and each video slice includes multiple video frames; training an initial video anomaly detection model using the normal video slices to obtain a prediction network, the prediction network being used to perform frame-level prediction on the video frames of all video slices in the video source domain, and labeling the video frames according to the frame-level prediction results to obtain a frame-level labeled source domain; inputting randomly selected samples from the frame-level labeled source domain into a generative adversarial network to output a first intermediate domain sample, wherein the semantic information of the intermediate domain sample is the same as that of the video source domain, and the pixel information of the intermediate domain sample is the same as that of the video target domain; training the initial video anomaly detection model using the first intermediate domain sample and the frame-level labeled source domain to obtain a teacher network, and constructing a student network based on the network structure and network parameters of the teacher network; inputting a second intermediate domain sample corresponding to the video target domain and the video target domain into the student network, inputting the first intermediate sample and the frame-level labeled source domain into the teacher network, and adjusting the network parameters of the student network based on the output results of the teacher network and the student network to obtain the video anomaly detection model.

[0005] According to an embodiment of the present invention, a normal video slice includes T+1 video frames, the T+1 video frames including the first T video frames as prediction input data, and the T+1th actual video frame as a label, where T is an integer greater than 0; training an initial video anomaly detection model using the normal video slice to obtain a prediction network includes: inputting the first T video frames into the initial video anomaly detection model and outputting the T+1th predicted video frame; iteratively adjusting the model parameters of the initial video anomaly detection model based on the prediction loss between the T+1th predicted video frame and the T+1th actual video frame until a preset training stopping condition is reached to obtain the prediction network.

[0006] According to an embodiment of the present invention, the prediction network annotates video frames of all video slices in the video source domain in the following manner: For the T+1 video frames of any video slice: the first T video frames are input into the prediction network, and the T+1 predicted video frame is output; based on the pixel-wise error between the T+1 predicted video frame and the T+1 actual video frame, the peak signal-to-noise ratio (PSNR) between the T+1 predicted video frame and the T+1 actual video frame is determined; the PNR is normalized to obtain a standard PNR; and the T+1 actual video frame is annotated according to the standard PNR.

[0007] According to an embodiment of the present invention, the frame-level annotation source domain is obtained by comparing a standard peak signal-to-noise ratio (PSNR) with a predetermined threshold; constructing a normal video frame set based on video frames with PSNR greater than or equal to the predetermined threshold; constructing an abnormal video frame set based on video frames with PSNR less than the predetermined threshold; and constructing a frame-level annotation source domain based on the normal video frame set and the abnormal video frame set.

[0008] According to an embodiment of the present invention, the samples randomly selected from the frame-level annotation source domain include a first normal video frame and a second normal video frame selected from a set of normal video frames, and an abnormal video frame selected from a set of abnormal video frames, wherein the second normal video frame serves as an anchor video frame; the samples randomly selected from the frame-level annotation source domain are input into the adversarial generative network to output a first intermediate domain sample, including: inputting the first normal video frame, the abnormal video frame, and the anchor video frame into the adversarial generative network respectively, and outputting an intermediate domain normal video frame, an intermediate domain abnormal video frame, and an intermediate domain anchor video frame.

[0009] According to an embodiment of the present invention, the initial video anomaly detection model includes a temporal feature extraction module for extracting temporal information between multiple video frames; training the initial video anomaly detection model using first intermediate domain samples and frame-level labeled source domains to obtain a teacher network includes: concatenating intermediate domain normal video frames, intermediate domain abnormal video frames, and intermediate domain anchor video frames with first normal video frames, abnormal video frames, and anchor video frames to obtain an input sequence; processing the input sequence using the temporal feature extraction module to obtain a first temporal feature corresponding to the first normal video frame, a second temporal feature corresponding to the abnormal video frame, and a third temporal feature corresponding to the anchor video frame; constructing a ternary feature loss function based on a first distance between the first and third temporal features, a second distance between the second and third temporal features, the standard peak signal-to-noise ratio of the first normal video frame, the standard peak signal-to-noise ratio of the abnormal video frame, and boundary parameters, wherein the boundary parameters are used to adjust the first and second distances; constructing a first objective function based on the ternary feature loss function and the prediction loss; adjusting the model parameters of the initial video anomaly detection model according to the first objective function until the first objective function converges to obtain the video anomaly detection model as the teacher network.

[0010] According to an embodiment of the present invention, a second intermediate domain sample corresponding to the video target domain and the video target domain are input into a student network, and a first intermediate domain sample and a frame-level labeled source domain are input into a teacher network. Based on the output results of the teacher network and the student network, the network parameters of the student network are adjusted to obtain a video anomaly detection model. This includes: constructing a distillation loss function for the teacher network and the student network based on the output results of the teacher network and the student network; constructing a second objective function based on the distillation loss function and the first objective function; and updating the model parameters of the student network using the second objective function until a preset training stopping condition is reached to obtain the video anomaly detection model.

[0011] According to an embodiment of the present invention, the Generative Adversarial Network (GAN) includes a first generator, a first discriminator, a second generator, and a second discriminator. The GAN is trained as follows: a predetermined number of video frames are selected from the video source domain and the video target domain, respectively, to obtain source domain video frames and target domain video frames; the first generator converts the source domain video frames into pseudo-target domain video frames, and a first loss function is constructed based on the discrimination result of the first discriminator on the pseudo-target domain video frames; the second generator converts the target domain video frames into pseudo-source domain video frames, and a second loss function is constructed based on the discrimination result of the second discriminator on the pseudo-source domain video frames; a cycle consistency loss function is constructed based on the restoration result of restoring the pseudo-target domain video frames to source domain video frames and the restoration result of restoring the pseudo-source domain video frames to target domain video frames; the initial GAN ​​is trained according to the first loss function, the second loss function, and the cycle consistency loss function to obtain the GAN.

[0012] An embodiment of the present invention also provides a video anomaly detection method, the method comprising: acquiring a video to be detected; inputting the video to be detected into a video anomaly detection model, and outputting a video anomaly detection result, wherein the video anomaly detection model is trained using the above-described training method.

[0013] Embodiments of the present invention also provide an electronic device, comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the above-described method.

[0014] The present invention also provides a computer-readable storage medium having a computer program or instructions stored thereon, which, when executed by a processor, implement the steps of the above-described method.

[0015] The present invention also provides a computer program product, including a computer program or instructions, wherein the computer program or instructions, when executed by a processor, implement the steps of the above-described method.

[0016] According to an embodiment of the present invention, a video source domain and a video target domain are obtained; an initial video anomaly detection model is trained using normal video slices in the video source domain to obtain a prediction network, which performs frame-level annotation on video frames in the video source domain to obtain a frame-level annotated source domain; samples randomly selected from the frame-level annotated source domain are input into a generative adversarial network to output first intermediate domain samples; the initial video anomaly detection model is trained using the first intermediate domain samples and the frame-level annotated source domain to obtain a teacher network; second intermediate domain samples and the video target domain are input into a student network, and the first intermediate domain samples and the frame-level annotated source domain are input into the teacher network; based on the output results of the teacher network and the student network, the teacher network is trained to obtain a video anomaly detection model. Since the prediction network is first trained using normal video frames during training, the prediction network can annotate video frames in the video source domain to obtain a frame-level annotated source domain, thereby providing refined frame-level annotation for the training of the video anomaly detection model. This enables the video anomaly detection model to accurately distinguish the boundaries between normal and abnormal frames during training, thus improving the model training accuracy. On the other hand, since the teacher network and student network are trained by combining intermediate domain samples, the intermediate domain samples retain the semantics of the source domain and are visually identical to the pixels of the target domain. By using the intermediate domain as a bridge between the source domain and the target domain, the domain difference between the source domain and the target domain can be reduced. This overcomes the problem that video anomaly detection models are unable to accurately detect anomalies when performing cross-domain detection due to the large difference between the source domain and the target domain, and improves the accuracy of video anomaly detection. Attached Figure Description

[0017] The above-described features, other objects, and advantages of the present invention will become clearer from the following description of embodiments of the invention with reference to the accompanying drawings, in which:

[0018] Figure 1 The diagram illustrates a training method for a video anomaly detection model according to an embodiment of the present invention, as well as an application scenario diagram of the video anomaly detection method.

[0019] Figure 2 A flowchart illustrating a training method for a video anomaly detection model according to an embodiment of the present invention is shown.

[0020] Figure 3 A training architecture diagram of a video anomaly detection model according to an embodiment of the present invention is shown.

[0021] Figure 4 A flowchart of a video anomaly detection method according to an embodiment of the present invention is shown.

[0022] Figure 5 A structural block diagram of a training apparatus for a video anomaly detection model according to an embodiment of the present invention is shown.

[0023] Figure 6 A structural block diagram of a video anomaly detection device according to an embodiment of the present invention is shown.

[0024] Figure 7 A block diagram of an electronic device suitable for implementing a video anomaly detection model and a video anomaly detection method according to an embodiment of the present invention is shown. Detailed Implementation

[0025] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the invention. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the invention for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concept of the invention.

[0026] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0027] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0028] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0029] In the technical solution of this invention, the user information (including but not limited to user personal information, user image information, user device information, such as location information) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of related data all comply with relevant laws, regulations, and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entry points for users to choose to authorize or refuse.

[0030] In the field of video anomaly detection, model performance largely depends on the quantity and diversity of anomalous samples in the training data. However, anomalous behavior itself is often infrequent and unpredictable, resulting in extremely sparse anomalous samples in the dataset. This problem is particularly evident in real-world scenarios, such as traffic monitoring, public safety monitoring, or industrial production inspection, where the probability of anomalous events is low, leading to insufficient anomalous data available in the training set, thus limiting the model's learning ability and generalization performance. Further complicating matters, the data distribution changes significantly when the model is applied to different environmental conditions (such as different weather, lighting, or camera quality). These degradation conditions include, but are not limited to, rainy, foggy, snowy, nighttime, or strong backlighting environments. These environmental factors can cause the loss of visual features, increased noise, or changes in the manifestation of anomalous behavior patterns, making the already sparse anomalous samples even scarcer in the new data domain. This cross-domain sparsity greatly increases the generalization difficulty of anomaly detection models and reduces the accuracy of video anomaly detection.

[0031] In view of this, the present invention provides a training method for a video anomaly detection model, a video anomaly detection method, and an apparatus to improve the accuracy of video anomaly detection. Specifically, the training method includes: acquiring a video source domain and a video target domain, wherein the video source domain includes normal video slices with normal labels, and each video slice includes multiple video frames; training an initial video anomaly detection model using the normal video slices to obtain a prediction network, which is used to perform frame-level prediction of video frames in all video slices in the video source domain, and annotating the video frames according to the frame-level prediction results to obtain a frame-level labeled source domain; inputting randomly selected samples from the frame-level labeled source domain into a generative adversarial network to output a first intermediate domain sample, wherein the semantic information of the intermediate domain sample is the same as that of the video source domain, and the pixel information of the intermediate domain sample is the same as that of the video target domain; training the initial video anomaly detection model using the first intermediate domain sample and the frame-level labeled source domain to obtain a teacher network, and constructing a student network based on the network structure and network parameters of the teacher network; inputting a second intermediate domain sample corresponding to the video target domain and the video target domain into the student network, and inputting the first intermediate sample and the frame-level labeled source domain into the teacher network; and adjusting the network parameters of the student network based on the output results of the teacher network and the student network to obtain the video anomaly detection model.

[0032] Figure 1 The diagram illustrates a training method for a video anomaly detection model according to an embodiment of the present invention, as well as an application scenario diagram of the video anomaly detection method.

[0033] like Figure 1 As shown, application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, a server 105, and a video anomaly detection model 106. The network 104 serves as a medium for providing communication links between the first terminal device 101, the second terminal device 102, the third terminal device 103, the video anomaly detection model 106, and the server 105. The network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc. The video anomaly detection model 106 may be a model capable of performing anomaly detection on videos in fields such as traffic monitoring, public safety monitoring, or industrial production inspection.

[0034] Users can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 via the network 104 to receive or send messages, such as sending a training request for training the video anomaly detection model 106, sending a video to be detected, receiving the training results of the video anomaly detection model 106, or receiving the detection results of the video anomaly detection model 106 for the video to be detected. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0035] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0036] Server 105 can be a server providing various services, such as a backend management server supporting information sent by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. For example, it can train the video anomaly detection model 106 according to the training request sent by the terminal device, and call the video anomaly detection model 106 to detect the video to be detected according to the video to be detected sent by the terminal device, and obtain the detection result. The backend management server can analyze and process the received user requests and other data, and feed back the processing results (such as training results, detection results, web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0037] It should be noted that the training method and video anomaly detection method of the video anomaly detection model provided in the embodiments of the present invention can generally be executed by server 105. Correspondingly, the training device and video anomaly detection device of the video anomaly detection model provided in the embodiments of the present invention can generally be located in server 105. The training method and video anomaly detection method of the video anomaly detection model provided in the embodiments of the present invention can also be executed by a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105. Correspondingly, the training device and video anomaly detection device of the video anomaly detection model provided in the embodiments of the present invention can also be located in a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105.

[0038] It should be understood that Figure 1The number of terminal devices, networks, servers, and video anomaly detection models shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, servers, and video anomaly detection models can be included.

[0039] The following will be based on Figure 1 The described scene, through Figures 2-4 The training method of the video anomaly detection model and the video anomaly detection method of the embodiments of the present invention are described in detail.

[0040] Figure 2 A flowchart illustrating a training method for a video anomaly detection model according to an embodiment of the present invention is shown.

[0041] like Figure 2 As shown, the training method of the video anomaly detection model in this embodiment includes operations S210 to S250.

[0042] In operation S210, the video source domain and the video target domain are obtained. The video source domain includes normal video slices with normal labels, and each video slice includes multiple video frames.

[0043] In operation S220, the initial video anomaly detection model is trained using normal video slices to obtain a prediction network. The prediction network is used to perform frame-level prediction of video frames of all video slices in the video source domain, and the video frames are labeled according to the frame-level prediction results to obtain the frame-level labeled source domain.

[0044] In operation S230, samples randomly selected from the frame-level labeled source domain are input into the adversarial generative network, and the first intermediate domain sample is output. The semantic information of the intermediate domain sample is the same as that of the video source domain, and the pixel information of the intermediate domain sample is the same as that of the video target domain.

[0045] In operation S240, the initial video anomaly detection model is trained using the first intermediate domain samples and frame-level labeled source domains to obtain the teacher network, and the student network is constructed based on the network structure and network parameters of the teacher network.

[0046] In operation S250, the second intermediate domain sample corresponding to the video target domain and the video target domain are input into the student network, and the first intermediate domain sample and the frame-level labeled source domain are input into the teacher network. Based on the output results of the teacher network and the student network, the network parameters of the student network are adjusted to obtain the video anomaly detection model.

[0047] In some embodiments, the video source domain may include videos from non-sparse scenes, such as videos taken on a sunny day. A non-sparse scene can refer to a scene without image degradation, i.e., a scene where anomaly detection can be performed without image restoration processing. The video source domain may include multiple video slices, each with a video-level label (e.g., normal or anomalous). Each video slice may include multiple video frames. The video source domain may include multiple normal video slices with normal labels and multiple anomalous video slices with anomalous labels.

[0048] The target domain for video can include videos taken in sparse scenes, such as those filmed in rainy, snowy, or foggy conditions. Sparse scenes can refer to scenes with image degradation, meaning that image restoration processing is required before anomaly detection.

[0049] Normal video slices can be used to train an initial anomaly detection model to obtain a prediction network. This trained network can predict video frames in all video slices in the video source domain based on the features of normal video slices, thereby labeling normal video frames and abnormal video frames, and obtaining a source domain with frame-level labeling.

[0050] Triple samples can be randomly selected from the frame-level labeled source domain, such as the first normal video frame, the anchor video frame, and the abnormal video frame. The anchor video frame can be a second normal video frame that is different from the first normal video frame.

[0051] Generative adversarial networks can generate intermediate domain samples based on the source and target domains. For example, intermediate domain samples may have the same semantics as the source domain but different environments, and different image details as the target domain but the same pixels. These intermediate domain samples can serve as a bridge between the source and target domains, alleviating the difficulty of aligning the source and target domains. These intermediate domain samples can also reduce the gap between the teacher network and the student network, making it easier for the exponent to be distilled in different networks.

[0052] By inputting randomly selected triplet samples from the frame-level labeled source domain into the adversarial generative network, corresponding intermediate domain triplet samples can be output. The randomly selected triplet samples and the intermediate domain triplet samples are then concatenated. The resulting concatenated data can be used to train an initial video anomaly detection model. After reaching a predetermined number of iterations or when the loss function constructed based on the concatenated data converges, a video anomaly detection model suitable as a teacher network is obtained. By replicating the model parameters and architecture of the teacher network, a student network can be derived.

[0053] When training the student network and the teacher network, the video target domain and the second intermediate domain samples (obtained by inputting the triplet samples randomly selected from the video target domain into the adversarial generative network) can be concatenated and input into the student network. The first intermediate domain samples and the frame-level labeled source domain can be input into the teacher network. When the loss function constructed based on the output results of the teacher network and the student network converges, the network parameters of the student network can be stopped, thus obtaining a video anomaly detection method that can be used as the student network for cross-domain detection.

[0054] According to an embodiment of the present invention, a video source domain and a video target domain are obtained; an initial video anomaly detection model is trained using normal video slices in the video source domain to obtain a prediction network, which performs frame-level annotation on video frames in the video source domain to obtain a frame-level annotated source domain; samples randomly selected from the frame-level annotated source domain are input into a generative adversarial network to output first intermediate domain samples; the initial video anomaly detection model is trained using the first intermediate domain samples and the frame-level annotated source domain to obtain a teacher network; second intermediate domain samples and the video target domain are input into a student network, and the first intermediate domain samples and the frame-level annotated source domain are input into the teacher network; based on the output results of the teacher network and the student network, the teacher network is trained to obtain a video anomaly detection model. Since the prediction network is first trained using normal video frames during training, the prediction network can annotate video frames in the video source domain to obtain a frame-level annotated source domain, thereby providing refined frame-level annotation for the training of the video anomaly detection model. This enables the video anomaly detection model to accurately distinguish the boundaries between normal and abnormal frames during training, thus improving the model training accuracy. On the other hand, since the teacher network and student network are trained by combining intermediate domain samples, the intermediate domain samples retain the semantics of the source domain and are visually identical to the pixels of the target domain. By using the intermediate domain as a bridge between the source domain and the target domain, the domain difference between the source domain and the target domain can be reduced. This overcomes the problem that video anomaly detection models are unable to accurately detect anomalies when performing cross-domain detection due to the large difference between the source domain and the target domain, and improves the accuracy of video anomaly detection.

[0055] Figure 3 A training architecture diagram of a video anomaly detection model according to an embodiment of the present invention is shown.

[0056] like Figure 3 As shown, the architecture may include a video source domain 301, a video target domain 302, a first intermediate domain sample 303, a second intermediate domain sample 304, an encoder 305, a temporal feature extraction module 306, and a decoder 307. The video source domain 301 can be a video under normal weather conditions, such as a video under sunny conditions, while the video target domain 302 can be a video under extreme weather conditions, such as a video under rainy, snowy, or foggy conditions.

[0057] The architecture of the initial video anomaly detection model may include an encoder 305, a temporal feature extraction module 306, and a decoder 307. The encoder can be a common 2D feature extraction network, such as a Visual GeometryGroup (VGG), Residual Network (ResNet), or VisionTransformer (ViT). The temporal network can be a Long Short-Term Memory (LSTM) network or a Convolutional Long Short-Term Memory (ConvLSTM) network. The decoder is a reconstruction network upsampled from the encoder. The encoder processes the input normal video frames, abnormal video frames, and anchor video frames into feature vectors, which are then input into the temporal feature extraction module. The temporal feature extraction module extracts the temporal information between multiple vectors, obtaining the first temporal feature of the normal video frame, the second temporal feature of the abnormal video frame, and the third temporal feature of the anchor video frame. The decoder is used to predict a first predicted frame corresponding to a first normal video frame based on a first temporal feature, a second predicted frame corresponding to an abnormal video frame based on a second video frame, and a third predicted frame corresponding to an anchor video frame based on a third temporal feature.

[0058] The following are Figure 3 The training architecture of the video anomaly detection model shown is described.

[0059] In training a video anomaly detection model, the video source domain in a given non-sparse scene is denoted as... ,in Indicates the first Individual video slices, Indicates the first Video-level annotation (normal or abnormal) of each source video slice. This indicates the number of video slices in the source domain. The target domain for video in sparse scenes is... , Indicates the first Video slices of the target domain, This indicates the number of video slices in the target domain.

[0060] Based on the normal video slices in the aforementioned video source domain, where each normal video slice includes T+1 frames, and these T+1 video frames include the first T video frames as prediction input data and the T+1th actual video frame as a label, the initial video anomaly detection model can be trained as follows to obtain the prediction network: input the first T video frames into the initial video anomaly detection model to output the T+1th predicted video frame; iteratively adjust the model parameters of the initial video anomaly detection model based on the prediction loss between the T+1th predicted video frame and the T+1th actual video frame until the preset training stopping condition is met to obtain the prediction network.

[0061] In some embodiments, a For continuous sampling from the video source domain, including A normal video slice of a frame, t≥0. From this normal video slice, there are two types of normal video frames: one type is used as anchor samples, and the other type is used as normal samples. Inputting the anchor samples and normal samples from the first T frames into the initial video detection model yields the prediction results for the anchor samples and normal samples of the (T+1)th frame. and The training objective of the initial video detection model here is to predict the results of anchor samples and normal samples. and The results closely approximate the true values ​​of anchor samples and normal samples. and Therefore, a predictive loss is constructed. Measure the difference between the predicted value and the actual value, predicting the loss. As shown in formula (1).

[0062] (1)

[0063] in, This can be the prediction result of the anchor sample in frame T+1. This can be the actual result of the anchor sample in frame T+1. The prediction result is for the normal sample in frame T+1. This represents the actual result for a normal sample in frame T+1. This can represent the L1 norm. It should be noted that the embodiments of this invention do not impose an L1 norm on the prediction loss. This constraint, This represents the prediction result for the abnormal video frame in frame T+1. This represents the actual result for the abnormal video frame at frame T+1. Because in any given scene, normal and abnormal video frames share the same background information, forcing abnormal data to fit this constraint with a larger prediction error would have adverse effects.

[0064] Based on this prediction loss, the model parameters of the initial video detection model can be iteratively adjusted until the prediction loss converges, thus obtaining the prediction network.

[0065] In some embodiments, the above operations result in a prediction network that can annotate video frames in all video slices within the video source domain, thus obtaining a frame-level annotated source domain. Specifically, this process may include the following operations: For the T+1 video frames of any video slice: input the first T video frames into the prediction network and output the T+1th predicted video frame; determine the peak signal-to-noise ratio (PSNR) between the T+1th predicted video frame and the T+1th actual video frame based on the pixel-wise error between the T+1th predicted video frame and the T+1th actual video frame; normalize the PNR to obtain a standard PNR; and annotate the T+1th actual video frame according to the standard PNR.

[0066] In some embodiments, prediction networks can be used to... The video frames are labeled, for example, the standard peak signal-to-noise ratio is labeled, to obtain A higher value indicates a greater likelihood that the frame represents a normal event. For video-level annotation, since the entire video is annotated as 1 or 0, the following values ​​are used: Indicates the first The frame's normalized PSNR. In the specific implementation, it is set based on the output of the prediction network trained using only normal video slices. .

[0067] Specifically, the aforementioned prediction network is used to predict future frames. For example, the first T video frames are input into the prediction network, and the (T+1)th predicted video frame is output. The peak signal-to-noise ratio (PSNR) is used to evaluate the quality of the predicted video frame. For example, for the t-th frame, a higher PSNR value indicates that the t-th frame is likely to be a normal frame.

[0068] Based on the pixel-wise error between the (T+1)th predicted video frame and the (T+1)th actual video frame, the peak signal-to-noise ratio between the (T+1)th predicted video frame and the (T+1)th actual video frame can be determined. This process can be shown in Equations (2) and (3).

[0069] (2)

[0070] (3)

[0071] in, Indicates the first One actual video frame, Indicates the first One predicted video frame, This represents the mean square error determined on a pixel-by-pixel basis. This indicates the line number of a single video frame. This indicates the column number of a video sequence. Represents the first frame in a video frame. OK, Represents the first frame in a video frame. List, It represents the peak signal-to-noise ratio between the (T+1)th predicted video frame and the (T+1)th actual video frame.

[0072] Since different scenarios may cause differences in the magnitude of PSNR, the peak signal-to-noise ratio can be normalized to obtain the standard peak signal-to-noise ratio. This process can be shown in formula (4).

[0073] (4)

[0074] in, This represents the standard peak signal-to-noise ratio, used for labeling the (T+1)th actual video frame. This indicates the maximum peak signal-to-noise ratio. This indicates the maximum peak signal-to-noise ratio. This represents the set of PSNR values ​​from all frames captured by the same single-view surveillance camera. Different cameras have significantly different viewpoints and angles, resulting in noticeable background variations.

[0075] When the peak signal-to-noise ratio (PSNR) of video frames is annotated, the frame-level annotation source domain can be constructed as follows: compare the standard PNR with a predetermined threshold; construct a set of normal video frames based on video frames with PNR greater than or equal to the predetermined threshold; construct a set of abnormal video frames based on video frames with PNR less than the predetermined threshold; and construct the frame-level annotation source domain based on the set of normal video frames and the set of abnormal video frames.

[0076] In some embodiments, the standard peak signal-to-noise ratio (PSNR) obtained from the above operations can be used as a confidence level for frame-level annotation of video frames. Specifically, the PSNR of each video frame in a video slice can be compared with a predetermined threshold (e.g., 0.5). Video frames with a normalized PSNR greater than or equal to 0.5 are designated as normal video frames, constructing a set of normal video frames. Video frames with a PSNR less than 0.5 are designated as abnormal video frames, constructing a set of abnormal video frames. A frame-level annotation source domain can be constructed based on the set of normal and abnormal video frames.

[0077] According to an embodiment of the present invention, a prediction network is obtained by training an initial prediction model using normal video slices. This prediction network can perform frame-level annotation on all video slices in the video source domain. This not only eliminates the need for manual frame-by-frame annotation, reducing data preparation costs, but also provides refined frame-level annotation information for subsequent anomaly detection model training.

[0078] In some embodiments, before training an initial video detection model based on frame-level labeled source and target domains, directly extracting domain-adaptive detection knowledge is challenging due to the significant inter-domain differences between the source and target domains. To alleviate this problem, embodiments of the present invention can obtain an intermediate domain through a Cycle-Consistent Generative Adversarial Network (CycleGAN), which consists of two symmetric mapping functions. and And two related discriminators and composition. As the first generator, As the first discriminator, As a second generator and As a second discriminator.

[0079] CycleGAN can be trained as follows: A predetermined number of video frames are selected from the source and target domains to obtain source domain video frames and target domain video frames, respectively. A first generator is used to convert the source domain video frames into pseudo-target domain video frames, and a first loss function is constructed based on the discrimination result of the first discriminator on the pseudo-target domain video frames. A second generator is used to convert the target domain video frames into pseudo-source domain video frames, and a second loss function is constructed based on the discrimination result of the second discriminator on the pseudo-source domain video frames. A cycle consistency loss function is constructed based on the restoration results of restoring the pseudo-target domain video frames to source domain video frames and the restoration results of restoring the pseudo-source domain video frames to target domain video frames. The initial adversarial generative network is trained using the first loss function, the second loss function, and the cycle consistency loss function to obtain the adversarial generative network.

[0080] Specifically, it can be done from the video source domain. and video target domain 5000 unpaired video frames were selected from each of the video source domains, with those from the source domain being used as source domain video frames. Video frames from the target domain .

[0081] First loss function It can be as shown in formula (5).

[0082] (5)

[0083] in, The first discriminator representing the target domain exist Log-likelihood expectation on the distribution Indicates that it follows a distribution as Similarly, for samples in the target domain. The first generator representing the target domain exist Log-likelihood expectation on the distribution Indicates that it follows a distribution as The source domain samples.

[0084] Based on formula (5), for the transformation function and its discriminator A second loss function can also be obtained. As shown in formula (6).

[0085] (6)

[0086] in, The second discriminator represents the source domain. exist Log-likelihood expectation on the distribution Indicates that it follows a distribution as Similarly, for the source domain samples... The second generator representing the source domain exist Log-likelihood expectation on the distribution Indicates that it follows a distribution as The target domain sample.

[0087] To further address the mode collapse problem and preserve the structural semantic information of the source image, a cycle consistency loss function can be constructed based on the restoration results of pseudo-target domain video frames to source domain video frames, and the restoration results of pseudo-source domain video frames to target domain video frames. As shown in formula (7).

[0088] (7)

[0089] in, , representing source domain samples sequentially through the mapping function and The output and source domain samples The mathematical expectation of the L1 norm, Similarly, to represent the L1 norm, Represents target domain samples sequentially through the mapping function and The output and target domain samples The mathematical expectation of the L1 norm.

[0090] Based on the first loss function, the second loss function, and the cycle consistency loss function, a loss function for adversarial generative networks can be constructed. As shown in formula (8).

[0091] (8)

[0092] in, This is used to control the relative importance of two objectives, one of which is the sum of the first and second loss functions, and the other is the cycle consistency loss function. The function shown in Equation (8) can be solved by adversarial training using Equation (9).

[0093] (9)

[0094] in, They represent the optimal mapping functions respectively. and . After adversarial optimization, the parameters that make the objective optimal are selected.

[0095] In the loss function If convergence is achieved, training the initial adversarial generative network can be stopped, resulting in an adversarial generative network used to output the intermediate domain.

[0096] After the above adversarial generative network is trained, it can be used... and Obtain video frames from both the source and target domains, respectively. The target domain-like video frames can be used as an intermediate domain. The source, intermediate, and target domains are denoted as follows: , and Because the intermediate domain preserves the structural semantic content of the source image, the source labels are transferred: . and They have the same semantic content, only differing in appearance. While S and T differ in image detail, they are similar in pixel-level distribution. Therefore, this intermediate domain acts as a bridge between the source and target domains, alleviating the difficulty of directly aligning significant domain differences between S and T. Furthermore, since the feature representations extracted from source and target domain data differ significantly across different networks, the intermediate domain can also reduce the gap between student and teacher networks, making knowledge distillation easier across different networks.

[0097] According to an embodiment of the present invention, bidirectional image transformation between the source domain and the target domain is realized by a first generator and a second generator respectively, and an adversarial generative network is trained by combining a cycle consistency loss function. This allows the generated intermediate domain samples to retain both the semantic content of the source domain and the visual style of the target domain without the need for paired video frames, thereby providing an effective bridge support for cross-domain anomaly detection.

[0098] In some embodiments, the CycleGAN constructed based on the above operations, and the constructed frame-level labeled source domain, can randomly select samples from the frame-level labeled source domain and input them into the CycleGAN to output the first intermediate domain samples.

[0099] Specifically, the samples randomly selected from the frame-level labeled source domain constructed above can include a first normal video frame and a second normal video frame selected from the set of normal video frames, and an abnormal video frame selected from the set of abnormal video frames, with the second normal video frame serving as the anchor video frame. The process of inputting the randomly selected samples from the frame-level labeled source domain into the adversarial generative network and outputting the first intermediate domain samples can include the following operations: inputting the first normal video frame, the abnormal video frame, and the anchor video frame into the adversarial generative network respectively, and outputting the intermediate domain normal video frame, the intermediate domain abnormal video frame, and the intermediate domain anchor video frame.

[0100] In some embodiments, based on the frame-level annotation source domain constructed above, a triplet sample can be randomly selected from the frame-level annotation source domain. As a triplet sample in boundary learning, Indicates an abnormal video frame. This represents the first normal video frame. This indicates the anchor point video frame.

[0101] The triple When input into a generative adversarial network, it can output a first intermediate domain sample, which may include intermediate domain normal video frames, intermediate domain abnormal video frames, and intermediate domain anchor point video frames.

[0102] In some embodiments, based on frame-level labeled source domain and first intermediate domain samples, an initial video anomaly detection model can be trained to obtain a teacher network as follows: First normal video frames, abnormal video frames, and anchor point video frames are randomly selected from the frame-level labeled source domain and concatenated with normal video frames, abnormal video frames, and anchor point video frames from the intermediate domain to obtain an input sequence. This input sequence can be sequentially input into the encoder and the temporal feature extraction module. The temporal feature extraction module processes the input sequence to obtain a first temporal feature corresponding to the first normal video frame, a second temporal feature corresponding to the abnormal video frame, and a third temporal feature corresponding to the anchor point video frame. A ternary feature loss function is constructed based on the first distance between the first and third temporal features, the second distance between the second and third temporal features, the standard peak signal-to-noise ratio (SNR) of the first normal video frame, the standard peak SNR of the abnormal video frame, and boundary parameters. The boundary parameters are used to adjust the first and second distances. A first objective function is constructed based on the ternary feature loss function and the prediction loss. The model parameters of the initial video anomaly detection model are adjusted according to the first objective function until the first objective function converges, resulting in a video anomaly detection model serving as the teacher network.

[0103] In some embodiments, by processing the input sequence using a temporal feature extraction module, the first temporal feature corresponding to the first normal video frame can be obtained. Second temporal features corresponding to abnormal video frames and the third temporal features corresponding to the anchor video frames. .

[0104] like Figure 3 As shown, in the boundary learning module, the goal is to reduce the distance between normal video frames by utilizing a small number of abnormal video frames and a large number of normal video frames, that is, to reduce the distance between normal video frames and anchor video frames, while increasing the distance between normal video frames and abnormal video frames. Since a large number of normal video frames are assumed to be provided, forcing normal video frames to maintain a small distance will make the distribution of normal video frames in the feature space more compact, thereby facilitating the separation of normal video frames from abnormal video frames. This further helps to detect unknown abnormal events under the open set setting. Based on this concept, the embodiments of the present invention construct a ternary feature loss function in the boundary learning method, as shown in formula (10).

[0105] (10)

[0106] in, Represents the ternary feature loss function. The standard peak signal-to-noise ratio of a normal video frame. The standard peak signal-to-noise ratio of the abnormal video frame. This represents the first distance between the first time series feature and the third time series feature. This represents the second distance between the second and third time series features. Denotes the square of the L2 norm. These are boundary parameters used to control the first and second distances. In formula (10), The larger and ( The smaller, or The smaller and ( The larger the value of the triplet loss, the greater its weight should be (i.e., ...). The larger (the larger).

[0107] According to an embodiment of the present invention, by using a temporal feature extraction module to extract temporal information between video frames, and constructing a ternary feature loss function based on the feature distance between the anchor video frame and normal video frames and abnormal video frames, and training a video anomaly detection model with the goal of increasing the distance between normal video frames and abnormal video frames and decreasing the distance between normal video frames, the video anomaly detection model can accurately determine the boundary between normal and abnormal, thereby improving the accuracy of anomaly detection.

[0108] Based on the prediction loss shown in formula (1) and the ternary feature loss function shown in formula (10), the first objective function can be constructed. As shown in formula (11).

[0109] (11)

[0110] in The weights are used to balance the ternary feature loss function. In the first objective function... Once convergence is achieved, the iterative adjustment of the initial video anomaly detection model can be stopped, resulting in a model that can serve as a teacher network. A video anomaly detection model. This is achieved through analysis of teacher networks. By copying the network structure and network parameters, the student network can be obtained. .

[0111] For training the student network, samples randomly selected from the video target domain can be input into CycleGAN to output second intermediate domain samples. Then, the second intermediate domain samples and the video target domain are input into the student network, and the first intermediate domain samples and the frame-level labeled source domain are input into the teacher network. Based on the output results of the teacher network and the student network, the network parameters of the student network are adjusted to obtain the video anomaly detection model.

[0112] In some embodiments, during the construction of the intermediate domain (Depend on Based on the generated target domain style image, to alleviate the source domain... With the target domain To address the significant differences in feature distribution between student and teacher networks and facilitate knowledge transfer between them, this invention proposes a feature-level cross-domain distillation module. The core idea of ​​this module is to utilize intermediate domains as anchor points for feature alignment, progressively distilling domain-invariant feature representations across different networks. Specifically, the student network is... Teacher network for During the training phase of the student network, the source domain samples extracted from the frame-level labeled source domain are: The intermediate domain samples drawn from the first intermediate domain sample or the second intermediate domain sample are: And the target domain samples extracted from the video target domain are Each of the multi-level features is extracted separately, as shown in formula (12).

[0113] (12)

[0114] in, Representing intermediate domain samples In the student network eigenvalues ​​of the layer Represents target domain samples In the teacher network eigenvalues ​​of the layer The first part represents the encoder or decoder section of the teacher network or student network. Each feature layer, Indicates from input to the... Mapping of layer features.

[0115] To make the feature representation of the student network in the target domain more closely resemble its distribution in the intermediate domain, while preserving the semantic information of the video source domain, a distillation loss function can be constructed for the teacher network and the student network based on their respective outputs. As shown in formula (13).

[0116] (13)

[0117] in, Indicates the first Weights of the layer distribution distance metric Indicates the first Layer distribution distance metric The distance measurement function can be determined using the mean squared error (MSE), for example... It can be determined by formula (14).

[0118] (14)

[0119] in, This indicates gradient stopping, meaning that the features of the teacher network are only used as targets and do not participate in backpropagation to prevent training instability. For the first The dimension normalization coefficient of the layer feature. This represents the square of the L2 norm.

[0120] Based on the distillation loss function shown in formula (13) and the first objective function shown in formula (11), a second objective function can be constructed. As shown in formula (15).

[0121] (15)

[0122] in, The first objective function , This is the distillation loss function.

[0123] If the second objective function converges, the iterative adjustment of the student network parameters can be stopped. The network parameters can be integrated using the Exponential Moving Average (EMA) method to integrate the teacher network. The network parameters are updated as shown in formula (16).

[0124] (16)

[0125] in, The attenuation coefficient is used to control the update process of the student network parameters. The student network parameters will not be completely replaced by the teacher network parameters, but will gradually approach the teacher network parameters (smooth transition).

[0126] According to embodiments of the present invention, in order to simultaneously address the sparsity issues of cross-domain detection and anomalous behavior, embodiments of the present invention propose a framework that combines boundary embedding prediction learning and cross-domain learning, aiming to improve the performance of cross-domain video anomaly detection.

[0127] The proposed embodiment of this invention combines boundary embedding prediction learning, which refers to a multi-layered boundary embedding prediction learning video anomaly detection model constructed by an encoder, a temporal feature extraction module, and a decoder. To address the sparsity of anomalous behavior, this model employs a multi-layered boundary embedding prediction learning strategy. Based on temporal features extracted by the temporal feature extraction module, and considering the distance between these features, the model is trained to reduce the distance between normal video frames while simultaneously increasing the distance between normal and anomalous video frames. This allows the model to learn a more compact distribution of normal data and effectively expand the discrimination interval between normal and anomalous events, thereby alleviating the detection difficulties caused by the sparsity of anomalous behavior.

[0128] To address the challenge of cross-domain detection, this invention provides a cross-domain learning framework comprising a student network and a teacher network. Since the differences in cross-domain detection primarily stem from the data level—for example, significant distributional discrepancies between training input data and actual detection input data—this invention, in its embodiments, utilizes a generative adversarial network (GAN) to generate intermediate domain samples between the video source domain and the video target domain. Then, cross-domain distillation loss is used to simultaneously train both the student and teacher networks, systematically fusing pixel-level (e.g., the generated intermediate domain samples are similar to pixels in the target domain) and feature-level (e.g., cross-domain feature distillation loss) domain adaptation methods. This gradually aligns the data distribution differences between different data domains, thereby mitigating the domain adaptation deficiency that may result from directly performing domain-invariant feature learning. Overall, the training method for the video anomaly detection model proposed in this invention enables the model to simultaneously handle anomaly detection in both sparse and cross-domain scenarios, improving the model's detection performance and generalization ability.

[0129] Figure 4 A flowchart of a video anomaly detection method according to an embodiment of the present invention is shown.

[0130] like Figure 4 As shown, the video anomaly detection method of this embodiment may include operations S410 to S420.

[0131] Using the S410, acquire the video to be tested.

[0132] In operation S420, the video to be detected is input into the video anomaly detection model, and the video anomaly detection result is output. The video anomaly detection model is trained using the training method described above.

[0133] In some embodiments, the video to be detected may be a video segment (which may be a video stream, a video file, or several consecutive frames) acquired from a camera device, storage device, or other device in an actual monitoring scenario and collected to determine whether it contains abnormal behavior.

[0134] A video anomaly detection model can include an encoder, a temporal feature extraction module, and a decoder. The encoder processes video frames in the video to be detected into feature vectors, which are then input into the temporal feature extraction module to extract the temporal information of the feature vectors. This temporal information is then input into the decoder to predict video frames. Based on the mean square error between the predicted and actual video frames, the standard peak signal-to-noise ratio (PSNR) of each video frame in the video to be detected is output as the video anomaly detection result.

[0135] According to embodiments of the present invention, by using the above-described video anomaly detection model to detect anomalies in the video to be detected, only the video to be detected needs to be input into the trained model, and the standard peak signal-to-noise ratio of each video frame can be output, eliminating the need for manual frame-by-frame review and improving the efficiency and automation of anomaly detection. Furthermore, because the video anomaly detection model overcomes the domain differences between the source and target domains during training through intermediate domain samples and feature-level knowledge distillation, it maintains high detection accuracy and robustness even under image degradation conditions such as sparse scenes and low lighting, further improving video detection efficiency.

[0136] Based on the training method of the video anomaly detection model described above, this invention also provides a training device for the video anomaly detection model. The following will combine... Figure 5 The device is described in detail.

[0137] Figure 5 A structural block diagram of a training apparatus for a video anomaly detection model according to an embodiment of the present invention is shown.

[0138] like Figure 5 As shown, the training device 500 for the video anomaly detection model in this embodiment includes a first acquisition module 510, a first training module 520, a first input module 530, a second training module 540, and a third training module 550.

[0139] The first acquisition module 510 is used to acquire the video source domain and the video target domain. The video source domain includes normal video slices with normal labels, and the video slices include multiple video frames.

[0140] The first training module 520 is used to train the initial video anomaly detection model using normal video slices to obtain a prediction network. The prediction network is used to perform frame-level prediction of video frames of all video slices in the video source domain, and to label the video frames according to the frame-level prediction results to obtain the frame-level labeled source domain.

[0141] The first input module 530 is used to input samples randomly selected from the frame-level labeled source domain into the adversarial generative network and output the first intermediate domain sample. The semantic information of the intermediate domain sample is the same as that of the video source domain, and the pixel information of the intermediate domain sample is the same as that of the video target domain.

[0142] The second training module 540 is used to train the initial video anomaly detection model using the first intermediate domain samples and frame-level labeled source domains to obtain the teacher network, and to construct the student network based on the network structure and network parameters of the teacher network.

[0143] The third training module 550 is used to input the second intermediate domain samples corresponding to the video target domain and the video target domain into the student network, and input the first intermediate domain samples and the frame-level labeled source domain into the teacher network. Based on the output results of the teacher network and the student network, the network parameters of the student network are adjusted to obtain the video anomaly detection model.

[0144] According to an embodiment of the present invention, a normal video slice includes T+1 video frames, the T+1 video frames include the first T video frames as prediction input data, and the T+1th actual video frame as a label, where T is an integer greater than 0, and the first training module 520 may include a first input unit and a first training unit.

[0145] The first input unit is used to input the first T video frames into the initial video anomaly detection model and output the (T+1)th predicted video frame.

[0146] The first training unit is used to iteratively adjust the model parameters of the initial video anomaly detection model based on the prediction loss between the (T+1)th predicted video frame and the (T+1)th actual video frame, until the preset training stopping condition is met, thus obtaining the prediction network.

[0147] According to an embodiment of the present invention, the training device 500 for the video anomaly detection model may further include a second input module, a determination module, a processing module, and a labeling module.

[0148] For any video slice, the T+1 video frames:

[0149] The second input module is used to input the first T video frames into the prediction network and output the t-th predicted video frame.

[0150] The determination module is used to determine the peak signal-to-noise ratio between the (T+1)th predicted video frame and the (T+1)th actual video frame based on the pixel-by-pixel error between the (T+1)th predicted video frame and the (T+1)th actual video frame.

[0151] The processing module is used to normalize the peak signal-to-noise ratio to obtain a standard peak signal-to-noise ratio.

[0152] The annotation module is used to annotate the (T+1)th actual video frame according to the standard peak signal-to-noise ratio.

[0153] According to an embodiment of the present invention, the training device 500 for the video anomaly detection model may further include a comparison module, a first construction module, a second construction module, and a third construction module.

[0154] The comparison module is used to compare the standard peak signal-to-noise ratio with a predetermined threshold.

[0155] The first construction module is used to construct a set of normal video frames based on video frames with a standard peak signal-to-noise ratio greater than or equal to a predetermined threshold.

[0156] The second construction module is used to construct a set of abnormal video frames based on video frames with a standard peak signal-to-noise ratio less than a predetermined threshold.

[0157] The third construction module is used to construct the frame-level annotation source domain based on the set of normal video frames and the set of abnormal video frames.

[0158] According to an embodiment of the present invention, the samples randomly selected from the frame-level annotation source domain include a first normal video frame and a second normal video frame selected from the set of normal video frames, and an abnormal video frame selected from the set of abnormal video frames. The second normal video frame serves as an anchor video frame. The first input module 530 can be used to input the first normal video frame, the abnormal video frame, and the anchor video frame into the generative adversarial network, respectively, and output intermediate domain normal video frames, intermediate domain abnormal video frames, and intermediate domain anchor video frames.

[0159] According to an embodiment of the present invention, the initial video anomaly detection model includes a temporal feature extraction module for extracting temporal information between multiple video frames, and the second training module 540 may include a splicing unit, a second input unit, a first construction unit, a second construction unit, and a second training unit.

[0160] The splicing unit is used to splice the intermediate domain normal video frames, intermediate domain abnormal video frames, and intermediate domain anchor point video frames with the first normal video frame, abnormal video frame, and anchor point video frame to obtain the input sequence.

[0161] The second input unit is used to process the input sequence using the temporal feature extraction module to obtain the first temporal feature corresponding to the first normal video frame, the second temporal feature corresponding to the abnormal video frame, and the third temporal feature corresponding to the anchor video frame.

[0162] The first construction unit is used to construct a ternary feature loss function based on the first distance between the first temporal feature and the third temporal feature, the second distance between the second temporal feature and the third temporal feature, the standard peak signal-to-noise ratio of the first normal video frame, the standard peak signal-to-noise ratio of the abnormal video frame, and boundary parameters. The boundary parameters are used to adjust the first distance and the second distance.

[0163] The second building unit is used to construct the first objective function based on the ternary feature loss function and the prediction loss.

[0164] The second training unit is used to adjust the model parameters of the initial video anomaly detection model according to the first objective function until the first objective function converges, thus obtaining the video anomaly detection model as the teacher network.

[0165] According to an embodiment of the present invention, the third training module 550 may include a third construction unit, a fourth construction unit, and a fifth training unit.

[0166] The third building unit is used to construct the distillation loss function of the teacher network and the student network based on the output results of the teacher network and the student network.

[0167] The fourth building unit is used to construct the second objective function based on the distillation loss function and the first objective function.

[0168] The fifth training unit is used to update the model parameters of the student network using the second objective function until the preset training stopping condition is met, thus obtaining the video anomaly detection model.

[0169] According to an embodiment of the present invention, the adversarial generative network includes a first generator, a first discriminator, a second generator, and a second discriminator, and the training device 500 for the video anomaly detection model may further include a selection unit, a first conversion unit, a second conversion unit, a fifth construction unit, and a fourth training unit.

[0170] The selection unit is used to select a preset number of video frames from the video source domain and the video target domain respectively, to obtain source domain video frames and target domain video frames.

[0171] The first conversion unit is used to convert source domain video frames into pseudo target domain video frames using a first generator, and to construct a first loss function based on the discrimination result of the first discriminator on the pseudo target domain video.

[0172] The second conversion unit is used to convert the target domain video frame into a pseudo source domain video frame using the second generator, and to construct a second loss function based on the discrimination result of the second discriminator on the pseudo source domain video.

[0173] The fifth construction unit is used to construct a cycle consistency loss function based on the restoration results of restoring pseudo-target domain video frames to source domain video frames and the restoration results of restoring pseudo-source domain video frames to target domain video frames.

[0174] The fourth training unit is used to train the initial adversarial generative network based on the first loss function, the second loss function, and the cycle consistency loss function to obtain the adversarial generative network.

[0175] Based on the above-described video anomaly detection method, this invention also provides a video anomaly detection device. The following will be combined with... Figure 6 The device is described in detail.

[0176] Figure 6 A structural block diagram of a video anomaly detection device according to an embodiment of the present invention is shown.

[0177] like Figure 6 As shown, the video anomaly detection device 600 of this embodiment includes a second acquisition module 610 and a third input module 620.

[0178] The second acquisition module 610 is used to acquire the video to be detected.

[0179] The third input module 620 is used to input the video to be detected into the video anomaly detection model and output the video anomaly detection result. The video anomaly detection model is trained using the training method of the video anomaly detection model described above.

[0180] According to embodiments of the present invention, any plurality of modules among the first acquisition module 510, first training module 520, first input module 530, second training module 540, third training module 550, second acquisition module 610, and third input module 620 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules can be combined with at least part of the functionality of other modules and implemented in one module. According to embodiments of the present invention, at least one of the first acquisition module 510, first training module 520, first input module 530, second training module 540, third training module 550, second acquisition module 610, and third input module 620 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable means of integrating or packaging the circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the first acquisition module 510, the first training module 520, the first input module 530, the second training module 540, the third training module 550, the second acquisition module 610, and the third input module 620 may be implemented at least partially as a computer program module, which can perform corresponding functions when the computer program module is run.

[0181] Figure 7 A block diagram of an electronic device suitable for implementing a video anomaly detection model and a video anomaly detection method according to an embodiment of the present invention is shown.

[0182] like Figure 7 As shown, an electronic device 700 according to an embodiment of the present invention includes a processor 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage portion 708 into a random access memory (RAM) 703. The processor 701 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 701 may also include onboard memory for caching purposes. The processor 701 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0183] Random access memory (RAM) 703 stores various programs and data required for the operation of electronic device 700. Processor 701, read-only memory (ROM) 702, and RAM 703 are interconnected via bus 704. Processor 701 performs various operations of the method flow according to embodiments of the present invention by executing programs in read-only memory (ROM) 702 and / or RAM 703. It should be noted that the programs may also be stored in one or more memories other than read-only memory (ROM) 702 and RAM 703. Processor 701 may also perform various operations of the method flow according to embodiments of the present invention by executing programs stored in said one or more memories.

[0184] According to an embodiment of the present invention, the electronic device 700 may further include an input / output (I / O) interface 705, which is also connected to a bus 704. The electronic device 700 may also include one or more of the following components connected to the input / output (I / O) interface 705: an input section 706 including a keyboard, mouse, etc.; an output section 707 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the input / output (I / O) interface 705 as needed. A removable medium 711, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 710 as needed so that computer programs read from it can be installed into the storage section 708 as needed.

[0185] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of the present invention.

[0186] According to embodiments of the present invention, a computer-readable storage medium may be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of the present invention, a computer-readable storage medium may include the read-only memory (ROM) 702 and / or random access memory (RAM) 703 described above and / or one or more memories other than read-only memory (ROM) 702 and random access memory (RAM) 703.

[0187] Embodiments of the present invention also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to cause the computer system to implement the methods provided in the embodiments of the present invention.

[0188] When the computer program is executed by the processor 701, it performs the functions defined in the system / apparatus of this invention. According to embodiments of the invention, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0189] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 709, and / or installed from a removable medium 711. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0190] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 709, and / or installed from the removable medium 711. When the computer program is executed by the processor 701, it performs the functions defined in the system of this embodiment of the invention. According to embodiments of the invention, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0191] According to embodiments of the present invention, program code for executing the computer programs provided in the embodiments of the present invention can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0192] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0193] Those skilled in the art will understand that the features described in the various embodiments of the present invention can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in the present invention. In particular, the features described in the various embodiments of the present invention can be combined and / or combined in various ways without departing from the spirit and teachings of the present invention. All such combinations and / or combinations fall within the scope of the present invention.

[0194] The embodiments of the present invention have been described above. However, these embodiments are merely illustrative and not intended to limit the scope of the invention. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of the invention, and all such substitutions and modifications should fall within the scope of the invention.

Claims

1. A training method for a video anomaly detection model, characterized in that, The method includes: Obtain the video source domain and the video target domain, wherein the video source domain includes normal video slices with normal labels, and the video slices include multiple video frames; Using the normal video slices, the initial video anomaly detection model is trained to obtain a prediction network. The prediction network is used to perform frame-level prediction of video frames of all video slices in the video source domain, and to annotate the video frames according to the frame-level prediction results to obtain the frame-level annotated source domain. Samples randomly selected from the frame-level labeled source domain are input into the adversarial generative network, and a first intermediate domain sample is output. The semantic information of the intermediate domain sample is the same as that of the video source domain, and the pixel information of the intermediate domain sample is the same as that of the video target domain. The initial video anomaly detection model is trained using the first intermediate domain samples and the frame-level labeled source domain to obtain a teacher network, and a student network is constructed based on the network structure and network parameters of the teacher network. The second intermediate domain sample corresponding to the video target domain and the video target domain are input into the student network, and the first intermediate domain sample and the frame-level labeled source domain are input into the teacher network. Based on the output results of the teacher network and the student network, the network parameters of the student network are adjusted to obtain the video anomaly detection model.

2. The method according to claim 1, characterized in that, The normal video slice includes T+1 video frames, which include the first T video frames as prediction input data and the T+1th actual video frame as a label, where T is an integer greater than 0. The step of training the initial video anomaly detection model using the normal video slices to obtain the prediction network includes: The first T video frames are input into the initial video anomaly detection model, and the T+1th predicted video frame is output. Based on the prediction loss between the (T+1)th predicted video frame and the (T+1)th actual video frame, the model parameters of the initial video anomaly detection model are iteratively adjusted until a preset training stopping condition is reached, thus obtaining the prediction network.

3. The method according to claim 2, characterized in that, The prediction network annotates video frames of all video slices in the video source domain in the following manner: For any video slice, the T+1 video frames: The first T video frames are input into the prediction network, and the (T+1)th predicted video frame is output. Based on the pixel-wise error between the (T+1)th predicted video frame and the (T+1)th actual video frame, the peak signal-to-noise ratio between the (T+1)th predicted video frame and the (T+1)th actual video frame is determined; The peak signal-to-noise ratio is normalized to obtain the standard peak signal-to-noise ratio; The (T+1)th actual video frame is labeled according to the standard peak signal-to-noise ratio.

4. The method according to claim 3, characterized in that, The frame-level annotation source domain is obtained in the following way: The standard peak signal-to-noise ratio is compared with a predetermined threshold. A set of normal video frames is constructed based on video frames with a standard peak signal-to-noise ratio greater than or equal to the predetermined threshold. A set of abnormal video frames is constructed based on video frames with a standard peak signal-to-noise ratio less than the predetermined threshold. The frame-level annotation source domain is constructed based on the set of normal video frames and the set of abnormal video frames.

5. The method according to claim 4, characterized in that, The samples randomly selected from the frame-level annotation source domain include a first normal video frame and a second normal video frame selected from the set of normal video frames, and an abnormal video frame selected from the set of abnormal video frames, wherein the second normal video frame is used as an anchor video frame. The step of inputting randomly selected samples from the frame-level labeled source domain into the adversarial generative network and outputting first intermediate domain samples includes: The first normal video frame, the abnormal video frame, and the anchor point video frame are respectively input into the adversarial generative network, and the intermediate domain normal video frame, the intermediate domain abnormal video frame, and the intermediate domain anchor point video frame are output.

6. The method according to claim 5, characterized in that, The initial video anomaly detection model includes a temporal feature extraction module, used to extract temporal information between multiple video frames; The step of training the initial video anomaly detection model using the first intermediate domain samples and the frame-level labeled source domain to obtain the teacher network includes: The intermediate domain normal video frame, intermediate domain abnormal video frame, and intermediate domain anchor point video frame are concatenated with the first normal video frame, abnormal video frame, and anchor point video frame to obtain the input sequence. The input sequence is processed using the temporal feature extraction module to obtain the first temporal feature corresponding to the first normal video frame, the second temporal feature corresponding to the abnormal video frame, and the third temporal feature corresponding to the anchor video frame. Based on the first distance between the first temporal feature and the third temporal feature, the second distance between the second temporal feature and the third temporal feature, the standard peak signal-to-noise ratio of the first normal video frame, the standard peak signal-to-noise ratio of the abnormal video frame, and boundary parameters, a ternary feature loss function is constructed, wherein the boundary parameters are used to adjust the first distance and the second distance; Based on the ternary feature loss function and the prediction loss, a first objective function is constructed; The model parameters of the initial video anomaly detection model are adjusted according to the first objective function until the first objective function converges, thus obtaining the video anomaly detection model as a teacher network.

7. The method according to claim 6, characterized in that, The process involves inputting a second intermediate domain sample corresponding to the video target domain and the video target domain into the student network, inputting the first intermediate domain sample and the frame-level labeled source domain into the teacher network, and adjusting the network parameters of the student network based on the outputs of the teacher network and the student network to obtain a video anomaly detection model, including: Based on the output results of the teacher network and the student network, a distillation loss function is constructed for the teacher network and the student network. Based on the distillation loss function and the first objective function, a second objective function is constructed; The model parameters of the student network are updated using the second objective function until a preset training stopping condition is met, thus obtaining the video anomaly detection model.

8. The method according to claim 1, characterized in that, The adversarial generative network includes a first generator, a first discriminator, a second generator, and a second discriminator; The adversarial generative network is trained in the following manner: A preset number of video frames are selected from the video source domain and the video target domain respectively to obtain source domain video frames and target domain video frames; The source domain video frame is converted into a pseudo target domain video frame using the first generator, and a first loss function is constructed based on the discrimination result of the pseudo target domain video by the first discriminator. The target domain video frame is converted into a pseudo source domain video frame using the second generator, and a second loss function is constructed based on the discrimination result of the pseudo source domain video by the second discriminator. Based on the restoration results of restoring the pseudo target domain video frame to the source domain video frame and the restoration results of restoring the pseudo source domain video frame to the target domain video frame, a cycle consistency loss function is constructed. The initial adversarial generative network is trained based on the first loss function, the second loss function, and the cycle consistency loss function to obtain the adversarial generative network.

9. A video anomaly detection method, characterized in that, The method includes: Obtain the video to be tested; The video to be detected is input into the video anomaly detection model, and the video anomaly detection result is output. The video anomaly detection model is trained using the training method described in any one of claims 1 to 8.

10. An electronic device, comprising: One or more processors; Memory, used to store one or more computer programs. The characteristic feature is that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 9.

11. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 9.

12. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 9.

Images