A multi-task driven wide-area video anomaly event detection method, system, device and medium

Abstract

The application provides a multi-task driven wide-area video abnormal event detection method, system, device and medium, relates to the technical field of video monitoring, and extracts spatial features of a current nth video frame through a spatial feature extraction module of a video abnormal event detection model, and stores the obtained spatial features in a cache; then extracts spatial features of adjacent continuous M-1 video frames from the cache, combines the spatial features with the spatial features of the current video frame in chronological order, and obtains a spatial feature sequence of an nth video segment; performs time convolution operation on the spatial feature sequence through a time feature extraction module, and obtains space-time features of the nth video segment; and performs abnormal event detection in parallel through multiple abnormal event detection modules by using the space-time features, and obtains abnormal event detection results. Not only can the calculation cost be reduced, but also the detection efficiency and accuracy of multi-type abnormal event detection can be improved.

Description

Technical Field

[0001] This application relates to the field of video surveillance technology, and in particular to a multi-task driven wide-area video anomaly event detection method, system, device and medium. Background Technology

[0002] With the rapid development of video surveillance technology, real-time video analysis based on high-position cameras (such as iron towers and communication towers) is playing an increasingly important role in public safety, environmental protection, and traffic management. Especially in mid-to-high-position real-time video applications, it is necessary to simultaneously monitor multiple abnormal events within a wide field of view, such as straw burning, illegal fishing, and landslides. These events are characterized by their suddenness, diversity, and complex background interference, posing significant technical challenges to abnormal event detection.

[0003] Most current mainstream anomaly detection methods are designed for single tasks, meaning a single model is responsible for detecting only one type of anomaly. However, in practical applications, there are numerous types of anomalies, and a single model cannot detect all event types simultaneously, making it difficult to meet the needs of multi-task concurrent detection. Furthermore, designing and deploying a separate model for each type of anomaly not only incurs huge storage overhead but also generates redundant calculations during inference, severely impacting the real-time performance and resource utilization efficiency of the monitoring system.

[0004] On the other hand, tower video surveillance scenarios place extremely high demands on processing speed. Because cameras are typically mounted at high points, they offer wide fields of view and extensive coverage, resulting in massive amounts of video stream data, and require real-time response to unexpected events. Traditional multi-model serial inference architectures are ill-suited to meet frame-level real-time requirements.

[0005] Therefore, there is an urgent need for a new multi-task-driven method for detecting anomalies in wide-area video. Summary of the Invention

[0006] In view of the above problems, embodiments of this application provide a multi-task driven wide-area video anomaly event detection method, system, device and medium to overcome the above problems or at least partially solve the above problems.

[0007] In a first aspect, this application provides a multi-task-driven wide-area video anomaly event detection method, the method comprising: Receive the nth video frame captured by the camera for the target scene, where n is an integer greater than 1; The spatial features of the nth video frame are extracted by the spatial feature extraction module of the video abnormal event detection model, and the spatial features of the nth video frame are obtained and stored in the cache. Extract the spatial features of M-1 consecutive video frames adjacent to the nth video frame from the cache, and combine them with the spatial features of the nth video frame in chronological order to obtain the spatial feature sequence of the nth video segment. The nth video segment includes M consecutive video frames and takes the nth video frame as the last video frame, where M is an integer greater than 1. The temporal feature extraction module of the video anomaly detection model performs temporal convolution on the spatial feature sequence of the nth video segment to obtain the spatiotemporal features of the nth video segment. By utilizing the spatiotemporal features of the nth video segment, anomaly detection is performed in parallel by multiple anomaly detection modules of the video anomaly detection model to obtain the anomaly detection result of the nth video segment. The anomaly detection result of the nth video segment includes at least: whether there is an anomaly in the nth video segment, and, if there is an anomaly in the nth video segment, the type of the anomaly in the nth video segment.

[0008] Optionally, the total number of multiple anomaly detection modules is T; utilizing the spatiotemporal features of the nth video segment, anomaly detection is performed in parallel by multiple anomaly detection modules of the video anomaly detection model to obtain the anomaly detection result of the nth video segment, including: Taking t sequentially from 1 to T, the task features of each of the T anomaly detection modules for the nth video segment are obtained according to the following steps: The spatiotemporal features of the nth video segment are input into the gating network of the tth anomaly detection module to obtain the tth feature selection weight; The spatiotemporal features of the nth video segment are weighted using the tth feature selection weight to obtain the tth task feature, which is used as the task feature of the tth anomaly detection module for the nth video segment; The T-anomaly detection module performs anomaly detection in parallel using the task features of each module for the nth video segment, and obtains the anomaly detection result for the nth video segment.

[0009] Optionally, the multiple abnormal event detection modules include: a straw burning event detection module, an illegal fishing event detection module, and a landslide event detection module; the method further includes: The spatial feature extraction module and the temporal feature extraction module are used to process the video clips of straw burning events to obtain the spatiotemporal features of the video clips of straw burning events. Input the spatiotemporal features of video clips from straw burning incident samples into T abnormal event detection modules to be trained; By performing abnormal event detection in parallel using the first detection module to be trained, the second detection module to be trained, and the third detection module to be trained, the first detection result, the second detection result, and the third detection result are obtained. Based on the first detection result and the tags carried by the video clips of the straw burning event sample, the value of the first loss function is determined; Based on the second detection result and the tags carried by the video clips of the straw burning event sample, the value of the second loss function is determined; The value of the third loss function is determined based on the third detection result and the tags carried by the video clips of the straw burning event samples. The total loss function value is determined based on the first loss function value, the second loss function value, and the third loss function value. Based on the total loss function value, the model parameters of the spatial feature extraction module, the temporal feature extraction module, the first detection module, the second detection module, and the third detection module to be trained are updated. Using video clips of illegal fishing incidents and landslide incidents, the above steps are repeated until training is complete, thus obtaining the video abnormal event detection model.

[0010] Optionally, the method further includes: Receive the (n+1)th video frame captured by the camera for the target scene; The spatial features of the (n+1)th video frame are obtained by extracting spatial features from the spatial feature extraction module of the video abnormal event detection model. Remove the spatial features of the video frame with the earliest cache time from the cache, and store the spatial features of the (n+1)th video frame into the cache; Extract the spatial features of M-1 consecutive video frames adjacent to the (n+1)th video frame from the cache, and combine them with the spatial features of the (n+1)th video frame in chronological order to obtain the spatial feature sequence of the (n+1)th video segment. The (n+1)th video segment includes M consecutive video frames and the (n+1)th video frame is the last video frame. The temporal feature extraction module of the video abnormal event detection model performs temporal convolution operation on the spatial feature sequence of the (n+1)th video segment to obtain the spatiotemporal features of the (n+1)th video segment. By utilizing the spatiotemporal features of the (n+1)th video segment, anomaly detection is performed in parallel by multiple anomaly detection modules of the video anomaly detection model to obtain the anomaly detection result of the (n+1)th video segment. The anomaly detection result of the (n+1)th video segment includes at least: whether there is an anomaly in the (n+1)th video segment, and, if there is an anomaly in the (n+1)th video segment, the type of the anomaly in the (n+1)th video segment.

[0011] Optionally, utilizing the spatiotemporal features of the (n+1)th video segment, anomaly detection is performed in parallel by multiple anomaly detection modules of the video anomaly detection model to obtain the anomaly detection result of the (n+1)th video segment, including: Taking t sequentially from 1 to T, the task features of each of the T anomaly detection modules for the (n+1)th video segment are obtained according to the following steps: The spatiotemporal features of the (n+1)th video segment are input into the gating network of the t-th anomaly detection module to update the t-th feature selection weights; the updated t-th feature selection weights are used to weight the spatiotemporal features of the (n+1)th video segment to obtain the updated t-th task features, which are used as the task features of the t-th anomaly detection module for the (n+1)th video segment. The T-anomaly detection module performs anomaly detection in parallel using the task features of each segment for the (n+1)th video segment, and obtains the anomaly detection result for the (n+1)th video segment.

[0012] A second aspect of this application provides a multi-task-driven wide-area video anomaly event detection system, the system comprising: The video frame receiving module is used to receive the nth video frame captured by the camera for the target scene, where n is an integer greater than 1; The first determining module is used to extract spatial features from the nth video frame through the spatial feature extraction module of the video abnormal event detection model, obtain the spatial features of the nth video frame and store them in the cache; The feature combination module is used to extract the spatial features of M-1 consecutive video frames adjacent to the nth video frame from the cache, and combine them with the spatial features of the nth video frame in chronological order to obtain the spatial feature sequence of the nth video segment. The nth video segment includes M consecutive video frames and takes the nth video frame as the last video frame, where M is an integer greater than 1. The second determining module is used to perform temporal convolution operation on the spatial feature sequence of the nth video segment through the temporal feature extraction module of the video abnormal event detection model to obtain the spatiotemporal features of the nth video segment; The third determining module is used to utilize the spatiotemporal features of the nth video segment, and to perform anomaly detection in parallel through multiple anomaly detection modules of the video anomaly detection model to obtain the anomaly detection result of the nth video segment. The anomaly detection result of the nth video segment includes at least: whether there is an anomaly in the nth video segment, and, if there is an anomaly in the nth video segment, the type of the anomaly in the nth video segment.

[0013] Optionally, the total number of multiple anomaly detection modules is T; utilizing the spatiotemporal features of the nth video segment, anomaly detection is performed in parallel by multiple anomaly detection modules of the video anomaly detection model to obtain the anomaly detection result of the nth video segment. The third determining module includes: The first input submodule is used to input the spatiotemporal features of the nth video segment into the gating network of the tth abnormal event detection module to obtain the tth feature selection weight; The weighted submodule is used to select weights based on the t-th feature to weight the spatiotemporal features of the n-th video segment, thereby obtaining the t-th task feature, which is used as the task feature of the t-th abnormal event detection module for the n-th video segment. The first determining submodule is used to perform anomaly detection in parallel using each of the T anomaly detection modules for the task characteristics of the nth video segment, and obtain the anomaly detection result of the nth video segment.

[0014] Optionally, the multiple abnormal event detection modules include: a straw burning event detection module, an illegal fishing event detection module, and a landslide event detection module; the system also includes: The video segment processing submodule is used to process the straw burning event sample video segments through the spatial feature extraction module and the temporal feature extraction module to obtain the spatiotemporal features of the straw burning event sample video segments. The second input submodule is used to input the spatiotemporal features of the straw burning event sample video clips into T abnormal event detection modules to be trained; The second determination submodule is used to perform abnormal event detection in parallel through the first detection module to be trained, the second detection module to be trained, and the third detection module to be trained, to obtain the first detection result, the second detection result, and the third detection result. The first loss function determination submodule is used to determine the value of the first loss function based on the first detection result and the tags carried by the straw burning event sample video clips; The second loss function determination submodule is used to determine the value of the second loss function based on the second detection result and the tags carried by the straw burning event sample video clips; The third loss function determination submodule is used to determine the value of the third loss function based on the third detection result and the tags carried by the straw burning event sample video clips; The third determining submodule is used to determine the total loss function value based on the first loss function value, the second loss function value, and the third loss function value; The parameter update submodule is used to update the model parameters of the spatial feature extraction module, the temporal feature extraction module, the first detection module, the second detection module, and the third detection module based on the total loss function value. The fourth determination submodule is used to repeat the above steps using video clips of illegal fishing incidents and video clips of landslide incidents until training is complete, thereby obtaining the video abnormal event detection model.

[0015] A third aspect of this application provides an electronic device, including a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the multi-task-driven wide-area video anomaly event detection method as described in the first aspect of this application.

[0016] A fourth aspect of this application provides a readable storage medium storing a program or instructions that, when executed by a processor, implement the steps of the multi-task-driven wide-area video anomaly event detection method as described in the first aspect of this application.

[0017] The beneficial effects of this application are: This application provides a multi-task-driven wide-area video anomaly event detection method, the method comprising: receiving an nth video frame captured by a camera for a target scene, where n is an integer greater than 1; extracting spatial features from the nth video frame using a spatial feature extraction module of a video anomaly event detection model, obtaining the spatial features of the nth video frame and storing them in a cache; extracting the spatial features of M-1 consecutive video frames adjacent to the nth video frame from the cache, and combining them with the spatial features of the nth video frame in chronological order to obtain a spatial feature sequence of an nth video segment, wherein the nth video segment comprises M consecutive video frames, with the nth video frame being the last one. The video frame M is an integer greater than 1; the temporal feature extraction module of the video anomaly detection model performs temporal convolution on the spatial feature sequence of the nth video segment to obtain the spatiotemporal features of the nth video segment; using the spatiotemporal features of the nth video segment, multiple anomaly detection modules of the video anomaly detection model are used to perform anomaly detection in parallel to obtain the anomaly detection result of the nth video segment. The anomaly detection result of the nth video segment includes at least: whether there is an anomaly in the nth video segment, and, if there is an anomaly in the nth video segment, the type of the anomaly in the nth video segment.

[0018] This application extracts spatial features from the current nth video frame using a spatial feature extraction module of a video anomaly detection model, and stores the obtained spatial features in a cache. Then, it extracts the spatial features of M-1 adjacent consecutive video frames from the cache and combines them with the spatial features of the current video frame in chronological order to obtain the spatial feature sequence of the nth video segment. Next, a temporal feature extraction module performs temporal convolution on this spatial feature sequence to extract temporal change information and motion features between consecutive video frames, obtaining the spatiotemporal features of the nth video segment. Finally, using these spatiotemporal features, multiple anomaly detection modules execute anomaly detection in parallel to obtain the anomaly detection result. This application reduces redundant feature extraction of video frames and lowers computational costs by reusing the spatial features of historical video frames. Simultaneously, it can characterize the dynamic changes of anomaly events through temporal convolution and improve the detection efficiency and accuracy of multiple types of anomaly events through parallel detection by multiple anomaly detection modules. Attached Figure Description

[0019] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating the steps of a multi-task-driven wide-area video anomaly event detection method provided in an embodiment of this application. Figure 2 This is a schematic diagram of a multi-task driven wide-area video anomaly event detection system provided in an embodiment of this application; Figure 3 This is a schematic diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0021] Exemplary embodiments of this application will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of this application are shown in the drawings, it should be understood that this application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of this application and to fully convey the scope of this application to those skilled in the art.

[0022] In a first aspect, this application provides a multi-task-driven wide-area video anomaly event detection method, such as... Figure 1 As shown, the method includes: S101, Receive the nth video frame captured by the camera for the target scene, where n is an integer greater than 1.

[0023] In S101, the nth video frame captured by the camera for the target scene is received, where n is an integer greater than 1. The camera can continuously capture the video stream of the target scene according to a preset frame rate. In this application, the nth video frame can be understood as the video frame that needs to be processed in the video stream.

[0024] S102, the spatial features of the nth video frame are extracted by the spatial feature extraction module of the video abnormal event detection model, and the spatial features of the nth video frame are obtained and stored in the cache.

[0025] In S102, after receiving the nth video frame, the spatial feature extraction module in the video abnormal event detection model extracts the spatial features of the nth video frame to obtain the spatial features of the nth video frame, and stores the spatial features of the nth video frame in the cache. In this application, the spatial feature extraction module can extract the spatial features of the nth video frame based on a two-dimensional convolutional network to obtain the spatial features of the nth video frame. The spatial features are features extracted from the image content of a single video frame to characterize spatial information such as the shape, color, region location and scene structure of objects in the target scene.

[0026] S103, extract the spatial features of M-1 consecutive video frames adjacent to the nth video frame from the cache, and combine them with the spatial features of the nth video frame in chronological order to obtain the spatial feature sequence of the nth video segment. The nth video segment includes M consecutive video frames and takes the nth video frame as the last video frame, where M is an integer greater than 1.

[0027] In S103, the spatial features of M-1 consecutive video frames adjacent to the nth video frame are extracted from the cache, and the spatial features of the M-1 video frames are combined with the spatial features of the nth video frame in chronological order to obtain the spatial feature sequence of the nth video segment. The nth video segment consists of M consecutive video frames, with the nth video frame as the last video frame, and M is an integer greater than 1.

[0028] S104, the temporal feature extraction module of the video abnormal event detection model performs temporal convolution operation on the spatial feature sequence of the nth video segment to obtain the spatiotemporal features of the nth video segment.

[0029] In step S104, after obtaining the spatial feature sequence of the nth video segment, this spatial feature sequence is input into the temporal feature extraction module of the video anomaly detection model. The temporal feature extraction module performs temporal convolution operations on the spatial features corresponding to each of the M consecutive video frames along the temporal dimension of the video frames to extract the temporal change information and motion features between adjacent video frames.

[0030] S105, utilizing the spatiotemporal features of the nth video segment, anomaly detection is performed in parallel by multiple anomaly detection modules of the video anomaly detection model to obtain the anomaly detection result of the nth video segment. The anomaly detection result of the nth video segment includes at least: whether there is an anomaly in the nth video segment, and, if there is an anomaly in the nth video segment, the type of the anomaly in the nth video segment. In step S105, after obtaining the spatiotemporal features of the nth video segment, these features are input into multiple anomaly detection modules of the video anomaly detection model. These multiple anomaly detection modules can each correspond to different types of anomalies and perform detection in parallel based on the same spatiotemporal features of the nth video segment. This step eliminates the need to repeatedly extract video features for different anomalies and to sequentially call multiple independent detection models, thereby reducing redundant computations and lowering computational costs. After each anomaly detection module completes its detection, the anomaly detection result for the nth video segment is obtained. This result includes at least whether an anomaly exists in the nth video segment, and if so, the type of anomaly it belongs to.

[0031] In one embodiment, the total number of multiple anomaly detection modules is T; utilizing the spatiotemporal features of the nth video segment, anomaly detection is performed in parallel by multiple anomaly detection modules of the video anomaly detection model to obtain the anomaly detection result of the nth video segment, including: Taking t sequentially from 1 to T, the task features of each of the T anomaly detection modules for the nth video segment are obtained according to the following steps: The spatiotemporal features of the nth video segment are input into the gating network of the tth anomaly detection module to obtain the tth feature selection weight; The spatiotemporal features of the nth video segment are weighted using the tth feature selection weight to obtain the tth task feature, which is used as the task feature of the tth anomaly detection module for the nth video segment; The T-anomaly detection module performs anomaly detection in parallel using the task features of each module for the nth video segment, and obtains the anomaly detection result for the nth video segment.

[0032] In this embodiment, the video anomaly event detection model may include T detection modules, where T is an integer greater than 1. Different detection modules may correspond to different types of anomaly events, such as straw burning events, illegal fishing events, and landslide events. Since different types of anomaly events appear differently in video footage, even if each detection module performs detection based on the spatiotemporal features of the same nth video segment, the specific features that each detection module needs to focus on during detection may differ.

[0033] Specifically, t is sequentially selected from 1 to T, and the spatiotemporal features of the nth video segment are input into the gating network of the t-th detection module to obtain the t-th feature selection weight. The t-th feature selection weight can be used to characterize the attention paid by the t-th detection module to different feature parts in the spatiotemporal features of the nth video segment when performing anomaly detection.

[0034] Furthermore, the spatiotemporal features of the nth video segment are weighted using the t-th feature selection weights to obtain the t-th task features, which are then used as the input features for the t-th detection module for the nth video segment. In this embodiment, the t-th detection module does not directly use the original spatiotemporal features for detection, but rather uses a gating network to determine task features that are more suitable for the current detection task based on the same spatiotemporal features.

[0035] Based on this embodiment, the task features of each of the T detection modules for the nth video segment can be obtained. Then, the T detection modules perform anomaly detection in parallel using their respective task features to obtain the anomaly detection result for the nth video segment. The anomaly detection result for the nth video segment can at least include whether an anomaly exists in the nth video segment, and if an anomaly exists, the type of the anomaly.

[0036] In this embodiment, multiple detection modules can share the spatiotemporal features of the same nth video segment, avoiding repeated feature extraction for different abnormal event types; at the same time, each detection module can obtain corresponding task features through its own gating network, enabling different detection modules to perform detection around the abnormal event types they are responsible for.

[0037] In one embodiment, the multiple abnormal event detection modules include: a straw burning event detection module, an illegal fishing event detection module, and a landslide event detection module; the method further includes: The spatial feature extraction module and the temporal feature extraction module are used to process the video clips of straw burning events to obtain the spatiotemporal features of the video clips of straw burning events. Input the spatiotemporal features of video clips from straw burning incident samples into T abnormal event detection modules to be trained; By performing abnormal event detection in parallel using the first detection module to be trained, the second detection module to be trained, and the third detection module to be trained, the first detection result, the second detection result, and the third detection result are obtained. Based on the first detection result and the tags carried by the video clips of the straw burning event sample, the value of the first loss function is determined; Based on the second detection result and the tags carried by the video clips of the straw burning event sample, the value of the second loss function is determined; The value of the third loss function is determined based on the third detection result and the tags carried by the video clips of the straw burning event samples. The total loss function value is determined based on the first loss function value, the second loss function value, and the third loss function value. Based on the total loss function value, the model parameters of the spatial feature extraction module, the temporal feature extraction module, the first detection module, the second detection module, and the third detection module to be trained are updated. Using video clips of illegal fishing incidents and landslide incidents, the above steps are repeated until training is complete, thus obtaining the video abnormal event detection model.

[0038] In this embodiment, multiple abnormal event detection modules include: a straw burning event detection module, an illegal fishing event detection module, and a landslide event detection module. The video abnormal event detection model is obtained by training a pre-constructed training model, which includes: a training spatial feature extraction module and a training temporal feature extraction module connected in series. After the training temporal feature extraction module, three training detection modules are connected in parallel: a training first detection module, a training second detection module, and a training third detection module. The training first detection module includes: a training first gating network and a training first classification network connected in series; the training second detection module includes: a training second gating network and a training second classification network connected in series; and the training third detection module includes: a training third gating network and a training third classification network connected in series.

[0039] The training samples for the video anomaly detection model are obtained by pre-annotating multiple sample video segments, each of which consists of M consecutive sample video frames. To enable the trained video anomaly detection model to adapt to the complex environment of wide-area video surveillance, the multiple sample video segments can be acquired under different haze conditions, different lighting conditions, or different degrees of occlusion.

[0040] Based on the different types of anomalous events, the sample video clips are divided into: straw burning event sample video clips, illegal fishing event sample video clips, and landslide event sample video clips. The tags carried by each sample video clip include at least two binary labels indicating whether the sample video clip contains different anomalous events.

[0041] The process of training a pre-built model to obtain a video anomaly detection model includes: The spatial feature extraction module and the temporal feature extraction module are used to process the video clips of straw burning events to obtain the spatiotemporal features of the video clips of straw burning events. The spatiotemporal features of the video clips of straw burning incident samples are input into T abnormal event detection modules to be trained. Abnormal event detection is performed in parallel by the first detection module, the second detection module, and the third detection module to be trained, and the first detection result, the second detection result, and the third detection result are obtained. Among them, the detection result output by the first detection module to be trained is the first detection result, which represents: the probability that the first detection module to be trained predicts the existence of abnormal straw burning events in the straw burning event sample video segments; and the first loss function value is obtained based on the labels carried by the straw burning event sample video segments and the first detection result.

[0042] Similarly, the detection result output by the second detection module to be trained is the second detection result, which represents the probability that the second detection module to be trained predicts the existence of an abnormal event such as illegal fishing in the straw burning event sample video clip; and the second loss function value is obtained based on the label carried by the straw burning event sample video clip and the second detection result. Similarly, the detection result output by the third detection module to be trained is the third detection result, which represents the probability that the third detection module to be trained predicts the existence of an abnormal event such as landslide in the straw burning event sample video clip; and the third loss function value is obtained based on the label carried by the straw burning event sample video clip and the third detection result.

[0043] Based on the first, second, and third loss function values, the total loss function value is determined, and the model parameters of the untrained spatial feature extraction module, the untrained temporal feature extraction module, the untrained first detection module, the untrained second detection module, and the untrained third detection module are updated. Similarly, using video clips of illegal fishing events, the first, second, and third loss function values ​​can be obtained, and the model parameters can be updated accordingly; using video clips of landslide events, the first, second, and third loss function values ​​can be obtained, and the model parameters can be updated accordingly, until training is complete, resulting in a video anomaly event detection model.

[0044] By updating the model parameters as described above, the spatial feature extraction module, the temporal feature extraction module, and the various abnormal event detection modules can be jointly trained in the same training process.

[0045] The process of updating the model parameters for the first detection module to be trained includes updating the model parameters of the first gating network and the first classification network to be trained. The trained first detection module is the straw burning event detection module. Similarly, updating the model parameters for the second detection module to be trained includes updating the model parameters of the second gating network and the second classification network to be trained. The trained second detection module is the illegal fishing event detection module. The process of updating the model parameters for the third detection module to be trained includes updating the model parameters of the third gating network and the third classification network to be trained. The trained third detection module is the landslide event detection module.

[0046] Through multiple model parameter updates, the spatial feature extraction module and the temporal feature extraction module to be trained gradually learn the spatial and temporal change information in the sample video clips. This also enables the t-th gating network and the t-th classification network to be trained to adapt to the detection task of the t-th abnormal event type.

[0047] Understandably, video clips labeled as indicating straw burning incidents are positive samples for the first detection module to be trained, but negative samples for the second and third detection modules to be trained. Similarly, video clips labeled as indicating illegal fishing incidents are positive samples for the second detection module to be trained, but negative samples for the first and third detection modules to be trained. Likewise, video clips labeled as indicating landslide incidents are positive samples for the third detection module to be trained, but negative samples for the first and second detection modules to be trained.

[0048] In one embodiment, the multi-task-driven wide-area video anomaly event detection method further includes: Receive the (n+1)th video frame captured by the camera for the target scene; The spatial features of the (n+1)th video frame are obtained by extracting spatial features from the spatial feature extraction module of the video abnormal event detection model. Remove the spatial features of the video frame with the earliest cache time from the cache, and store the spatial features of the (n+1)th video frame into the cache; Extract the spatial features of M-1 consecutive video frames adjacent to the (n+1)th video frame from the cache, and combine them with the spatial features of the (n+1)th video frame in chronological order to obtain the spatial feature sequence of the (n+1)th video segment. The (n+1)th video segment includes M consecutive video frames and the (n+1)th video frame is the last video frame. The temporal feature extraction module of the video abnormal event detection model performs temporal convolution operation on the spatial feature sequence of the (n+1)th video segment to obtain the spatiotemporal features of the (n+1)th video segment. By utilizing the spatiotemporal features of the (n+1)th video segment, anomaly detection is performed in parallel by multiple anomaly detection modules of the video anomaly detection model to obtain the anomaly detection result of the (n+1)th video segment. The anomaly detection result of the (n+1)th video segment includes at least: whether there is an anomaly in the (n+1)th video segment, and, if there is an anomaly in the (n+1)th video segment, the type of the anomaly in the (n+1)th video segment.

[0049] In this embodiment, the multi-task-driven wide-area video anomaly event detection method further includes: Receive the (n+1)th video frame captured by the camera for the target scene; and after receiving the (n+1)th video frame, extract the spatial features of the (n+1)th video frame through the spatial feature extraction module of the video abnormal event detection model to obtain the spatial features of the (n+1)th video frame.

[0050] In this embodiment, since the spatial features of previous video frames are already stored in the cache, to ensure that the spatial features stored in the cache always correspond to the video segment to be constructed, the spatial features of the earliest cached video frame can be deleted from the cache, and the spatial features of the (n+1)th video frame can be stored in the cache. That is, the cache can be updated as new video frames are stored, so that the cache retains the spatial features of consecutive video frames adjacent to the current video frame.

[0051] Furthermore, spatial features of the M-1 consecutive video frames adjacent to the (n+1)th video frame are extracted from the updated cache. These spatial features are then combined with the spatial features of the (n+1)th video frame in chronological order to obtain the spatial feature sequence of the (n+1)th video segment. The (n+1)th video segment comprises M consecutive video frames, with the (n+1)th video frame being the last video frame.

[0052] Furthermore, the temporal feature extraction module of the video anomaly detection model performs temporal convolution on the spatial feature sequence of the (n+1)th video segment to obtain the spatiotemporal features of the (n+1)th video segment. In this application, the temporal feature extraction module can perform convolution calculations on the spatial features corresponding to consecutive video frames along the temporal dimension of the video frames, thereby extracting temporal change information and motion features in the video sequence.

[0053] Furthermore, utilizing the spatiotemporal features of the (n+1)th video segment, anomaly detection is performed in parallel by multiple anomaly detection modules of the video anomaly detection model to obtain the anomaly detection result for the (n+1)th video segment. This anomaly detection result for the (n+1)th video segment includes at least whether anomaly exists in the (n+1)th video segment, and if anomaly exists, also includes the type of anomaly in the (n+1)th video segment.

[0054] Through the above embodiments, the spatial features of video frames in the cache can be updated as new video frames are continuously input into the video stream. The updated cache is then used to quickly construct a new sequence of spatial features for video segments, avoiding repeated extraction of spatial features from overlapping video frames in adjacent video segments. Simultaneously, by performing temporal convolution on the spatial feature sequence through the temporal feature extraction module, spatiotemporal features characterizing the video sequence's change process can be further obtained. Based on these spatiotemporal features, multiple anomaly detection modules can perform anomaly detection in parallel. Furthermore, by reusing the spatial features of M-1 consecutive video frames, repeated extraction of spatial features from overlapping video frames can be avoided when constructing adjacent video segments, reducing the computational load in the spatial feature extraction stage and thus improving the processing efficiency of anomaly detection in continuous video streams.

[0055] In one embodiment, utilizing the spatiotemporal features of the (n+1)th video segment, anomaly detection is performed in parallel by multiple anomaly detection modules of a video anomaly detection model to obtain the anomaly detection result of the (n+1)th video segment, including: Taking t sequentially from 1 to T, the task features of each of the T anomaly detection modules for the (n+1)th video segment are obtained according to the following steps: The spatiotemporal features of the (n+1)th video segment are input into the gating network of the t-th anomaly detection module to update the t-th feature selection weights; the updated t-th feature selection weights are used to weight the spatiotemporal features of the (n+1)th video segment to obtain the updated t-th task features, which are used as the task features of the t-th anomaly detection module for the (n+1)th video segment. The T-anomaly detection module performs anomaly detection in parallel using the task features of each segment for the (n+1)th video segment, and obtains the anomaly detection result for the (n+1)th video segment.

[0056] In this embodiment, t is taken sequentially from 1 to T. For the t-th abnormal event detection module, the spatiotemporal features of the (n+1)-th video segment are input into the gating network of the t-th abnormal event detection module to update the selection weight of the t-th feature.

[0057] Furthermore, the spatiotemporal features of the (n+1)th video segment are weighted using the updated t-th feature selection weights to obtain the updated t-th task features, and the updated t-th task features are used as the task features of the t-th abnormal event detection module for the (n+1)-th video segment.

[0058] In this embodiment, T anomaly detection modules can be obtained, each targeting the (n+1)th video segment, for their respective task features. Then, the T anomaly detection modules perform anomaly detection in parallel using their respective task features to obtain the anomaly detection result for the (n+1)th video segment. This anomaly detection result can at least include whether an anomaly exists in the (n+1)th video segment, and, if so, the type of the anomaly in the (n+1)th video segment.

[0059] Based on the same inventive concept, a second aspect of the embodiments of this application provides a multi-task-driven wide-area video anomaly event detection system, such as... Figure 2 As shown, the system includes: The video frame receiving module 501 is used to receive the nth video frame captured by the camera for the target scene, where n is an integer greater than 1; The first determining module 502 is used to extract spatial features from the nth video frame through the spatial feature extraction module of the video abnormal event detection model, obtain the spatial features of the nth video frame and store them in the cache; The feature combination module 503 is used to extract the spatial features of M-1 consecutive video frames adjacent to the nth video frame from the cache, and combine them with the spatial features of the nth video frame in chronological order to obtain the spatial feature sequence of the nth video segment. The nth video segment includes M consecutive video frames and takes the nth video frame as the last video frame, where M is an integer greater than 1. The second determining module 504 is used to perform temporal convolution operation on the spatial feature sequence of the nth video segment through the temporal feature extraction module of the video abnormal event detection model to obtain the spatiotemporal features of the nth video segment. The third determining module 505 is used to utilize the spatiotemporal features of the nth video segment, and perform anomaly detection in parallel through multiple anomaly detection modules of the video anomaly detection model to obtain the anomaly detection result of the nth video segment. The anomaly detection result of the nth video segment includes at least: whether there is an anomaly in the nth video segment, and, if there is an anomaly in the nth video segment, the type of the anomaly in the nth video segment.

[0060] Optionally, the total number of multiple anomaly detection modules is T; utilizing the spatiotemporal features of the nth video segment, anomaly detection is performed in parallel by multiple anomaly detection modules of the video anomaly detection model to obtain the anomaly detection result of the nth video segment. The third determining module 505 includes: The first input submodule is used to input the spatiotemporal features of the nth video segment into the gating network of the tth abnormal event detection module to obtain the tth feature selection weight; The weighted submodule is used to select weights based on the t-th feature to weight the spatiotemporal features of the n-th video segment, thereby obtaining the t-th task feature, which is used as the task feature of the t-th abnormal event detection module for the n-th video segment. The first determining submodule is used to perform anomaly detection in parallel using each of the T anomaly detection modules for the task characteristics of the nth video segment, and obtain the anomaly detection result of the nth video segment.

[0061] Optionally, the multiple abnormal event detection modules include: a straw burning event detection module, an illegal fishing event detection module, and a landslide event detection module; the system also includes: The video segment processing submodule is used to process the straw burning event sample video segments through the spatial feature extraction module and the temporal feature extraction module to obtain the spatiotemporal features of the straw burning event sample video segments. The second input submodule is used to input the spatiotemporal features of the straw burning event sample video clips into T abnormal event detection modules to be trained; The second determination submodule is used to perform abnormal event detection in parallel through the first detection module to be trained, the second detection module to be trained, and the third detection module to be trained, to obtain the first detection result, the second detection result, and the third detection result. The first loss function determination submodule is used to determine the value of the first loss function based on the first detection result and the tags carried by the straw burning event sample video clips; The second loss function determination submodule is used to determine the value of the second loss function based on the second detection result and the tags carried by the straw burning event sample video clips; The third loss function determination submodule is used to determine the value of the third loss function based on the third detection result and the tags carried by the straw burning event sample video clips; The third determining submodule is used to determine the total loss function value based on the first loss function value, the second loss function value, and the third loss function value; The parameter update submodule is used to update the model parameters of the spatial feature extraction module, the temporal feature extraction module, the first detection module, the second detection module, and the third detection module based on the total loss function value. The fourth determination submodule is used to repeat the above steps using video clips of illegal fishing incidents and video clips of landslide incidents until training is complete, thereby obtaining the video abnormal event detection model.

[0062] Based on the same inventive concept, a third aspect of the embodiments of this application provides a method as follows: Figure 3 The electronic device 100 shown includes a processor 120, a memory 110, and a program or instructions stored in the memory 110 and executable on the processor 120. When the program or instructions are executed by the processor 120, they implement the steps of the multi-task driven wide-area video anomaly event detection method as described in the first aspect of this application.

[0063] Based on the same inventive concept, in a fourth aspect of this application, a readable storage medium is provided, on which a program or instructions are stored, which, when executed by a processor, implement the steps of the multi-task driven wide-area video anomaly event detection method as described in the first aspect of this application.

[0064] Each embodiment in this specification focuses on the differences from other embodiments. For the same or similar parts between the embodiments, please refer to each other.

[0065] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, embodiments of this application can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of this application can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0066] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0067] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0068] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0069] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0070] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0071] The above provides a detailed description of a multi-task-driven wide-area video anomaly event detection method, system, device, and medium. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A multi-task-driven wide-area video anomaly event detection method, characterized in that, The method includes: Receive the nth video frame captured by the camera for the target scene, where n is an integer greater than 1; The spatial features of the nth video frame are extracted by the spatial feature extraction module of the video abnormal event detection model, and the spatial features of the nth video frame are obtained and stored in the cache. Extract the spatial features of M-1 consecutive video frames adjacent to the nth video frame from the cache, and combine them with the spatial features of the nth video frame in chronological order to obtain the spatial feature sequence of the nth video segment. The nth video segment includes M consecutive video frames and takes the nth video frame as the last video frame, where M is an integer greater than 1. The temporal feature extraction module of the video anomaly detection model performs temporal convolution on the spatial feature sequence of the nth video segment to obtain the spatiotemporal features of the nth video segment. By utilizing the spatiotemporal features of the nth video segment, anomaly detection is performed in parallel by multiple anomaly detection modules of the video anomaly detection model to obtain the anomaly detection result of the nth video segment. The anomaly detection result of the nth video segment includes at least: whether there is an anomaly in the nth video segment, and, if there is an anomaly in the nth video segment, the type of the anomaly in the nth video segment.

2. The multi-task-driven wide-area video anomaly event detection method according to claim 1, characterized in that, The total number of anomaly detection modules is T; utilizing the spatiotemporal features of the nth video segment, anomaly detection is performed in parallel by multiple anomaly detection modules of the video anomaly detection model to obtain the anomaly detection results of the nth video segment, including: Taking t sequentially from 1 to T, the task features of each of the T anomaly detection modules for the nth video segment are obtained according to the following steps: The spatiotemporal features of the nth video segment are input into the gating network of the tth anomaly detection module to obtain the tth feature selection weight; The spatiotemporal features of the nth video segment are weighted using the tth feature selection weight to obtain the tth task feature, which is used as the task feature of the tth anomaly detection module for the nth video segment; The T-anomaly detection module performs anomaly detection in parallel using the task features of each module for the nth video segment, and obtains the anomaly detection result for the nth video segment.

3. The multi-task-driven wide-area video anomaly event detection method according to claim 1, characterized in that, The multiple abnormal event detection modules include: a straw burning event detection module, an illegal fishing event detection module, and a landslide event detection module; the method also includes: The spatial feature extraction module and the temporal feature extraction module are used to process the video clips of straw burning events to obtain the spatiotemporal features of the video clips of straw burning events. Input the spatiotemporal features of video clips from straw burning incident samples into T abnormal event detection modules to be trained; By performing abnormal event detection in parallel using the first detection module to be trained, the second detection module to be trained, and the third detection module to be trained, the first detection result, the second detection result, and the third detection result are obtained. Based on the first detection result and the tags carried by the video clips of the straw burning event sample, the value of the first loss function is determined; Based on the second detection result and the tags carried by the video clips of the straw burning event sample, the value of the second loss function is determined; The value of the third loss function is determined based on the third detection result and the tags carried by the video clips of the straw burning event samples. The total loss function value is determined based on the first loss function value, the second loss function value, and the third loss function value. Based on the total loss function value, the model parameters of the spatial feature extraction module, the temporal feature extraction module, the first detection module, the second detection module, and the third detection module to be trained are updated. Using video clips of illegal fishing incidents and landslide incidents, the above steps are repeated until training is complete, thus obtaining the video abnormal event detection model.

4. The multi-task-driven wide-area video anomaly event detection method according to claim 1, characterized in that, The method further includes: Receive the (n+1)th video frame captured by the camera for the target scene; The spatial features of the (n+1)th video frame are obtained by extracting spatial features from the spatial feature extraction module of the video abnormal event detection model. Remove the spatial features of the video frame with the earliest cache time from the cache, and store the spatial features of the (n+1)th video frame into the cache; Extract the spatial features of M-1 consecutive video frames adjacent to the (n+1)th video frame from the cache, and combine them with the spatial features of the (n+1)th video frame in chronological order to obtain the spatial feature sequence of the (n+1)th video segment. The (n+1)th video segment includes M consecutive video frames and the (n+1)th video frame is the last video frame. The temporal feature extraction module of the video abnormal event detection model performs temporal convolution operation on the spatial feature sequence of the (n+1)th video segment to obtain the spatiotemporal features of the (n+1)th video segment. By utilizing the spatiotemporal features of the (n+1)th video segment, anomaly detection is performed in parallel by multiple anomaly detection modules of the video anomaly detection model to obtain the anomaly detection result of the (n+1)th video segment. The anomaly detection result of the (n+1)th video segment includes at least: whether there is an anomaly in the (n+1)th video segment, and, if there is an anomaly in the (n+1)th video segment, the type of the anomaly in the (n+1)th video segment.

5. The method according to claim 4, characterized in that, Utilizing the spatiotemporal features of the (n+1)th video segment, anomaly detection is performed in parallel by multiple anomaly detection modules of the video anomaly detection model to obtain the anomaly detection results for the (n+1)th video segment, including: Taking t sequentially from 1 to T, the task features of each of the T anomaly detection modules for the (n+1)th video segment are obtained according to the following steps: The spatiotemporal features of the (n+1)th video segment are input into the gating network of the t-th anomaly detection module to update the t-th feature selection weights; the updated t-th feature selection weights are used to weight the spatiotemporal features of the (n+1)th video segment to obtain the updated t-th task features, which are used as the task features of the t-th anomaly detection module for the (n+1)th video segment. The T-anomaly detection module performs anomaly detection in parallel using the task features of each segment for the (n+1)th video segment, and obtains the anomaly detection result for the (n+1)th video segment.

6. A multi-task-driven wide-area video anomaly event detection system, characterized in that, The system includes: The video frame receiving module is used to receive the nth video frame captured by the camera for the target scene, where n is an integer greater than 1; The first determining module is used to extract spatial features from the nth video frame through the spatial feature extraction module of the video abnormal event detection model, obtain the spatial features of the nth video frame and store them in the cache; The feature combination module is used to extract the spatial features of M-1 consecutive video frames adjacent to the nth video frame from the cache, and combine them with the spatial features of the nth video frame in chronological order to obtain the spatial feature sequence of the nth video segment. The nth video segment includes M consecutive video frames and takes the nth video frame as the last video frame, where M is an integer greater than 1. The second determining module is used to perform temporal convolution operation on the spatial feature sequence of the nth video segment through the temporal feature extraction module of the video abnormal event detection model to obtain the spatiotemporal features of the nth video segment. The third determining module is used to utilize the spatiotemporal features of the nth video segment, and to perform anomaly detection in parallel through multiple anomaly detection modules of the video anomaly detection model to obtain the anomaly detection result of the nth video segment. The anomaly detection result of the nth video segment includes at least: whether there is an anomaly in the nth video segment, and, if there is an anomaly in the nth video segment, the type of the anomaly in the nth video segment.

7. The multi-task-driven wide-area video anomaly event detection system according to claim 6, characterized in that, The total number of multiple anomaly detection modules is T; utilizing the spatiotemporal features of the nth video segment, anomaly detection is performed in parallel by multiple anomaly detection modules of the video anomaly detection model to obtain the anomaly detection result of the nth video segment. The third determining module includes: The first input submodule is used to input the spatiotemporal features of the nth video segment into the gating network of the tth abnormal event detection module to obtain the tth feature selection weight; The weighted submodule is used to select weights based on the t-th feature to weight the spatiotemporal features of the n-th video segment, thereby obtaining the t-th task feature, which is used as the task feature of the t-th abnormal event detection module for the n-th video segment. The first determining submodule is used to perform anomaly detection in parallel using each of the T anomaly detection modules for the task characteristics of the nth video segment, and obtain the anomaly detection result of the nth video segment.

8. The multi-task-driven wide-area video anomaly event detection system according to claim 6, characterized in that, Multiple abnormal event detection modules include: a straw burning event detection module, an illegal fishing event detection module, and a landslide event detection module; the system also includes: The video segment processing submodule is used to process the straw burning event sample video segments through the spatial feature extraction module and the temporal feature extraction module to obtain the spatiotemporal features of the straw burning event sample video segments. The second input submodule is used to input the spatiotemporal features of the straw burning event sample video clips into T abnormal event detection modules to be trained; The second determination submodule is used to perform abnormal event detection in parallel through the first detection module to be trained, the second detection module to be trained, and the third detection module to be trained, to obtain the first detection result, the second detection result, and the third detection result. The first loss function determination submodule is used to determine the value of the first loss function based on the first detection result and the tags carried by the straw burning event sample video clips; The second loss function determination submodule is used to determine the value of the second loss function based on the second detection result and the tags carried by the straw burning event sample video clips; The third loss function determination submodule is used to determine the value of the third loss function based on the third detection result and the tags carried by the straw burning event sample video clips; The third determining submodule is used to determine the total loss function value based on the first loss function value, the second loss function value, and the third loss function value; The parameter update submodule is used to update the model parameters of the spatial feature extraction module, the temporal feature extraction module, the first detection module, the second detection module, and the third detection module based on the total loss function value. The fourth determination submodule is used to repeat the above steps using video clips of illegal fishing incidents and video clips of landslide incidents until training is complete, thereby obtaining the video abnormal event detection model.

9. An electronic device, characterized in that, The method includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the multi-task driven wide-area video anomaly event detection method as described in any one of claims 1-5.

10. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the multi-task driven wide-area video anomaly event detection method as described in any one of claims 1-5.

Images