Abnormal behavior detection method, device, electronic device and storage medium

Abstract

This application belongs to the field of security detection technology and discloses a method, apparatus, electronic device, and storage medium for abnormal behavior detection. The method includes obtaining a video frame sequence of a video to be detected; and obtaining abnormal behavior detection results for the video based on the video frame sequence and a pre-trained abnormal behavior detection model. This reduces the time cost of abnormal behavior detection and improves the accuracy of abnormal behavior detection when performing abnormal behavior detection on the video frame sequence to be detected.

Description

Technical Field

[0001] The present application relates to the field of security detection technology, and in particular to a method, device, electronic device and storage medium for abnormal behavior detection. Background Art

[0002] In financial fields such as insurance, great attention is paid to the privacy and security of customer data. When labeling this data, labelers are required to do so in a strictly controlled office environment. To prevent illegal labelers from using their mobile phones to secretly photograph the data content on the computer screen, it is necessary to detect abnormal mobile phone photography behavior.

[0003] Under the existing technology, when detecting abnormal behavior of mobile phone hidden photography in a video, it is usually determined whether there is abnormal behavior of mobile phone hidden photography based on the positional relationship between the person and the mobile phone in a series of video frames.

[0004] However, in special scenarios, the accuracy of judging abnormal behavior of mobile phone secret photography based on the positional relationship between people and mobile phones is low.

[0005] Therefore, when detecting abnormal behavior, how to improve the accuracy of abnormal behavior detection is a technical problem that needs to be solved. Summary of the Invention

[0006] The purpose of this application is to provide a method, device, electronic device and storage medium for abnormal behavior detection, which can improve the accuracy of abnormal behavior detection when detecting abnormal behavior.

[0007] On the one hand, a method for abnormal behavior detection is provided, including: obtaining a video frame sequence of a video to be detected; obtaining an abnormal behavior detection result of the video based on the video frame sequence and a pre-trained abnormal behavior detection model, wherein the abnormal behavior detection model is constructed based on a three-dimensional convolutional neural network and a self-attention network.

[0008] In the above implementation process, an abnormal behavior detection model constructed by a three-dimensional convolutional neural network and a self-attention network is used to perform abnormal behavior detection on the video frame sequence of the video to be detected, and the detection results of the video to be detected are obtained, thereby improving the accuracy of abnormal behavior detection.

[0009] In one embodiment, before obtaining abnormal behavior detection results of a video based on a video frame sequence and a pre-trained abnormal behavior detection model, the method further includes:

[0010] Extracting features from the video frame sequence to obtain a plurality of first feature images corresponding to the video frame sequence;

[0011] Performing abnormality analysis on the plurality of first feature images to obtain at least one abnormal candidate region;

[0012] The plurality of first feature images and at least one abnormal candidate region are input into an abnormal behavior detection model to obtain at least one target abnormal region output by the abnormal behavior detection model.

[0013] In the above implementation process, the target abnormal area is screened out according to the first feature image and the candidate abnormal areas, which narrows the scope of abnormal behavior detection and reduces the time cost.

[0014] In one embodiment, performing abnormality analysis on a plurality of first feature images to obtain at least one abnormal candidate region includes:

[0015] Performing feature extraction on the plurality of first feature images to obtain second feature images corresponding to the plurality of first feature images;

[0016] Performing abnormality analysis on each second feature image to obtain position information and confidence information of each second feature image;

[0017] According to the position information and confidence information of each second feature image, an abnormal candidate region of the video frame sequence is obtained.

[0018] In the above implementation process, candidate abnormal regions for abnormal behavior detection can be directly obtained based on the obtained position information and confidence information, thereby improving the accuracy of abnormal behavior detection.

[0019] In one embodiment, obtaining abnormal behavior detection results for a video based on a video frame sequence and a pre-trained abnormal behavior detection model includes:

[0020] Performing linear mapping processing on at least one abnormal candidate region to obtain a key vector, a value vector, and a query vector of the at least one abnormal candidate region;

[0021] Based on the obtained key vector and query vector of the at least one abnormal candidate region, screening the at least one abnormal candidate region to obtain at least one target abnormal region;

[0022] According to the target abnormal area, the abnormal behavior detection results of the video are obtained.

[0023] In the above implementation process, candidate abnormal regions can be screened according to their key vectors, value vectors, and query vectors to obtain target abnormal regions. The attention mechanism in the self-attention network model is used to focus on video frames with abnormal behaviors, while ignoring most video frames without abnormal behaviors, thereby obtaining abnormal behavior detection results and improving the accuracy of abnormal behavior detection.

[0024] In one embodiment, based on the key vector and query vector of the at least one abnormal candidate region obtained, screening the at least one abnormal candidate region to obtain at least one target abnormal region includes:

[0025] Normalizing a key vector and a query vector of at least one abnormal candidate region to obtain a plurality of self-attention features of the at least one abnormal candidate region;

[0026] At least one abnormal candidate region is screened according to multiple self-attention features, a value vector, and a query vector of at least one abnormal candidate region to obtain at least one target abnormal region.

[0027] In the above implementation process, multiple abnormal candidate areas are screened to obtain the target abnormal area for abnormal behavior detection, thereby improving the accuracy of abnormal behavior detection.

[0028] In one embodiment, obtaining abnormal behavior detection results of a video based on a target abnormal area includes:

[0029] Obtaining self-attention features, value vectors, and key vectors of at least one target abnormal region;

[0030] The self-attention features, value vectors, and key vectors of at least one target abnormal area are input into a feedforward neural network model to obtain abnormal behavior features, wherein the abnormal behavior features are abnormal behavior detection results including location information and confidence information. The feedforward neural network model is constructed based on the feedforward neural network.

[0031] In the above implementation process, the self-attention features, value vectors and key vectors of the target abnormal area can be directly used as input parameters of the feedforward neural network model to obtain the abnormal behavior detection results output by the feedforward neural network model, thereby improving the accuracy of abnormal behavior detection.

[0032] On the one hand, a device for abnormal behavior detection is provided, including: an acquisition unit, used to obtain a video frame sequence of a video to be detected; a detection unit, used to obtain an abnormal behavior detection result of the video based on the video frame sequence and a pre-trained abnormal behavior detection model, wherein the abnormal behavior detection model is constructed based on a three-dimensional convolutional neural network and a self-attention network.

[0033] In one embodiment, the detection unit is further used to:

[0034] Extracting features from the video frame sequence to obtain a plurality of first feature images corresponding to the video frame sequence;

[0035] Performing abnormality analysis on the plurality of first feature images to obtain at least one abnormal candidate region;

[0036] The plurality of first feature images and at least one abnormal candidate region are input into an abnormal behavior detection model to obtain at least one target abnormal region output by the abnormal behavior detection model.

[0037] In one embodiment, the detection unit is used to:

[0038] Performing feature extraction on the plurality of first feature images to obtain second feature images corresponding to the plurality of first feature images;

[0039] Performing abnormality analysis on each second feature image to obtain position information and confidence information of each second feature image;

[0040] According to the position information and confidence information of each second feature image, an abnormal candidate region of the video frame sequence is obtained.

[0041] In one embodiment, the detection unit is used to:

[0042] Performing linear mapping processing on at least one abnormal candidate region to obtain a key vector, a value vector, and a query vector of the at least one abnormal candidate region;

[0043] Based on the obtained key vector and query vector of the at least one abnormal candidate region, screening the at least one abnormal candidate region to obtain at least one target abnormal region;

[0044] According to the target abnormal area, the abnormal behavior detection results of the video are obtained.

[0045] In one embodiment, the detection unit is used to:

[0046] Normalizing a key vector and a query vector of at least one abnormal candidate region to obtain a plurality of self-attention features of the at least one abnormal candidate region;

[0047] At least one abnormal candidate region is screened according to multiple self-attention features, a value vector, and a query vector of at least one abnormal candidate region to obtain at least one target abnormal region.

[0048] In one embodiment, the detection unit is used to:

[0049] Obtaining self-attention features, value vectors, and key vectors of at least one target abnormal region;

[0050] The self-attention features, value vectors, and key vectors of at least one target abnormal area are input into a feedforward neural network model to obtain abnormal behavior features, wherein the abnormal behavior features are abnormal behavior detection results including location information and confidence information. The feedforward neural network model is constructed based on the feedforward neural network.

[0051] On the one hand, an electronic device is provided, including a processor and a memory, wherein the memory stores computer-readable instructions. When the computer-readable instructions are executed by the processor, the steps of the method provided in any of the various optional implementations of abnormal behavior detection described above are executed.

[0052] On the one hand, a readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method provided in any of the various optional implementations of abnormal behavior detection described above are executed.

[0053] In one aspect, a computer program product is provided. When the computer program product is run on a computer, the computer is caused to execute the steps of the method provided in any of the various optional implementations of abnormal behavior detection described above.

[0054] In an embodiment of the present application, an abnormal behavior detection model constructed by a three-dimensional convolutional neural network and a self-attention network is used to extract features from a video frame sequence of a video to be detected, and a target abnormal area is screened out based on a series of features, thereby narrowing the scope of abnormal behavior detection and reducing the time cost. In addition, the abnormal behavior detection results of the video to be detected are obtained based on the self-attention features, value vectors and key vectors of the screened target abnormal area, thereby improving the accuracy of abnormal behavior detection.

[0055] Other features and advantages of the present application will be described in the following description, and in part will become apparent from the description, or will be understood by practicing the present application. The purposes and other advantages of the present application can be realized and obtained by the structures particularly pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0057] Figure 1 A schematic diagram of the architecture of an abnormal behavior detection system provided in an embodiment of the present application;

[0058] Figure 2 A flowchart of an implementation method for abnormal behavior detection provided in an embodiment of the present application;

[0059] Figure 3 A schematic diagram of video frame sequence feature extraction provided in an embodiment of the present application;

[0060] Figure 4 A schematic diagram of extracting abnormal candidate regions provided in an embodiment of the present application;

[0061] Figure 5 A schematic diagram of abnormal behavior detection provided in an embodiment of the present application;

[0062] Figure 6 A detailed implementation flow chart of a method for detecting abnormal behavior provided in an embodiment of the present application;

[0063] Figure 7 A schematic diagram of the structure of an abnormal behavior detection device provided in an embodiment of the present application;

[0064] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0065] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work fall within the scope of protection of the present application.

[0066] It should be noted that similar reference numerals and letters represent similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and should not be understood as indicating or implying relative importance.

[0067] First, some of the terms involved in the embodiments of the present application are explained to facilitate understanding by those skilled in the art.

[0068] Terminal device: can be a mobile terminal, fixed terminal or portable terminal, such as a mobile phone, station, unit, device, multimedia computer, multimedia tablet, Internet node, communicator, desktop computer, laptop computer, notebook computer, netbook computer, tablet computer, personal communication system device, personal navigation device, personal digital assistant, audio / video player, digital camera / camcorder, positioning device, television receiver, radio broadcast receiver, e-book device, gaming device or any combination thereof, including accessories and peripherals of these devices or any combination thereof. It is also foreseeable that the terminal device can support any type of user interface (such as wearable device), etc.

[0069] Server: It can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, and big data and artificial intelligence platforms.

[0070] Hyper Text Transfer Protocol (HTTP): is a simple request-response protocol that specifies what messages a client may send to a server and what responses it may receive.

[0071] Reference box (Anchor): It is a manually defined rectangular box of various sizes and aspect ratios, and is used as a candidate box for detection.

[0072] Risk Priority Number: It is the product of the frequency, severity and detection level of an event. It is called the risk coefficient or risk sequence number. The larger the value, the more serious the potential problem.

[0073] Region of Interest Pooling (RoI Pooling): is an operation widely used in convolutional neural network object detection tasks.

[0074] Downsampling: It is to sample a sample value sequence once every few sample values, so that the new sequence obtained is the downsampling of the original sequence.

[0075] In order to improve the accuracy of abnormal behavior detection when detecting abnormal behavior, embodiments of the present application provide a method, apparatus, electronic device, and storage medium for abnormal behavior detection.

[0076] See Figure 1, which is a schematic diagram of the architecture of an abnormal behavior detection system according to an embodiment of the present application, wherein the abnormal behavior detection system includes a terminal device and a detection device.

[0077] Terminal device: used to provide a video frame sequence of the video to be detected;

[0078] Detection equipment: used to obtain the video frame sequence of the video to be detected, and based on the video frame sequence and a pre-trained abnormal behavior detection model, perform abnormal behavior detection on the video frame sequence to obtain the abnormal behavior detection result of the video.

[0079] In the embodiment of the present application, the execution subject can be Figure 1 The detection device in the abnormal behavior detection system shown in the figure can be an electronic device such as a terminal device or a server in actual application, and is not limited here.

[0080] See Figure 2 The figure is a flowchart of an abnormal behavior detection method provided by the embodiment of the present application, combined with Figure 1 The specific implementation process of the method is as follows:

[0081] Step 200: Obtain a video frame sequence of a video to be detected.

[0082] Specifically, the detection device obtains a video frame sequence of the video to be detected through a predetermined transmission protocol.

[0083] Optionally, the predetermined transmission protocol may be the HTTP transmission protocol or other transmission protocols. In actual applications, the transmission protocol may be set according to actual application scenarios and is not limited here.

[0084] It should be noted that the length of the video frame sequence of the video to be detected can be any length.

[0085] In one implementation, when the length of the video to be detected is short, the detection device obtains each frame sequence of the video to be detected through the HTTP transmission protocol.

[0086] In one implementation, when the length of the video to be detected is long, the detection device periodically obtains the video frame sequence of the video to be detected through the HTTP transmission protocol.

[0087] In this way, a sequence of video frames of a video of arbitrary length can be obtained.

[0088] Step 201: Obtain abnormal behavior detection results of the video based on the video frame sequence and a pre-trained abnormal behavior detection model.

[0089] It should be noted that the abnormal behavior detection model is built based on a three-dimensional convolutional neural network and a self-attention network.

[0090] Specifically, before executing step 201, the following steps may be further executed:

[0091] Step 1: Extract features from a video frame sequence to obtain a plurality of first feature images corresponding to the video frame sequence.

[0092] Specifically, the detection device performs feature extraction on the video frame sequence through a series of convolution and pooling operations to obtain multiple first feature images corresponding to the video frame sequence.

[0093] In one embodiment, see Figure 3 As shown in the figure, a schematic diagram of feature extraction of a video frame sequence provided by an embodiment of the present application is provided. The detection device performs convolution operations and pooling operations with convolution kernels of 64×3×3×3 and 1×2×2 on the acquired video frame sequence of size 3×L×H×W, respectively, to obtain a video frame sequence of size The feature map a is obtained by performing convolution operations and pooling operations with kernels of 128×3×3×3 and 2×2×2, respectively, to obtain a feature map of size The feature map b is respectively subjected to convolution operations and pooling operations with convolution kernels of 256×3×3×3, 256×3×3×3 and 2×2×2 to obtain a size of The feature map c is convolved with kernels of 512×3×3×3 and 512×3×3×3 respectively, and the size is obtained. The feature map d (ie, the first feature image) is obtained, where L represents the length of the video to be detected, H represents the height of the video to be detected, and W represents the width of the video to be detected.

[0094] In this way, by extracting features from video frame sequences, the scope of abnormal behavior detection is narrowed and the time cost of detecting abnormal behavior is reduced.

[0095] Step 2: Perform abnormality analysis on the multiple first feature images to obtain at least one abnormal candidate region.

[0096] Specifically, when executing step 2, you can perform the following steps:

[0097] Step a: performing feature extraction on the plurality of first feature images to obtain each second feature image corresponding to the plurality of first feature images.

[0098] Specifically, the abnormal behavior detection device performs a series of convolution operations and pooling operations on the multiple first feature images to extract features, and obtains each second feature image corresponding to the multiple first feature images.

[0099] In one embodiment, see Figure 4 As shown in the figure, a schematic diagram of abnormal candidate region extraction provided by an embodiment of the present application is provided. The abnormal behavior detection device sets K anchors of different sizes at every L / 8 position of the video frame sequence to be detected, with a total of (L / 8)×K anchors, where K is an integer; the RPN candidate region extraction network is used to extract the abnormal candidate region. Figure 3 The feature map d in the image is subjected to two convolution operations with a convolution kernel of 512×3×3×3, and the size is The feature map e is convolved using ROI Pool. The maximum pooling operation is used to downsample and obtain the size of The feature map f (i.e., the second feature image) that only contains temporal information.

[0100] In this way, feature extraction is performed on the first feature image, which reduces the dimension of the image, narrows the scope of abnormal behavior detection, and further improves the accuracy of abnormal behavior detection.

[0101] Step b: Perform an abnormality analysis on each second feature image to obtain position information and confidence information of each second feature image.

[0102] Specifically, the anomaly detection device performs a series of convolution operations on each second feature image to obtain position information and confidence information of each second feature image.

[0103] In one embodiment, a convolution operation with a convolution kernel of 1×1×1 is performed on the feature map f to obtain the offset (i.e., position information) and score (confidence information) of the region of interest (i.e., the abnormal candidate region).

[0104] In this way, in the subsequent step of detecting abnormal behavior, the candidate abnormal region for abnormal behavior detection can be directly obtained based on the obtained position information and confidence information, thereby improving the accuracy of abnormal behavior detection.

[0105] Step c: Obtain abnormal candidate regions of the video frame sequence based on the position information and confidence information of each second feature image.

[0106] In one embodiment, if the offset (i.e., position information) value of a portion of the image in the second feature image is greater than a preset offset threshold and the confidence is greater than a preset confidence threshold, this portion of the image will be regarded as an abnormal candidate area of the video frame sequence.

[0107] It should be noted that the preset offset threshold and the preset confidence threshold can be set according to the actual application scenario. For example, the preset offset threshold is 60% and the preset confidence threshold is 75%, which are not limited here.

[0108] In this way, candidate abnormal regions are screened out from the video frame sequence based on the position information and confidence information, which narrows the scope of abnormal behavior detection and improves the accuracy of abnormal behavior detection.

[0109] Step 3: Input the multiple first feature images and at least one abnormal candidate region into the abnormal behavior detection model to obtain at least one target abnormal region output by the abnormal behavior detection model.

[0110] Specifically, the abnormal behavior detection device inputs multiple first feature images and at least one abnormal candidate area into the abnormal behavior detection model to obtain at least one target abnormal area output by the abnormal behavior detection model.

[0111] In this way, multiple abnormal candidate areas are screened through the abnormal behavior detection model, and the target abnormal area for abnormal behavior detection is obtained, thereby improving the accuracy of abnormal behavior detection.

[0112] Furthermore, when executing step 201, the following steps may be performed:

[0113] S2011: Perform linear mapping processing on at least one abnormal candidate region to obtain a key vector, a value vector, and a query vector of the at least one abnormal candidate region.

[0114] Specifically, the abnormal behavior detection device performs linear mapping processing on at least one abnormal candidate region to obtain a key vector, a value vector, and a query vector of the at least one abnormal candidate region.

[0115] In one embodiment, the abnormal behavior detection device performs linear mapping processing on at least one abnormal candidate area to obtain the shape and size of the key vector and value vector of the abnormal candidate area. The shape of the query vector is 1×1×512.

[0116] In this way, in the subsequent screening process, the key vector, value vector, and query vector of the candidate abnormal region can be obtained to screen the candidate abnormal region.

[0117] S2012: Based on the key vector and query vector of the at least one abnormal candidate region obtained, screen the at least one abnormal candidate region to obtain at least one target abnormal region.

[0118] Specifically, when executing step S2012, the following steps may be performed:

[0119] Step A: Normalize the key vector and query vector of at least one abnormal candidate region to obtain multiple self-attention features of the at least one abnormal candidate region.

[0120] Specifically, the key vector of at least one abnormal candidate region is multiplied by the query vector and then divided by After normalization, multiple self-attention features of at least one abnormal candidate region are obtained.

[0121] In this way, in the subsequent abnormal behavior detection process, the obtained self-attention features can further accurately determine the target area to be detected, ignoring video frames without abnormal behavior, thereby reducing the time cost of abnormal behavior detection.

[0122] Step B: Screen at least one abnormal candidate region based on multiple self-attention features, value vectors, and query vectors of at least one abnormal candidate region to obtain at least one target abnormal region.

[0123] Specifically, multiple self-attention features and value vectors of at least one abnormal candidate region are multiplied, added to the query vector, and then normalized. At least one abnormal candidate region is screened according to the normalized value to obtain at least one target abnormal region.

[0124] In this way, the abnormal candidate areas are screened to obtain the target abnormal area, which narrows the scope of abnormal behavior detection and further reduces the time cost of abnormal behavior detection.

[0125] S2013: Obtain abnormal behavior detection results of the video based on the target abnormal area.

[0126] Specifically, when executing step S2013, the following steps may be performed:

[0127] Step 1: Obtain the self-attention features, value vector, and key vector of at least one target abnormal region.

[0128] In this way, in the subsequent abnormal behavior detection process, the abnormal behavior detection results can be obtained directly based on the self-attention features, value vectors, and key vectors of the target abnormal area.

[0129] Step 2: Input the self-attention features, value vectors, and key vectors of at least one target abnormal region into the feedforward neural network model to obtain abnormal behavior features.

[0130] Specifically, the abnormal behavior detection device inputs the self-attention features, value vectors, and key vectors of at least one target abnormal area into a feedforward neural network model, normalizes the values output by the feedforward neural network model, and obtains the abnormal behavior features.

[0131] In one embodiment, see Figure 5As shown, a schematic diagram of abnormal behavior detection provided by an embodiment of the present application is provided. The self-attention feature, value vector and key vector of at least one target abnormal area are input into a feedforward neural network model. The feedforward neural network model multiplies the self-attention feature with the value vector and then adds it to the key vector. The added value is normalized and output to obtain the abnormal behavior result.

[0132] It should be noted that the abnormal behavior feature is the abnormal behavior detection result including location information and confidence information, and the feedforward neural network model is constructed based on the feedforward neural network.

[0133] In this way, the self-attention features, value vectors, and key vectors of the target abnormal area can be directly used as input parameters of the feedforward neural network model to obtain the abnormal behavior detection results output by the feedforward neural network model, thereby improving the accuracy of abnormal behavior detection.

[0134] See Figure 6 FIG. 1 is a detailed flow chart of a method for detecting abnormal behavior provided by an embodiment of the present application. The specific implementation process of the method is as follows:

[0135] Step 600: Obtain a video frame sequence of a video to be detected.

[0136] Step 601: extract features from a video frame sequence to obtain a plurality of first feature images corresponding to the video frame sequence.

[0137] Step 602: performing feature extraction on the plurality of first feature images to obtain each second feature image corresponding to the plurality of first feature images.

[0138] Step 603: Perform an abnormality analysis on each second feature image to obtain position information and confidence information of each second feature image.

[0139] Step 604: Obtain at least one abnormal candidate region of the video frame sequence according to the position information and confidence information of each second feature image.

[0140] Step 605: Perform linear mapping processing on at least one abnormal candidate region to obtain a key vector, a value vector, and a query vector of the at least one abnormal candidate region.

[0141] Step 606: Based on the key vector and query vector of the at least one abnormal candidate region obtained, screen the at least one abnormal candidate region to obtain at least one target abnormal region.

[0142] Step 607: Obtain the self-attention features, value vector, and key vector of at least one target abnormal region.

[0143] Step 608: Input the self-attention features, value vectors, and key vectors of at least one target abnormal region into a feedforward neural network model to obtain abnormal behavior features.

[0144] Specifically, when executing steps 600 to 608 , the specific steps refer to the above steps 200 to 201 , which are not described in detail here.

[0145] In an embodiment of the present application, an abnormal behavior detection model constructed by a three-dimensional convolutional neural network and a self-attention network is used to extract features from a video frame sequence of a video to be detected, and a target abnormal area is screened out based on a series of features, thereby narrowing the scope of abnormal behavior detection and reducing the time cost. In addition, the abnormal behavior detection results of the video to be detected are obtained based on the self-attention features, value vectors and key vectors of the screened target abnormal area, thereby improving the accuracy of abnormal behavior detection.

[0146] See Figure 7 FIG. 1 is a schematic diagram of a structure of an abnormal behavior detection device provided in an embodiment of the present application, including:

[0147] Acquisition unit 701: used to acquire a video frame sequence of a video to be detected;

[0148] Detection unit 702: used to obtain abnormal behavior detection results of the video based on the video frame sequence and a pre-trained abnormal behavior detection model.

[0149] In one embodiment, the detection unit 702 is used to:

[0150] Extracting features from the video frame sequence to obtain a plurality of first feature images corresponding to the video frame sequence;

[0151] Performing abnormality analysis on the plurality of first feature images to obtain at least one abnormal candidate region;

[0152] The plurality of first feature images and at least one abnormal candidate region are input into an abnormal behavior detection model to obtain at least one target abnormal region output by the abnormal behavior detection model.

[0153] In one embodiment, the detection unit 702 is used to:

[0154] Performing feature extraction on the plurality of first feature images to obtain second feature images corresponding to the plurality of first feature images;

[0155] Performing abnormality analysis on each second feature image to obtain position information and confidence information of each second feature image;

[0156] According to the position information and confidence information of each second feature image, an abnormal candidate region of the video frame sequence is obtained.

[0157] In one embodiment, the detection unit 702 is used to:

[0158] Performing linear mapping processing on at least one abnormal candidate region to obtain a key vector, a value vector, and a query vector of the at least one abnormal candidate region;

[0159] Based on the obtained key vector and query vector of the at least one abnormal candidate region, screening the at least one abnormal candidate region to obtain at least one target abnormal region;

[0160] According to the target abnormal area, the abnormal behavior detection results of the video are obtained.

[0161] In one embodiment, the detection unit 702 is used to:

[0162] Normalizing a key vector and a query vector of at least one abnormal candidate region to obtain a plurality of self-attention features of the at least one abnormal candidate region;

[0163] At least one abnormal candidate region is screened according to multiple self-attention features, a value vector, and a query vector of at least one abnormal candidate region to obtain at least one target abnormal region.

[0164] In one embodiment, the detection unit 702 is used to:

[0165] Obtaining self-attention features, value vectors, and key vectors of at least one target abnormal region;

[0166] The self-attention features, value vectors, and key vectors of at least one target abnormal area are input into a feedforward neural network model to obtain abnormal behavior features, wherein the abnormal behavior features are abnormal behavior detection results including location information and confidence information. The feedforward neural network model is constructed based on the feedforward neural network.

[0167] In an embodiment of the present application, an abnormal behavior detection model constructed by a three-dimensional convolutional neural network and a self-attention network is used to extract features from a video frame sequence of a video to be detected, and a target abnormal area is screened out based on a series of features, thereby narrowing the scope of abnormal behavior detection and reducing the time cost. In addition, the abnormal behavior detection results of the video to be detected are obtained based on the self-attention features, value vectors and key vectors of the screened target abnormal area, thereby improving the accuracy of abnormal behavior detection.

[0168] See Figure 8 , which is a schematic structural diagram of an electronic device provided in an embodiment of the present application.

[0169] The electronic device 8000 includes: a processor 8080 and a memory 8020 , and optionally, may also include a power supply 8030 , a display unit 8040 , and an input unit 8050 .

[0170] The processor 8080 is the control center of the electronic device 8000. It uses various interfaces and lines to connect various components, and performs various functions of the electronic device 8000 by running or executing software programs and / or data stored in the memory 8020, thereby monitoring the electronic device 8000 as a whole.

[0171] In the embodiment of the present application, when the processor 8080 calls the computer program stored in the memory 8020, the following is executed: Figure 2 The embodiment shown in provides a method for detecting abnormal behavior.

[0172] Optionally, the processor 8080 may include one or more processing units. Preferably, the processor 8080 may integrate an application processor and a modem processor, wherein the application processor primarily processes the operating system, user interface, and applications, and the modem processor primarily processes wireless communications. It is understood that the modem processor may not be integrated into the processor 8080. In some embodiments, the processor and memory may be implemented on a single chip. In some embodiments, they may also be implemented on separate chips.

[0173] The memory 8020 may mainly include a program storage area and a data storage area. The program storage area may store an operating system, various applications, etc., and the data storage area may store data created based on the use of the electronic device 8000. In addition, the memory 8020 may include a high-speed random access memory and a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage device.

[0174] The electronic device 8000 also includes a power supply 8030 (such as a battery) for supplying power to various components. The power supply can be logically connected to the processor 8080 through a power management system, thereby managing functions such as charging, discharging, and power consumption through the power management system.

[0175] The display unit 8040 can be used to display information input by the user or information provided to the user, as well as various menus of the electronic device 8000. In the embodiment of the present invention, it is mainly used to display the display interface of each application in the electronic device 8000 and objects such as text and pictures displayed on the display interface. The display unit 8040 may include a display panel 8041. The display panel 8041 can be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), etc.

[0176] The input unit 8050 can be used to receive information such as numbers or characters input by the user. The input unit 8050 may include a touch panel 8051 and other input devices 8052. The touch panel 8051, also known as a touch screen, can receive user touch operations on or near it (for example, operations performed by the user using a finger, a stylus, or any other suitable object or accessory on or near the touch panel 8051).

[0177] Specifically, the touch panel 8051 can detect user touch operations and the signals generated by the touch operations, convert these signals into touch point coordinates, and send them to the processor 8080. It can also receive and execute commands sent by the processor 8080. In addition, the touch panel 8051 can be implemented using various types, such as resistive, capacitive, infrared, and surface acoustic wave. Other input devices 8052 can include, but are not limited to, one or more of a physical keyboard, function keys (such as volume control buttons, power buttons, etc.), a trackball, a mouse, a joystick, etc.

[0178] Of course, the touch panel 8051 can cover the display panel 8041. When the touch panel 8051 detects a touch operation on or near it, it transmits it to the processor 8080 to determine the type of touch event. Then the processor 8080 provides corresponding visual output on the display panel 8041 according to the type of touch event. Figure 8 In the embodiment, the touch panel 8051 and the display panel 8041 are two independent components to realize the input and output functions of the electronic device 8000, but in some embodiments, the touch panel 8051 and the display panel 8041 can be integrated to realize the input and output functions of the electronic device 8000.

[0179] The electronic device 8000 may also include one or more sensors, such as a pressure sensor, a gravity acceleration sensor, a proximity light sensor, etc. Of course, according to the needs of specific applications, the electronic device 8000 may also include other components such as a camera. Since these components are not the key components used in the embodiments of this application, Figure 8 It is not shown and will not be described in detail.

[0180] Those skilled in the art will understand that Figure 8 The electronic device is merely an example and does not limit the electronic device. The electronic device may include more or fewer components than shown in the figure, or may include a combination of certain components or different components.

[0181] In an embodiment of the present application, a readable storage medium stores a computer program. When the computer program is executed by a processor, the communication device can perform the various steps in the above embodiment.

[0182] For the convenience of description, the above parts are divided into modules (or units) according to their functions and described separately. Of course, when implementing this application, the functions of each module (or unit) can be implemented in the same or multiple software or hardware.

[0183] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0184] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0185] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0186] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0187] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present application.

[0188] Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.

Claims

1. A method for detecting abnormal behavior, characterized in that: include: Obtain a video frame sequence of the video to be detected; Performing feature extraction on the video frame sequence to obtain a plurality of first feature images corresponding to the video frame sequence; Performing feature extraction on the plurality of first feature images using a candidate region extraction network to obtain each second feature image corresponding to the plurality of first feature images; Obtaining at least one abnormal candidate region of the video frame sequence according to the position information and confidence information of each second feature image; Inputting the plurality of first feature images and the at least one abnormal candidate region into an abnormal behavior detection model to obtain at least one target abnormal region output by the abnormal behavior detection model; wherein the abnormal behavior detection model is constructed based on a three-dimensional convolutional neural network and a self-attention network; The self-attention features, value vectors, and key vectors of the at least one target abnormal area are input into a feedforward neural network model to obtain abnormal behavior features; wherein the abnormal behavior features are abnormal behavior detection results including location information and confidence information, and the feedforward neural network model is constructed based on the feedforward neural network; the abnormal behavior detection results indicate the detection results of individual abnormal behavior.

2. The method according to claim 1, characterized in that The step of obtaining at least one target abnormal area output by the abnormal behavior detection model includes: Performing linear mapping processing on the at least one abnormal candidate region to obtain a key vector, a value vector, and a query vector for the at least one abnormal candidate region; Normalizing the key vector and the query vector of the at least one abnormal candidate region to obtain a plurality of self-attention features of the at least one abnormal candidate region; The at least one abnormal candidate region is screened according to the multiple self-attention features, the value vector, and the query vector of the at least one abnormal candidate region to obtain the at least one target abnormal region output by the abnormal behavior detection model.

3. A device for detecting abnormal behavior, characterized in that: include: An acquisition unit, configured to acquire a video frame sequence of a video to be detected; a detection unit, configured to perform feature extraction on the video frame sequence to obtain a plurality of first feature images corresponding to the video frame sequence; The detection unit is further configured to perform feature extraction on the plurality of first feature images using a candidate region extraction network to obtain each second feature image corresponding to each of the plurality of first feature images; The detection unit is further configured to obtain at least one abnormal candidate region of the video frame sequence based on the position information and confidence information of each second feature image; The detection unit is further configured to input the plurality of first feature images and the at least one abnormal candidate region into an abnormal behavior detection model to obtain at least one target abnormal region output by the abnormal behavior detection model; wherein the abnormal behavior detection model is constructed based on a three-dimensional convolutional neural network and a self-attention network; The detection unit is further used to input the self-attention features, value vectors and key vectors of the at least one target abnormal area into a feedforward neural network model to obtain abnormal behavior features; wherein the abnormal behavior features are abnormal behavior detection results including location information and confidence information, and the feedforward neural network model is constructed based on the feedforward neural network; the abnormal behavior detection results indicate the detection results of individual abnormal behavior.

4. An electronic device, characterized in that: The invention comprises a processor and a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 2 is executed.

5. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 2 is executed.

Images