Abnormal behavior detection device and method

Abstract

The present invention provides an abnormal behavior detection device and method. The abnormal behavior detection device comprises: a suspicious gesture detection module, which detects a suspicious gesture for an image of a frame in a video and determines a suspicious gesture detection frame according to the suspicious gesture; and an abnormal behavior detection module, which captures images in the suspicious gesture detection frame of consecutive predetermined frames before and after a frame from the video and determines whether it is an abnormal behavior according to the captured images of the predetermined frames.

Description

Technical Field

[0001] The present invention relates to the field of video image processing, and in particular, to an abnormal behavior detection device and method based on video images. Background Art

[0002] Public safety is related to the stability of the country and society, and directly affects the safety of people's lives and property. In order to ensure public safety, it is very necessary to detect abnormal behaviors in security inspection scenes in real time and issue warnings in time to ensure the safety of people's travel.

[0003] In recent years, using deep learning technology to detect abnormal behaviors in images and identify continuous actions has become one of the research hotspots in this technical field. Among them, how to effectively, quickly and in real time identify the behavior actions in the video is still the core work of many researchers. Summary of the invention

[0004] In recent years, research on behavior recognition algorithms based on deep learning has made major breakthroughs, mainly due to the significant improvement in the recognition rate of convolutional neural networks in the field of computer vision.

[0005] At present, some video action recognition methods in the field of deep learning can be divided into two directions: one direction is to use 3D convolutional neural networks or 3D convolutional neural networks combined with dual streams to recognize the actions appearing in video images, however, this method cannot identify the person who performs the action. The other direction is to use the human skeleton plus RNN neural network to perform action recognition on pedestrians appearing in the video, which can identify the person who performs the action.

[0006] However, the above two methods are not effective in identifying abnormal behaviors in the entire scene when the ROI area is not specified in video surveillance of large scenes. When there are many people, the amount of calculation for action recognition based on the human skeleton is relatively large and the real-time performance is poor. When using the 3D convolutional neural network method, action recognition must be performed on multiple consecutive frames of images each time, which consumes a lot of computer resources.

[0007] The inventors of the present invention discovered that behavioral action is a continuous process that needs to be identified from multiple frames of continuous video images. It is difficult to identify the action to be taken through a single image. However, the posture of the pedestrian when performing the action can be better detected on a single image.

[0008] Therefore, the present invention provides an abnormal behavior detection device and method that can accurately identify abnormal behaviors while reducing the amount of calculation and improving the calculation speed.

[0009] As an embodiment of the present invention, an abnormal behavior detection device is provided, including: a suspicious posture detection module, which detects a suspicious posture for an image of a frame in a video, and determines a suspicious posture detection frame according to the suspicious posture; and an abnormal behavior detection module, which captures images within the suspicious posture detection frame of consecutive predetermined frames before and after a frame from the video, and determines whether it is an abnormal behavior based on the captured images of the predetermined frames.

[0010] In the above-mentioned abnormal behavior detection device, the suspicious posture detection module uses a suspicious posture detection model to detect suspicious postures. The suspicious posture detection model is a target detection convolutional neural network model. The abnormal behavior detection module uses an abnormal behavior judgment model to judge whether it is an abnormal behavior. The abnormal behavior judgment model is a 3D convolutional neural network model.

[0011] In the above-mentioned abnormal behavior detection device, the suspicious posture detection module includes: a frame image extraction unit, which extracts a frame of image from the video; a suspicious posture detection unit, which inputs the frame of image into a suspicious posture detection model to determine whether it is a suspicious posture; and a suspicious posture detection frame determination unit, which determines a suspicious posture detection frame in the image when the suspicious posture detection unit detects a suspicious posture.

[0012] In the above-mentioned abnormal behavior detection device, the suspicious posture detection model is a YOLOV3 target detection network model.

[0013] In the above-mentioned abnormal behavior detection device, the abnormal behavior detection module includes: a predetermined frame image capture unit, which captures the image of the predetermined frame; and an abnormal behavior judgment unit, which inputs the image of the predetermined frame into an abnormal behavior judgment model to judge whether it is an abnormal behavior; and a result output unit, which outputs the type of abnormal behavior and the image of the suspicious posture detection box as a result.

[0014] In the above-mentioned abnormal behavior detection device, the abnormal behavior judgment model is a 3D neural network model based on the vgg16 framework.

[0015] In the above-mentioned abnormal behavior detection device, the suspicious posture detection model also detects suspicious persons who make suspicious gestures, and the suspicious posture detection module also includes: a tracking frame determination unit, which determines the suspicious person tracking frame based on the detected suspicious person, and the abnormal behavior detection module also includes: a suspicious person tracking unit, which tracks the suspicious person tracking frame using a tracking algorithm for an image of a predetermined frame, and the result output unit also outputs an image of the suspicious person tracking frame.

[0016] In the abnormal behavior detection device described above, the image of the suspicious person tracking frame outputted by the result output unit includes the image of the suspicious person tracking frame in the image of each frame in the predetermined frames.

[0017] In the abnormal behavior detection device described above, the image of the suspicious person tracking frame outputted by the result output unit includes a moving image constituted by the image of the suspicious person tracking frame in each of the predetermined frames.

[0018] In the above-mentioned abnormal behavior detection device, the labeled suspicious posture training data is imported into the suspicious posture detection model to train the suspicious posture detection model; the trained suspicious posture detection model is used to detect the collected video data to generate abnormal behavior training data; and the abnormal behavior training data is used to train the abnormal behavior judgment model.

[0019] As another embodiment of the present invention, an abnormal behavior detection method is provided, including: a suspicious posture detection step, detecting a suspicious posture for an image of a frame in a video, and determining a suspicious posture detection frame according to the suspicious posture; and an abnormal behavior detection step, capturing images within suspicious posture detection frames of consecutive predetermined frames before and after a frame in the video, and determining whether it is an abnormal behavior based on the captured images of the predetermined frames.

[0020] In the above-mentioned abnormal behavior detection method, in the suspicious posture detection step, a suspicious posture detection model is used to detect suspicious postures, and the suspicious posture detection model is a target detection convolutional neural network model. In the abnormal behavior detection step, an abnormal behavior judgment model is used to judge whether it is an abnormal behavior, and the abnormal behavior judgment model is a 3D convolutional neural network model.

[0021] In the above abnormal behavior detection method, the suspicious posture detection step includes: extracting an image frame from the video;

[0022] Inputting a frame of image into a suspicious gesture detection model to determine whether it is a suspicious gesture; and when the suspicious gesture detection unit detects a suspicious gesture, determining a suspicious gesture detection frame in the image.

[0023] In the above abnormal behavior detection method, the suspicious posture detection model is the YOLOV3 target detection network model.

[0024] In the above-mentioned abnormal behavior detection method, the abnormal behavior detection step includes: capturing an image of a predetermined frame; and inputting the image of the predetermined frame into an abnormal behavior judgment model to determine whether it is an abnormal behavior; and a result output unit, as a result, outputs the type of abnormal behavior and the image of the suspicious posture detection box.

[0025] In the above abnormal behavior detection method, the abnormal behavior judgment model is a 3D neural network model based on the vgg16 framework.

[0026] In the above-mentioned abnormal behavior detection method, the suspicious posture detection model also detects suspicious persons who make suspicious gestures, and the suspicious posture detection step also includes: determining a suspicious person tracking frame based on the detected suspicious person, and the abnormal behavior detection step also includes: tracking the suspicious person tracking frame using a tracking algorithm for an image of a predetermined frame, and in the result output step, also outputting an image of the suspicious person tracking frame.

[0027] In the above abnormal behavior detection method, the image of the suspicious person tracking frame output in the result output step includes: the image of the suspicious person tracking frame in the image of each frame in the predetermined frames.

[0028] In the abnormal behavior detection method described above, the image of the suspicious person tracking frame output in the result output step includes: a moving image composed of the image of the suspicious person tracking frame in each frame of the predetermined frames.

[0029] In the above-mentioned abnormal behavior detection method, the labeled suspicious posture training data is imported into the suspicious posture detection model to train the suspicious posture detection model; the trained suspicious posture detection model is used to detect the collected video data to generate abnormal behavior training data; and the abnormal behavior training data is used to train the abnormal behavior judgment model.

[0030] As another embodiment of the present invention, an abnormal behavior detection device is provided, including: a memory; and a processor, executing a program stored in the memory, so that the computer performs the following processing: detecting a suspicious gesture for an image of a frame in a video, and determining a suspicious gesture detection frame based on the suspicious gesture; and capturing images within suspicious gesture detection frames of consecutive predetermined frames before and after a frame from the video, and determining whether it is an abnormal behavior based on the captured images of the predetermined frames.

[0031] As another embodiment of the present invention, a computer-readable storage medium is provided, which stores an abnormal behavior detection program. The abnormal behavior detection program enables a computer to perform the following processing: detecting a suspicious gesture for an image of a frame in a video, and determining a suspicious gesture detection frame based on the suspicious gesture; and capturing images within suspicious gesture detection frames of consecutive predetermined frames before and after a frame from the video, and determining whether it is an abnormal behavior based on the captured images of the predetermined frames.

[0032] In the present invention, by detecting suspicious gestures for an image frame in a video, determining a suspicious gesture detection frame, and acquiring images of consecutive predetermined frames before and after the suspicious gesture detection frame, abnormal behavior can be accurately identified while reducing the amount of calculation and improving the calculation speed. In addition, in the present invention, the suspicious gesture detection frame can be automatically determined by detecting an image frame, without the need to manually define the ROI area, etc., thereby reducing the workload of operators, improving detection efficiency, and being more conducive to real-time detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 is a flowchart showing an abnormal behavior detection method according to a first embodiment of the present invention;

[0034] Figure 2 is a flowchart showing an abnormal behavior detection method according to Example 1 of the first embodiment;

[0035] Figure 3 is a schematic diagram showing a YOLOV3 network model as an example of a suspicious posture detection model;

[0036] Figure 4 is a schematic diagram showing a 3D convolutional neural network model based on a vgg16 framework as an example of an abnormal behavior judgment model;

[0037] Figure 5 is a flowchart showing an abnormal behavior detection method according to Example 2 of the first embodiment;

[0038] Figure 6 is a diagram showing an example of determining a suspicious gesture detection frame and a suspicious person tracking frame;

[0039] Figure 7 is a diagram showing an example of an image within a suspicious gesture detection frame;

[0040] Figure 8 is a schematic diagram showing the structure of an abnormal behavior detection device according to a second embodiment of the present invention;

[0041] Fig. 9 is a schematic diagram showing the structure of an abnormal behavior detection device according to Example 1 of the first embodiment;

[0042] Fig.10 This is a schematic diagram showing the structure of an abnormal behavior detection device according to Example 2 of the first embodiment. DETAILED DESCRIPTION

[0043] Below will be described in more detail with reference to the accompanying drawings exemplary embodiments or embodiments of the present invention. Although the accompanying drawings show exemplary embodiments of the present invention, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments or embodiments set forth herein. On the contrary, these embodiments or embodiments are provided in order to be able to more clearly understand the present invention.

[0044] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, rather than to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments or examples of the present application described herein can be implemented in an order other than that shown or described. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not limited to the steps or units clearly listed, but may include other steps or units that are not clearly listed.

[0045] <First Embodiment>

[0046] Hereinafter, an abnormal behavior detection method will be described as a first embodiment of the present invention.

[0047] Figure 1 This is a flowchart showing the abnormal behavior detection method according to the first embodiment of the present invention.

[0048] like Figure 1 As shown, in step S10, a suspicious gesture is detected for an image of a frame in a video, and a suspicious gesture detection frame is determined according to the suspicious gesture.

[0049] In step S20, for the suspicious gesture detection frame obtained in step S10, the images in the suspicious gesture detection frame of the consecutive predetermined frames before and after the frame are intercepted from the video, and whether it is an abnormal behavior is determined based on the intercepted images of the predetermined frames. Here, the type of abnormal behavior can be left behind a package, passing objects across the fence, fighting, etc. There is no special limitation on the type of abnormal behavior, which can be set according to the application scenario.

[0050] It can be seen that in this embodiment, first, the suspicious posture detection model is used to detect suspicious postures that may be part of abnormal behavior in a certain frame of the video, and a suspicious posture detection frame is obtained. Then, the suspicious posture detection frame is used to intercept multiple consecutive images and pass them to the abnormal behavior judgment model to identify abnormal behavior. In this way, there is no need to manually delineate the ROI area, and the detection frame can be automatically obtained, and only the image data of the predetermined frame within the detection frame is intercepted, so that the abnormal behavior can be accurately identified through multiple images while reducing the amount of calculation and improving the calculation speed, so that the abnormal behavior occurring in the video can be quickly and accurately detected.

[0051] In addition, in this embodiment, in step S10, a suspicious posture detection model composed of a target detection convolutional neural network model can be used to detect suspicious postures for a frame of image, and in step S20, an abnormal behavior judgment model composed of a 3D convolutional neural network model can be used to judge whether the above-mentioned suspicious posture-related behavior is abnormal behavior for a predetermined frame (for example, n frames) of images. By using the target detection convolutional neural network model as a suspicious posture detection model, suspicious postures can be quickly found in a single image, thereby determining a suspicious posture detection frame. By using a 3D convolutional neural network model as an abnormal behavior judgment model, abnormal behavior can be accurately identified from multiple images.

[0052] Hereinafter, specific examples of step S10 and step S20 will be described.

[0053] (Example 1)

[0054] Figure 2 This is a flowchart showing the abnormal behavior detection method according to Example 1 of the first embodiment.

[0055] First, step S10 will be described in detail.

[0056] In step S11, an image frame is extracted from the video.

[0057] In the application scenario of video surveillance, the video is a surveillance video, which can be obtained from a surveillance camera or the like, and the image extraction in step S11 can be performed in real time or periodically.

[0058] In step S12, the image of the one frame is input into the suspicious gesture detection model.

[0059] In step S13, the suspicious gesture detection model is used to determine whether there is a suspicious gesture in the image of the frame. In step S13, if the judgment is "yes", that is, a suspicious gesture is detected, then enter step S14, if the judgment is "no", that is, no suspicious gesture is detected, then return to step S11.

[0060] Here, the suspicious posture detection model is a target detection convolutional neural network model. The model is trained by pre-labeled suspicious posture training data. The type of suspicious posture training data can be posture types related to abnormal behaviors such as left-behind packages, passing objects between fences, fighting, etc., which can be set according to the application scenario and the behavior of interest. Suspicious postures are postures that may be related to abnormal behaviors. That is, suspicious postures represent postures that may be a certain state of the abnormal behavior.

[0061] Specifically, for example, the model can be a YOLOV3 target detection network model. The training is performed by importing the previously labeled suspicious posture training data into the YOLOV3 target detection network to obtain a trained suspicious posture detection model. The suspicious posture detection model of the present invention is not particularly limited, as long as it is a convolutional neural network model that can perform target detection. Preferably, a 2D convolutional neural network model for target detection can make the calculation speed faster.

[0062] As an example, the YOLOV3 network model can adopt the following model: 3×3 and 1×1 convolutions are stacked, and the pooling layer is removed. The convolution layer is used to shrink the image. The network architecture is as follows: Figure 3 When the suspicious gesture detection model is constructed in this way, the calculation speed can be made faster, which is conducive to the rapid detection of suspicious gestures.

[0063] In step S14, a suspicious gesture detection frame is determined in the image according to the suspicious gesture detected in step S13. In step S14, the image of the suspicious gesture detection frame can also be directly cut out.

[0064] Next, step S20 will be described in detail.

[0065] In step S21, the image in the suspicious gesture detection frame of the predetermined frames (for example, n frames) before and after the image of the frame extracted in the above-mentioned step S11 is intercepted from the video. In other words, for the images of the predetermined frames before and after the above-mentioned frame in the video, the images in the suspicious gesture detection frame are all intercepted. Here, the predetermined frames before and after the image of a frame refer to a total of predetermined frames (n frames) containing the image of the frame. There is no special limitation on the number n of the predetermined frames, as long as the range of the behavior can be identified. n can be appropriately set according to the type of abnormal behavior to be identified. For example, the abnormal behavior of fighting can be set to a larger value because it lasts for a long time, while the abnormal behavior of throwing things can be set to a larger value because it lasts for a short time. If you want to identify multiple abnormal behaviors, select n that can identify all abnormal behaviors of concern.

[0066] Here, the predetermined frames before and after one frame of image include the predetermined frames starting from the one frame extracted in step S11, that is, n frames with this one frame as the first frame.

[0067] Here, since only the image within the suspicious posture detection frame is captured, the amount of calculation can be reduced compared to the case where the entire image range is calculated. At the same time, since the images of predetermined frames before and after the collection area are collected, abnormal behavior can also be identified through multiple images.

[0068] In step S22, the images of the predetermined frames (n frames) obtained in step S21 are input into the abnormal behavior judgment model.

[0069] In step S23, whether it is an abnormal behavior is determined by using an abnormal behavior determination model.

[0070] Here, the abnormal behavior judgment model is a 3D convolutional neural network model. The model is trained using abnormal behavior training data. The abnormal behavior training data can be generated by detecting the collected video data using the trained suspicious posture detection model. The types of abnormal behavior training data can be divided into background, left-behind packages, passing objects between fences, fighting, etc. It can be set according to the application scenario and the behavior of concern. The type of abnormal behavior is related to the type of suspicious posture mentioned above.

[0071] The abnormal behavior judgment model of the present invention is not particularly limited, as long as it is a 3D convolutional neural network model that can perform behavior detection.

[0072] Specifically, for example, the model can be a 3D convolutional neural network model based on the vgg16 framework.

[0073] As an example, the following model can be used: replace the 2D convolutional layer and pooling layer in the vgg16 framework with 3D convolutional layer and 3D pooling layer, and change the input layer to 16×3×112×112, that is, 16 consecutive images, 3 channels, and the width and height of the image are 112×112, so that the network architecture becomes as follows Figure 4 In this case, the calculation speed can be fast and the calculation accuracy can be high, which is conducive to the rapid and accurate identification of abnormal behavior.

[0074] In step S24, the content to be output as a result is determined.

[0075] Here, as a result, the type of abnormal behavior and the image of the suspicious gesture detection box can be output.

[0076] Here, the type of abnormal behavior is obtained through the calculation result of the abnormal behavior judgment model mentioned above.

[0077] The image of the suspicious gesture detection frame may be the image of the suspicious gesture detection frame in the image of the one frame determined in the above step S14.

[0078] In addition, an alarm may be output as a result. The alarm may be a text message or a control signal that controls a speaker to sound an alarm.

[0079] In Example 1, by applying a suspicious posture detection model composed of a target detection convolutional neural network model to a single frame image to identify suspicious postures, a suspicious posture detection frame can be determined, so that suspicious postures can be quickly found from a single frame image. Then, for the range of the suspicious posture detection frame, the images of n consecutive frames before and after the frame are intercepted, and the abnormal behavior judgment model composed of a 3D convolutional neural network model is applied to the n frames to identify abnormal behaviors. Thus, by only intercepting n frames of images within the range of the suspicious posture detection frame, the amount of image data to be processed is reduced, thereby reducing the amount of calculation, thereby reducing the occupied computing resources, and at the same time, by performing abnormal behavior recognition on n frames of images, the accuracy of abnormal behavior recognition can be guaranteed.

[0080] (Example 2)

[0081] The difference between Example 2 and Example 1 is that a person who makes a suspicious gesture or abnormal behavior is also identified.

[0082] Figure 5 This is a flowchart showing the abnormal behavior detection method according to Example 2 of the first embodiment.

[0083] Specifically, if Figure 5 As shown, step S10 also includes step S15, and step S20 also includes step S25.

[0084] That is, in step S13 of step S10, the suspicious gesture detection model detects not only the suspicious gesture but also the person making the suspicious gesture, ie, the suspicious person. Accordingly, in the training of the suspicious gesture detection model, training data on the suspicious person can be added.

[0085] In step S15, a suspicious person tracking frame is determined based on the suspicious person determined in step S13.

[0086] In step S25, the suspicious person tracking frame is tracked using a tracking algorithm.

[0087] Here, there is no particular limitation on the tracking algorithm, as long as it can track specific targets in multiple images. As a tracking algorithm, a KCF (Kernel Correlation Filter) target tracking algorithm can be used, which can achieve better tracking effect.

[0088] In step S24, as output content, an image of the suspicious person tracking frame is also output.

[0089] Here, the image of the suspicious person tracking frame can be the image of the suspicious person tracking frame in each frame (the 1st frame to the nth frame) among the n frames. It can also be a moving image composed of the images of the suspicious person tracking frame in the 1st frame to the nth frame. By outputting the moving image, the user can observe the abnormal behavior process more intuitively.

[0090] In the second embodiment, not only can abnormal behavior be detected quickly and accurately as in the first embodiment, but also the person who has performed the abnormal behavior can be identified.

[0091] The abnormal behavior detection methods of Embodiment 1 and Embodiment 2 are described below by taking a scenario of passing objects through a fence as an example. Figure 6 is a diagram showing an example of determining a suspicious gesture detection frame and a suspicious person tracking frame. Figure 7 : is a diagram showing an example of an image within a suspicious gesture detection frame.

[0092] Figure 6 As shown, in step S10, in an image frame, when a suspicious gesture C that may be related to an abnormal behavior of passing an object through a fence is detected by the suspicious gesture detection model, a suspicious gesture detection frame K is determined in the image frame, and the image within the suspicious gesture detection frame K is cut out, thereby obtaining the following Figure 7 Image shown.

[0093] In the above-mentioned embodiment 2, the person A and the person B who made the suspicious gesture C are also identified by the suspicious gesture detection model, and the suspicious person tracking frame KA and the suspicious person tracking frame KB respectively including the heads of the person A and the person B are determined.

[0094] In step S20, for the range of the suspicious posture detection frame K cut out, the images of the continuous n frames before and after the frame are obtained, and based on the images of the suspicious posture detection frame K of the n frames, abnormal behavior detection is performed through the abnormal behavior judgment model. In Example 2, target tracking is also performed on the suspicious person tracking frame KA and the suspicious person tracking frame KB.

[0095] As a result, the type of abnormal behavior judged by the abnormal behavior judgment model, the image of the suspicious posture detection frame K, the images of the suspicious person tracking frame KA, and the suspicious person tracking frame KB can be output.

[0096] Here, the output image of the suspicious posture detection box K can be as follows Figure 7 As shown, the output images of the suspicious person tracking frame KA and the suspicious person tracking frame KB can be the images of the suspicious person tracking frame KA and the suspicious person tracking frame KB of the 1st frame to the nth frame, and can be output in the form of multiple static images or in the form of dynamic images.

[0097] <Second Embodiment>

[0098] Hereinafter, an abnormal behavior detection device will be described as a second embodiment of the present invention.

[0099] The abnormal behavior detection device of this embodiment may have a processor and a memory. The abnormal behavior detection device may implement the above-mentioned functional modules or functional units by executing the abnormal behavior detection program stored in the memory through the processor. These functional modules or functional units are divided according to the functions they perform, and may be physically integrated or separated.

[0100] The processor may include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), etc. The processor may include multiple processors, or a multi-core processor, each of which performs part of the processing, thereby reducing the processing load of a single processor or a single processing unit. Multiple processors or a multi-core processor may perform the above-mentioned part of the processing in parallel, thereby increasing the computing speed.

[0101] The memory includes, for example, a ROM (Read Only Memory) storing programs and various data, and a RAM (Random Access Memory) used as a work area of ​​the CPU.

[0102] The abnormal behavior detection device may also include hardware such as FPGA (Field Programmable Gate Array), ASIC (Application Specific Integrated Circuit), etc. Part or all of the processing performed by the abnormal behavior detection device may be implemented by FPGA, ASIC, etc.

[0103] Therefore, the abnormal behavior detection device of this embodiment can be implemented by software, a combination of software and hardware, or hardware.

[0104] As an application scenario, the abnormal behavior detection device of this embodiment can be applied to, for example, security video surveillance, etc. In this case, the abnormal behavior detection device can be used as a part of the on-site security inspection equipment, or can be installed in the monitoring host of the monitoring room. However, the application scenario of the abnormal behavior detection device of the embodiment is not limited to this.

[0105] Hereinafter, each functional module and functional unit constituting the abnormal behavior detection device will be described in detail.

[0106] Figure 8 1 is a schematic diagram showing the configuration of an abnormal behavior detection device 100 according to a second embodiment of the present invention.

[0107] like Figure 8 As shown, the abnormal behavior detection device 100 may include a suspicious posture detection module 10 and an abnormal behavior detection module 20 .

[0108] The suspicious gesture detection module 10 detects a suspicious gesture for an image frame in a video, and determines a suspicious gesture detection frame according to the suspicious gesture.

[0109] The abnormal behavior detection module 20 captures images within suspicious gesture detection frames of consecutive predetermined frames before and after the frame from the video, and determines whether it is an abnormal behavior based on the captured images of the predetermined frames.

[0110] In addition, in this embodiment, the suspicious posture detection module 10 can use a suspicious posture detection model composed of a target detection convolutional neural network model to detect suspicious postures for a frame of image, and the abnormal behavior detection module 20 can use an abnormal behavior judgment model composed of a 3D convolutional neural network model to judge whether the above-mentioned suspicious posture-related behavior is abnormal behavior for images of predetermined frames (for example, n frames).

[0111] According to this embodiment, the effect of quickly and accurately identifying abnormal behavior can be achieved without defining an ROI area, which is similar to the first embodiment, and the amount of calculation can be reduced.

[0112] (Example 1)

[0113] Fig. 9 It is a schematic diagram showing the structure of the abnormal behavior detection device 100 according to Example 1 of the second embodiment.

[0114] like Fig. 9As shown, the suspicious gesture detection module 10 may include: a frame image extraction unit 11 , a suspicious gesture detection unit 12 , and a suspicious gesture detection frame determination unit 13 .

[0115] The one-frame image extraction unit 11 extracts an image of one frame from the video. That is, the one-frame image extraction unit 11 may execute the process of step S11 in the first embodiment.

[0116] The suspicious gesture detection unit 12 inputs one frame of image into the suspicious gesture detection model to determine whether it is a suspicious gesture. That is, the suspicious gesture detection unit 12 can execute the processing of steps S12 and S13 in the first embodiment.

[0117] The suspicious gesture detection frame determination unit 13 determines a suspicious gesture detection frame in the image according to the suspicious gesture detected by the suspicious gesture detection unit 12. At the same time, the suspicious gesture detection frame determination unit 13 can also directly cut out the image of the suspicious gesture detection frame. That is, the suspicious gesture detection frame determination unit 13 can perform the processing of step S14 in the first embodiment.

[0118] In addition, if Fig. 9 As shown, the abnormal behavior detection module 20 may include: a predetermined frame image interception unit 21, an abnormal behavior determination unit 22, and a result output unit 23.

[0119] The predetermined frame image extraction unit 21 extracts from the video the images within the suspicious gesture detection frame of predetermined frames (e.g., n frames) before and after the frame extracted by the above-mentioned one frame image extraction unit 11. That is, the predetermined frame image extraction unit 21 can perform the processing of step S21 in the first embodiment.

[0120] The abnormal behavior determination unit 22 inputs the image of the predetermined frame extracted by the predetermined frame image cutting unit 21 into the abnormal behavior determination model to determine whether it is an abnormal behavior. That is, the abnormal behavior determination unit 22 can execute the processing of steps S22 and S23 in the first embodiment.

[0121] The result output unit 23 determines the content to be output as a result. Here, as a result, the type of abnormal behavior and the image of the suspicious gesture detection frame can be output. That is, the result output unit 23 can perform the processing of step S24 in the first embodiment.

[0122] This first embodiment can also obtain the same effects as those of the first embodiment of the first embodiment.

[0123] (Example 2)

[0124] Fig.101 is a schematic diagram showing the structure of the abnormal behavior detection device involved in Example 2 of the first embodiment. The difference between Example 2 and Example 1 is that it further includes: a component for identifying a person who has made a suspicious gesture or abnormal behavior. Fig.10 As shown, the suspicious gesture detection module 10 may further include a tracking frame determination unit 14, and the abnormal behavior detection module 20 may further include a suspicious person tracking unit.

[0125] In the second embodiment, the suspicious gesture detection unit 12 further detects a suspicious person who makes a suspicious gesture using a suspicious gesture detection model.

[0126] The tracking frame determination unit 14 determines a suspicious person tracking frame based on the suspicious person detected by the suspicious posture detection unit 12. That is, the tracking frame determination unit 14 may execute the process of step S15 in the first embodiment.

[0127] The suspicious person tracking unit 24 tracks the suspicious person tracking frame using a tracking algorithm for the image of a predetermined frame. That is, the tracking frame determination unit 24 may perform the process of step S25 in the first embodiment.

[0128] At this time, the result output unit 23 also outputs the image of the suspicious person tracking frame.

[0129] In the second embodiment, not only can abnormal behavior be detected quickly and accurately as in the first embodiment, but also the person who has performed the abnormal behavior can be identified.

[0130] In the second embodiment, the details of the processing performed by the frame image extraction unit 11, the suspicious posture detection unit 12, the suspicious posture detection frame determination unit 13, the tracking frame determination unit 14, the predetermined frame image interception unit 21, the abnormal behavior judgment unit 22, the result output unit 23, and the suspicious person tracking unit 24 can refer to the relevant records in the first embodiment.

[0131] As described above, in the present embodiment, the above-mentioned suspicious posture detection module 10, abnormal behavior detection module 20, one-frame image extraction unit 11, suspicious posture detection unit 12, suspicious posture detection frame determination unit 13, tracking frame determination unit 14, predetermined frame image interception unit 21, abnormal behavior judgment unit 22, result output unit 23, and suspicious person tracking unit 24 can be implemented by a processor executing an abnormal behavior detection program stored in a memory.

[0132] Although the implementation modes and specific examples of the present invention are described above in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the claims.

Claims

1. An abnormal behavior detection device, comprising: A suspicious gesture detection module detects a suspicious gesture for an image frame in a video, determines a suspicious gesture detection frame according to the suspicious gesture, detects a suspicious person who makes the suspicious gesture, and determines a suspicious person tracking frame according to the detected suspicious person; as well as The abnormal behavior detection module captures the image in the suspicious gesture detection frame of the continuous predetermined frames before and after the one frame from the video, determines whether it is an abnormal behavior based on the captured image of the predetermined frame, tracks the suspicious person tracking frame using a tracking algorithm for the image of the predetermined frame, and outputs the type of abnormal behavior, the image of the suspicious gesture detection frame, and the image of the suspicious person tracking frame. The image of the suspicious person tracking frame includes a moving image constituted by the image of the suspicious person tracking frame in each of the predetermined frames.

2. The abnormal behavior detection device according to claim 1, wherein: The suspicious gesture detection module detects suspicious gestures and suspicious persons making the suspicious gestures using a suspicious gesture detection model, wherein the suspicious gesture detection model is a target detection convolutional neural network model. The abnormal behavior detection module uses an abnormal behavior judgment model to judge whether it is an abnormal behavior, and the abnormal behavior judgment model is a 3D convolutional neural network model.

3. The abnormal behavior detection device according to claim 2, wherein: The suspicious gesture detection module comprises: A frame image extraction unit extracts the image of the frame from the video; A suspicious gesture detection unit, inputting the image of the one frame into a suspicious gesture detection model to determine whether it is a suspicious gesture; a suspicious gesture detection frame determining unit, which determines a suspicious gesture detection frame in the image when the suspicious gesture detecting unit detects a suspicious gesture; and The tracking frame determination unit determines a suspicious person tracking frame based on the detected suspicious person.

4. The abnormal behavior detection device according to claim 3, wherein: The suspicious posture detection model is a YOLOV3 target detection network model.

5. The abnormal behavior detection device according to claim 2, wherein: The abnormal behavior detection module includes: a predetermined frame image intercepting unit, which intercepts the image of the predetermined frame; and An abnormal behavior judgment unit inputs the image of the predetermined frame into an abnormal behavior judgment model to judge whether it is an abnormal behavior; A suspicious person tracking unit, for an image of a predetermined frame, tracks the suspicious person tracking frame using a tracking algorithm; and The result output unit outputs the type of abnormal behavior, the image of the suspicious gesture detection frame, and the image of the suspicious person tracking frame as a result.

6. The abnormal behavior detection device according to claim 5, wherein: The abnormal behavior judgment model is a 3D neural network model based on the vgg16 framework.

7. The abnormal behavior detection device according to claim 5, wherein: Importing the labeled suspicious posture training data into the suspicious posture detection model to train the suspicious posture detection model; Use the trained suspicious posture detection model to detect the collected video data and generate abnormal behavior training data; as well as The abnormal behavior training data is used to train the abnormal behavior judgment model.

8. A method for detecting abnormal behavior, comprising: A suspicious gesture detection step, for an image of a frame in the video, detects a suspicious gesture, determines a suspicious gesture detection frame according to the suspicious gesture, detects a suspicious person who makes the suspicious gesture, and determines a suspicious person tracking frame according to the detected suspicious person; as well as The abnormal behavior detection step includes capturing images in the suspicious gesture detection frame of consecutive predetermined frames before and after the one frame from the video, determining whether it is an abnormal behavior based on the captured images of the predetermined frames, tracking the suspicious person tracking frame using a tracking algorithm for the images of the predetermined frames, and outputting the type of the abnormal behavior, the image of the suspicious gesture detection frame, and the image of the suspicious person tracking frame. The image of the suspicious person tracking frame includes a moving image constituted by the image of the suspicious person tracking frame in each of the predetermined frames.

9. The abnormal behavior detection method according to claim 8, wherein: In the suspicious gesture detection step, a suspicious gesture detection model is used to detect suspicious gestures and suspicious persons making the suspicious gestures, wherein the suspicious gesture detection model is a target detection convolutional neural network model. In the abnormal behavior detection step, an abnormal behavior judgment model is used to judge whether it is an abnormal behavior, and the abnormal behavior judgment model is a 3D convolutional neural network model.

10. The abnormal behavior detection method according to claim 9, wherein: The suspicious gesture detection step comprises: Extracting the image of the one frame from the video; Inputting the image of the one frame into a suspicious gesture detection model to determine whether it is a suspicious gesture; When a suspicious gesture is detected in the suspicious gesture detection step, determining a suspicious gesture detection frame in the image; and A suspicious person tracking frame is determined based on the detected suspicious person.

11. The abnormal behavior detection method according to claim 10, wherein: The suspicious posture detection model is a YOLOV3 target detection network model.

12. The abnormal behavior detection method according to claim 9, wherein: The abnormal behavior detection step comprises: capturing an image of the predetermined frame; and Inputting the image of the predetermined frame into an abnormal behavior judgment model to judge whether it is an abnormal behavior; Tracking the suspicious person tracking frame using a tracking algorithm for the image of a predetermined frame; and The result output step outputs the type of abnormal behavior, the image of the suspicious gesture detection frame, and the image of the suspicious person tracking frame as a result.

13. The abnormal behavior detection method according to claim 12, wherein: The abnormal behavior judgment model is a 3D neural network model based on the vgg16 framework.

14. The abnormal behavior detection method according to claim 12, wherein: Importing the labeled suspicious posture training data into the suspicious posture detection model to train the suspicious posture detection model; Use the trained suspicious posture detection model to detect the collected video data and generate abnormal behavior training data; as well as The abnormal behavior training data is used to train the abnormal behavior judgment model.

15. An abnormal behavior detection device, comprising: Memory: and The processor executes the program stored in the memory so that the computer performs the following processing: For an image of a frame in a video, a suspicious gesture is detected, a suspicious gesture detection frame is determined according to the suspicious gesture, a suspicious person making the suspicious gesture is detected, and a suspicious person tracking frame is determined according to the detected suspicious person; as well as The image in the suspicious gesture detection frame of the continuous predetermined frames before and after the one frame is intercepted from the video, and whether it is an abnormal behavior is determined according to the intercepted image of the predetermined frame, and the suspicious person tracking frame is tracked by using a tracking algorithm for the image of the predetermined frame, and the type of the abnormal behavior, the image of the suspicious gesture detection frame, and the image of the suspicious person tracking frame are output. The image of the suspicious person tracking frame includes a moving image constituted by the image of the suspicious person tracking frame in each of the predetermined frames.

16. A computer-readable storage medium storing an abnormal behavior detection program, wherein the abnormal behavior detection program causes a computer to perform the following processing: For an image of a frame in the video, a suspicious gesture is detected, a suspicious gesture detection frame is determined according to the suspicious gesture, a suspicious person making the suspicious gesture is detected, and a suspicious person tracking frame is determined according to the detected suspicious person; and The image in the suspicious gesture detection frame of the continuous predetermined frames before and after the one frame is intercepted from the video, and whether it is an abnormal behavior is determined according to the intercepted image of the predetermined frame, and the suspicious person tracking frame is tracked by using a tracking algorithm for the image of the predetermined frame, and the type of the abnormal behavior, the image of the suspicious gesture detection frame, and the image of the suspicious person tracking frame are output. The image of the suspicious person tracking frame includes a moving image constituted by the image of the suspicious person tracking frame in each of the predetermined frames.

Images