Abnormal behavior detection method, device, equipment and medium

Abstract

The present disclosure provides a method for detecting abnormal behavior, which can be applied to the fields of artificial intelligence and financial technology. The abnormal behavior detection method includes: obtaining a target surveillance video to be detected, wherein the target surveillance video includes multiple video clips; extracting behavioral features of the video clips from the target surveillance video; inputting the behavioral features of the video clips into an abnormal behavior detection model, and outputting abnormal values ​​of the video clips; wherein the abnormal behavior detection model is trained based on a target loss function, and the target loss function is constructed based on multiple abnormal value parameters; and based on the abnormal values ​​of the multiple video clips, obtaining a detection result for the target surveillance video. The present disclosure also provides an abnormal behavior detection device, equipment, storage medium, and program product.

Description

Technical Field

[0001] The present disclosure relates to the fields of artificial intelligence and financial technology, and in particular to abnormal behavior detection methods, devices, equipment, media, and program products. Background Art

[0002] In related technologies, video can be used to detect abnormal behavior. When abnormal behavior is detected, an immediate warning can be issued, and security personnel can be notified to intervene and handle the abnormal situation in a timely manner, reducing the possibility of the abnormal incident escalating.

[0003] During the implementation of the present disclosure, it was found that abnormal behavior detection still has problems such as complicated detection process and low detection accuracy. Summary of the Invention

[0004] In view of the above problems, the present disclosure provides an abnormal behavior detection method, abnormal behavior detection apparatus, device, medium and program product.

[0005] According to a first aspect of the present disclosure, a method for detecting abnormal behavior is provided, comprising:

[0006] Acquire a target surveillance video to be detected, wherein the target surveillance video includes multiple video clips;

[0007] Extracting behavioral features of video clips from target surveillance videos;

[0008] Inputting behavioral features of the video clip into an abnormal behavior detection model and outputting an abnormal value of the video clip; wherein the abnormal behavior detection model is trained based on a target loss function, and the target loss function is constructed based on multiple abnormal value parameters;

[0009] Based on the outliers of multiple video clips, the detection results of the target surveillance video are obtained.

[0010] According to an embodiment of the present disclosure, the outlier parameters include: the maximum outlier parameter of the normal video clip, the first-order outlier parameter of the abnormal video clip, the second-order outlier parameter of the abnormal video clip, the third-order outlier parameter of the abnormal video clip, and the minimum outlier parameter of the abnormal video clip.

[0011] According to an embodiment of the present disclosure, the target loss function is constructed based on multiple outlier parameters and includes:

[0012] Constructing a first sub-loss function according to the maximum outlier parameter of the normal video segment and the first order outlier parameter of the abnormal video segment;

[0013] Constructing a second sub-loss function according to the first order outlier parameter of the abnormal video clip and the minimum outlier parameter of the abnormal video clip;

[0014] Constructing a third sub-loss function based on the maximum outlier parameter of the normal video clip and the second-order outlier parameter of the abnormal video clip;

[0015] Constructing a fourth sub-loss function based on the maximum outlier parameter of the normal video clip and the third-order outlier parameter of the abnormal video clip;

[0016] The first sub-loss function, the second sub-loss function, the third sub-loss function and the fourth sub-loss function are combined to construct the target loss function.

[0017] According to an embodiment of the present disclosure, the target loss function is constructed based on multiple outlier parameters and further includes:

[0018] Construct a sparsity-constrained loss function based on the outlier parameters of normal video clips;

[0019] According to the difference between the outlier parameter of the i-th abnormal video clip and the outlier parameter of the i+1-th abnormal video clip, a temporal smoothness constraint loss function is constructed, where the i-th abnormal video clip and the i+1-th abnormal video clip are adjacent in time sequence, and i is a positive integer;

[0020] The first sub-loss function, the second sub-loss function, the third sub-loss function, the fourth sub-loss function, the sparsity constraint loss function and the time smoothness constraint loss function are combined to construct the target loss function.

[0021] According to an embodiment of the present disclosure, extracting behavioral features of a video clip from a target surveillance video includes:

[0022] A deep convolutional neural network with a spatiotemporal 3D kernel is used to extract behavioral features from video clips in surveillance videos.

[0023] According to an embodiment of the present disclosure, before obtaining the target surveillance video to be detected, the method further includes:

[0024] Obtain the initial surveillance video to be detected;

[0025] The initial surveillance video is divided into segments to obtain the target surveillance video.

[0026] According to an embodiment of the present disclosure, the abnormal behavior detection model is trained based on the target loss function and includes:

[0027] Obtaining a training data set, wherein the training data set includes a plurality of video clip samples, and at least one of the plurality of video clip samples includes abnormal behavior;

[0028] Extracting behavioral features of video clip samples from the training dataset;

[0029] The behavioral features of the video clip samples are input into the classifier, and the abnormal value of the video clip samples is output;

[0030] The parameters of the classifier are adjusted based on the outliers and the target loss function, and the adjusted classifier is used as the abnormal behavior detection model.

[0031] A second aspect of the present disclosure provides an abnormal behavior detection device, comprising:

[0032] An acquisition module is used to acquire a target surveillance video to be detected, wherein the target surveillance video includes multiple video clips;

[0033] A feature extraction module is used to extract behavioral features of video clips from target surveillance videos;

[0034] a classification module, configured to input the behavioral features of the video clip into an abnormal behavior detection model and output an abnormal value of the video clip; wherein the abnormal behavior detection model is trained based on a target loss function, and the target loss function is constructed based on multiple abnormal value parameters; and

[0035] The result determination module is used to obtain the detection result of the target monitoring video based on the abnormal values ​​of multiple video clips.

[0036] A third aspect of the present disclosure provides an electronic device, comprising: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the above-mentioned abnormal behavior detection method.

[0037] A fourth aspect of the present disclosure further provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to execute the above-mentioned abnormal behavior detection method.

[0038] A fifth aspect of the present disclosure further provides a computer program product, comprising a computer program, which implements the above-mentioned abnormal behavior detection method when executed by a processor.

[0039] According to an embodiment of the present disclosure, after extracting the behavioral features of a video clip, the behavioral features are detected by an abnormal behavior detection model, and the abnormal value of the video clip can be output. The detection result of the video can be obtained based on the abnormal values ​​of multiple video clips. By treating abnormal behavior detection as a binary classification problem, that is, detecting whether abnormal behavior has occurred in a video, the problem of complicated detection process can be reduced. The embodiment of the present disclosure constructs a target loss function based on multiple outlier value parameters, and the abnormal behavior detection model trained using the target loss function has high detection accuracy. The embodiment of the present disclosure can detect abnormal events in a timely manner when an abnormal event occurs, thereby effectively reducing the economic losses caused by abnormal events. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The above contents and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0041] Figure 1 Schematically illustrates an application scenario diagram of the abnormal behavior detection method, apparatus, device, medium, and program product according to an embodiment of the present disclosure;

[0042] Figure 2 The following schematically shows a flow chart of an abnormal behavior detection method according to an embodiment of the present disclosure;

[0043] Figure 3 A flowchart schematically illustrates a method for constructing a target loss function based on multiple outlier parameters according to an embodiment of the present disclosure;

[0044] Figure 4 Schematically illustrates a structural diagram of a deep convolutional neural network with a spatiotemporal three-dimensional kernel according to an embodiment of the present disclosure;

[0045] Figure 5 A flowchart of a method for training an abnormal behavior detection model based on a target loss function according to an embodiment of the present disclosure is schematically shown;

[0046] Figure 6 Schematically shows a structural block diagram of an abnormal behavior detection device according to an embodiment of the present disclosure; and

[0047] Figure 7 A block diagram of an electronic device suitable for implementing the abnormal behavior detection method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0048] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present disclosure. In the detailed description below, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0049] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise," "include," etc. used herein indicate the presence of the features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0050] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0051] When expressions such as "at least one of A, B and C, etc." are used, they should generally be interpreted in accordance with the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0052] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved comply with the provisions of relevant laws and regulations, take necessary confidentiality measures, and do not violate public order and good morals.

[0053] In the technical solution of the embodiment of the present disclosure, the user's authorization or consent is obtained before obtaining or collecting the user's personal information.

[0054] During the implementation of the present disclosure, it was discovered that abnormal behavior detection can primarily detect abnormal behaviors such as falls and collisions. The Surendra moving object detection algorithm, based on inter-frame-background differences, was used to detect moving targets in the ATM self-service hall and extract foreground targets. The CanShift tracking algorithm with Kalman filtering was then used to track and locate the moving targets. Feature information was then extracted from the moving targets in the current frame, and a behavior detector was established using an SVM (support vector machine) for classification and judgment. The modeling process is complex, requiring multiple steps to extract valid motion feature information. Undefined abnormal behaviors cannot be effectively identified, and the algorithm lacks generalization capabilities. Furthermore, the algorithm requires manual definition of possible abnormal behaviors and pre-training, making it ineffective in responding to unexpected abnormal events and exhibiting certain limitations. Furthermore, abnormal behavior detection still suffers from problems such as a complex detection process and low detection accuracy.

[0055] An embodiment of the present disclosure provides an abnormal behavior detection method, comprising: obtaining a target surveillance video to be detected, wherein the target surveillance video includes multiple video clips; extracting behavioral features of the video clips from the target surveillance video; inputting the behavioral features of the video clips into an abnormal behavior detection model, and outputting abnormal values ​​of the video clips; wherein the abnormal behavior detection model is trained based on a target loss function, and the target loss function is constructed based on multiple abnormal value parameters; and obtaining a detection result of the target surveillance video based on the abnormal values ​​of the multiple video clips.

[0056] Figure 1 The application scenario diagram of the abnormal behavior detection method, apparatus, device, medium and program product according to the embodiments of the present disclosure is schematically shown.

[0057] like Figure 1 As shown, the application scenario 100 according to this embodiment may include terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is used as a medium for providing a communication link between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired or wireless communication links or optical fiber cables.

[0058] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only as examples).

[0059] The terminal devices 101 , 102 , and 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.

[0060] The server 105 may be a server that provides various services, such as a background management server (for example only) that supports websites browsed by users using the terminal devices 101, 102, and 103. The background management server may analyze and process received data such as user requests, and feed back processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0061] It should be noted that the abnormal behavior detection method provided in the embodiments of the present disclosure can generally be executed by the server 105. Accordingly, the abnormal behavior detection apparatus provided in the embodiments of the present disclosure can generally be set in the server 105. The abnormal behavior detection method provided in the embodiments of the present disclosure can also be executed by a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105. Accordingly, the abnormal behavior detection apparatus provided in the embodiments of the present disclosure can also be set in a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105.

[0062] The abnormal behavior detection method provided in the embodiments of the present disclosure may also be executed by terminal devices 101, 102, and 103. Accordingly, the abnormal behavior detection apparatus provided in the embodiments of the present disclosure may also generally be disposed in terminal devices 101, 102, and 103. The abnormal behavior detection method provided in the embodiments of the present disclosure may also be executed by other terminals different from terminal devices 101, 102, and 103. Accordingly, the abnormal behavior detection apparatus provided in the embodiments of the present disclosure may also be disposed in other terminals different from terminal devices 101, 102, and 103.

[0063] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.

[0064] The following will be based on Figure 1 The scene described by Figures 2 to 5 The abnormal behavior detection method of the embodiment of the present disclosure is described in detail.

[0065] Figure 2 The flowchart of the abnormal behavior detection method according to the embodiment of the present disclosure is schematically shown.

[0066] like Figure 2As shown, the abnormal behavior detection method 200 of this embodiment includes operations S201 to S204.

[0067] In operation S201 , a target surveillance video to be detected is acquired, where the target surveillance video includes multiple video clips.

[0068] According to an embodiment of the present disclosure, the initial surveillance video to be detected can be divided into multiple video segments according to a preset number of frames to obtain the target surveillance video to be detected. The preset number of frames can be, for example, 32 frames.

[0069] In operation S202 , behavioral features of a video segment are extracted from a target surveillance video.

[0070] According to an embodiment of the present disclosure, a pre-trained 3D ResNet network can be used to extract behavioral features of a video clip from a target surveillance video, such as the dynamic features of a person in a video clip.

[0071] In operation S203, the behavior features of the video clip are input into an abnormal behavior detection model, and an abnormal value of the video clip is output; wherein the abnormal behavior detection model is trained based on a target loss function, and the target loss function is constructed according to multiple abnormal value parameters.

[0072] According to embodiments of the present disclosure, a target loss function can be constructed based on multiple outlier parameters. Deep multiple instance learning (MIL) is used to train the network in a weakly supervised manner. In the absence of frame-level or temporal-level labels, MIL can train the network using the target loss function by providing video-level labels, resulting in an abnormal behavior detection model.

[0073] It should be noted that in multiple instance learning (MIL), the input is a series of labeled "packages", each of which contains multiple "examples". In the embodiments of the present disclosure, it can be regarded as a binary classification problem, that is, there are positive packages and negative packages. A positive package is defined as one in which there are at least one or more positive examples, and a negative package is defined as one in which all the examples are negative examples. A surveillance video can be regarded as a package, and the multiple video clips divided from the surveillance video are examples. If a surveillance video contains one or more abnormal video clips, the surveillance video is regarded as a positive package, and the abnormal video clips are positive examples. If the surveillance video is composed entirely of normal video clips, it is a negative package, and the normal video clips are negative examples.

[0074] In operation S204 , a detection result of the target surveillance video is obtained based on the outliers of the plurality of video clips.

[0075] According to embodiments of the present disclosure, the abnormality value of a video clip can be compared with a preset threshold. If the abnormality value exceeds the preset threshold, the video clip is considered abnormal, and the detection result of the target surveillance video can be considered to indicate that an abnormal event has occurred. If the abnormality value is less than the preset threshold, the video clip is considered normal, and the detection result of the target surveillance video can be considered to indicate that no abnormal event has occurred. The preset threshold can be manually determined based on the actual model.

[0076] For example, if the preset threshold is 4, then scores below 4 may be considered normal events, and scores above 4 may be considered abnormal events, and the detection results of the target surveillance video can be obtained accordingly.

[0077] According to an embodiment of the present disclosure, after extracting the behavioral features of a video clip, the behavioral features are detected by an abnormal behavior detection model, and the abnormal value of the video clip can be output. The detection result of the video can be obtained based on the abnormal values ​​of multiple video clips. By treating abnormal behavior detection as a binary classification problem, that is, detecting whether abnormal behavior has occurred in a video, the problem of complicated detection process can be reduced. The embodiment of the present disclosure constructs a target loss function based on multiple outlier value parameters, and the abnormal behavior detection model trained using the target loss function has high detection accuracy. The embodiment of the present disclosure can detect abnormal events in a timely manner when an abnormal event occurs, thereby effectively reducing the economic losses caused by abnormal events.

[0078] Figure 3 A flowchart of a method for constructing a target loss function based on multiple outlier parameters according to an embodiment of the present disclosure is schematically shown.

[0079] like Figure 3 As shown, the method 300 of constructing the target loss function according to the multiple outlier parameters in this embodiment includes operations S301 to S305. It should be noted that the outlier parameters may include: the maximum outlier parameter of the normal video segment, the first order outlier parameter of the abnormal video segment, the second order outlier parameter of the abnormal video segment, the third order outlier parameter of the abnormal video segment, and the minimum outlier parameter of the abnormal video segment.

[0080] In operation S301 , a first sub-loss function is constructed according to a maximum outlier parameter of a normal video segment and a first-order outlier parameter of an abnormal video segment.

[0081] According to the embodiment of the present disclosure, the abnormal behavior detection problem can be treated as a regression problem, so all abnormal video clips can be assigned a higher abnormality value than normal video clips. For example, the ranking function relationship shown in the following formula (1) can be obtained:

[0082] S(I a )>S(I n) (1)

[0083] Among them, I a and I n Represent abnormal video clips and normal video clips respectively; function S(I a ) and S(I n ) represent the abnormal value of abnormal video clips and the abnormal value of normal video clips respectively; the abnormal value can be between 0 and 1.

[0084] It should be noted that there may be two false situations in the abnormal behavior detection task: ① predicting normal video clips as abnormal video clips, that is, false abnormal video clip situation; ② predicting abnormal video clips as normal video clips, that is, false normal video clip situation. In order to reduce various error rates and complete the abnormal behavior detection task, the following sorting conditions can be proposed as shown in formulas (2) and (3):

[0085]

[0086]

[0087] Among them, C a and C n represent the abnormal video clip group and the normal video clip group respectively; formula (2) compares the rankings of the video clips with the largest abnormal values ​​in the abnormal video clip group and the normal video clip group, where the maximum-ranked video clip in the abnormal video clip group is most likely to be a true abnormal video clip, while the maximum-ranked video clip in the normal video clip group may be a false abnormal video clip; formula (3) compares the maximum-ranked video clip and the minimum-ranked video clip in the abnormal video clip group, where the maximum-ranked video clip in the abnormal video clip group is most likely to be a true abnormal video clip, and the minimum-ranked video clip may be a false abnormal video clip.

[0088] Therefore, the outlier values ​​of the video clips in the abnormal video clip group can be arranged in descending order, as shown in formula (4):

[0089]

[0090] Among them, M n Indicates the nth order abnormal value of the abnormal video clip; n is a positive integer and n also represents the total number of abnormal video clips.

[0091] It should be noted that since the video scale of the training dataset is large and there may be multiple video clips of abnormal behavior in the video, it is necessary to maximize the outlier values ​​of other video clips in the abnormal video clip group to avoid false situations②; it is also necessary to minimize the outlier values ​​of all video clips in the normal video clip group.

[0092] Therefore, in order to avoid the occurrence of the above two false situations, for example, formula (5) and formula (6) can be used to respectively compare the second-ranked and third-ranked video clips of the abnormal video clip group with the highest-ranked video clip of the abnormal video clip group.

[0093] As shown below:

[0094]

[0095]

[0096] According to the embodiment of the present disclosure, the first sub-loss function l1(C a ,C n ) can be expressed as shown in formula (7):

[0097]

[0098] In operation S302 , a second sub-loss function is constructed according to the first order outlier parameter of the abnormal video segment and the minimum outlier parameter of the abnormal video segment.

[0099] According to the embodiment of the present disclosure, the second sub-loss function l2(C a ,C n ) can be expressed as shown in formula (8):

[0100]

[0101] In operation S303 , a third sub-loss function is constructed according to the maximum outlier parameter of the normal video segment and the second-order outlier parameter of the abnormal video segment.

[0102] According to the embodiment of the present disclosure, the third sub-loss function l3 (C a ,C n ) can be expressed as shown in formula (9):

[0103]

[0104] In operation S304 , a fourth sub-loss function is constructed according to the maximum outlier parameter of the normal video segment and the third-order outlier parameter of the abnormal video segment.

[0105] According to the embodiment of the present disclosure, the fourth sub-loss function l4 (C a ,C n ) can be expressed as shown in formula (10):

[0106]

[0107] In operation S305 , the first sub-loss function, the second sub-loss function, the third sub-loss function, and the fourth sub-loss function are combined to construct a target loss function.

[0108] According to an embodiment of the present disclosure, the objective loss function L(w) can be expressed as shown in Formula (11):

[0109] L(w)=l(C a ,C n )+μ1||w1|| 2 (11)

[0110] Among them, l(C a ,C n )=l1(C a ,C n )+l2(C a ,C n )+l3(C a ,C n )+l4(C a ,C n ); w1 represents the weight of the abnormal behavior detection model; μ1 represents the hyperparameter of the abnormal behavior detection model.

[0111] According to the embodiment of the present disclosure, when constructing the target loss function, the two situations of false abnormal video clips and false normal video clips are fully considered to avoid the occurrence of these two false situations, and all truly abnormal video clips are kept as far away from all truly normal video clips as possible. The model trained in this way can improve the accuracy of the model.

[0112] According to another embodiment of the present disclosure, the method for constructing the target loss function based on multiple outlier parameters may further consider sparsity constraints and time smoothness constraints in addition to the above operations S301 to S305, including operations S1 to S3.

[0113] In operation S1, a sparsity-constrained loss function is constructed based on outlier parameters of normal video clips.

[0114] According to an embodiment of the present disclosure, the sparsity constraint loss function can be expressed as shown in formula (12):

[0115]

[0116] Among them, μ2 represents the hyperparameter of the abnormal behavior detection model.

[0117] In operation S2, a temporal smoothness constraint loss function is constructed based on the difference between the outlier parameter of the i-th abnormal video clip and the outlier parameter of the i+1-th abnormal video clip, where the i-th abnormal video clip and the i+1-th abnormal video clip are adjacent in time sequence and i is a positive integer.

[0118] According to an embodiment of the present disclosure, the temporal smoothness constraint loss function can be expressed as shown in Formula (13):

[0119]

[0120] Among them, μ3 represents the hyperparameter of the abnormal behavior detection model.

[0121] In operation S3, the first sub-loss function, the second sub-loss function, the third sub-loss function, the fourth sub-loss function, the sparsity constraint loss function, and the time smoothness constraint loss function are combined to construct a target loss function.

[0122] According to an embodiment of the present disclosure, the objective loss function L(w)′ can be expressed as shown in Formula (14):

[0123] L(w)′=l(C a ,C n )′+μ4||w2|| 2 (14)

[0124] Among them, l(C a ,C n )′=l1(C a ,C n )+l2(C a ,C n )+l3(C a ,C n )+l4(C a ,C n )+sparsity constraint +temporal constraint ; w2 represents the weight of the abnormal behavior detection model; μ4 represents the hyperparameter of the abnormal behavior detection model.

[0125] According to the embodiments of the present disclosure, in addition to considering the two situations of false abnormal video clips and false normal video clips when constructing the target loss function, the temporal smoothness and sparsity of the abnormal values ​​of the video clips are also considered. The trained model has high accuracy, which at least partially solves the problem of low detection accuracy of abnormal behavior detection models in the prior art.

[0126] According to an embodiment of the present disclosure, before obtaining the target surveillance video to be detected, the method further includes:

[0127] Obtain the initial surveillance video to be detected;

[0128] The initial surveillance video is divided into segments to obtain the target surveillance video.

[0129] According to an embodiment of the present disclosure, an initial surveillance video may be acquired by a video acquisition device, and then the initial surveillance video may be divided into multiple video segments according to time periods or a preset number of video frames to obtain a target surveillance video.

[0130] Figure 4 The schematic diagram shows the structure of a deep convolutional neural network with a spatiotemporal three-dimensional kernel according to an embodiment of the present disclosure.

[0131] According to an embodiment of the present disclosure, extracting behavioral features of a video clip from a target surveillance video includes:

[0132] A deep convolutional neural network with a spatiotemporal 3D kernel is used to extract behavioral features from video clips in surveillance videos.

[0133] The method of extracting behavioral features of video clips from target surveillance videos in this embodiment can utilize a deep convolutional neural network (3D ResNet network) with a spatiotemporal three-dimensional kernel. Compared with a simple 3D CNN network, it has an additional residual structure. When a traditional 3D CNN performs multiple convolutions on an image, the deeper layers will cause the loss of image features from the earlier layers. Moreover, as the degree of data duplication increases, simply increasing the network depth of the 3D CNN will also cause model degradation.

[0134] like Figure 4 As shown in the figure, the residual network also borrows the cross-layer connection idea of ​​the high-speed network, but the difference is that the residual term originally has a weight, but in the residual network it is replaced by an identity mapping. Among them, the activation function ReLU is used as the pre-activation processing of the weight layer. Assuming that the network input is x and the expected output is H(x), which is the expected complex potential mapping, Figure 4The shortcut connections in the residual network directly transfer x to the output as the initial result, resulting in H(x) = F(x) + x. When F(x) = 0, H(x) = x, which is an identity mapping. Residual networks no longer target a complete output, but instead target the difference between the target value H(x) and x. Therefore, the subsequent training goal is to bring the residual result close to 0. This ensures that accuracy does not decrease even as the network becomes deeper. Compared to traditional neural networks, where the output of the previous layer only serves as input to that layer, this architecture can address gradient vanishing and model degradation while increasing network depth. This architecture not only optimizes network speed but also improves parameter utilization and reduces model size. Shortcut connections transfer high-dimensional features extracted by shallow layers to deeper layers for feature reuse, effectively extracting deep features from images while preventing gradient vanishing. This allows for more efficient feature extraction and enhances the network's recognition capabilities.

[0135] According to an embodiment of the present disclosure, a pre-trained deep convolutional neural network with a spatiotemporal three-dimensional kernel is used to extract behavioral features from video clips in surveillance videos. The pre-trained deep convolutional neural network with a spatiotemporal three-dimensional kernel can be pre-trained on the UCF-101 dataset.

[0136] Figure 5 The flowchart of the method for training an abnormal behavior detection model based on a target loss function according to an embodiment of the present disclosure is schematically shown.

[0137] like Figure 5 As shown, the method 500 of obtaining the abnormal behavior detection model based on target loss function training in this embodiment includes operations S501 to S504.

[0138] In operation S501, a training dataset is obtained, wherein the training dataset includes a plurality of video clip samples, at least one of which includes abnormal behavior. The training dataset can be divided into a normal video clip group and an abnormal video clip group, wherein at least one of the abnormal video clip samples includes abnormal behavior.

[0139] According to an embodiment of the present disclosure, a training dataset may be constructed using indoor surveillance video data in a real surveillance video dataset UCF-CRIME dataset.

[0140] In operation S502 , behavioral features of video clip samples are extracted from a training dataset.

[0141] According to embodiments of the present disclosure, 3D ResNet can be used to extract behavioral features of video clip samples in a training dataset. The behavioral features include behavioral features within normal and abnormal video clips. For example, the behavioral features can include dynamic features of characters within the video clips.

[0142] In operation S503 , the behavior feature of the video clip sample is input into a classifier, and an abnormal value of the video clip sample is output.

[0143] According to an embodiment of the present disclosure, the classifier may be a multi-instance learning model. As described in the above operation S501, the training data set has grouped all video clips, namely, the normal video clip group C a and abnormal video clip group C n Although at least one video clip sample in the abnormal video clip group includes abnormal behavior, the exact label of the video clip is unknown. The following optimization function (15) can be used to perform the binary classification task and output the abnormal value of the video clip sample.

[0144]

[0145] in, represents the hinge loss function, Indicates normal video clip group C a and abnormal video clip group C n , φ(x) represents the behavioral feature representation of the video clip sample, b represents the bias term, m represents the total number of video clip samples in the training set, and w represents the weight of the multi-instance learning model.

[0146] In operation S504, the parameters of the classifier are adjusted based on the outliers and the target loss function, and the adjusted classifier is used as an abnormal behavior detection model.

[0147] According to an embodiment of the present disclosure, the parameters of the classifier can be adjusted according to the abnormal value of the video clip sample output in operation S503 and the target loss function obtained through the above-mentioned embodiment of the present disclosure, and the classifier after parameter adjustment can be used as an abnormal behavior detection model.

[0148] According to the embodiments of the present disclosure, abnormal behavior detection is performed by extracting behavioral features from video clips. This eliminates the need to individually model and detect targets in surveillance videos, simplifying the entire process, reducing computational overhead, and making it more suitable for lightweight, low-computing detection equipment. Furthermore, the abnormal behavior detection model trained using the embodiments of the present disclosure exhibits strong generalization capabilities and high accuracy. It can be applied to detect abnormal events within ATM self-service halls, enabling timely and effective detection and reducing the economic losses caused by these events.

[0149] Based on the above abnormal behavior detection method, the present disclosure also provides an abnormal behavior detection device. Figure 6 The device is described in detail.

[0150] Figure 6 The following schematically shows a structural block diagram of an abnormal behavior detection device according to an embodiment of the present disclosure.

[0151] like Figure 6 As shown, the abnormal behavior detection device 600 of this embodiment includes an acquisition module 610 , a feature extraction module 620 , a classification module 630 and a result determination module 640 .

[0152] The acquisition module 610 is used to acquire the target surveillance video to be detected, wherein the target surveillance video includes multiple video clips. In one embodiment, the acquisition module 610 can be used to perform the operation S201 described above, which will not be repeated here.

[0153] The feature extraction module 620 is used to extract the behavioral features of the video clip from the target surveillance video. In one embodiment, the feature extraction module 620 can be used to perform the operation S202 described above, which will not be repeated here.

[0154] Classification module 630 is configured to input the behavioral features of the video clip into an abnormal behavior detection model and output an abnormality value for the video clip. The abnormal behavior detection model is trained based on a target loss function constructed based on multiple outlier value parameters. In one embodiment, classification module 630 can be configured to perform operation S203 described above and will not be further described here.

[0155] The result determination module 640 is used to obtain the detection result of the target monitoring video based on the abnormal values ​​of the multiple video clips. In one embodiment, the result determination module 640 can be used to perform the operation S204 described above, which will not be repeated here.

[0156] According to an embodiment of the present disclosure, any multiple modules among the acquisition module 610, the feature extraction module 620, the classification module 630 and the result determination module 640 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present disclosure, at least one of the acquisition module 610, the feature extraction module 620, the classification module 630 and the result determination module 640 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware and firmware or in an appropriate combination of any of them. Alternatively, at least one of the acquisition module 610 , the feature extraction module 620 , the classification module 630 and the result determination module 640 may be at least partially implemented as a computer program module, which may perform corresponding functions when executed.

[0157] Figure 7 A block diagram of an electronic device suitable for implementing the abnormal behavior detection method according to an embodiment of the present disclosure is schematically shown.

[0158] like Figure 7 As shown, the electronic device 700 according to an embodiment of the present disclosure includes a processor 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage part 708 into a random access memory (RAM) 703. The processor 701 may, for example, include a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 701 may also include an onboard memory for caching purposes. The processor 701 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0159] Various programs and data required for the operation of the electronic device 700 are stored in the RAM 703. The processor 701, ROM 702, and RAM 703 are connected to each other via a bus 704. The processor 701 executes the various operations of the method flow according to the embodiment of the present disclosure by executing the programs in the ROM 702 and / or RAM 703. It should be noted that the programs may also be stored in one or more memories other than the ROM 702 and RAM 703. The processor 701 may also execute the various operations of the method flow according to the embodiment of the present disclosure by executing the programs stored in the one or more memories.

[0160] According to an embodiment of the present disclosure, the electronic device 700 may further include an input / output (I / O) interface 705, which is also connected to the bus 704. The electronic device 700 may further include one or more of the following components connected to the I / O interface 705: an input portion 706 including a keyboard, a mouse, etc.; an output portion 707 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage portion 708 including a hard disk; and a communication portion 709 including a network interface card such as a LAN card or a modem. The communication portion 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed in the drive 710 as needed, so that a computer program read therefrom can be installed into the storage portion 708 as needed.

[0161] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments, or may exist independently and not be incorporated into the device / apparatus / system. The computer-readable storage medium carries one or more programs, and when executed, implements the method according to the embodiments of the present disclosure.

[0162] According to an embodiment of the present disclosure, a computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, it may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, apparatus, or device. For example, according to an embodiment of the present disclosure, a computer-readable storage medium may include the ROM 702 and / or RAM 703 described above and / or one or more memories other than ROM 702 and RAM 703.

[0163] The embodiments of the present disclosure also include a computer program product, which includes a computer program containing program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the method provided by the embodiments of the present disclosure.

[0164] The computer program executes the above functions defined in the system / device of the embodiment of the present disclosure when the processor 701 executes the computer program. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0165] In one embodiment, the computer program may be stored on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may be transmitted and distributed in the form of a signal on a network medium, downloaded and installed via the communication portion 709, and / or installed from a removable medium 711. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to wireless, wired, or any suitable combination thereof.

[0166] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 709, and / or installed from a removable medium 711. When the computer program is executed by the processor 701, the above-described functions defined in the system of the embodiment of the present disclosure are performed. According to the embodiment of the present disclosure, the systems, devices, means, modules, units, etc. described above can be implemented by computer program modules.

[0167] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computer programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, python, "C" or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect via the Internet).

[0168] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the above-mentioned module, program segment, or a part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0169] Those skilled in the art will appreciate that the features described in the various embodiments and / or claims of this disclosure may be combined and / or coupled in various ways, even if such combinations and / or couplings are not explicitly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure may be combined and / or coupled in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or couplings are intended to fall within the scope of this disclosure.

[0170] The embodiments of the present disclosure are described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Although each embodiment has been described separately above, this does not mean that the measures in each embodiment cannot be used in combination to advantage. The scope of the present disclosure is defined by the appended claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A method for detecting abnormal behavior, comprising: Acquire a target surveillance video to be detected, wherein the target surveillance video includes multiple video clips; extracting behavioral features of the video clip from the target surveillance video; Inputting the behavioral features of the video clip into an abnormal behavior detection model, and outputting an abnormal value of the video clip; wherein the abnormal behavior detection model is trained based on a target loss function, and the target loss function is constructed according to a plurality of abnormal value parameters, the abnormal value parameters including: a maximum abnormal value parameter of a normal video clip, a first-order abnormal value parameter of an abnormal video clip, a second-order abnormal value parameter of the abnormal video clip, a third-order abnormal value parameter of the abnormal video clip, and a minimum abnormal value parameter of the abnormal video clip, wherein the first-order abnormal value parameter, the second-order abnormal value parameter, and the third-order abnormal value parameter are arranged in descending order according to the abnormal value of the abnormal video clip; Based on the abnormal values ​​of the plurality of video clips, a detection result of the target monitoring video is obtained.

2. The method according to claim 1, wherein The target loss function is constructed based on multiple outlier parameters and includes: constructing a first sub-loss function according to the maximum outlier parameter of the normal video segment and the first order outlier parameter of the abnormal video segment; constructing a second sub-loss function according to the first order outlier parameter of the abnormal video segment and the minimum outlier parameter of the abnormal video segment; constructing a third sub-loss function according to the maximum outlier parameter of the normal video segment and the second-order outlier parameter of the abnormal video segment; constructing a fourth sub-loss function according to the maximum outlier parameter of the normal video segment and the third-order outlier parameter of the abnormal video segment; The first sub-loss function, the second sub-loss function, the third sub-loss function and the fourth sub-loss function are combined to construct the target loss function.

3. The method according to claim 2, wherein: The target loss function is constructed based on multiple outlier parameters and further includes: Construct a sparsity-constrained loss function based on the outlier parameters of normal video clips; Constructing a temporal smoothness constraint loss function according to the difference between the outlier parameter of the i-th abnormal video segment and the outlier parameter of the i+1-th abnormal video segment, wherein the i-th abnormal video segment and the i+1-th abnormal video segment are adjacent in time sequence, and i is a positive integer; The first sub-loss function, the second sub-loss function, the third sub-loss function, the fourth sub-loss function, the sparsity constraint loss function and the time smoothness constraint loss function are combined to construct the target loss function.

4. The method according to claim 1, wherein Extracting the behavioral features of the video clip from the target surveillance video includes: The behavioral features in the video clips in the surveillance video are extracted using a deep convolutional neural network with a spatiotemporal three-dimensional kernel.

5. The method according to claim 1, wherein Before acquiring the target surveillance video to be detected, the method further includes: Obtain the initial surveillance video to be detected; The initial surveillance video is divided into segments to obtain the target surveillance video.

6. The method according to claim 1, wherein The abnormal behavior detection model is trained based on the target loss function and includes: Acquire a training data set, wherein the training data set includes a plurality of video clip samples, and at least one of the plurality of video clip samples includes abnormal behavior; extracting behavioral features of the video clip samples from the training data set; Inputting the behavioral features of the video clip sample into a classifier and outputting an abnormal value of the video clip sample; The parameters of the classifier are adjusted based on the outlier value and the target loss function, and the adjusted classifier is used as the abnormal behavior detection model.

7. An abnormal behavior detection device, comprising: An acquisition module is used to acquire a target surveillance video to be detected, wherein the target surveillance video includes multiple video clips; A feature extraction module, configured to extract behavioral features of the video clip from the target surveillance video; a classification module, configured to input the behavioral features of the video clip into an abnormal behavior detection model and output an abnormal value of the video clip; wherein the abnormal behavior detection model is trained based on a target loss function, and the target loss function is constructed based on a plurality of abnormal value parameters, the abnormal value parameters including: a maximum abnormal value parameter of a normal video clip, a first-order abnormal value parameter of an abnormal video clip, a second-order abnormal value parameter of the abnormal video clip, a third-order abnormal value parameter of the abnormal video clip, and a minimum abnormal value parameter of the abnormal video clip, wherein the first-order abnormal value parameter, the second-order abnormal value parameter, and the third-order abnormal value parameter are arranged in descending order according to the abnormal value of the abnormal video clip; and The result determination module is used to obtain the detection result of the target monitoring video based on the abnormal values ​​of the multiple video clips.

8. An electronic device comprising: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors are enabled to execute the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to perform the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images