Weak supervision video abnormal fragment detection method based on corrosion-expansion

Abstract

The invention relates to the technical field of video analysis, in particular to a weak supervision video abnormal fragment detection method based on corrosion-expansion, comprising the following steps: acquiring a plurality of video packets, and establishing a video fragment training set; constructing an abnormal regression network, taking the video clip training set as input of the abnormal regression network, and outputting video clip features through a full connection layer of the abnormal regression network; mapping the abnormal score of the video frame in the video clip through an abnormal regression layer of the abnormal regression network; establishing a typical abnormal example set; establishing a typical normal example set; calculating the contrast loss of the typical normal example set and the typical abnormal example set according to the positive matching pair and the negative matching pair; and constructing an abnormal fragment detection model, and carrying out video abnormal fragment detection by adopting the abnormal fragment detection model. By adopting the scheme, the detection accuracy of the abnormal segment of the weak supervision video can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of video analysis, and in particular to a weakly supervised video abnormal segment detection method based on corrosion-expansion. Background Art

[0002] In the current technological context, obtaining fully annotated video data is usually costly and time-consuming. To solve this problem, weakly supervised video anomaly detection methods have emerged. When using weakly supervised video anomaly detection methods for anomaly detection, only video-level labels (normal video / abnormal video) are required, and deep learning technology can be used to detect abnormal clips in the video to be tested, which effectively reduces the cost of data annotation and improves the efficiency of anomaly detection.

[0003] In the field of weakly supervised video anomaly detection, deep learning technology plays a key role. Specifically, existing weakly supervised video anomaly detection methods usually use convolutional neural networks (CNNs) and recurrent neural networks (RNNs) to map complex video data into feature space, thereby obtaining feature information of the video; then, video anomaly detection is defined as a regression problem, and anomaly scores are provided for samples in the feature space to determine whether a video clip or frame is abnormal. This way of scoring samples in the feature space is usually regarded as multi-instance learning. For example, first extract video features through the C3D network, and output the anomaly score of the video frame with a fully connected network; then, optimize the model based on the Hinge loss of the sample with the highest anomaly score in each video and the video label; after obtaining the anomaly score, optimize the AR layer through a multi-instance ranking loss function, which is the mainstream method of video anomaly detection under weak supervision. However, since the top-k method is prone to misselection, it is easy to introduce errors if only the top-k method is used to discover the top k examples with the anomaly score as typical anomalies.

[0004] It can be seen that the existing methods focus on using the samples with the highest anomaly scores for multi-instance learning, while ignoring the use of limited label information in weakly supervised learning to distinguish the distribution relationship between normal samples and abnormal samples in the feature space, resulting in low detection accuracy of weakly supervised video abnormal segments. Therefore, it is urgent to provide a weakly supervised video abnormal segment detection method based on corrosion-dilation to improve the accuracy of weakly supervised video abnormal segment detection. Summary of the invention

[0005] The present invention provides a weakly supervised video abnormal segment detection method based on erosion-expansion, which can improve the feature distinguishability of normal segments and abnormal segments of a video, can more accurately extract normal segments and abnormal segments in the weakly supervised video abnormal segment detection method, and improve the accuracy of weakly supervised video abnormal segment detection.

[0006] In order to achieve the above objectives, this application provides the following technical solutions:

[0007] A weakly supervised video abnormal segment detection method based on corrosion-expansion includes the following steps:

[0008] S100, obtaining a plurality of video packages, the video packages including a plurality of positive packages and negative packages, the positive packages and negative packages each including a plurality of video clips, the video clips in the positive packages are all positive samples, and the video clips in the negative packages include positive samples and negative samples; integrating the positive samples in the positive packages and the positive samples and negative samples in the negative packages to establish a video clip training set;

[0009] S200, constructing an abnormal regression network, taking the video clip training set as input of the abnormal regression network, and outputting video clip features through a fully connected layer of the abnormal regression network;

[0010] S300, mapping anomaly scores of each video frame in the video clip through an anomaly regression layer of an anomaly regression network according to the video clip features;

[0011] S400, binarizing the anomaly score, performing an erosion-expansion operation on the binarized anomaly score, generating a typical anomaly example, and establishing a collection of typical anomaly examples;

[0012] S500, selecting a plurality of video clips in the video clip training set as typical normal examples, and establishing a collection of typical normal examples;

[0013] S600, performing contrast loss constraints on a typical normal example collection and a typical abnormal example collection; constructing positive matching pairs within each of the typical normal example collection and the typical abnormal example collection, constructing negative matching pairs between the typical normal example collection and the typical abnormal example collection, and calculating contrast losses of the typical normal example collection and the typical abnormal example collection based on the positive matching pairs and the negative matching pairs;

[0014] S700: construct an abnormal segment detection model based on the contrast loss of a typical normal example collection and a typical abnormal example collection, and use the abnormal segment detection model to perform video abnormal segment detection.

[0015] The principles and advantages of the present invention are:

[0016] For weakly supervised video anomaly detection, the present invention proposes a weakly supervised video abnormal segment detection method based on expansion-erosion, obtains several video segments in a video package, establishes a video segment training set, constructs an abnormal regression network, uses the video segment training set as the input of the abnormal regression network, outputs video segment features through the fully connected layer of the abnormal regression network, and maps the abnormal scores of video frames in the video segment through the abnormal regression layer of the abnormal regression network according to the video segment features; screens positive samples and negative samples in several video segment training sets as typical normal examples, and establishes a collection of typical normal examples; binarizes the abnormal scores through an erosion-expansion abnormal segment selection algorithm, which helps to clearly distinguish normal and abnormal states, performs an erosion-expansion operation on the binarized abnormal scores, generates typical abnormal examples, and establishes a collection of typical abnormal examples; selects representative typical abnormal segments through the erosion-expansion algorithm, can improve the positioning ability of abnormal video frames, the erosion-expansion operation helps to reduce false detection and missed detection, improves the accuracy of anomaly detection, and helps to filter out random fluctuations in abnormal scores; thereby completing the establishment of a collection of typical normal examples and a collection of typical abnormal examples.

[0017] Then, the contrast loss constraint is applied to the typical normal example collection and the typical abnormal example collection; positive matching pairs are constructed within each of the typical normal example collection and the typical abnormal example collection, negative matching pairs are constructed between the typical normal example collection and the typical abnormal example collection, and the contrast loss of the typical normal example collection and the typical abnormal example collection is constructed based on the positive matching pairs and the negative matching pairs. In this way, the samples in the same example collection can be more clustered, and the samples in different example collections can be more distinguished. By optimizing the separation of normal samples and abnormal samples through the contrast loss constraint of typical normal examples and typical abnormal examples, the normal state and abnormal state can be distinguished more accurately; the contrast learning loss goal is to expand the inter-class distance of the examples in the bag.

[0018] According to the contrast loss of the typical normal example collection and the typical abnormal example collection, an abnormal segment detection model is constructed, and the dynamic multi-example loss function and sparse constraints are used to optimize the abnormal segment detection model to generate an optimized abnormal segment detection model; the goal of using dynamic multi-example loss is to expand the distance between positive packages and negative packages in the video segment training set, which helps the abnormal segment detection model to better handle the differences between different video packages. Sparse constraints are also introduced to keep the abnormal scores of typical normal examples low. The addition of sparse constraints makes the abnormal scores of typical normal examples as low as possible, which helps to reduce misjudgments and missed judgments; based on The contrast loss of the typical normal example collection and the typical abnormal example collection expands the inter-class distance of examples in the positive bag and the negative bag. The dynamic multi-example loss function is combined to expand the distance between the positive bag and the negative bag in the video clip training set. Sparse constraints are also introduced to keep the typical normal examples at a low abnormality score. This can strengthen the guiding role of abnormal and normal clips during training, making the examples in the positive and negative bags closer to the typical normal samples and abnormal samples in the feature space, effectively improving the accuracy, robustness and generalization ability of anomaly detection, while reducing false positives and false negatives, and improving the training efficiency and real-time processing capability of the abnormal clip detection model.

[0019] Furthermore, S200 includes:

[0020] S201, construct an abnormal regression network, use the rectified linear unit function as the activation function of the fully connected layer in the abnormal regression network, and introduce a Dropout operation in the fully connected layer;

[0021] S202, using the video clip training set as the input of the anomaly regression network, and outputting the video clip features through the fully connected layer of the anomaly regression network

[0022]

[0023] Where D′ represents the Dropout operation; W FC is the first fully connected layer parameter, representing the weight matrix weight of the fully connected layer; b FC is the parameter of the second fully connected layer, indicating the bias of the fully connected layer; R F×F represents a matrix of shape [F,F], R F×1 represents a matrix with a shape of [F,1], where R is a real number;

[0024] X i is the i-th video clip of the input, is the corresponding output video segment feature after inputting the i-th video segment.

[0025] Beneficial effects: The rectified linear unit function (ReLU) is used as the activation function of the fully connected layer in the abnormal regression network. The ReLU activation function can effectively introduce nonlinear characteristics into the neural network through its mathematical expression of f(x) = max(0,x). Compared with other activation functions, the calculation of ReLU is very efficient. It only needs to compare the size of the input value with zero and take the larger value, which greatly reduces the consumption of computing resources and is particularly suitable for the training of large-scale data sets and complex models. In addition, ReLU can effectively alleviate the problem of gradient disappearance. In order to avoid overfitting, the dropout operation (random inactivation: a regularization technology that prevents deep neural networks from overfitting, which increases the generalization ability of the model by randomly shutting down some neurons during training) is introduced in the fully connected layer. By randomly "discarding" the outputs of some neurons during training, overfitting is prevented, the complex co-adaptation relationship between neurons is reduced, and the remaining neurons are forced to work together with other randomly selected neurons, thereby improving the generalization ability of the model. This scheme effectively improves the performance of the anomaly regression network by adopting the ReLU activation function and the Dropout operation, ensuring the accuracy and reliability of the video clip characteristics output by the fully connected layer of the anomaly regression network, thereby further improving the efficiency and accuracy of anomaly detection.

[0026] Further, in S300, the abnormal regression layer of the abnormal regression network adopts a fully connected structure and an S-type logical operation; the calculation formula of the abnormal score is as follows:

[0027]

[0028] In the formula, s i Video clip features The anomaly score of the corresponding video frame; exp represents the exponent; W AR is the first abnormal regression layer parameter, which represents the weight matrix weight of the abnormal regression layer; b AR is the parameter of the second abnormal regression layer, indicating the bias of the abnormal regression layer; R 1×F Represents a matrix with a shape of [1,F]; R 1 Represents a matrix with shape [1]; R is a real number.

[0029] Beneficial effects: S-type logical operation (Sigmoid operation refers to an activation function widely used in machine learning and deep learning, which maps the input value to between 0 and 1 and is often used for binary classification problems); when constructing an abnormal regression network, the feature output of the fully connected layer is mapped to an abnormal score through the fully connected structure and the Sigmoid operation. This process ensures that the abnormal regression network model can effectively convert features into abnormal probabilities, so that the abnormal regression layer of the abnormal regression network can effectively output the abnormal scores corresponding to the video clip features. Specifically, the fully connected layer plays the role of a "classifier" in the neural network. It integrates all the features of the previous layer so that each input can affect the final output, ensuring the effective transmission and comprehensive utilization of information; the output range of the Sigmoid function is limited, the optimization is stable, and it can be used as an output layer, which makes the model output more interpretable. The Sigmoid function is a continuous function and is easy to derive, which is particularly important for gradient-based optimization methods and can ensure that parameters are effectively updated during training.

[0030] Further, the erosion-expansion operation includes an erosion operation and an expansion operation;

[0031] S400 includes:

[0032] S401, sorting the anomaly scores corresponding to the video clip features in the video clip training set in descending order, and generating a sorting result;

[0033] S402, based on the sorting results, select k i video clips, and based on the selected k i The anomaly score of each video clip generates a binary threshold θ;

[0034]

[0035] Where, t i is the total number of video clips in the video clip training set, k is a constant, and sort is a sorting function;

[0036] S403, binarize the abnormality score corresponding to each video segment feature according to the binarization threshold, denoted as

[0037] The binarized anomaly score It is expressed as:

[0038]

[0039] ε is a step function, which converts the anomaly score s into i Mapped to 0 or 1;

[0040] S404, based on the binarized anomaly score Through the erosion function E, combined with the erosion structure parameter m, the anomaly score after binarization is Perform corrosion operation to generate the index E after corrosion i ;

[0041] S405, using the expansion function D and the expansion structure parameter M to erode the index E i Perform expansion operation to generate the expanded structural parameters D i ;

[0042] S406, according to the structural parameter D after expansion i For the anomaly score s i Index and generate a collection of typical exception examples

[0043] The index E after corrosion i , structural parameters after expansion D i And a collection of typical exception examples Respectively expressed as:

[0044]

[0045] D i =D(E i ; M)

[0046]

[0047] In the formula, E i is the index after the corrosion operation, m is the corrosion structure parameter, D i is the structural parameter after the expansion operation, M is the expansion structural parameter, E is the corrosion function, and D is the expansion function. Indicates the structural parameter D i A video clip of 1, This is a typical exception example.

[0048] Beneficial effects: This solution binarizes the anomaly score by setting a binarization threshold, which helps to clearly distinguish between normal and abnormal states, thereby reducing misjudgments and missed judgments, and can eliminate small changes and noise in the anomaly score. The binarized anomaly score more clearly indicates the abnormal state, reduces false positives and missed negatives caused by score ambiguity, and makes the model more robust to random fluctuations in the data;

[0049] The binarized anomaly score is then processed by the corrosion-dilation operation. The binarized anomaly score is first corroded and then dilated, and a set of typical anomaly examples is generated, which can be used to remove small noise in video clips, separate adhesion objects, smooth boundaries, etc., and can keep the area of ​​the main object in the video clip unchanged while removing small protruding parts, remove noise in the anomaly score, and more accurately identify the real typical anomaly examples, making the generated collection of typical anomaly examples more accurate and representative; the corrosion-dilation operation of the index of the video frame can improve the positioning ability of the abnormal video frame; the corrosion-dilation operation helps to reduce false detection and missed detection, improve the accuracy of anomaly detection, and help filter out random fluctuations in the anomaly score, so that the abnormal segment detection model is more robust to small changes in the data, thereby enhancing the robustness of the abnormal segment detection model.

[0050] Further, in S500, according to the ranking result, α video clips with the lowest abnormality scores are selected as typical normal examples. And establish a collection of typical normal examples

[0051] p i =sort(-s i )

[0052]

[0053] In the formula, sort is the sorting function, p i To sort the results, is the jth typical normal example selected from the i-th video clip.

[0054] Beneficial effects: The typical normal examples selected from the positive and negative samples in the video clip training set increase the diversity of the training data. These typical normal examples represent different normal situations, which help to improve the model's adaptability to various scenarios; the fixed selection method ensures the consistency of the data, avoids the subjective bias that may be caused by manually selecting examples, and further enhances the generalization ability of the model; the α examples with the lowest anomaly scores corresponding to the sorting results are selected as typical normal examples, and the typical normal examples are used as training data, which can further train the model to distinguish subtle abnormal features and reduce the false negative rate; the fixed selection of the α typical normal examples with the lowest anomaly scores can reduce the amount of data required for model training and improve training efficiency.

[0055] Furthermore, S600 includes:

[0056] S601, the positive matching pairs include normal positive matching pairs and abnormal positive matching pairs; constructing a number of normal positive matching pairs within the typical normal example collection, wherein the positive matching pairs include normal positive matching subjects and normal positive matching objects, wherein the normal positive matching subjects are any typical normal examples in the typical normal example collection, and the normal positive matching objects are the average of the video clip features of each typical normal example; constructing a number of abnormal positive matching pairs within the typical abnormal example collection, wherein the abnormal positive matching pairs include abnormal positive matching subjects and abnormal positive matching objects, wherein the abnormal positive matching subjects are any typical abnormal examples in the typical abnormal example collection, and the abnormal positive matching objects are the average of the video clip features of each typical abnormal example;

[0057] S602, constructing a plurality of negative matching pairs between a typical normal example collection and a typical abnormal example collection, wherein the negative matching pairs include a negative matching subject and a negative matching object; if the negative matching subject is any typical normal example in the typical normal example collection, then the negative matching object is any typical abnormal example in the typical abnormal example collection; if the negative matching subject is any typical abnormal example in the typical abnormal example collection, then the negative matching object is any typical normal example in the typical normal example collection;

[0058] S603, based on the constructed positive matching pairs and negative matching pairs, calculating the contrast loss of each typical normal example in the typical normal example collection and the contrast loss of each typical abnormal example in the typical abnormal example collection;

[0059] S604, calculating the contrast loss of the typical normal example collection and the typical abnormal example collection according to the contrast loss of each typical normal example in the typical normal example collection and the contrast loss of each typical abnormal example in the typical abnormal example collection.

[0060] Beneficial effects: Improve the accuracy of anomaly detection: By constructing positive matching pairs within the typical normal example collection and the typical abnormal example collection, and constructing negative matching pairs between the two, it is possible to more accurately distinguish the normal and abnormal states of video clips. The purpose of contrast loss is to make the video clips in the same example set more clustered, and the video clips in different example sets more distinguishable, that is, to separate the two categories of typical normal examples and typical abnormal examples as much as possible, while at the same time moving closer to their respective positive and negative samples. This method helps to reduce misjudgments and missed judgments, and facilitates the abnormal clip detection model to learn the difference between normal and abnormal more clearly through contrast loss.

[0061] Furthermore, the contrast loss between the typical normal example and the typical abnormal example The calculation formula is as follows:

[0062]

[0063] Where x is any typical normal example or typical abnormal example, x + is the mean of the video clip features of each typical normal example or the mean of the video clip features of each typical abnormal example; if x is a typical normal example, then x + is the mean of the video clip features of each typical normal example, x - is any typical anomaly example; if x is a typical anomaly example, then x + is the mean of the video clip features of each typical abnormal example, s - is any typical normal example; S represents a negative matching object s - An example set of τ is the temperature parameter, exp is the exponential function, x T is the transpose of the x argument.

[0064] Beneficial effects: By calculating the contrast loss of typical normal examples and typical abnormal examples, the ability of the abnormal fragment detection model to distinguish normal examples from abnormal examples can be enhanced. This formula maximizes the similarity between positive matching pairs and minimizes the similarity between negative matching pairs, thereby widening the distance between normal and abnormal samples in the feature space; the introduction of the temperature parameter τ can adjust the behavior of the contrast loss function. A smaller τ can make the similarity of positive matching pairs more prominent, thereby enhancing the model's selectivity for positive matching pairs; a larger τ can make the loss function smoother and reduce the risk of overfitting.

[0065] Furthermore, the contrast loss L between the typical normal example set and the typical abnormal example set is contrast The calculation formula is as follows:

[0066]

[0067] In the formula, Represents a typical normal example set The contrast loss is A typical normal example set The mean of the video clip features of each typical normal example in is taken as the typical normal example set The positive matching object x + , with a collection of typical exception examples Any typical abnormal example in is used as a negative match object x - ;

[0068] In the formula, Represents a collection of typical exception examples The contrast loss, A collection of typical exception examples The mean of the video clip features of each typical abnormal example in is used as the typical abnormal example set The positive matching object x +, with a typical normal example set Any typical normal example in is used as a negative matching object x - .

[0069] Beneficial effect: By calculating the contrast loss L of a typical normal example collection and a typical abnormal example collection contrast The comparison can be regarded as strengthening the ability of the common segment detection model to understand and distinguish between typical normal examples and typical abnormal examples.

[0070] Furthermore, S700 also includes optimizing the abnormal segment detection model by using a dynamic multi-instance loss function and sparse constraints, generating an optimized abnormal segment detection model, and using the optimized abnormal segment detection model to perform video abnormal segment detection.

[0071] Beneficial effects: This scheme uses a dynamic multi-instance loss function to optimize the abnormal segment detection model, so that the abnormal score in the negative package is as low as possible, and the abnormal score in the positive package is as high as possible, so as to more accurately identify abnormal and normal states and expand the distance between positive and negative packages in the video segment training set. This helps the abnormal segment detection model to better handle the differences between different video packages and enhance the robustness of the model; the addition of sparse constraints makes the abnormal score of typical normal examples as low as possible, which helps to reduce misjudgments and missed judgments.

[0072] Furthermore, the use of a dynamic multi-instance loss function and sparse constraints to optimize the abnormal segment detection model includes:

[0073] According to the sorting results, select the q with the highest anomaly score i video clips;

[0074]

[0075] p i = sort(s i )

[0076] In the formula, l i is the length of the i-th video packet, d is a constant, sort is the sorting function;

[0077] Get the q with the highest anomaly score i The anomaly scores corresponding to the video clips are used to construct a dynamic multi-instance set SD for multi-instance learning. i :

[0078]

[0079] In the formula, is the anomaly score corresponding to the jth video clip selected from the i-th video package;

[0080] Based on dynamic multi-instance set SD i , calculate the dynamic multi-instance loss L DMIL :

[0081]

[0082] In the formula, y i is the video packet label, when y i When it is 0, the video packet i is a positive packet, and the dynamic multi-instance loss function is used to make Approaching 0; when y i When it is 1, the video packet i is a negative packet, and the dynamic multi-instance loss function is used to make Approaching 1;

[0083] By using the sparse constraint L norm Constraint processing is performed on a collection of typical normal examples:

[0084]

[0085] In the formula, y i is the video packet label, when y i When is 0, video package i is a positive package and belongs to the typical normal example collection. The anomaly scores corresponding to each video clip in the positive package in the typical normal example collection are accumulated;

[0086] Optimized abnormal segment detection model L total It is expressed as:

[0087] L total =L comtrast +λ 1 L DMIL +λ 2 L norm

[0088] Among them, L contrast is the contrast loss between a typical normal example set and a typical abnormal example set, λ 1 is the dynamic multi-instance loss function L DMIL The hyperparameter, λ 2 is the sparse constraint L norm Hyperparameters of .

[0089] Beneficial effect: This scheme adopts a dynamic multi-instance loss function to select the top 1 / d examples with anomaly scores from the video clip training set to optimize the model, so that the anomaly score in the negative bag is as low as possible and the anomaly score in the positive bag is as high as possible. Specifically, y i is the video packet label, when y i When it is 0, the video packet i is a positive packet. According to the calculation result of the dynamic multi-instance loss function, adjust the parameters and let Approaching 0, the anomaly score corresponding to each video clip in the positive bag is reduced, that is, the anomaly score corresponding to each video clip in the positive bag is made as low as possible, so that each video clip in the positive bag is more likely to be judged as a positive sample; when y i When it is 1, the video packet i is a negative packet. Specifically, according to the calculation result of the dynamic multi-instance loss function, adjust the parameters and let Approaching 1, reducing the anomaly scores corresponding to each video clip in the negative bag, that is, making the anomaly scores corresponding to each video clip in the negative bag as high as possible, making it easier for each video clip in the negative bag to be identified as a negative sample;

[0090] Based on the contrast loss of the collection of typical normal examples and the collection of typical abnormal examples, the inter-class distance of examples in the positive bag and the negative bag is expanded. The distance between the positive bag and the negative bag in the video clip training set is expanded in combination with the dynamic multi-example loss function. Sparse constraints are also introduced to keep the typical normal examples at a low abnormality score. This can strengthen the guiding role of abnormal and normal clips during training, making the examples in the positive and negative bags closer to the typical normal samples and abnormal samples in the feature space, effectively improving the accuracy, robustness and generalization ability of anomaly detection, while reducing false positives and false negatives, and improving the training efficiency and real-time processing capability of the abnormal clip detection model. BRIEF DESCRIPTION OF THE DRAWINGS

[0091] Figure 1 The present invention is a flowchart of an embodiment of a weakly supervised video abnormal segment detection method based on corrosion-dilation.

[0092] Figure 2 The present invention is a schematic diagram of the overall data processing logic of an embodiment of a weakly supervised video abnormal segment detection method based on corrosion-expansion.

[0093] Figure 3 The present invention is a data processing logic diagram of an abnormal regression network in an embodiment of a weakly supervised video abnormal segment detection method based on corrosion-expansion.

[0094] Figure 4 The present invention is a schematic diagram of the data processing flow of a corrosion-expansion abnormal segment selection algorithm in an embodiment of a weakly supervised video abnormal segment detection method based on corrosion-expansion.

[0095] Figure 5 This is a schematic diagram of data processing results of a corrosion-expansion abnormal segment selection algorithm in an embodiment of a weakly supervised video abnormal segment detection method based on corrosion-expansion of the present invention.

[0096] Figure 6 The figure is a schematic diagram of typical normal example contrast loss and typical abnormal example contrast loss in an embodiment of a weakly supervised video abnormal segment detection method based on erosion-dilation of the present invention.

[0097] Figure 7 It is a schematic diagram of the experimental effect of the erosion-dilation operation in an embodiment of the weakly supervised video abnormal segment detection method based on erosion-dilation of the present invention.

[0098] Figure 8 This is a schematic diagram of the distribution of typical normal examples and typical abnormal examples in the feature space when using segment contrast loss in an embodiment of a weakly supervised video abnormal segment detection method based on erosion-dilation of the present invention.

[0099] Fig. 9 This is a schematic diagram of the distribution of typical normal examples and typical abnormal examples in the feature space when the segment contrast loss is not used in an embodiment of a weakly supervised video abnormal segment detection method based on erosion-dilation of the present invention. DETAILED DESCRIPTION

[0100] The following is further described in detail through specific implementation methods:

[0101] Embodiment 1:

[0102] Embodiment 1 is basically as attached Figures 1 to 9 As shown:

[0103] A weakly supervised video abnormal segment detection method based on corrosion-dilation, such as Figure 1 As shown, the following steps are included:

[0104] S100, obtain several video packages with video level labels, the video packages include several positive packages and negative packages, the positive packages and negative packages each include several video clips, the video clips in the positive packages are all positive samples, and the video clips in the negative packages include positive samples and negative samples; integrate the positive samples in the positive packages and the positive samples and negative samples in the negative packages to establish a video clip training set. In this embodiment, all video clips in the video package with a video level label of 0 are positive samples, and there are no abnormal features in the video clips, and there are abnormal features in the video clips of the video package with a video level label greater than 0.

[0105] S200, construct an abnormal regression network, use the video clip training set as the input of the abnormal regression network, and output the video clip features through the fully connected layer of the abnormal regression network; in this embodiment, the architecture of the abnormal regression network is as follows Figure 2 and Figure 3 As shown in Figure 2, the anomaly regression network includes a fully connected (FC) layer and an anomaly regression (AR) layer, and only video-level labels are required for video anomaly detection.

[0106] S200 includes:

[0107] S201, construct an abnormal regression network, and use the rectified linear unit function (ReLU) as the activation function of the fully connected layer in the abnormal regression network. The ReLU activation function can effectively introduce nonlinear characteristics into the neural network through its mathematical expression of f(x) = max(0,x). Compared with other activation functions, the calculation of ReLU is very efficient; it only needs to compare the size of the input value with zero and take the larger value, which greatly reduces the consumption of computing resources and is particularly suitable for the training of large-scale data sets and complex models. In addition, ReLU can effectively alleviate the problem of gradient disappearance.

[0108] Dropout operation is introduced in the fully connected layer to avoid overfitting. The dropout operation (random deactivation) refers to a regularization technique that prevents overfitting of deep neural networks. It increases the generalization ability of the model by randomly shutting down some neurons during the training process. In other words, this solution can prevent overfitting by randomly "discarding" the outputs of some neurons during the training process, reducing the complex co-adaptation relationship between neurons and forcing the remaining neurons to work together with other randomly selected neurons, thereby improving the generalization ability of the model.

[0109] S202, using the video clip training set as the input of the anomaly regression network, and outputting the video clip features through the fully connected layer of the anomaly regression network

[0110]

[0111] Where D′ represents the Dropout operation; W FC is the first fully connected layer parameter, representing the weight matrix weight of the fully connected layer; b FC is the parameter of the second fully connected layer, indicating the bias of the fully connected layer; R F×F represents a matrix of shape [F,F], R F×1 represents a matrix with a shape of [F,1], where R is a real number;

[0112] X i is the i-th video clip of the input, is the corresponding output video segment feature after inputting the i-th video segment.

[0113] This scheme effectively improves the performance of the anomaly regression network by adopting the ReLU activation function and the Dropout operation, ensuring the accuracy and reliability of the video clip characteristics output by the fully connected layer of the anomaly regression network, thereby further improving the efficiency and accuracy of anomaly detection.

[0114] S300, mapping anomaly scores of each video frame in the video clip through an anomaly regression layer of an anomaly regression network according to the video clip features.

[0115] In S300, the abnormal regression layer of the abnormal regression network adopts a fully connected structure and an S-type logical operation; the calculation formula of the abnormal score is as follows:

[0116]

[0117] In the formula, s i Video clip features The anomaly score of the corresponding video frame; exp represents the exponent; W AR is the first abnormal regression layer parameter, which represents the weight matrix weight of the abnormal regression layer; b AR is the parameter of the second abnormal regression layer, indicating the bias of the abnormal regression layer; R 1×F Represents a matrix with a shape of [1,F]; R 1 Represents a matrix with shape [1]; R is a real number.

[0118] S-type logical operation (Sigmoid operation refers to an activation function widely used in machine learning and deep learning. Its function is to map the input value to between 0 and 1, and is often used for binary classification problems); when constructing the abnormal regression network, the feature output of the fully connected layer is mapped to the abnormal score through the fully connected structure and the Sigmoid operation. This process ensures that the abnormal regression network model can effectively convert the features into abnormal probabilities, so that the abnormal regression layer of the abnormal regression network can effectively output the abnormal scores corresponding to the video clip features. Specifically, the fully connected layer plays the role of a "classifier" in the neural network. It integrates all the features of the previous layer so that each input can affect the final output, ensuring the effective transmission and comprehensive utilization of information; the output range of the Sigmoid function is limited, the optimization is stable, and it can be used as an output layer, which makes the model output more interpretable. The Sigmoid function is a continuous function and is easy to derive, which is particularly important for gradient-based optimization methods and can ensure that parameters are effectively updated during training.

[0119] S400, such as Figure 2 As shown, the anomaly score is binarized, and an erosion-expansion operation is performed on the binarized anomaly score to generate a typical anomaly example, and a collection of typical anomaly examples is established; the erosion-expansion operation includes an erosion operation and an expansion operation. In this embodiment, S400, the anomaly score is binarized by using an erosion-expansion anomaly segment selection algorithm.

[0120] S400 includes:

[0121] S401 , sorting the anomaly scores corresponding to the video clip features in the video clip training set in descending order, and generating a sorting result.

[0122] S402, based on the sorting results, select k i video clips, and based on the selected k i The anomaly score of each video clip generates a binary threshold θ;

[0123]

[0124] Where, t i is the total number of video clips in the video clip training set, k is a constant, and sort is the sorting function.

[0125] S403, binarize the abnormality score corresponding to each video segment feature according to the binarization threshold, denoted as

[0126] Anomaly score after binarization It is expressed as:

[0127]

[0128] ε is a step function, which converts the anomaly score s into i Mapped to 0 or 1;

[0129] In order to perform the erosion-expansion operation to more accurately select abnormal examples and avoid large-scale misselection that may be caused by directly setting the binary threshold, the erosion-expansion abnormal fragment selection algorithm proposed in the present invention first selects abnormal examples according to the abnormal score s i The number of the highest 1 / k samples is rounded and set as the binarization threshold, and the anomaly score corresponding to each video clip feature is binarized.

[0130] S404, based on the binarized anomaly score Through the erosion function E, combined with the erosion structure parameter m, the anomaly score after binarization is Perform corrosion operation to generate the index E after corrosion i The principle is to directly use the binarized anomaly score Using it for comparative learning will lead to more misselection phenomena. Therefore, in this scheme, Figure 4 and Figure 5 As shown, according to the anomaly score after binarization The corrosion function E is used to corrode the structure parameter m to eliminate the examples of abnormal values ​​that are misjudged as high in the normal picture, and the index E after corrosion is obtained. i .

[0131] S405, using the expansion function D and the expansion structure parameter M to erode the index E i Perform expansion operation to generate the expanded structural parameters D i .

[0132] S406, according to the structural parameter D after expansion i For the anomaly score s i Index and generate a collection of typical exception examples

[0133] The index E after corrosion i , structural parameters after expansion D i And a collection of typical exception examples Respectively expressed as:

[0134]

[0135] D i =D(E i ; M)

[0136]

[0137] In the formula, E i is the index after the corrosion operation, m is the corrosion structure parameter, D i is the structural parameter after the expansion operation, M is the expansion structural parameter, E is the corrosion function, and D is the expansion function. Indicates the structural parameter D i A video clip of 1, This is a typical exception example.

[0138] This solution binarizes the anomaly score by setting a binarization threshold, which helps to clearly distinguish between normal and abnormal states, thereby reducing misjudgments and missed judgments, and can eliminate slight changes and noise in the anomaly score. The binarized anomaly score more clearly indicates the abnormal state, reduces false positives and missed negatives caused by score ambiguity, and makes the model more robust to random fluctuations in the data.

[0139] The binarized anomaly score is then processed by the corrosion-dilation operation. The binarized anomaly score is first corroded and then dilated, and a set of typical anomaly examples is generated, which can be used to remove small noise in video clips, separate adhesion objects, smooth boundaries, etc., and can keep the area of ​​the main object in the video clip unchanged while removing small protruding parts, remove noise in the anomaly score, and more accurately identify the real typical anomaly examples, making the generated collection of typical anomaly examples more accurate and representative; the corrosion-dilation operation of the index of the video frame can improve the positioning ability of the abnormal video frame; the corrosion-dilation operation helps to reduce false detection and missed detection, improve the accuracy of anomaly detection, and help filter out random fluctuations in the anomaly score, so that the abnormal segment detection model is more robust to small changes in the data, thereby enhancing the robustness of the abnormal segment detection model.

[0140] S500, select several video clips in the video clip training set as typical normal examples, and establish a collection of typical normal examples. Specifically, according to the sorting result, select α video clips with the lowest abnormality scores as typical normal examples And establish a collection of typical normal examples

[0141] p v =sort(-s i )

[0142]

[0143] In the formula, sort is the sorting function, p i To sort the results, is the jth typical normal example selected from the i-th video clip.

[0144] The typical normal examples selected from the positive and negative samples in the video clip training set increase the diversity of the training data. These typical normal examples represent different normal situations, which help to improve the model's adaptability to various scenarios. The fixed selection method ensures the consistency of the data, avoids the subjective bias that may be caused by manual selection of examples, and further enhances the generalization ability of the model. The α examples with the lowest anomaly scores in the sorting results are selected as typical normal examples, and the typical normal examples are used as training data, which can further train the model to distinguish subtle abnormal features and reduce the false negative rate. The fixed selection of the α typical normal examples with the lowest anomaly scores can reduce the amount of data required for model training and improve training efficiency.

[0145] S600, such as Figure 6As shown, contrast loss constraints are applied to the typical normal example collection and the typical abnormal example collection; positive matching pairs are constructed within the typical normal example collection and the typical abnormal example collection, negative matching pairs are constructed between the typical normal example collection and the typical abnormal example collection, and the contrast loss of the typical normal example collection and the typical abnormal example collection is calculated based on the positive matching pairs and the negative matching pairs.

[0146] S600 includes:

[0147] S601, the positive match pairs include normal positive match pairs and abnormal positive match pairs; a number of normal positive match pairs are constructed within the typical normal example collection, the positive match pairs include normal positive match subjects and normal positive match objects, the normal positive match subjects are any typical normal examples in the typical normal example collection, and the normal positive match objects are the average of the video clip features of each typical normal example; a number of abnormal positive match pairs are constructed within the typical abnormal example collection, the abnormal positive match pairs include abnormal positive match subjects and abnormal positive match objects, the abnormal positive match subjects are any typical abnormal examples in the typical abnormal example collection, and the abnormal positive match objects are the average of the video clip features of each typical abnormal example; in this embodiment, the video clip features are vectors.

[0148] S602, constructing several negative matching pairs between a typical normal example collection and a typical abnormal example collection, wherein the negative matching pairs include a negative matching subject and a negative matching object; if the negative matching subject is any typical normal example in the typical normal example collection, then the negative matching object is any typical abnormal example in the typical abnormal example collection; if the negative matching subject is any typical abnormal example in the typical abnormal example collection, then the negative matching object is any typical normal example in the typical normal example collection.

[0149] S603: Based on the constructed positive matching pairs and negative matching pairs, calculate the contrast loss of each typical normal example in the typical normal example collection and the contrast loss of each typical abnormal example in the typical abnormal example collection.

[0150] The contrast loss of the typical normal example and the typical abnormal example The calculation formula is as follows:

[0151]

[0152] Where x is any typical normal example or typical abnormal example, x + is the mean of the video clip features of each typical normal example or the mean of the video clip features of each typical abnormal example; if x is a typical normal example, then x + is the mean of the video clip features of each typical normal example, x - is any typical anomaly example; if x is a typical anomaly example, then x+ is the mean of the video clip features of each typical abnormal example, x - is any typical normal example; S represents a negative matching object x - An example set of τ is the temperature parameter, exp is the exponential function, x T is the transpose of the x parameter. The example set S of negative matching objects and the temperature parameter τ can make the optimization objective more focused on difficult negative matching pairs.

[0153] S604, calculating the contrast loss of the typical normal example collection and the typical abnormal example collection according to the contrast loss of each typical normal example in the typical normal example collection and the contrast loss of each typical abnormal example in the typical abnormal example collection;

[0154] The contrast loss L between the typical normal example collection and the typical abnormal example collection contrast The calculation formula is as follows:

[0155]

[0156] In the formula, Represents a typical normal example set The contrast loss, A typical normal example set The mean of the video clip features of each typical normal example in is taken as the typical normal example set The positive matching object x + , with a collection of typical exception examples Any typical abnormal example in is used as a negative match object x - ;

[0157] In the formula, Represents a collection of typical exception examples The contrast loss, A collection of typical exception examples The mean of the video clip features of each typical abnormal example in is used as the typical abnormal example set The positive matching object x + , with a typical normal example set Any typical normal example in is used as a negative matching object x - .

[0158] Improve the accuracy of anomaly detection: By constructing positive matching pairs within the typical normal example collection and the typical abnormal example collection, and constructing negative matching pairs between the two, it is possible to distinguish normal and abnormal states more accurately. The purpose of contrast loss is to make samples in the same example set more clustered and samples in different example sets more distinguishable, that is, to separate the two categories of typical normal examples and typical abnormal examples as much as possible, while at the same time moving closer to their respective positive and negative samples. This method helps to reduce misjudgments and missed judgments, and facilitates the abnormal segment detection model to learn the difference between normal and abnormal more clearly through contrast loss.

[0159] In this embodiment, if Figure 6 As shown in FIG. 1 , the normal images and abnormal images in the same video are more closely clustered. and The mean of the values ​​is taken as the positive match of the examples in the respective example sets, denoted as x + , taking all the examples in the corresponding other typical example set as negative matching objects, denoted as x - , then the contrast loss of each example in the set can be expressed as The purpose of contrast loss is to make samples in the same example set more clustered and samples in different example sets more distinguishable.

[0160] S700, construct an abnormal segment detection model based on the contrast loss of a typical normal example collection and a typical abnormal example collection; use a dynamic multi-example loss function and sparse constraints to optimize the abnormal segment detection model, generate an optimized abnormal segment detection model, and use the optimized abnormal segment detection model to perform video abnormal segment detection.

[0161] The use of dynamic multi-instance loss function and sparse constraints to optimize the abnormal segment detection model includes:

[0162] According to the sorting results, select the q with the highest anomaly score i video clips;

[0163]

[0164] p i = sort(s i )

[0165] In the formula, l i is the length of the i-th video packet, d is a constant, 1 / d is the selection ratio of the dynamic multi-instance loss function, and sort is the sorting function.

[0166] Get the q with the highest anomaly score i The anomaly scores corresponding to the video clips are used to construct a dynamic multi-instance set SD for multi-instance learning.i :

[0167]

[0168] In the formula, is the anomaly score corresponding to the j-th video clip selected from the i-th video package.

[0169] Based on dynamic multi-instance set SD i , calculate the dynamic multi-instance loss L DMIL :

[0170]

[0171] In the formula, y i is the video packet label, when y i When it is 0, the video packet i is a positive packet. Specifically, according to the calculation result of the dynamic multi-instance loss function, adjust the parameters and let Approaching 0, the anomaly score corresponding to each video clip in the positive bag is reduced, that is, the anomaly score corresponding to each video clip in the positive bag is made as low as possible, so that each video clip in the positive bag is more likely to be judged as a positive sample; when y i When it is 1, the video packet i is a negative packet. Specifically, according to the calculation result of the dynamic multi-instance loss function, adjust the parameters and let Approaching 1, the anomaly score corresponding to each video clip in the negative bag is reduced, that is, the anomaly score corresponding to each video clip in the negative bag is made as high as possible, so that each video clip in the negative bag is easier to be identified as a negative sample.

[0172] By using the sparse constraint L norm Constraint processing is performed on a collection of typical normal examples:

[0173]

[0174] In the formula, y i is the video packet label, when y i When is 0, video package i is a positive package and belongs to the typical normal example collection. The anomaly scores corresponding to each video clip of the positive package in the typical normal example collection are accumulated.

[0175] The constraint object of the sparse constraint is the collection of typical normal examples. The implementation method is to accumulate the abnormal scores corresponding to each video clip in the typical normal example collection, and train so that L norm As small as possible, so that the anomaly score of the typical normal example is as low as possible, and it is easier to be identified as a normal example.

[0176] Optimized abnormal segment detection model L total It is expressed as:

[0177] Ltotal =L contrast +λ 1 L DMIL +λ 2 L norm

[0178] Among them, L contrast is the contrast loss between a typical normal example set and a typical abnormal example set, λ 1 is the dynamic multi-instance loss function L DMIL The hyperparameter, λ 2 is the sparse constraint L norm Hyperparameters of .

[0179] This scheme adopts a dynamic multi-instance loss function, selects the top 1 / d examples with anomaly scores from the video clip training set to optimize the model, so that the anomaly score in the negative bag is as low as possible, and the anomaly score in the positive bag is as high as possible, so as to more accurately identify the abnormal and normal states, and expand the distance between the positive bag and the negative bag in the video clip training set, which helps the abnormal clip detection model to better handle the differences between different video bags and enhance the robustness of the model; the addition of sparse constraints makes the anomaly score of typical normal examples as low as possible, which helps to reduce misjudgment and missed judgment; minimize the loss through the dynamic multi-instance loss function, that is, reduce the difference between the predicted results of the abnormal clip detection model and the actual label through the dynamic multi-instance loss function optimization process, and minimizing the loss means making the value of the loss function as small as possible by continuously adjusting the model parameters;

[0180] Based on the contrast loss of the collection of typical normal examples and the collection of typical abnormal examples, the inter-class distance of examples in the positive bag and the negative bag is expanded. The distance between the positive bag and the negative bag in the video clip training set is expanded in combination with the dynamic multi-example loss function. Sparse constraints are also introduced to keep the typical normal examples at a low abnormality score. This can strengthen the guiding role of abnormal and normal clips during training, making the examples in the positive and negative bags closer to the typical normal samples and abnormal samples in the feature space, effectively improving the accuracy, robustness and generalization ability of anomaly detection, while reducing false positives and false negatives, and improving the training efficiency and real-time processing capability of the abnormal clip detection model.

[0181] The principles and advantages of the present invention are:

[0182] For weakly supervised video anomaly detection, the present invention proposes a weakly supervised video abnormal segment detection method based on expansion-erosion, obtains several video segments in a video package, establishes a video segment training set, constructs an abnormal regression network, uses the video segment training set as the input of the abnormal regression network, outputs video segment features through the fully connected layer of the abnormal regression network, and maps the abnormal scores of video frames in the video segment through the abnormal regression layer of the abnormal regression network according to the video segment features; screens positive samples and negative samples in several video segment training sets as typical normal examples, and establishes a collection of typical normal examples; binarizes the abnormal scores through the corrosion-expansion abnormal segment selection algorithm, which helps to clearly distinguish between normal and abnormal states, performs corrosion-expansion operations on the binarized abnormal scores, generates typical abnormal examples, and establishes a collection of typical abnormal examples; selects representative typical abnormal segments through the corrosion-expansion algorithm, which can improve the positioning capability of abnormal video frames, and the corrosion-expansion operation helps to reduce false detection and missed detection, improve the accuracy of abnormal detection, and help filter out random fluctuations in abnormal scores; thereby completing the establishment of a collection of typical normal examples and a collection of typical abnormal examples;

[0183] Then, a contrast loss constraint is applied to the typical normal example collection and the typical abnormal example collection; a positive matching pair is constructed within each of the typical normal example collection and the typical abnormal example collection, a negative matching pair is constructed between the typical normal example collection and the typical abnormal example collection, and a contrast loss of the typical normal example collection and the typical abnormal example collection is constructed based on the positive matching pair and the negative matching pair; the purpose of the contrast loss is to make the samples in the same example collection more clustered and the samples in different example collections more distinguishable, and the contrast loss constraint of the typical normal example and the typical abnormal example is used to optimize the separation of normal samples and abnormal samples, so as to more accurately distinguish the normal state and the abnormal state; the contrast learning loss goal is to expand the inter-class distance of the examples in the bag;

[0184] According to the contrast loss of the typical normal example collection and the typical abnormal example collection, an abnormal segment detection model is constructed, and the dynamic multi-example loss function and sparse constraints are used to optimize the abnormal segment detection model to generate an optimized abnormal segment detection model; the goal of using dynamic multi-example loss is to expand the distance between positive packages and negative packages in the video segment training set, which helps the abnormal segment detection model to better handle the differences between different video packages. Sparse constraints are also introduced to keep the abnormal scores of typical normal examples low. The addition of sparse constraints makes the abnormal scores of typical normal examples as low as possible, which helps to reduce misjudgments and missed judgments; based on The contrast loss of the typical normal example collection and the typical abnormal example collection expands the inter-class distance of examples in the positive bag and the negative bag. The dynamic multi-example loss function is combined to expand the distance between the positive bag and the negative bag in the video clip training set. Sparse constraints are also introduced to keep the typical normal examples at a low abnormality score. This can strengthen the guiding role of abnormal and normal clips during training, making the examples in the positive and negative bags closer to the typical normal samples and abnormal samples in the feature space, effectively improving the accuracy, robustness and generalization ability of anomaly detection, while reducing false positives and false negatives, and improving the training efficiency and real-time processing capability of the abnormal clip detection model.

[0185] Detection method:

[0186] The present invention conducts experiments on the ShanghaiTech dataset. Since it is an unsupervised dataset, the present invention re-partitions the ShanghaiTech dataset according to the existing weak supervision method to conduct experiments under the weak supervision setting.

[0187] The present invention uses the area under the receiver operating characteristic curve (ROC) based on the video frame, AUC (Area Under Receiver Operating Characteristic Curve), as an evaluation index for video anomaly detection. In the video anomaly detection task, the higher the AUC, the better the model performance. In addition, since the lower the FAR value means the stronger the robustness of the anomaly detection method, FAR is also used as a performance evaluation reference in the invention.

[0188] The feature extractor of the present invention uses I3D RGB-optical-flow Model, specifically by I3D RGB and I3D Optical-FlowThe I3D feature extractor used was not fine-tuned, and the optical flow of each video clip was generated by the TV-L1 algorithm. In addition, the DropOut in the fully connected layer of the anomaly regression network of this method was set to 0.7. The specific experimental parameters are shown in Table 1. The Adam optimizer was used during training. During training, for a batch size of 60, the specific structure during training was 30 positive packets and 30 negative packets.

[0189]

[0190]

[0191] Table 1 Experimental parameter settings

[0192] Experimental results:

[0193] To evaluate the accuracy and robustness of the algorithm, the present invention tested the performance data of AUC and FAR on the ShanghaiTech dataset: AUC: 94.25%, FAR: 0.02%.

[0194] The present invention uses the ndimage package in the Python open source algorithm library scipy, and implements the erosion and dilation operations through the ndimage.binary_erosion and ndimage.binary_dilation functions. Compared with multiple groups of settings, the present invention sets the erosion structure parameter m to 2 and the dilation structure parameter M to 4 for the best performance. The visualization results of the erosion-dilation segment selection are shown in Figure 2. Figure 7 As shown in the figure, it can be seen that the original anomaly score is processed by the binary threshold ( Figure 7 After the erosion-expansion operation, the index of the selected fragments is more concentrated on the typical abnormal fragments ( Figure 7 The green part, Figure 7 In the figure, the overlapping parts of green and red appear brown due to the overlap of green and red colors. The resulting fragment is more accurate and closer to the GT value.

[0195] To further verify the impact of the representative fragment contrast loss, a comparative experiment was conducted between the present invention and the case where the fragment contrast loss is not used (only sparse loss and dynamic multi-instance loss are used), and the results were visualized. Figure 7 The proposed method uses the representative segment contrast loss Figure 8 Comparison loss with not using representative fragments Fig. 9 In the case of , the feature space is visualized using the t-SNE method for comparison.

[0196] in Figure 8 and Fig. 9 In the figure, the purple and yellow parts represent the distribution of normal examples and abnormal examples in the feature space, respectively. Figure 8 and Fig. 9 In contrast, after using the representative fragment contrast loss, normal examples and abnormal examples are more distinguishable in the feature space.

[0197] The experimental results on the three test videos 01_0015, 01_0025 and 01_055 of the ShanghaiTech dataset show that in the feature space constrained by the contrast loss, the normal examples and abnormal examples in each video are more clearly distinguished, and the normal examples and abnormal examples are more clustered. The experiment shows that the representative segment contrast loss helps to separate the normal examples and abnormal examples more accurately, and by clustering the normal examples and abnormal examples separately, the separability of the normal examples and abnormal examples is increased.

[0198] Since the present invention better considers the influence of abnormal fragments, it demonstrates good accuracy (AUC: 94.25%) and robustness (FAR: 0.02) on the ShanghaiTech and UCF-Crime datasets. In future work, the temporal relationship between examples can be considered for modeling to obtain stronger robustness.

[0199] The above are only embodiments of the present invention. Common knowledge such as the known specific structures and characteristics in the scheme is not described in detail here. Ordinary technicians in the relevant field know all the common technical knowledge in the technical field to which the invention belongs before the application date or priority date, can obtain all the existing technologies in the field, and have the ability to apply conventional experimental means before that date. Ordinary technicians in the relevant field can improve and implement this scheme in combination with their own abilities under the enlightenment given by this application. Some typical known structures or known methods should not become obstacles for ordinary technicians in the relevant field to implement this application. It should be pointed out that for those skilled in the art, without departing from the structure of the present invention, several deformations and improvements can be made, which should also be regarded as the scope of protection of the present invention, and these will not affect the effect of the implementation of the present invention and the practicality of the patent. The scope of protection required by this application shall be based on the content of its claims, and the specific implementation methods and other records in the specification can be used to interpret the content of the claims.

Claims

1. A weakly supervised video abnormal segment detection method based on corrosion-dilation, characterized by: The following steps are involved: S100, obtaining a plurality of video packages, the video packages including a plurality of positive packages and negative packages, the positive packages and negative packages each including a plurality of video clips, the video clips in the positive packages are all positive samples, and the video clips in the negative packages include positive samples and negative samples; integrating the positive samples in the positive packages and the positive samples and negative samples in the negative packages to establish a video clip training set; S200, constructing an abnormal regression network, taking the video clip training set as input of the abnormal regression network, and outputting video clip features through a fully connected layer of the abnormal regression network; S300, mapping anomaly scores of each video frame in the video clip through an anomaly regression layer of an anomaly regression network according to the video clip features; S400, binarizing the anomaly score, performing an erosion-expansion operation on the binarized anomaly score, generating a typical anomaly example, and establishing a collection of typical anomaly examples; S500, selecting a plurality of video clips in the video clip training set as typical normal examples, and establishing a collection of typical normal examples; S600, performing contrast loss constraints on a typical normal example collection and a typical abnormal example collection; constructing positive matching pairs within each of the typical normal example collection and the typical abnormal example collection, constructing negative matching pairs between the typical normal example collection and the typical abnormal example collection, and calculating contrast losses of the typical normal example collection and the typical abnormal example collection based on the positive matching pairs and the negative matching pairs; S700: construct an abnormal segment detection model based on the contrast loss of a typical normal example collection and a typical abnormal example collection, and use the abnormal segment detection model to perform video abnormal segment detection.

2. The method for detecting abnormal video segments based on weak supervision based on corrosion and expansion according to claim 1 is characterized in that: S200 includes: S201, construct an abnormal regression network, use the rectified linear unit function as the activation function of the fully connected layer in the abnormal regression network, and introduce a Dropout operation in the fully connected layer; S202, using the video clip training set as the input of the anomaly regression network, and outputting the video clip features through the fully connected layer of the anomaly regression network Where D′ represents the Dropout operation; W FC is the first fully connected layer parameter, representing the weight matrix weight of the fully connected layer; b FC is the parameter of the second fully connected layer, indicating the bias of the fully connected layer; R F×F represents a matrix of shape [F,F], R F ×1 represents a matrix with a shape of [F,1], where R is a real number; X i is the i-th video clip of the input, is the corresponding output video segment feature after inputting the i-th video segment.

3. The method for detecting abnormal video segments based on weak supervision based on corrosion and expansion according to claim 2 is characterized in that: In S300, the abnormal regression layer of the abnormal regression network adopts a fully connected structure and an S-type logical operation; the calculation formula of the abnormal score is as follows: In the formula, s i Video clip features The anomaly score of the corresponding video frame; exp represents the exponent; W AR is the first abnormal regression layer parameter, which represents the weight matrix weight of the abnormal regression layer; b AR is the parameter of the second abnormal regression layer, indicating the bias of the abnormal regression layer; R 1×F Represents a matrix with a shape of [1,F]; R 1 Represents a matrix with shape [1]; R is a real number.

4. The method for detecting abnormal video segments based on weak supervision based on erosion-expansion according to claim 3 is characterized in that: The corrosion-expansion operation includes a corrosion operation and a expansion operation; S400 includes: S401, sorting the anomaly scores corresponding to the video clip features in the video clip training set in descending order, and generating a sorting result; S402, based on the sorting results, select k i video clips, and based on the selected k i The anomaly score of each video clip generates a binary threshold θ; In the formula, t i is the total number of video clips in the video clip training set, k is a constant, and sort is a sorting function; S403, binarize the abnormality score corresponding to each video segment feature according to the binarization threshold, denoted as The binarized anomaly score It is expressed as: ε is a step function, which converts the anomaly score s into i Mapped to 0 or 1; S404, based on the binarized anomaly score Through the erosion function E, combined with the erosion structure parameter m, the anomaly score after binarization is Perform corrosion operation to generate the index E after corrosion i ; S405, using the expansion function D and the expansion structure parameter M to erode the index E i Perform expansion operation to generate the expanded structural parameters D i ; S406, according to the structural parameter D after expansion i For the anomaly score s i Index and generate a collection of typical exception examples The index E after corrosion i , structural parameters after expansion D i And a collection of typical exception examples Respectively expressed as: D i =D(E i ;M) In the formula, E i is the index after the corrosion operation, m is the corrosion structure parameter, D i is the structural parameter after the expansion operation, M is the expansion structural parameter, E is the corrosion function, and D is the expansion function. Indicates the structural parameter D i A video clip of 1, This is a typical exception example.

5. The method for detecting abnormal video segments based on weak supervision based on erosion-expansion according to claim 4 is characterized in that: In S500, according to the ranking result, α video clips with the lowest abnormality scores are selected as typical normal examples. And establish a collection of typical normal examples p i =sort(-s i ) In the formula, sort is the sorting function, p i To sort the results, is the jth typical normal example selected from the i-th video clip.

6. The method for detecting abnormal video segments based on weak supervision based on erosion-expansion according to claim 5, characterized in that: S600 includes: S601, the positive matching pairs include normal positive matching pairs and abnormal positive matching pairs; constructing a number of normal positive matching pairs within the typical normal example collection, wherein the positive matching pairs include normal positive matching subjects and normal positive matching objects, wherein the normal positive matching subjects are any typical normal examples in the typical normal example collection, and the normal positive matching objects are the average of the video clip features of each typical normal example; constructing a number of abnormal positive matching pairs within the typical abnormal example collection, wherein the abnormal positive matching pairs include abnormal positive matching subjects and abnormal positive matching objects, wherein the abnormal positive matching subjects are any typical abnormal examples in the typical abnormal example collection, and the abnormal positive matching objects are the average of the video clip features of each typical abnormal example; S602, constructing a plurality of negative matching pairs between a typical normal example collection and a typical abnormal example collection, wherein the negative matching pairs include a negative matching subject and a negative matching object; if the negative matching subject is any typical normal example in the typical normal example collection, then the negative matching object is any typical abnormal example in the typical abnormal example collection; if the negative matching subject is any typical abnormal example in the typical abnormal example collection, then the negative matching object is any typical normal example in the typical normal example collection; S603, based on the constructed positive matching pairs and negative matching pairs, calculating the contrast loss of each typical normal example in the typical normal example collection and the contrast loss of each typical abnormal example in the typical abnormal example collection; S604, calculating the contrast loss of the typical normal example collection and the typical abnormal example collection according to the contrast loss of each typical normal example in the typical normal example collection and the contrast loss of each typical abnormal example in the typical abnormal example collection.

7. The method for detecting abnormal video segments based on weak supervision based on erosion-dilation according to claim 6 is characterized in that: The contrast loss l(x,x) between the typical normal example and the typical abnormal example + ,x - ) is calculated as follows: Where x is any typical normal example or typical abnormal example, x + is the mean of the video clip features of each typical normal example or the mean of the video clip features of each typical abnormal example; if x is a typical normal example, then x + is the mean of the video clip features of each typical normal example, x - is any typical anomaly example; if x is a typical anomaly example, then x + is the mean of the video clip features of each typical abnormal example, x - is any typical normal example; S represents a negative matching object x - An example set of τ is the temperature parameter, exp is the exponential function, x T is the transpose of the x argument.

8. The method for detecting abnormal video segments based on weak supervision based on erosion-dilation according to claim 7, characterized in that: The contrast loss L between the typical normal example collection and the typical abnormal example collection contrast The calculation formula is as follows: In the formula, Represents a typical normal example set The contrast loss is A typical normal example set The mean of the video clip features of each typical normal example in is taken as the typical normal example set The positive matching object x + , with a collection of typical exception examples Any typical abnormal example in is used as a negative match object x - ; In the formula, Represents a collection of typical exception examples The contrast loss is A collection of typical exception examples The mean of the video clip features of each typical abnormal example in is used as the typical abnormal example set The positive matching object x + , with a typical normal example set Any typical normal example in is used as a negative matching object x - .

9. The method for detecting abnormal video segments based on weak supervision based on erosion-dilation according to claim 8, characterized in that: S700 also includes optimizing the abnormal segment detection model by using a dynamic multi-instance loss function and sparse constraints, generating an optimized abnormal segment detection model, and performing video abnormal segment detection by using the optimized abnormal segment detection model.

10. The method for detecting abnormal video segments based on weak supervision based on erosion-dilation according to claim 9, characterized in that: The use of dynamic multi-instance loss function and sparse constraints to optimize the abnormal segment detection model includes: According to the sorting results, select the q with the highest anomaly score i video clips; p i =sort(s i ) In the formula, l i is the length of the i-th video packet, d is a constant, sort is the sorting function; Get the q with the highest anomaly score i The anomaly scores corresponding to the video clips are used to construct a dynamic multi-instance set SD for multi-instance learning. i : In the formula, is the anomaly score corresponding to the jth video clip selected from the i-th video package; Based on dynamic multi-instance set SD i , calculate the dynamic multi-instance loss L DMIL : In the formula, y i is the video packet label, when y i When it is 0, the video packet i is a positive packet, and the dynamic multi-instance loss function is used to make Approaching 0; when y i When it is 1, the video packet i is a negative packet, and the dynamic multi-instance loss function is used to make Approaching 1; By using the sparse constraint L norm Constraint processing is performed on a collection of typical normal examples: In the formula, y i is the video packet label, when y i When is 0, video package i is a positive package and belongs to the typical normal example collection. The anomaly scores corresponding to each video clip in the positive package in the typical normal example collection are accumulated; Optimized abnormal segment detection model L total It is expressed as: L total =L contrast +λ1L DMIL +λ2L norm Among them, L contrast is the contrast loss between a typical normal example set and a typical abnormal example set, and λ1 is the dynamic multi-instance loss function L DMIL is a hyperparameter, λ2 is the sparse constraint L norm Hyperparameters of .

Images