Abnormal event detection method and device, computer device, and storage medium

Abstract

The present disclosure relates to an abnormal event detection method and device, a computer device and a storage medium. The method comprises: acquiring at least two image sequences; wherein each image sequence comprises at least one image frame; dividing each image sequence into at least two scales to obtain an image block set composed of image blocks at the same position in all image frames at the same scale; determining correlation features between the image sequences based on the image block sets of the image sequences; and determining a target image sequence with an abnormal event in the at least two image sequences according to the correlation features between the image sequences. The method can improve the accuracy of abnormal event detection.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer vision, and particularly relates to an abnormal event detection method and device, a computer device, and a storage medium. BACKGROUND

[0002] A video abnormal event detection method aims to capture an abnormal event in a video and determine a time interval in which the abnormal event occurs. The abnormal event refers to an unexpected and rarely occurring behavior. How to improve the accuracy of abnormal event detection has always been a concern. SUMMARY

[0003] The present disclosure provides an abnormal event detection method and device, a computer device, and a storage medium.

[0004] According to a first aspect of an embodiment of the present disclosure, an abnormal event detection method is provided, including: acquiring at least two image sequences; wherein each of the image sequences includes at least one image frame; performing division on each of the image sequences in at least two scales to obtain an image block set composed of image blocks at a same position in all image frames under a same scale; determining a correlation feature between the image sequences based on the image block sets of the image sequences; and determining a target image sequence in which an abnormal event exists in the at least two image sequences according to the correlation feature between the image sequences.

[0005] According to a second aspect of an embodiment of the present disclosure, an abnormal event detection device is provided, including: an acquisition module configured to acquire at least two image sequences; wherein each of the image sequences includes at least one image frame; a division module configured to perform division on each of the image sequences in at least two scales to obtain an image block set composed of image blocks at a same position in all image frames under a same scale; a first determination module configured to determine a correlation feature between the image sequences based on the image block sets of the image sequences; and a second determination module configured to determine a target image sequence in which an abnormal event exists in the at least two image sequences according to the correlation feature between the image sequences.

[0006] According to a third aspect of an embodiment of the present disclosure, a computer device is provided, including: a processor; and a memory configured to store processor-executable instructions; wherein the processor is configured to execute an abnormal event detection method as described in the first aspect.

[0007] According to a fourth aspect of an embodiment of the present disclosure, a storage medium is provided, including: when instructions in the storage medium are executed by a processor of a device, the device is enabled to execute an abnormal event detection method as described in the first aspect.

[0008] The technical solution provided by the embodiments of the present disclosure can include the following beneficial effects:

[0009] In the embodiments of the present disclosure, considering that some abnormal events occur in a small area of the image frame, and some abnormal events may run through the entire picture, neither directly taking the image frame as a whole nor carrying out single-scale area division can cope with various abnormal events, thus the present disclosure carries out multi-scale division for each image in the image sequence, which can improve the scale robustness in abnormal event detection. In addition, the present disclosure determines the correlation features between the image sequences based on the image block set of each image sequence, so that the abnormal event detection device can combine the correlation between the image sequences on the basis of multi-scale, and improve the detection accuracy of abnormal events.

[0010] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0011] The accompanying drawings incorporated in and forming a part of the specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the disclosure.

[0012] Figure 1 An abnormal event detection method flowchart shown in an embodiment of the present disclosure Figure One .

[0013] Figure 2 An example diagram of scale division shown in an embodiment of the present disclosure.

[0014] Figure 3 An abnormal event detection method flowchart shown in an embodiment of the present disclosure Figure Two .

[0015] Figure 4 A principle example diagram for obtaining a first feature based on a first splicing feature in an embodiment of the present disclosure.

[0016] Figure 5 A principle example diagram of feature fusion in an embodiment of the present disclosure.

[0017] Figure 6 An abnormal event detection method flowchart shown in an embodiment of the present disclosure Figure Three .

[0018] Figure 7 An abnormal event detection method flowchart shown in an embodiment of the present disclosure Figure Four .

[0019] Figure 8A A principle diagram of an abnormal event detection method shown in an embodiment of the present disclosure.

[0020] Figure 8B An abnormal event detection method shown in an embodiment of the present disclosureFigure 8A Process diagram of the middle part of the module.

[0021] Figure 9 An abnormal event detection device diagram shown in embodiments of the present disclosure.

[0022] Figure 10 A hardware entity diagram of a computer device in embodiments of the present disclosure. DETAILED DESCRIPTION

[0023] The exemplary embodiments will be described in detail herein with reference to the attached drawings. In the following description, unless otherwise indicated, like numbers in the attached drawings refer to the same or similar elements. The following exemplary embodiments described in the following description are not meant to be construed in a limiting sense as to the scope of the present disclosure. Rather, they are described in such a way as to illustrate at least some of the aspects of the present disclosure.

[0024] The abnormal event detection method provided by the embodiments of the present disclosure can be executed by an abnormal event detection device, for example, the abnormal event detection method can be executed by a terminal device or a server or other electronic device, wherein the terminal device can be a user equipment (User Equipment, UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (Personal Digital Assistant, PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. In some possible implementation manners, the abnormal event detection method can be realized by a processor calling computer readable instructions stored in a memory.

[0025] In the embodiments of the present disclosure, the abnormal event detection device can include an image acquisition component, so as to acquire continuous frame images of a scene by using the image acquisition component, and at least two image sequences are divided. For example, the image acquisition component is a camera, which can acquire a video of a fixed position, and the abnormal event detection device including the camera can divide the video into at least two image sequences in the time dimension, one image sequence can be referred to as one video segment, and the image frames included in different video segments can not overlap. In addition, the abnormal event detection device can also not include the image acquisition component, and the abnormal event detection device can receive transmitted at least two divided image sequences; or after acquiring multiple videos of the same scene by using independently arranged cameras at different angles, the videos are transmitted to the abnormal event detection device, and one video received by the abnormal event detection device can be referred to as one image sequence. In the embodiments of the present disclosure, one image sequence can be a sequence in a time window, that is, the image frames in the image sequence are adjacent in time.

[0026] It should be noted that in the embodiments of the present disclosure, the specific image sequence acquisition method and the content of at least one image included in the image sequence can be determined according to actual needs and application scenarios, and the embodiments of the present disclosure are not limited.

[0027] Figure 1 An abnormal event detection method flow shown in an embodiment of the present disclosure Figure One As shown in Figure 1 , the abnormal event detection method comprises the following steps:

[0028] S11, acquiring at least two image sequences; wherein each of the image sequences comprises at least one image frame;

[0029] S12, dividing each of the image sequences into at least two scales to obtain an image block set composed of image blocks at the same position in all image frames under the same scale;

[0030] S13, determining the correlation features between the image sequences based on the image block sets of the image sequences;

[0031] S14, determining a target image sequence in which an abnormal event exists in the at least two image sequences according to the correlation features between the image sequences.

[0032] In the formula embodiment, after the abnormal event detection device acquires at least two image sequences, at least two scales are divided for each image sequence, which means that at least two scales are divided for each image frame included in the image sequence. After the image sequence is divided into multiple scales, the image block set composed of image blocks at the same position in all image frames under the same scale.

[0033] For example, the abnormal event detection device divides the video V into T non-overlapping image sequences For each image sequence, each image frame is divided using R different sliding window sizes . Figure 2 An example diagram of scale division shown in an embodiment of the present disclosure is shown in Figure 2 , a certain image sequence is divided into 3 scales (R=3), and the number of corresponding image blocks in each image frame is 1, 6 and 15 respectively. In the embodiments of the present disclosure, the image blocks at the same position in different image frames of the image sequence are formed as a whole to form an image block set, that is, a small cube shown in Figure 2 . The diagram shown in includes a cube, that is, it corresponds to an image block set under this scale; includes 6 cubes, that is, it corresponds to 6 image block sets under this scale; It includes 15 cubes, which are 15 image patch sets corresponding to this scale.

[0034] In this embodiment of the disclosure, a set of image patches of the same scale can be represented as: Where, N r This represents the number of image patch sets corresponding to the scale. For example... Figure 2 As shown, the first scale division N r =1; the second scale division N r The third scale is 6; the division N r It is 15.

[0035] It should be noted that in the embodiments of this disclosure, when dividing each frame of an image sequence into corresponding scales, the image blocks corresponding to the same scale have the same size. Furthermore, when using a sliding window to divide each frame of an image into non-overlapping image blocks, the number of image blocks in each frame at the corresponding scale can be the result of rounding down the ratio of the size of each frame to the size of the sliding window. That is, when dividing image blocks, if the size of the frame image and the size of the sliding window are not divisible, no additional image blocks are obtained by padding with "0"s or "1"s. In this disclosure, the content of each image block belongs to the content of the frame image before division.

[0036] In step S13, after the abnormal event detection device obtains the set of image blocks for each image sequence, it can obtain the features that can characterize each image sequence, and then obtain the correlation features between the image sequences based on the features that characterize the image sequences.

[0037] In one embodiment, when obtaining features characterizing an image sequence based on sets of image patches, for example, the features of each image patch in multiple sets of image patches at different scales can be directly concatenated as the features of the image sequence. As in the previous example, each image sequence includes 3 frames, and each frame includes sets of image patches at 3 different scales. Assuming each image patch corresponds to one feature, the number of features for this image sequence is: the number of image patches after multi-scale division of each frame * the number of frames, i.e., (1+6+15)*3, a total of 66 features. In this embodiment, if at least two image sequences are obtained by dividing the same video in the time dimension, the correlation features between the image sequences obtained based on the features of each image sequence can be called temporal correlation features.

[0038] In another embodiment, the features of the image sequence are obtained based on the image block sets of each image in the image sequence. For example, the correlation features between different image block sets of the same scale can be determined, and then the features of the image sequence are obtained based on the correlation features between different image block sets of the same scale. Alternatively, the correlation features between image blocks of each frame are obtained first, and then the features of the image sequence are obtained based on the correlation features between image blocks.

[0039] It can be understood that, since the image blocks have the position attribute, the correlation features between different image block sets of the same scale or the correlation features between image blocks of each frame have the spatial attribute, and can be represented as the correlation in space. In the embodiments of the present disclosure, if the at least two image sequences are obtained by dividing the same video in the time dimension, the correlation features between the image sequences obtained based on the features of the image sequences can be referred to as the spatio-temporal correlation features.

[0040] Of course, if the at least two image sequences in the embodiments of the present disclosure are image sequences of different angles of the same scene, the correlation features between the at least two image sequences can be understood as the spatial correlation features. In addition, if the correlation features between different image block sets of the same scale or the correlation features between image blocks of each frame are obtained first, and then the correlation features of the image sequences are obtained based on the correlation features, the correlation features of the image sequences can be understood as including the features of the local spatial correlation and the global spatial correlation. The local spatial correlation is related to the position attribute of the image blocks, and the global spatial correlation is related to the acquisition angle attribute of the image sequence.

[0041] It should be noted that the correlation features between the image sequences are used to represent the association relationship between the image sequences, for example, can include the features of each image sequence after being weighted by different weights, and the association relationship between different image sequences is reflected by the allocation of the weights. In addition, the correlation features of the image sequences can also include the features of any image sequence by fusing part of the features of other image sequences, that is, the association relationship between the image sequences is reflected by feature fusion. It should be noted that the present disclosure does not make specific limitations on the manner of obtaining the correlation features.

[0042] In the embodiments of the present disclosure, if there are T image sequences, the correlation features between the image sequences are represented as φ ST , and φ ST includes the features corresponding to the T image sequences, but the features corresponding to each image sequence are related to the features of other image sequences.

[0043] In step S14, after obtaining the correlation features between the image sequences, the abnormal event detection apparatus can determine the target image sequence in which the abnormal event exists in the at least two image sequences according to the correlation features, for example, by using a traditional feature recognition method or a trained model.

[0044] It can be understood that in the embodiments of the present disclosure, considering that some abnormal events occur in a small area of an image frame, and some abnormal events may run through the entire picture, neither directly taking the image frame as a whole nor performing single-scale area division can cope with various abnormal events, and therefore the present disclosure performs multi-scale division on each image in each image sequence, which can improve the scale robustness in abnormal event detection. In addition, the present disclosure determines the correlation features between the image sequences based on the image block sets of the image sequences, for example, the aforementioned correlation in time and / or space obtained by weight allocation, so that the abnormal event detection apparatus can combine the correlation between the image sequences on a multi-scale basis to improve the detection accuracy of abnormal events.

[0045] Figure 3 An abnormal event detection method flow shown in the embodiments of the present disclosure Figure Two As shown in Figure 3 , step S13 in Figure 1 may include the following steps:

[0046] S13a, for each image sequence, obtaining a first feature corresponding to a scale based on the image block set at the same scale; wherein the first feature includes the correlation between the image block sets at the same scale;

[0047] S13b, fusing the first features corresponding to each scale in the same image sequence to obtain a second feature of each image sequence;

[0048] S13c, determining the correlation features between the image sequences based on the second features of the image sequences.

[0049] In step S13a, after determining the image block set corresponding to the scale, the first feature including the correlation between the image block sets at the same scale can be obtained, as shown in the correlation features between each small cubic block in Figure 2 . It can be understood that since the image blocks in the image block set have a position attribute, each image block set also has a position attribute, and therefore the obtained first feature is a feature including the spatial correlation between the image block sets.

[0050] For example, if the abnormal event detection apparatus performs R sets of scale division, the first feature is represented by If the first feature is represented as φ, then the scale corresponding first feature obtained by the abnormal event detection device The R groups are shared.

[0051] In step S13b, the first features corresponding to each scale in the same image sequence are fused to obtain the second feature of each image sequence. If there are T image sequences, the second feature is represented as φ' t If the first feature is represented as φ, then the T groups of φ' t .

[0052] In step S13c, the correlation features between the image sequences are determined based on the second features of the image sequences. Since the first features include the spatial correlation features between the image block sets, if at least two image sequences are image sequences of different time periods of the same video, the correlation features between the image sequences obtained in this step can be spatio-temporal correlation features. In addition, as previously analyzed, if at least two image sequences are image sequences of different angles of the same scene, the correlation features between the image sequences can also include local spatial correlation and global spatial correlation.

[0053] It can be understood that in the embodiments of the present disclosure, the first features are obtained by taking the image block set composed of the image blocks at the same position in all frame images of an image sequence as a processing unit, without focusing on one image block of each frame image, so that the calculation amount can be relatively reduced when further obtaining the correlation features between the image sequences based on the first features; and the correlation features between the obtained image block sets include multi-dimensional correlation features, so that the accuracy of abnormal event detection can be improved.

[0054] In an embodiment, the first feature corresponding to a scale is obtained based on the image block sets at the same scale, including:

[0055] Feature extraction is performed on each of the image block sets at the same scale to obtain a feature corresponding to the image block set;

[0056] The features of the image block sets at the same scale are spliced to obtain a first spliced feature corresponding to the scale;

[0057] Based on the first spliced feature corresponding to the scale, a self-attention mechanism and convolution processing are used to construct the correlation relationship between the image block sets at the same scale represented by the first spliced feature, to obtain the first feature corresponding to the scale.

[0058] In this embodiment, the features of the image block sets are obtained as a whole, and then the features of the image block sets at the same scale are spliced to obtain a first spliced feature corresponding to the scale.

[0059] In the stitching of the features of the image block set of the same scale, the image block set can be stitched horizontally as a whole. For example, if the dimension of the feature corresponding to each image block set after feature extraction is D dimensions, the first stitched feature is represented as . The dimension of the first stitched feature is: the number of image block sets of the same scale * D, i.e., N r *D.

[0060] In an embodiment, the feature extraction on each image block set of the same scale to obtain the feature corresponding to the image block set comprises:

[0061] The feature extraction on each image block set of the same scale to obtain the feature corresponding to the image block set comprises the time sequence information between the image blocks in the image block set.

[0062] As described above, each image frame in the image sequence is adjacent in time, i.e., there is time sequence information between the image frames in the image sequence, and thus there is time sequence information between the image blocks in the image set. In this embodiment, the feature extraction on the image block set can obtain the feature comprising the time sequence information between the image blocks in the image block set.

[0063] For example, the present disclosure can use a preset I3D feature encoder to extract the features of each image block set of the same scale to obtain the features comprising the time sequence information between the image blocks in the image block set. It can be understood that, since the network structure of the I3D feature encoder is deep and a 3-dimensional convolution kernel is used, and the image block set comprises the time sequence information, the use of the 3-dimensional convolution kernel can include the time sequence information of the image block set, so that the feature extraction is more complete.

[0064] In the present embodiment of the present disclosure, after the first stitched feature corresponding to the scale is obtained, the correlation between the image block sets of the same scale represented by the first stitched feature can be constructed, so as to obtain the first feature corresponding to the scale.

[0065] It should be noted that, in the present embodiment of the present disclosure, as described above, the first feature corresponding to the scale can be represented as , and then the dimension of the obtained first feature corresponding to the scale is the same as that of the first stitched feature, but the first feature comprises the correlation between the image sequences of the same scale, The dimension of the first feature can also be N r *D.

[0066] It can be understood that the disclosure constructs the association relationship between the image block sets of the same scale represented by the first spliced features through the self-attention mechanism and the convolution processing, and based on the machine vision theory, the obtained first features have a good enhancement effect, for example, selectively highlighting the parts of interest (i.e., parts that may exist abnormally) in each image block set of the same scale, thereby further improving the detection effect of abnormal events.

[0067] In an embodiment, based on the first spliced features corresponding to the scale, the association relationship between the image block sets of the same scale represented by the first spliced features is constructed using the self-attention mechanism and the convolution processing to obtain the first features corresponding to the scale, including:

[0068] Based on the self-attention mechanism and the first spliced features, a weight matrix is determined; wherein the weight matrix includes weight values representing the probability of each image block set of the same scale existing abnormally;

[0069] Based on the weight matrix and the first spliced features, a weighted feature is obtained;

[0070] The first spliced features are subjected to convolution processing to obtain a convolution feature;

[0071] Based on the weighted feature, the convolution feature, and the first spliced features, the first features are obtained.

[0072] In this embodiment, the weight matrix is first determined based on the self-attention mechanism, and the weight values in the weight matrix represent the probability of each image block set of the same scale existing abnormally. The greater the weight value, the greater the probability of the image block set existing abnormally.

[0073] In this embodiment, the first spliced features are also subjected to convolution processing, for example, using non-dilated convolution or dilated convolution to process the first spliced features. When the first spliced features are subjected to convolution processing, the features of each image block set of the same scale are included in the first spliced features, and thus the features of multiple image block sets can be associated through convolution operations of the convolution kernel.

[0074] In an embodiment, the first spliced features are subjected to convolution processing to obtain a convolution feature, including:

[0075] At least two dilated convolution kernels are used to respectively convolve the first spliced features to obtain convolution results corresponding to each dilated convolution kernel; wherein the dilated rates of the at least two dilated convolution kernels are different;

[0076] The convolution results corresponding to each dilated convolution kernel are spliced to obtain the convolution feature.

[0077] In this embodiment, dilated convolution is used to process the first spliced ​​feature. For example, at least two dilated convolution kernels are used, including three kernels, each of which is a one-dimensional convolution kernel with dilation rates of 1, 2, and 4, respectively. If the first spliced ​​feature... The dimension is N r *D, then after processing with three dilated convolution kernels, the dimension of the convolution result corresponding to each dilated convolution kernel can be N. r *D / 4, the convolutional features obtained by concatenating the convolution results corresponding to each dilated convolution kernel are N. r *3D / 4.

[0078] In this embodiment of the disclosure, the result of the convolution can be used This indicates that DC1, DC2, and DC3 are the convolution results corresponding to the dilated convolution kernels, respectively.

[0079] Of course, this disclosure is not limited to the above three one-dimensional dilated convolution kernels. Since the final weighted features, convolutional features, and first spliced ​​features need to work together to form the first feature, the number, size, and corresponding dilation rate of dilated convolution kernels can be set according to actual needs.

[0080] It is understandable that dilated convolution can expand the receptive field, and when multiple dilated convolution kernels with different dilation rates are superimposed, different receptive fields will bring multi-scale information. Therefore, the features obtained after dilated convolution after convolution with multiple dilated convolution kernels and concatenation of the convolution results enhance the first concatenated features.

[0081] In one embodiment, determining the weight matrix based on the self-attention mechanism and the first concatenation feature includes:

[0082] The first splicing feature is subjected to dimensionality reduction processing to obtain the dimensionality-reduced first splicing feature;

[0083] The first spliced ​​feature after dimensionality reduction is convolved using a preset first convolution kernel to obtain the first convolution result;

[0084] The first spliced ​​feature after dimensionality reduction is convolved using a preset second convolution kernel to obtain the second convolution result;

[0085] The weight matrix is ​​determined by multiplying the transposes of the first and second convolution results using the self-attention mechanism.

[0086] In embodiments of this disclosure, the first concatenated feature is first subjected to dimensionality reduction processing to reduce subsequent computational load. For example, dimensionality reduction can be performed using a one-dimensional convolution matrix. For example, the dimensionality-reduced first concatenated feature can be... denotes, dimension is N r D / 4. Of course, the present disclosure is not limited to reducing the feature dimension of each image block set to 1 / 4 of the original feature dimension.

[0087] In the embodiments of the present disclosure, the self-attention mechanism is based on predicting the covariance between any image block set and other image block sets in the same scale, and each image block set is regarded as a random variable. The weight value in the obtained weight matrix is the correlation between each image block set and all image block sets.

[0088] In this embodiment, the preset first convolution kernel and the preset second convolution kernel can both be one-dimensional convolution kernels. The first convolution result and the second convolution result obtained by performing convolution on the reduced first spliced feature using the preset first convolution kernel and the preset second convolution kernel can both be one-dimensional vectors. The product of the transposes of the first convolution result and the second convolution result is the attention map, i.e., the weight matrix, obtained by the normalization exponential function (softmax) of the self-attention mechanism. The weight matrix is essentially a covariance matrix.

[0089] For example, if the dimension of the first convolution result is N r D / 4, and the dimension of the second convolution result is D / 4*N r , then the dimension of the weight matrix is N r *N r .

[0090] In one embodiment, the obtaining of the weighted feature based on the weight matrix and the first spliced feature comprises:

[0091] performing convolution on the reduced first spliced feature using a preset third convolution kernel to obtain a third convolution result;

[0092] multiplying the weight matrix and the third convolution result to obtain a weighted matrix;

[0093] determining the sum of the weighted matrix and the result obtained by performing convolution on the reduced first spliced feature using a preset fourth convolution kernel as the weighted feature.

[0094] In this embodiment, the preset third convolution kernel and the preset fourth convolution kernel can also be one-dimensional convolution kernels. The weighted matrix obtained by multiplying the third convolution result obtained by performing convolution on the reduced first spliced feature using the preset third convolution kernel and the weight matrix is the weighted sum of the image block sets in the reduced first spliced feature, and the weight is the covariance between the image block sets in the same scale included in the reduced first spliced feature.

[0095] For example, the dimension of the third convolution result can be N r D / 4, and the dimension of the weighted matrix is Nr D / 4, the dimension of the weighted feature can be N r D / 4.

[0096] In the embodiments of the present disclosure, the result obtained by convolving the weight matrix with the preset fourth convolution is added to the first spliced feature after dimension reduction, that is, residual connection is performed, and the obtained weighted feature has stronger representation ability for each image block set.

[0097] In the present disclosure, the above process of obtaining the weight matrix and the weighted feature can be represented by the following formulas (1) and (2):

[0098]

[0099]

[0100] In the above formulas (1) and (2), W θ is a preset first convolution kernel, is a preset second convolution kernel, W g is a preset third convolution kernel, W z is a preset fourth convolution kernel, is the first spliced feature after dimension reduction. The weight matrix obtained by the softmax part is is the weighted feature, is the weighted feature.

[0101] In an embodiment, the first feature is obtained based on the weighted feature, the convolution result and the first spliced feature.

[0102] The weighted feature is spliced with the convolution result, and then added to the first spliced feature to obtain the first feature.

[0103] In this embodiment, the first feature can be represented by the following formula (3):

[0104]

[0105] wherein, is the weighted feature, is the convolution result, is the first spliced feature, is the first feature, and the dimension is N r D.

[0106] Figure 4 is a principle example diagram for obtaining the first feature based on the first spliced feature in the embodiments of the present disclosure, as Figure 4 shown, the right branch is to determine the weight matrix based on the self-attention mechanism and the first spliced feature, that is,Figure 4 The left branch is a process of obtaining a convolutional feature by processing the first spliced feature based on a cavity convolution, and the specific description can refer to the foregoing description, which will not be described here.

[0107] In an embodiment, the first features corresponding to each scale in the same image sequence are fused to obtain second features of each image sequence, including:

[0108] The first features of the same scale are reconstructed according to the positional relationship of each image block set to obtain scale corresponding reconstruction features;

[0109] After the scale corresponding reconstruction features are convolved by using a preset fifth convolution kernel, the scale corresponding reconstruction features are converted into a one-dimensional feature vector by a full connection layer;

[0110] The one-dimensional feature vectors of each scale are accumulated to obtain second features of each image sequence.

[0111] In this embodiment, since the first features corresponding to each scale are obtained by splicing the image block sets of the same scale, and the dimensions of the first features corresponding to each scale and the first spliced features are the same, the first features can be understood as the result of horizontally splicing the correlation features of the image block sets of the same scale. Since the image blocks included in the image block set have a position attribute, the reconstruction features corresponding to each scale can be obtained by reconstructing according to the positional relationship of the image blocks in each image block set, and it can be understood that the reconstruction features are a three-dimensional vector, which can be represented by in the embodiment of the present disclosure. Each element in the reconstruction features represents an image block set, and the feature dimension is D.

[0112] After the reconstruction features are obtained by reconstructing according to the positional relationship of the image blocks in the image block set, the reconstruction features are converted into a one-dimensional feature vector by using a preset fifth convolution kernel and a full connection layer, wherein the preset fifth convolution kernel can be a two-dimensional convolution kernel, which is used for convolution processing of feature dimension reduction of the reconstruction features. The one-dimensional feature vector converted after the two-dimensional convolution and the full connection layer can be represented by, and the feature dimension can be D. It can be understood that the one-dimensional feature vector is a feature representing the image block set of the same scale.

[0113] Since the second features of the image sequence are obtained by accumulating the one-dimensional feature vectors of each scale, it can be understood that the second features of the image sequence are features fused from multiple scales.

[0114] Figure 5 ​​This is a schematic diagram illustrating the principle of feature fusion in an embodiment of this disclosure. The explanation uses a first feature corresponding to a scale as an example. Figure 5 As shown, the dashed box L51a represents the first feature corresponding to a scale, which includes the correlation between image patch sets at the same scale. The cube L52a in the diagram represents the reconstructed feature obtained after reconstructing this first feature according to the positional relationships of the image patches in each image patch set. The reconstructed feature is then transformed into a one-dimensional feature vector after passing through a two-dimensional convolutional layer L53a and a fully connected layer L54a. (The last sentence appears to be incomplete and possibly refers to a different feature vector.) Figure 5 As shown, each first feature corresponds to a reconstructed feature. After converting each reconstructed feature into a one-dimensional feature vector through a two-dimensional convolutional layer and a fully connected layer, the vectors are accumulated to obtain L50, which is the second feature corresponding to the image sequence. The two-dimensional convolutional layer may include the preset fifth convolutional kernel disclosed herein. It should be noted that... Figure 5 The L53a, L53b and L53c shown can be the same two-dimensional convolutional layer, and the L54a, L54b and L54c can also be the same fully connected layer. This disclosure does not limit the embodiments in this way.

[0115] Understandably, this disclosure enables the anomaly detection device to have a perception of image frames in an image sequence from local to global by fusing features of image patch sets at all scales, thereby improving robustness to anomalies at different scales.

[0116] In one embodiment, determining the correlation feature between the image sequences based on the second feature of each image sequence includes:

[0117] The second features of each of the image sequences are spliced ​​together to obtain the second spliced ​​feature;

[0118] Based on the second stitching feature, the association relationship between different image sequences represented by the second stitching feature is constructed, and the correlation feature between each image sequence is determined.

[0119] In the embodiments of this disclosure, the correlation features between each image sequence can be determined based on the method of obtaining the correlation features between image patch sets of the same scale, that is, the method of obtaining the first feature corresponding to the scale.

[0120] In this embodiment, the second features of each image sequence can be stitched together, for example, horizontally, to obtain the second stitched feature, and then based on the aforementioned... Figure 4According to the principle, the weight matrix of the image sequence is determined based on the self-attention mechanism and the second spliced feature, and the weight matrix of the image sequence includes a weight value representing the probability of each image sequence being abnormal. Subsequently, the weighted features corresponding to all image sequences are obtained based on the weight matrix of the image sequence and the second spliced feature. When the self-attention mechanism is used for processing, the second spliced feature can be first processed by dimension reduction, for example, by using one-dimensional convolution for dimension reduction. In addition, the second spliced feature is processed by convolution to obtain the convolution features corresponding to all image sequences, and the correlation features between the image sequences are further determined based on the weighted features corresponding to all image sequences, the convolution features corresponding to all image sequences, and the second spliced feature.

[0121] For example, if the second feature of the image sequence is represented by φ′ t , then the T groups of image sequences have T groups of φ′ t The second features of the image sequences are spliced to obtain the second spliced feature, which can be represented by .

[0122] The above process can be represented by the following formulas (4)-(6):

[0123]

[0124]

[0125]

[0126] wherein, is the feature after dimension reduction of the second spliced feature; W θ , W g and W z can refer to the description in the aforementioned formulas (1) and (2), and the softmax part obtains the weight matrix of the image sequence; is the weight matrix corresponding to all image sequences, is the weighted feature of all image sequences; φ *,A is the convolution feature corresponding to all image sequences, φ ST is used to represent the correlation features between the image sequences.

[0127] It should be noted that in the embodiments of the present disclosure, the dimension of φ ST may be the number of image sequences * the feature dimension of each image sequence, that is, T*D.

[0128] Figure 6 is a flow of an abnormal event detection method in the embodiments of the present disclosure Figure Three , for example, Figure 3As shown, Figure 1 Step S14 in the method can include the following steps:

[0129] S14a, detecting correlation features between each image sequence based on a preset anomaly prediction model to obtain a prediction result of each image sequence; wherein the preset anomaly prediction model is a model obtained by training using a weak supervision training method;

[0130] S14b, determining the target image sequence in which the abnormal event exists according to the prediction result of each image sequence.

[0131] As described above, the target image sequence in which the abnormal event exists can be determined in at least two image sequences according to the correlation features between each image sequence using a traditional feature recognition method or a trained model. In this embodiment, an anomaly detection model obtained by a weak supervision training method is used.

[0132] When performing weak supervision training, a loss function needs to be constructed, which is used to estimate the degree of inconsistency between the predicted value of the model and the true value. Generally, the smaller the loss function value, the better the robustness of the model. During the training process, the parameters of the model can be adjusted by constraining the loss function to obtain a better model.

[0133] In an embodiment of the present disclosure, the training samples are obtained according to the foregoing description in Figures 1 to 5 Then, based on the obtained features of the training samples and the sample labels, a loss function is constructed, and the parameters of the model are continuously corrected to obtain a model with better detection effect. In an embodiment of the present disclosure, the initial model is, for example, a Convolutional Neural Networks (CNN) model, a Deep Neural Networks (DNN) model, etc., which is not limited here.

[0134] In an embodiment, the method further includes:

[0135] For the positive samples and the negative samples in the training sample set, K sample image sequences with larger feature gradients are selected respectively to calculate the average feature gradient; wherein K is a positive integer greater than 1.

[0136] According to the average feature gradient corresponding to the positive samples and the average feature gradient corresponding to the negative samples, a loss function is constructed;

[0137] Based on the loss function, the preset anomaly prediction model is trained and obtained.

[0138] In the embodiments of the present disclosure, the training sample set includes positive samples and negative samples, wherein the positive sample refers to a sample in which no abnormal event exists in the image sequence included by the sample, and the negative sample refers to a sample in which an abnormal event exists in the image sequence included by the sample. One sample can be a video, the video is divided into different image sequences, one video corresponds to one label, but the image sequence has no label. In the embodiments of the present disclosure, each video can be regarded as a “package”, and the image sequence is regarded as an “instance”, that is, the “package” is labeled, but the “instance” is not labeled.

[0139] In the embodiments of the present disclosure, for the positive samples and the negative samples, K sample image sequences with larger feature gradients are selected to calculate average feature gradients, and then a loss function is constructed based on the average feature gradients corresponding to the positive samples and the average feature gradients corresponding to the negative samples.

[0140] Suppose that the sample feature of the T image sequences included in a video in the training sample set is obtained based on the foregoing method The specific method for constructing the loss function is as follows:

[0141] A. The first K image sequences with larger feature gradients are selected from all image sequences, and the average feature gradient is calculated according to the following formula (7):

[0142]

[0143] wherein, ||φ″ t is the 2-norm of the feature, and the feature gradient is obtained by calculating the 2-norm of the feature in the present disclosure.

[0144] B. The average feature gradient of the first K image sequences in the abnormal video φ ST ′ + and the average feature gradient of the first K image sequences in the normal video φ ST ′ - are calculated according to the following formula (8):

[0145]

[0146] wherein, g(φ ST ′ + ) is the average feature gradient of the first K image sequences in the normal video, and g(φ ST ′ - ) is the average feature gradient of the first K image sequences in the abnormal video.

[0147] C. The features of the first K image sequences included in each video are input into the original model to obtain (an image sequence corresponds to one prediction score), and the cross-entropy loss is calculated based on the predicted abnormal score and the label corresponding to the video, as shown in the following formula (9):

[0148]

[0149] wherein s represents a predicted anomaly score, and y represents a label corresponding to the video, for example, the label value of an abnormal video is 1, and the label value of a normal video is 0.

[0150] D, a sparse constraint and a time smoothing constraint are introduced, and a total loss function is determined as formula (10) as follows:

[0151]

[0152] wherein λ fm , λ1, λ2 are factors for balancing the loss of each term, represents a sparse constraint, represents a time smoothing constraint.

[0153] The disclosure can construct a loss function based on the above steps to preset an anomaly detection model. The correlation feature φ ST between each image sequence is input into the preset anomaly detection model, and a prediction result of each image sequence can be obtained, for example, the prediction result is a prediction score. The disclosure compares each prediction score with a preset score threshold, for example, an image sequence with a prediction score greater than the preset score threshold is determined as a target image sequence with an abnormal event.

[0154] It can be understood that, compared with the traditional method, the generalization ability of the preset anomaly event detection model is better in the way that the disclosure uses an anomaly event detection model obtained based on a weakly supervised training method to process the correlation feature of the image sequence to determine the target image sequence with an abnormal event. In addition, compared with a model obtained by unsupervised training, the accuracy of anomaly event detection is better because the supervised training method has the guidance of training labels.

[0155] Figure 7 An anomaly event detection method flow of an embodiment of the disclosure Figure Four As shown in FIG. 1, Figure 7 step S11 in the embodiment can include the following steps: Figure 1

[0156] S11a, acquiring a video to be detected;

[0157] S11b, determining a difference value between adjacent frame images in the video to be detected;

[0158] S11c, determining an image frame earlier in time as a tail frame of one image sequence and an image frame later in time as a head frame of an image sequence adjacent to one image sequence among the adjacent frame images with a difference value greater than a preset difference threshold.

[0159] ​In this embodiment, at least two image sequences come from the same video, namely the video to be detected. When dividing the image sequences based on the video to be detected, this disclosure uses clustering to detect the difference values ​​between adjacent frames in the video to be detected, and groups some image frames with similar content as an image sequence. This method can ensure that the content of each image sequence does not repeat, improve the difference between different image sequences, and thus improve the accuracy of anomaly localization.

[0160] It should be noted that when determining the difference value between adjacent frames in the video to be detected, this disclosure may, for example, determine the difference value by performing a difference operation on two adjacent frames, but this disclosure is not limited to this method. Furthermore, the method by which the abnormal event detection device acquires at least two image sequences is not limited to the method of this embodiment; it may also involve, for example, dividing the video into image sequences of equal duration based on time, etc., which will not be detailed here.

[0161] Figure 8A This is a schematic diagram illustrating the principle of an abnormal event detection method according to an embodiment of this disclosure. Figure 8B As shown in the embodiments of this disclosure Figure 8A A schematic diagram of the processing procedure for some modules in the middle section. Figure 8A The video segments identified by L81 are image sequences, and three image sequences are shown in total. Each image sequence is input into the multi-scale patch generator L82, and the resulting patch is a set of image patches mentioned in this disclosure. After extracting features from each patch using the pre-trained feature encoder L83, the spatial relationships between the patches can be modeled based on the modules identified by L84. For example... Figure 8B As shown, for an image sequence, after inputting the image sequence into the multi-scale patch generator L82 (with a total of R scales), the scale-corresponding values ​​are obtained. It can include multiple image patch sets. (For scale-corresponding...) The first concatenated feature corresponding to the scale is obtained after using the pre-trained feature encoder with L3 identifier. Subsequently, by modeling the spatial relationship of patches identified by L84 using the first stitching feature corresponding to the scale, the correlation between image patch sets at the same scale can be obtained, i.e., the first feature corresponding to the scale, such as... Figure 8B As shown The first feature, after modeling the spatial relationship of patches at each scale, can be aggregated using the L85-identified patch aggregation module to combine the first features at different scales of the same image sequence. By concatenating the images, we obtain the second feature corresponding to the image sequence, which is... Figure 8Aone of the T characteristic segments shown in L86. Subsequently, the second characteristic of all image sequences, i.e., the T characteristic segments shown in L86, is identified by the video time relationship module L87 to obtain the spatio-temporal modeling characteristic, i.e., the correlation characteristic between the image sequences mentioned in the disclosure. Finally, the correlation characteristic is input into the pre-trained classifier L88 to obtain the prediction score of each image sequence, and based on the prediction score of each image sequence, it can be determined whether the image sequence has an abnormal event. The pre-trained classifier can be obtained based on a weakly supervised training method, a loss function of the model is constructed by using the video-level label of the training sample and the prediction score of the training sample, and the model parameters are fixed when the loss meets the convergence condition to obtain the trained classifier.

[0162] Figure 9 An abnormal event detection device diagram is shown for the embodiments of the disclosure. Referring to Figure 9 , the abnormal event detection device 900 includes:

[0163] The acquisition module 901 is configured to acquire at least two image sequences; wherein each of the image sequences includes at least one image frame;

[0164] The division module 902 is configured to divide each of the image sequences into at least two scales to obtain an image block set composed of image blocks at the same position in all image frames at the same scale;

[0165] The first determination module 903 is configured to determine a correlation characteristic between the image sequences based on the image block sets of the image sequences;

[0166] The second determination module 904 is configured to determine a target image sequence with an abnormal event from the at least two image sequences according to the correlation characteristic between the image sequences.

[0167] In some embodiments, the first determination module 903 is configured to, for each of the image sequences, obtain a first characteristic corresponding to a scale based on the image block sets at the same scale; wherein the first characteristic includes the correlation between the image block sets at the same scale; fuse the first characteristics corresponding to each scale in the same image sequence to obtain a second characteristic of each of the image sequences; and determine the correlation characteristic between the image sequences based on the second characteristics of the image sequences.

[0168] In some embodiments, the first determining module 903 is configured to: perform feature extraction on each of the image block sets at the same scale to obtain features corresponding to the image block sets; splice the features of the image block sets at the same scale to obtain first spliced features corresponding to the scale; and based on the first spliced features corresponding to the scale, construct a correlation between the image block sets at the same scale represented by the first spliced features by using a self-attention mechanism and convolution processing to obtain first features corresponding to the scale.

[0169] In some embodiments, the first determining module 903 is configured to: determine a weight matrix based on the self-attention mechanism and the first spliced features, wherein the weight matrix includes weight values representing probabilities of existence of abnormalities in each of the image block sets at the same scale; obtain weighted features based on the weight matrix and the first spliced features; perform convolution processing on the first spliced features to obtain convoluted features; and obtain the first features based on the weighted features, the convoluted features, and the first spliced features.

[0170] In some embodiments, the first determining module 903 is configured to: perform dimension reduction processing on the first spliced features to obtain reduced-dimension first spliced features; perform convolution on the reduced-dimension first spliced features by using a preset first convolution kernel to obtain a first convolution result; perform convolution on the reduced-dimension first spliced features by using a preset second convolution kernel to obtain a second convolution result; and determine the weight matrix by using the self-attention mechanism on a result of multiplication of a transpose of the first convolution result and the second convolution result.

[0171] In some embodiments, the first determining module 903 is configured to: perform convolution on the reduced-dimension first spliced features by using a preset third convolution kernel to obtain a third convolution result; multiply the weight matrix and the third convolution result to obtain a weighted matrix; and determine a sum of the weighted matrix and a result of convolution of a preset fourth convolution kernel as the weighted features.

[0172] In some embodiments, the first determining module 903 is configured to: perform convolution on the first spliced features by using at least two hole convolution kernels respectively to obtain convolution results corresponding to the at least two hole convolution kernels, wherein the at least two hole convolution kernels have different hole rates; and splice the convolution results corresponding to the at least two hole convolution kernels to obtain the convoluted features.

[0173] In some embodiments, the first determining module 903 is configured to: splice the weighted features and the convoluted features and add the spliced features to the first spliced features to obtain the first features.

[0174] In some embodiments, the first determining module 903 is configured to perform feature extraction on each of the image block sets in the same scale to obtain features corresponding to the image block sets, the features including time sequence information between image blocks in the image block sets.

[0175] In some embodiments, the first determining module 903 is configured to reconstruct the first features in the same scale according to the positional relationship of each of the image block sets to obtain reconstructed features corresponding to the scale; after convolution of the reconstructed features corresponding to the scale by using a preset fifth convolution kernel, convert the reconstructed features into one-dimensional feature vectors by using a full connection layer; and accumulate the one-dimensional feature vectors of each scale to obtain second features of each of the image sequences.

[0176] In some embodiments, the first determining module 903 is configured to splice the second features of each of the image sequences to obtain second spliced features; and based on the second spliced features, construct a correlation relationship between different image sequences represented by the second spliced features based on a self-attention mechanism and convolution processing to determine the correlation features between the image sequences.

[0177] In some embodiments, the second determining module 904 is configured to detect the correlation features between the image sequences based on a preset anomaly prediction model to obtain prediction results of the image sequences; wherein the preset anomaly prediction model is a model trained by using a weakly supervised training method; and determine the target image sequence in which the anomaly event exists according to the prediction results of the image sequences.

[0178] In some embodiments, the apparatus further comprises:

[0179] The computing module 905 is configured to calculate average feature gradients of K sample image sequences with larger feature gradients for positive samples and negative samples in a training sample set respectively; wherein K is a positive integer greater than 1.

[0180] The constructing module 906 is configured to construct a loss function according to the average feature gradients corresponding to the positive samples and the average feature gradients corresponding to the negative samples.

[0181] The training module 907 is configured to train the preset anomaly prediction model based on the loss function.

[0182] In some embodiments, the obtaining module 901 is configured to obtain a video to be detected; determine difference values between adjacent frame images in the video to be detected; determine an image frame earlier in time as a tail frame of an image sequence and an image frame later in time as a head frame of an image sequence adjacent to the image sequence among the adjacent frame images with a difference value greater than a preset difference threshold.

[0183] The description of the above device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects as the method embodiments. For technical details not disclosed in the device embodiments of the present disclosure, please refer to the description of the method embodiments of the present disclosure for understanding.

[0184] It should be noted that, in the embodiments of the present disclosure, if the above-mentioned abnormal event detection method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such understanding, the technical solutions of the embodiments of the present disclosure can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in the embodiments of the present disclosure. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a magnetic disk or an optical disk, and various program code storage media. Thus, the embodiments of the present disclosure are not limited to any specific hardware and software combination.

[0185] Correspondingly, the embodiments of the present disclosure provide a computer device, including a memory and a processor, the memory stores a computer program capable of running on the processor, and the processor implements the steps in the above method when executing the program.

[0186] Correspondingly, the embodiments of the present disclosure provide a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the steps in the above method. The computer readable storage medium can be transitory or non-transitory.

[0187] Correspondingly, the embodiments of the present disclosure provide a computer program product, which includes a non-transitory computer readable storage medium storing a computer program, and the computer program is read and executed by a computer to implement some or all steps in the above method. The computer program product can be specifically implemented by hardware, software or a combination thereof. In one optional embodiment, the computer program product is specifically embodied as a computer storage medium, and in another optional embodiment, the computer program product is specifically embodied as a software product, such as a software development kit (Software Development Kit, SDK) and the like.

[0188] It should be noted that the descriptions of the above storage medium, computer program product and device embodiments are similar to the descriptions of the above method embodiments, and have similar beneficial effects to the method embodiments. For technical details of the storage medium, computer program product and device embodiments of the present disclosure that are not disclosed, please refer to the description of the method embodiments of the present disclosure for understanding.

[0189] It should be noted that, Figure 10 A hardware entity diagram of a computer device in the embodiments of the present disclosure is shown in FIG. 10, which includes a processor 1001, a communication interface 1002 and a memory 1003. Figure 10 The hardware entity of the computer device 1000 includes a processor 1001, a communication interface 1002 and a memory 1003, wherein:

[0190] The processor 1001 generally controls the overall operation of the computer device 1000.

[0191] The communication interface 1002 can enable the computer device to communicate with other terminals or servers through a network.

[0192] The memory 1003 is configured to store instructions and applications executable by the processor 1001, and can also cache data to be processed by the processor 1001 and modules in the computer device 1000 (for example, image data, audio data, voice communication data and video communication data) that have been processed or have been processed, which can be realized by FLASH or RAM. The processor 1001, the communication interface 1002 and the memory 1003 can transmit data through the bus 1004.

[0193] It should be understood that the "one embodiment" or "an embodiment" mentioned throughout the specification means that the specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present disclosure. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present disclosure, the size of the sequence number of each process does not mean the execution order, and the execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present disclosure. The above sequence number of the embodiments of the present disclosure is only for description, not representing the advantages and disadvantages of the embodiments.

[0194] It should be noted that, in the present document, the terms "comprising", "containing" or any other similar term are intended to encompass non-exclusive inclusion, such that processes, methods, articles, or apparatuses that comprise a list of elements are not limited to those elements, but can also include other elements not expressly listed, or inherent to such processes, methods, articles, or apparatuses. Without further limitation, an element defined by the phrase "comprising a" does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0195] In several embodiments provided by the present disclosure, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative, for example, the division of the units is only a logical function division, and actual implementation can have another division manner, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the various components shown or discussed can be through some interface, indirect coupling or communication connection between devices or units, which can be electrical, mechanical or other forms.

[0196] The units described above as separate components can or can not be physically separate, and the components shown as units can or can not be physical units; they can be located in one place or distributed on multiple network units; and part or all of the units can be selected according to actual needs to achieve the purpose of the present embodiment.

[0197] In addition, each functional unit in each embodiment of the present disclosure can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be realized in the form of hardware or in the form of hardware plus software functional unit.

[0198] Those of ordinary skill in the art can understand that all or part of the steps of the above method embodiments can be completed by program instruction related hardware, and the aforementioned program can be stored in a computer readable storage medium, and the program executes the steps of the above method embodiments when executed; and the aforementioned storage medium includes mobile storage devices, ROM, magnetic discs or optical discs, and various storage medium that can store program codes.

[0199] Alternatively, the above-mentioned integrated units of the present disclosure, if implemented in the form of software function modules and sold or used as independent products, can also be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present disclosure can be embodied in the form of a software product, and the computer software product is stored in a storage medium, and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in the various embodiments of the present disclosure. The aforementioned storage medium includes: mobile storage devices, ROM, magnetic disks or optical disks, and various other media that can store program codes.

[0200] The above is only an embodiment of the present disclosure, but the protection scope of the present disclosure is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present disclosure, which should be covered within the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be subject to the protection scope of the claims.

Claims

1. An abnormal event detection method characterized by, The method comprises: obtaining at least two image sequences; wherein each of the image sequences comprises at least one image frame; dividing each of the image sequences into at least two scales to obtain image block sets composed of image blocks at the same position in all image frames at the same scale; for each of the image sequences, obtaining a first feature corresponding to the scale based on the image block sets at the same scale; wherein the first feature comprises the correlation between the image block sets at the same scale; reconstructing the first feature at the same scale according to the positional relationship of the image block sets to obtain a reconstructed feature corresponding to the scale; converting the reconstructed feature corresponding to the scale into a one-dimensional feature vector through a full connection layer after convolution using a preset fifth convolution kernel; accumulating the one-dimensional feature vectors of each scale to obtain a second feature of each of the image sequences; determining the correlation feature between the image sequences based on the second features of the image sequences; determining a target image sequence with an abnormal event in the at least two image sequences according to the correlation feature between the image sequences.

2. The method of claim 1, wherein, The method comprises: extracting features from the image block sets at the same scale to obtain features corresponding to the image block sets; splicing the features of the image block sets at the same scale to obtain a first spliced feature corresponding to the scale; constructing the association relationship between the image block sets at the same scale represented by the first spliced feature using a self-attention mechanism and convolution processing based on the first spliced feature corresponding to the scale to obtain the first feature corresponding to the scale.

3. The method of claim 2, wherein, The method comprises: determining a weight matrix based on the self-attention mechanism and the first spliced feature; wherein the weight matrix comprises weight values representing the probability of abnormality of the image block sets at the same scale; obtaining a weighted feature based on the weight matrix and the first spliced feature; performing convolution processing on the first spliced feature to obtain a convolution result; obtaining the first feature based on the weighted feature, the convolution result, and the first spliced feature.

4. The method of claim 3, wherein, The method comprises: performing dimension reduction processing on the first spliced feature to obtain a reduced first spliced feature; performing convolution on the reduced first spliced feature using a preset first convolution kernel to obtain a first convolution result; performing convolution on the reduced first spliced feature using a preset second convolution kernel to obtain a second convolution result; multiplying the transposed first convolution result and the transposed second convolution result using the self-attention mechanism to determine the weight matrix.

5. The method of claim 4, wherein, The method comprises: convolve the first spliced feature after dimension reduction by using a preset third convolution kernel to obtain a third convolution result; multiply the weight matrix and the third convolution result to obtain a weighted matrix; determine the weighted feature as the sum of the weighted matrix and the result of the convolution of the preset fourth convolution kernel and the first spliced feature after dimension reduction.

6. The method of claim 3, wherein, The convolution processing of the first spliced feature to obtain the convolution feature includes: convolve the first spliced feature by using at least two hole convolution kernels to obtain the convolution result corresponding to each hole convolution kernel; wherein the hole rates of the at least two hole convolution kernels are different; splicing the convolution results corresponding to each hole convolution kernel to obtain the convolution feature.

7. The method of claim 3, wherein, The first feature is obtained based on the weighted feature, the convolution feature, and the first spliced feature, including: splicing the weighted feature and the convolution feature, and then adding the first spliced feature to obtain the first feature.

8. The method of claim 2, wherein, The feature extraction of each image block set under the same scale to obtain the feature corresponding to the image block set includes: performing feature extraction on each image block set under the same scale to obtain the feature corresponding to the image block set, which includes the temporal information between each image block in the image block set.

9. The method of claim 1, wherein, The correlation feature between each image sequence is determined based on the second feature of each image sequence, including: splicing the second features of each image sequence to obtain a second spliced feature; based on the second spliced feature, based on the self-attention mechanism and the hole convolution, the correlation between different image sequences represented by the second spliced feature is constructed to determine the correlation feature between each image sequence.

10. The method according to any one of claims 1 to 9, characterized in that, The target image sequence in which the abnormal event exists is determined from the at least two image sequences according to the correlation feature between each image sequence, including: detecting the correlation feature between each image sequence based on a preset abnormal prediction model to obtain a prediction result of each image sequence; wherein the preset abnormal prediction model is a model trained by using a weak supervision training method; determine the target image sequence in which the abnormal event exists according to the prediction result of each image sequence.

11. The method of claim 10, wherein, The method further includes: selecting K sample image sequences with larger feature gradients for each of the positive samples and the negative samples in the training sample set to calculate the average feature gradient; wherein K is a positive integer greater than 1; constructing a loss function according to the average feature gradient corresponding to the positive sample and the average feature gradient corresponding to the negative sample; training the preset abnormal prediction model based on the loss function.

12. The method according to any one of claims 1 to 9, characterized in that, The at least two image sequences are obtained, including: obtaining a video to be detected; determining the difference value between adjacent frame images in the video to be detected; determining the image frame earlier in time in the adjacent frame images with a difference value greater than a preset difference threshold as the tail frame of one image sequence, and determining the image frame later in time as the head frame of an image sequence adjacent to one image sequence.

13. An abnormal event detection apparatus characterized by comprising: The device includes: An acquisition module is configured to acquire at least two image sequences; each of the image sequences includes at least one image frame; A division module is configured to divide each of the image sequences into at least two scales to obtain image block sets composed of image blocks at the same position in all image frames at the same scale; A first determination module is configured to, for each of the image sequences, obtain a first feature corresponding to a scale based on the image block sets at the same scale; the first feature includes the correlation between the image block sets at the same scale; reconstruct the first feature at the same scale according to the positional relationship of the image block sets to obtain a reconstructed feature corresponding to the scale; after the reconstructed feature corresponding to the scale is convolved using a preset fifth convolution kernel, convert the reconstructed feature into a one-dimensional feature vector through a full connection layer; accumulate the one-dimensional feature vectors of all scales to obtain a second feature of each of the image sequences; determine the correlation feature between the image sequences based on the second features of the image sequences; A second determination module is configured to determine a target image sequence in which an abnormal event exists from the at least two image sequences according to the correlation feature between the image sequences.

14. A computer device, comprising: comprise: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to perform the abnormal event detection method of any one of claims 1 to 12.

15. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the abnormal event detection method of any one of claims 1 to 12.

Images