Video anomaly detection method, device and computer readable storage medium

Abstract

The present specification provides a video anomaly detection method, device and computer readable storage medium. Based on the method, before implementation, a preset feature processing model including at least a cross-modal attention mechanism module is trained in advance. During implementation, the target video obtained is first divided into a plurality of target image segments and a plurality of target optical flow segments; then the plurality of target image segments and the plurality of target optical flow segments are processed by using a preset feature extraction network to extract corresponding target appearance features and target motion features; then the preset feature processing model is used to remove redundant information of the target appearance features and the target motion features through the cross-modal attention mechanism module to obtain and fuse the optimized target appearance features and the optimized target motion features; and then the preset feature processing model is used to process the fused target features to obtain a target processing result, so that whether the target video has an abnormal target can be accurately and efficiently determined.

Description

Technical Field

[0001] This specification belongs to the field of video data processing technology, and in particular relates to video anomaly detection methods, devices and computer-readable storage media. Background Technology

[0002] In some application scenarios involving video data processing, such as road traffic monitoring, it is usually necessary to train and utilize corresponding video processing models to help staff detect and identify anomalies in the video in a timely manner.

[0003] However, when video processing models trained based on existing methods are applied to video anomaly detection, they often need to extract and process video features containing a large amount of redundant information at the same time. This leads to technical problems such as low detection efficiency and easy detection errors in actual implementation.

[0004] There is currently no effective solution to the above problems. Summary of the Invention

[0005] This specification provides a video anomaly detection method, apparatus, and computer-readable storage medium. By training and utilizing a pre-defined feature processing model that includes at least a cross-modal attention mechanism module, it is possible to accurately and efficiently detect and identify the presence of abnormal targets in a target video.

[0006] This manual provides a video anomaly detection method, including:

[0007] Acquire the target video;

[0008] Based on the target video, multiple target image segments and multiple target optical flow segments are obtained;

[0009] By using a pre-defined feature extraction network to process multiple target image segments and multiple target optical flow segments, the corresponding target appearance features and target motion features are obtained.

[0010] The target appearance features and target motion features are processed using a preset feature processing model to obtain target processing results for the target video; wherein, the preset feature processing model includes at least a cross-modal attention mechanism module; the cross-modal attention mechanism module is used to remove redundant information from the target appearance features and target motion features to obtain corresponding optimized target appearance features and optimized target motion features; the cross-modal attention mechanism module is also used to fuse the optimized target appearance features and optimized target motion features to obtain fused target features;

[0011] Based on the target processing results, it is determined whether there are any abnormal targets in the target video.

[0012] In one embodiment, the preset feature processing model further includes a fully connected neural network; wherein the fully connected neural network is used to process the fused target features and output the corresponding target processing result.

[0013] In one embodiment, the cross-modal attention mechanism module removes redundant information from the target appearance features in the following manner to obtain the corresponding optimized target appearance features:

[0014] The target appearance features are determined as the first primary modal features, and the target motion features are determined as the first auxiliary modal features;

[0015] Based on the first primary modality feature, first global information is obtained, and based on the first auxiliary modality feature, first local information is obtained;

[0016] Based on the first global information and the first local information, redundant information of the target appearance features is detected and removed to obtain the corresponding optimized target appearance features.

[0017] In one embodiment, the cross-modal attention mechanism module includes at least: an average pooling layer, a first convolutional layer, a second convolutional layer, and a sigmoid function.

[0018] In one embodiment, obtaining first global information based on the first dominant modality features includes:

[0019] The average pooling layer processes the first master modality features and outputs corresponding initial first global information; the first convolutional layer processes the initial first global information, obtains and utilizes the dependencies of the corresponding channels, and outputs corresponding first global awareness descriptors as the first global information.

[0020] In one embodiment, obtaining first local information based on the first auxiliary modal features includes:

[0021] The second convolutional layer processes the first auxiliary modality features and outputs the corresponding first local perceptual descriptor as the first local information.

[0022] In one embodiment, based on the first global information and the first local information, redundant information of the target appearance features is detected and removed to obtain corresponding optimized target appearance features, including:

[0023] Multiply the first global information and the first local information to obtain the first channel descriptor;

[0024] By using the first optimization weights at the channel level generated based on the Sigmoid function and the first channel descriptor, and processing the first master mode features, the corresponding optimized target appearance features are obtained.

[0025] In one embodiment, the method further includes:

[0026] Construct an initial model; wherein the initial model includes at least an initial cross-modal attention mechanism module and an initial fully connected neural network;

[0027] Obtain and construct the positive and negative packets of the samples based on the sample videos;

[0028] Based on the positive and negative packets of the samples, and the preset feature extraction network, the corresponding sample appearance features and sample motion features are obtained.

[0029] Based on a preset loss function, the initial model is trained using the appearance features and motion features of the samples to obtain a preset feature processing model that meets the requirements.

[0030] This specification also provides a video anomaly detection device, including:

[0031] The acquisition module is used to acquire the target video;

[0032] The segmentation module is used to segment the target video into multiple target image segments and multiple target optical flow segments;

[0033] The extraction module is used to obtain the corresponding target appearance features and target motion features by processing multiple target image segments and multiple target optical flow segments through a preset feature extraction network;

[0034] A processing module is used to process the target appearance features and target motion features using a preset feature processing model to obtain a target processing result for the target video; wherein, the preset feature processing model includes at least a cross-modal attention mechanism module; the cross-modal attention mechanism module is used to remove redundant information from the target appearance features and target motion features to obtain corresponding optimized target appearance features and optimized target motion features; the cross-modal attention mechanism module is also used to fuse the optimized target appearance features and optimized target motion features to obtain fused target features;

[0035] The determination module is used to determine whether there are abnormal targets in the target video based on the target processing results.

[0036] This specification also provides a computer-readable storage medium storing computer instructions that, when executed by a processor, perform the following steps: acquiring a target video; dividing the target video into multiple target image segments and multiple target optical flow segments; using a preset feature extraction network to process the multiple target image segments and multiple target optical flow segments to obtain corresponding target appearance features and target motion features; processing the target appearance features and target motion features using a preset feature processing model to obtain a target processing result for the target video; wherein the preset feature processing model includes at least a cross-modal attention mechanism module; the cross-modal attention mechanism module is used to remove redundant information from the target appearance features and target motion features to obtain corresponding optimized target appearance features and optimized target motion features; the cross-modal attention mechanism module is also used to fuse the optimized target appearance features and optimized target motion features to obtain fused target features; and determining whether there are abnormal targets in the target video based on the target processing result.

[0037] Based on the video anomaly detection method, apparatus and computer-readable storage medium provided in this specification, before implementation, an initial model including at least an initial cross-modal attention mechanism module and an initial fully connected neural network can be constructed; and using positive and negative sample packets, as well as a preset feature extraction network, the initial model can be trained through weakly supervised learning to obtain a preset feature processing model that is compatible with the existing feature extraction network and includes at least a cross-modal attention mechanism module. In practice, after acquiring the target video to be processed, the acquired target video can first be divided into multiple target image segments and multiple target optical flow segments. Then, a preset feature extraction network is used to process the multiple target image segments and multiple target optical flow segments to extract video features of two different dimensions: target appearance features and target motion features. Next, a preset feature processing model is used to remove redundant information of the target appearance features and target motion features through a cross-modal attention mechanism module, resulting in corresponding optimized target appearance features and optimized target motion features for video anomaly detection. By fusing the above two different dimensions of features, a fused target feature with rich and comprehensive feature information for video anomaly detection is obtained. Then, the preset feature processing model is used to process the fused target feature through a fully connected neural network to obtain the corresponding target processing result, so that the presence of abnormal targets in the target video can be determined based on the target processing result. This allows for accurate and efficient detection and identification of abnormal targets in the target video, effectively improving the detection efficiency and accuracy of video anomaly detection. Attached Figure Description

[0038] To more clearly illustrate the embodiments of this specification, the accompanying drawings used in the embodiments will be briefly introduced below. The drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 This is a flowchart illustrating a video anomaly detection method provided in one embodiment of this specification;

[0040] Figure 2 This is a schematic diagram illustrating an embodiment of the video anomaly detection method provided in this specification, applied in a scenario example.

[0041] Figure 3 This is a schematic diagram illustrating an embodiment of the video anomaly detection method provided in this specification, applied in a scenario example.

[0042] Figure 4 This is a schematic diagram illustrating an embodiment of the video anomaly detection method provided in this specification, applied in a scenario example.

[0043] Figure 5 This is a schematic diagram illustrating an embodiment of the video anomaly detection method provided in this specification, applied in a scenario example.

[0044] Figure 6 This is a schematic diagram illustrating an embodiment of the video anomaly detection method provided in this specification, applied in a scenario example.

[0045] Figure 7 This is a schematic diagram illustrating an embodiment of the video anomaly detection method provided in this specification, applied in a scenario example.

[0046] Figure 8 This is a flowchart illustrating a model training method provided in one embodiment of this specification;

[0047] Figure 9 This is a schematic diagram of the structural composition of a server provided in one embodiment of this specification;

[0048] Figure 10 This is a schematic diagram of the structural composition of a video anomaly detection device provided in one embodiment of this specification. Detailed Implementation

[0049] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0050] See Figure 1 As shown in the embodiments of this specification, a video anomaly detection method is provided. In specific implementation, this method may include the following:

[0051] S101: Acquire the target video;

[0052] S102: Based on the target video, multiple target image segments and multiple target optical flow segments are obtained;

[0053] S103: By using a preset feature extraction network to process multiple target image segments and multiple target optical flow segments, the corresponding target appearance features and target motion features are obtained;

[0054] S104: Process the target appearance features and target motion features using a preset feature processing model to obtain target processing results for the target video; wherein, the preset feature processing model includes at least: a cross-modal attention mechanism module; the cross-modal attention mechanism module is used to remove redundant information from the target appearance features and target motion features to obtain corresponding optimized target appearance features and optimized target motion features; the cross-modal attention mechanism module is also used to fuse the optimized target appearance features and optimized target motion features to obtain fused target features;

[0055] S105: Based on the target processing result, determine whether there is an abnormal target in the target video.

[0056] In some embodiments, the target video can be specifically understood as video data to be detected for the presence of abnormal targets.

[0057] Specifically, depending on the application scenario, the target video mentioned above can be different types of video data. For example, see [link to relevant documentation]. Figure 2 In the context of road traffic monitoring, (a), (b), (c), and (d) specifically refer to video data containing road surface conditions collected by road monitoring cameras. For example, see [link to relevant documentation]. Figure 2In the context of supermarket security, (e) and (f) can specifically refer to video data containing information about the interior of the supermarket, collected by security cameras installed in the supermarket. It should be noted that the target videos listed above are merely illustrative. In practice, depending on the specific application scenario and processing requirements, the target videos may include other types of video data. This specification does not limit this.

[0058] Specifically, the aforementioned abnormal targets may include abnormal objects, and / or abnormal behaviors, etc.

[0059] Specifically, the aforementioned abnormal objects can be understood as items that require attention or pose a risk in the targeted application scenario. For example, see [link to relevant documentation]. Figure 2 In the context of road traffic monitoring, the abnormal objects mentioned in (a), (b), and (c) can be items that are not allowed to enter the sidewalk, such as bicycles, skateboards, and trucks.

[0060] The aforementioned abnormal behaviors can be specifically understood as actions or behaviors that require attention or pose a risk in the targeted application scenario. For example, see [link to relevant documentation]. Figure 2 In (d) of the above, in a road traffic monitoring scenario, the aforementioned abnormal behavior could be a traffic accident occurring on the road. For example, see [reference needed]. Figure 2 In the context of supermarket security, (e) and (f) refer to abnormal behaviors that may occur within the supermarket.

[0061] In some embodiments, as specifically implemented, refer to Figure 3 As shown, based on the target video, multiple target image segments (which can be abbreviated as RGB) and multiple target optical flow segments (which can be abbreviated as Optical Flow) can be obtained for the target video. Among them, the target image segments contain at least appearance feature information about the target video, and the target optical flow segments contain at least motion feature information about the target video.

[0062] In some embodiments, during specific implementation, a preset feature extraction network can be used to extract the corresponding target appearance features by processing multiple target image segments; and to extract the corresponding target motion features by processing multiple target optical flow segments.

[0063] In some embodiments, the preset feature extraction network described above includes a pre-trained I3D network. Specifically, the preset feature extraction network may be an I3D network pre-trained using the Kinetics dataset. In some cases, the preset feature extraction network may also include a pre-trained C3D network.

[0064] Specifically, the Kinetics dataset mentioned above is designed for video action classification. Correspondingly, when using a pre-defined feature extraction network trained on the Kinetics dataset to directly extract features, it is more suitable for video action classification. However, when used for video anomaly detection, it often contains a large amount of redundant information. This redundant information increases the overall data processing volume, affecting the detection efficiency of video anomaly detection; furthermore, it can mask and interfere with key feature data in video anomaly detection, thus affecting the detection accuracy.

[0065] In some embodiments, see specific examples. Figure 4 As shown, the aforementioned I3D network may specifically include the following structure: convolutional layers (e.g., Conv), pooling layers (e.g., Max-Pool, Avg-Pool), and Inc layers (e.g., Inc.). Here, Rec.field represents the receptive field.

[0066] Further reading Figure 5 As shown, the Inc layer can increase the network's adaptability to scale by using 1×1×1 and 3×3×3 convolutions, so that appearance and motion features of different scales can be extracted through the preset feature extraction network, resulting in relatively better feature extraction performance.

[0067] In specific implementation, the above-mentioned use of a preset feature extraction network to process multiple target image segments and multiple target optical flow segments to obtain corresponding target appearance features and target motion features may include: using an I3D network to perform convolution and pooling operations on the target image segments and target optical flow segments in the spatiotemporal domain using convolution kernels and pooling kernels respectively, to obtain a 1024-dimensional spatial appearance feature vector and a 1024-dimensional temporal motion feature vector, which are respectively used as the target appearance features and the target motion features.

[0068] In some embodiments, the aforementioned preset feature processing model can be understood as a neural network model that can detect and identify abnormal targets in a video, obtained through weakly supervised training based on a cross-modal attention mechanism. The aforementioned preset feature processing model includes at least a cross-modal attention module (CAM).

[0069] Based on the aforementioned preset feature processing model, on the one hand, redundant information of target appearance features and target motion features can be effectively removed to obtain optimized target appearance features and optimized target motion features suitable for video anomaly detection; on the other hand, the two different dimensions of feature data, the optimized target appearance features and optimized target motion features, can be fully integrated to obtain fused target features that are rich and comprehensive in information for video anomaly detection. Thus, the fused target features can be used to accurately and efficiently realize video anomaly detection to determine whether there are abnormal targets.

[0070] In some embodiments, the preset feature processing model may further include a fully connected neural network; wherein the fully connected neural network is specifically used to process the fused target features and output the corresponding target processing result.

[0071] In some embodiments, the cross-modal attention mechanism module can remove redundant information of the target appearance features in the following manner to obtain the corresponding optimized target appearance features:

[0072] S1: The target appearance feature (e.g., F) is then... appearance ) is identified as the first principal modality feature, and the target motion feature (e.g., F) is determined as the first principal modality feature. motion This was identified as the first auxiliary modal feature;

[0073] S2: Based on the first primary modality feature, obtain the first global information (e.g., M). G And based on the first auxiliary modal features, obtain the first local information (e.g., M). L );

[0074] S3: Based on the first global information and the first local information, detect and remove redundant information of the target appearance features to obtain the corresponding optimized target appearance features (e.g., ).

[0075] Similarly, the cross-modal attention mechanism module can remove redundant information from the target motion features in the following manner to obtain the corresponding optimized target motion features (e.g., ):

[0076] S1: The target motion feature is determined as the second primary modal feature, and the target appearance feature is determined as the second auxiliary modal feature;

[0077] S2: Based on the second primary modality feature, obtain the second global information, and based on the second auxiliary modality feature, obtain the second local information;

[0078] S3: Based on the second global information and the second local information, detect and remove redundant information of the target motion features to obtain the corresponding optimized target motion features.

[0079] In some embodiments, see Figure 6 As shown, the cross-modal attention mechanism module may include at least: an average pooling layer (which may be denoted as AvgPool), a first convolutional layer (which may be denoted as F... G ), the second convolutional layer (which can be denoted as F) L Structures such as ), and the Sigmoid function.

[0080] The following example illustrates how to optimize target appearance features by using a cross-modal attention mechanism module to remove redundant information. For details on how to optimize target appearance features using the cross-modal attention mechanism module in a pre-defined feature processing model, please refer to the embodiments on optimizing target appearance features; these will not be repeated here.

[0081] In some embodiments, obtaining the first global information based on the first master modality feature may specifically include: the average pooling layer processing the first master modality feature to output corresponding initial first global information; and the first convolutional layer processing the initial first global information to obtain and utilize the dependencies of corresponding channels to output a corresponding first global awareness descriptor as the first global information.

[0082] Specifically, for example, see Figure 6 As shown, the preset feature processing model can first process the first principal modality feature F appearance The input is fed into the average pooling layer AvgPool in the cross-modal attention mechanism module to obtain the initial first global information X of the first primary modality feature. g Then utilize the first convolutional layer F G By processing this initial global information, the dependencies of the channels are obtained and utilized to generate and output the first global awareness descriptor M. G As the primary global information.

[0083] The specific operation can be represented by the following formula: X g =AvgPool(F appearance M G =F G (X g ).

[0084] Furthermore, the kernel of the first convolutional layer can be 3×3 with a stride of 1. Additionally, before performing the convolution operation using the first convolutional layer, the outer edges of the feature map (e.g., the initial first global information) can be padded with zeros of scale 1. This allows the first convolutional layer to achieve relatively better processing results during operation.

[0085] In some embodiments, considering that multiple modalities provide relevant feature information from different perspectives, cross-modal local information obtained from the first auxiliary modal features can be used to detect and remove redundant information in the first primary modal features that is irrelevant to the video anomaly detection task.

[0086] Based on the above considerations, the above-mentioned acquisition of first local information based on the first auxiliary modality features may specifically include: the second convolutional layer processing the first auxiliary modality features and outputting a corresponding first local perceptual descriptor as the first local information.

[0087] Specifically, for example, see Figure 6 As shown, the preset feature processing model can input the first auxiliary modality features into the second convolutional layer F. L To generate a first local perceptual descriptor M across modalities L This serves as the first local information.

[0088] The specific operation can be represented by the following formula: M L =F L (F motion ).

[0089] Furthermore, the kernel of the second convolutional layer can be 3×3 with a stride of 1. Additionally, before performing the convolution operation using the second convolutional layer, the outer edges of the feature map (e.g., the first auxiliary modality feature) can be padded with zeros of scale 1. This allows the second convolutional layer to achieve relatively better processing results during operation.

[0090] In some embodiments, the above-mentioned detection and removal of redundant information of the target appearance features based on the first global information and the first local information to obtain the corresponding optimized target appearance features may specifically include: multiplying the first global information and the first local information to obtain a first channel descriptor; and using the first optimization weight at the channel level generated based on the Sigmoid function and the first channel descriptor to process the first master mode feature to obtain the corresponding optimized target appearance features.

[0091] Specifically, for example, see Figure 6As shown, firstly, the preset feature processing model can multiply the first global information and the first local information to obtain the first channel descriptor M used to optimize appearance features. The specific operation can be represented as follows: in, The multiplication operator represents elements.

[0092] Next, the pre-defined feature processing model can first generate the corresponding channel-level first optimized weight σ(M) based on the Sigmoid function and the first channel descriptor; then, σ(M) is used to detect and remove redundant information of the target appearance features to obtain the optimized target appearance features. The specific operation can be expressed as follows: Where σ(·) represents the Sigmoid function.

[0093] Based on the above embodiments, through the cross-modal attention mechanism module in the preset feature processing model, contextual global information specific to the main modality can be obtained from the main modality features, while cross-modal specific local information can be obtained from the auxiliary modality features. Then, by aggregating the global and local information from different modalities, redundant information irrelevant to the video anomaly detection task in the target appearance features and target motion features can be filtered out accurately and effectively, thereby achieving optimized processing of the target appearance features and target motion features, resulting in optimized target appearance features and optimized target motion features with better performance for video anomaly detection.

[0094] In some embodiments, the cross-modal attention mechanism module can obtain the required fused target features by fusing the optimized target appearance features and the optimized target motion features according to the following formula:

[0095]

[0096] Where v represents the fused target features, and cat(.) represents the splicing operation between the optimized target appearance features and the optimized target motion features.

[0097] Based on the above embodiments, the cross-modal attention mechanism module in the preset feature processing model can effectively fuse two different dimensions of feature data: optimized target appearance features and optimized target motion features. This results in a fused target feature with rich and comprehensive feature information and relatively good performance for video anomaly detection. Subsequently, the preset feature processing model can accurately determine whether there is an abnormal target in the target video based on the fused target feature using a fully connected neural network.

[0098] In some embodiments, see Figure 7 As shown, in specific implementations, the method may also include the following:

[0099] S1: Construct an initial model; wherein the initial model includes at least an initial cross-modal attention mechanism module and an initial fully connected neural network;

[0100] S2: Obtain and construct the positive and negative packets of the samples based on the sample videos;

[0101] S3: Based on the positive and negative packets of the samples, and the preset feature extraction network, obtain the corresponding sample appearance features and sample motion features;

[0102] S4: Based on the preset loss function, the initial model is trained using the appearance features and motion features of the sample to obtain a preset feature processing model that meets the requirements.

[0103] For specific implementation, please refer to Figure 7 As shown, after acquiring the sample video, firstly, the sample video can be marked as a normal sample video (e.g., Normal Video) or an abnormal sample video (e.g., Anomaly Video) based on whether there are abnormal targets in the sample video; then, the normal sample videos are divided into negative packets (e.g., B). n ), dividing the abnormal sample video into positive sample packets (e.g., B a Dropouts represents random deactivation.

[0104] Next, the sample videos in the positive and negative sample packets can be divided to obtain multiple (for example) sample image segments and sample optical flow segments, which can be denoted as follows: and Then, the pre-trained I3D feature extraction network is used to extract the corresponding sample appearance features and sample motion features, which can be denoted as follows: and

[0105] Furthermore, based on a preset loss function (e.g., Loss), the initial model can be trained using the sample appearance features and sample motion features. The model parameters can be continuously adjusted to calculate the anomaly score (e.g., Anomaly Score) of each sample video in the positive and negative sample packets based on the model. The feature processing model that meets the requirements is obtained when the difference between the abnormal scores of normal sample videos and abnormal sample videos exceeds a preset difference threshold.

[0106] In some embodiments, the aforementioned preset loss function may specifically be a loss function based on an improved MIL ranking algorithm.

[0107] Specifically, the aforementioned preset loss function can be used to calculate the MIL ranking loss by using the highest anomaly score of the sample video in the positive and negative sample packets, and then combine this ranking loss with the hinge loss function.

[0108] Based on the aforementioned preset loss function, the gap between the abnormal scores of abnormal sample videos and normal sample videos can be widened more effectively, so that the model can more accurately distinguish between normal and abnormal videos.

[0109] Specifically, the aforementioned preset loss function can be expressed in the following form:

[0110]

[0111] Among them, v a v represents the fused features of abnormal sample videos within the positive sample package. n f(v) represents the fused features of normal sample videos within the negative sample packet. a ) and f(v n ) represent the anomaly scores of abnormal sample videos in the positive sample packet and the anomaly scores of normal sample videos in the negative sample packet, respectively. and These represent the highest abnormal score of abnormal sample videos in the positive sample packet and the highest abnormal score of normal sample videos in the negative sample packet, respectively.

[0112] In some embodiments, on the one hand, considering the continuity of video data in terms of data information, the differences between adjacent segments arranged in sequence should be as small as possible and have smoothness. Therefore, a smoothing term 'a' can be further introduced into the loss function for control.

[0113] The smoothing term 'a' can be set according to the following formula:

[0114] On the other hand, considering that in actual application scenarios, abnormal targets often occur infrequently and for short durations in the acquired videos, resulting in relatively sparse abnormal scores in the positive sample packets, a sparsity term b can be further introduced into the loss function.

[0115] The sparse term b can be specifically set according to the following formula:

[0116] Furthermore, to avoid overfitting during model training, a regularization term ||W|| can be introduced into the preset loss function. F To take control.

[0117] Taking the above considerations into account, the preset loss function can be expressed in the following form:

[0118] L(W)=l(B a B n )+λ1a+λ2b+||W|| F

[0119] Where λ1 and λ2 are the weight coefficients of the smoothing term and the sparse term, respectively.

[0120] Based on the aforementioned preset loss function, a preset feature processing model that meets the requirements can be trained more quickly.

[0121] In some embodiments, after determining whether an abnormal target exists in the target video based on the target processing result, the method further includes: if an abnormal target is determined to exist in the target video, generating an alarm notification about the abnormal target to alert relevant personnel. Furthermore, if an abnormal target is determined to exist in the target video, the target video can be archived in a designated abnormal video database for later retrospective querying.

[0122] As can be seen from the above, the video anomaly detection method provided in this specification can be implemented by constructing an initial model that includes at least an initial cross-modal attention mechanism module and an initial fully connected neural network before implementation. The initial model is then trained using positive and negative sample packets and a preset feature extraction network to obtain a preset feature processing model that includes at least the cross-modal attention mechanism module. In specific implementation, after acquiring the target video to be processed, the acquired target video can be divided into multiple target image segments and multiple target optical flow segments. Then, the preset feature extraction network is used to process the multiple target image segments and multiple target optical flow segments to obtain video features of two different dimensions: target appearance features and target motion features. Next, the preset feature processing model is used to remove redundant information from the target appearance features and target motion features through the cross-modal attention mechanism module to obtain corresponding optimized target appearance features and optimized target motion features for video anomaly detection. These two features are then fused to obtain a fused target feature that is relatively rich and comprehensive for video anomaly detection. Finally, the preset feature processing model is used to process the fused target feature through a fully connected neural network to obtain the corresponding target processing result, which is used to determine whether an abnormal target exists in the target video. This allows for the accurate and efficient detection and identification of abnormal targets in target videos, effectively improving the detection efficiency and accuracy of video anomaly detection.

[0123] See Figure 8 As shown, this specification also provides a model training method suitable for video anomaly detection, which may include the following in its implementation:

[0124] S801: Construct an initial model; wherein the initial model includes at least an initial cross-modal attention mechanism module and an initial fully connected neural network;

[0125] S802: Obtain and construct the positive and negative packets of the samples based on the sample videos;

[0126] S803: Based on the positive and negative packets of the samples and the preset feature extraction network, obtain the corresponding sample appearance features and sample motion features;

[0127] S804: Based on a preset loss function, the initial model is trained using the appearance features and motion features of the samples to obtain a preset feature processing model that meets the requirements.

[0128] The pre-defined feature processing model trained using the above method can accurately and efficiently determine whether there are abnormal targets in the target video.

[0129] This specification also provides a server, including a processor and a memory for storing processor-executable instructions. Specifically, the processor can perform the following steps according to the instructions: acquiring a target video; dividing the target video into multiple target image segments and multiple target optical flow segments; using a preset feature extraction network to process the multiple target image segments and multiple target optical flow segments to obtain corresponding target appearance features and target motion features; processing the target appearance features and target motion features using a preset feature processing model to obtain a target processing result for the target video; wherein the preset feature processing model includes at least a cross-modal attention mechanism module; the cross-modal attention mechanism module is used to remove redundant information from the target appearance features and target motion features to obtain corresponding optimized target appearance features and optimized target motion features; the cross-modal attention mechanism module is also used to fuse the optimized target appearance features and optimized target motion features to obtain fused target features; and determining whether there are abnormal targets in the target video based on the target processing result.

[0130] To execute the above instructions more accurately, please refer to... Figure 9 As shown in the embodiments of this specification, another specific server is also provided, wherein the server includes a network communication port 901, a processor 902, and a memory 903. The above structures are connected by internal cables so that the various structures can perform specific data interaction.

[0131] Specifically, the network communication port 901 can be used to acquire the target video.

[0132] The processor 902 is specifically configured to: divide a target video into multiple target image segments and multiple target optical flow segments; use a preset feature extraction network to process the multiple target image segments and multiple target optical flow segments to obtain corresponding target appearance features and target motion features; use a preset feature processing model to process the target appearance features and target motion features to obtain a target processing result for the target video; wherein the preset feature processing model includes at least a cross-modal attention mechanism module; the cross-modal attention mechanism module is used to remove redundant information from the target appearance features and target motion features to obtain corresponding optimized target appearance features and optimized target motion features; the cross-modal attention mechanism module is also used to fuse the optimized target appearance features and optimized target motion features to obtain fused target features; and determine whether there are abnormal targets in the target video based on the target processing result.

[0133] The memory 903 can be used to store the corresponding instruction program.

[0134] In this embodiment, the network communication port 901 can be a virtual port bound to different communication protocols, thereby enabling the sending or receiving of different data. For example, the network communication port can be a port responsible for web data communication, a port responsible for FTP data communication, or a port responsible for email data communication. Furthermore, the network communication port can also be a physical communication interface or communication chip. For example, it can be a wireless mobile network communication chip, such as GSM or CDMA; it can also be a Wi-Fi chip; or it can be a Bluetooth chip.

[0135] In this embodiment, the processor 902 can be implemented in any suitable manner. For example, the processor can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers, etc. This specification is not limiting.

[0136] In this embodiment, the memory 903 may include multiple layers. In a digital system, anything that can store binary data can be a memory. In an integrated circuit, a circuit with storage function but no physical form is also called a memory, such as RAM, FIFO, etc. In a system, a storage device with a physical form is also called a memory, such as a memory stick, TF card, etc.

[0137] This specification also provides a computer-readable storage medium based on the above-described video anomaly detection method. The computer-readable storage medium stores computer program instructions that, when executed, implement the following: acquiring a target video; dividing the target video into multiple target image segments and multiple target optical flow segments; using a preset feature extraction network to process the multiple target image segments and multiple target optical flow segments to obtain corresponding target appearance features and target motion features; processing the target appearance features and target motion features using a preset feature processing model to obtain a target processing result for the target video; wherein the preset feature processing model includes at least a cross-modal attention mechanism module; the cross-modal attention mechanism module is used to remove redundant information from the target appearance features and target motion features to obtain corresponding optimized target appearance features and optimized target motion features; the cross-modal attention mechanism module is also used to fuse the optimized target appearance features and optimized target motion features to obtain fused target features; and determining whether an abnormal target exists in the target video based on the target processing result.

[0138] In this embodiment, the storage medium includes, but is not limited to, Random Access Memory (RAM), Read-Only Memory (ROM), cache, hard disk drive (HDD), or memory card. The memory can be used to store computer program instructions. The network communication unit can be an interface configured according to standards specified in the communication protocol for network connection communication.

[0139] In this embodiment, the specific functions and effects implemented by the program instructions stored in the computer-readable storage medium can be explained in comparison with other embodiments, and will not be repeated here.

[0140] See Figure 10 As shown, at the software level, this specification also provides a video anomaly detection device, which may specifically include the following structural modules:

[0141] Module 1001 can be used to acquire target videos.

[0142] The segmentation module 1002 can be used to segment the target video into multiple target image segments and multiple target optical flow segments.

[0143] The extraction module 1003 can be used to obtain the corresponding target appearance features and target motion features by using a preset feature extraction network to process multiple target image segments and multiple target optical flow segments.

[0144] The processing module 1004 is specifically used to process the target appearance features and target motion features using a preset feature processing model to obtain target processing results for the target video; wherein, the preset feature processing model includes at least a cross-modal attention mechanism module; the cross-modal attention mechanism module is used to remove redundant information from the target appearance features and target motion features to obtain corresponding optimized target appearance features and optimized target motion features; the cross-modal attention mechanism module is also used to fuse the optimized target appearance features and optimized target motion features to obtain fused target features;

[0145] The determination module 1005 can be used to determine whether there are abnormal targets in the target video based on the target processing results.

[0146] In some embodiments, the preset feature processing model may further include a fully connected neural network, etc.; wherein the fully connected neural network is used to process the fused target features and output the corresponding target processing result.

[0147] In some embodiments, the cross-modal attention mechanism module can specifically remove redundant information of the target appearance features in the following manner to obtain corresponding optimized target appearance features: determining the target appearance features as a first primary modal feature and determining the target motion features as a first auxiliary modal feature; obtaining first global information based on the first primary modal feature and obtaining first local information based on the first auxiliary modal feature; detecting and removing redundant information of the target appearance features based on the first global information and the first local information to obtain corresponding optimized target appearance features.

[0148] In some embodiments, the cross-modal attention mechanism module may include at least the following structures: an average pooling layer, a first convolutional layer, a second convolutional layer, and a sigmoid function.

[0149] In some embodiments, the cross-modal attention mechanism module can specifically obtain first global information based on the first master modality feature in the following manner: by processing the first master modality feature through the average pooling layer, the corresponding initial first global information is output; by processing the initial first global information, the first convolutional layer obtains and utilizes the dependency relationship of the corresponding channel, and outputs the corresponding first global awareness descriptor as the first global information.

[0150] In some embodiments, the cross-modal attention mechanism module can obtain first local information based on the first auxiliary modal features in the following manner: by processing the first auxiliary modal features through the second convolutional layer, the corresponding first local perceptual descriptor is output as the first local information.

[0151] In some embodiments, the cross-modal attention mechanism module can specifically detect and remove redundant information of the target appearance features based on the first global information and the first local information to obtain the corresponding optimized target appearance features in the following manner: multiply the first global information and the first local information to obtain a first channel descriptor; and use the first optimization weight at the channel level generated based on the Sigmoid function and the first channel descriptor to process the first dominant modal features to obtain the corresponding optimized target appearance features.

[0152] In some embodiments, the device may further include a training model, which can be used to construct an initial model; wherein the initial model includes at least an initial cross-modal attention mechanism module and an initial fully connected neural network.

[0153] The sample positive and negative packets are constructed based on the sample videos. Based on the sample positive and negative packets and a preset feature extraction network, the corresponding sample appearance features and sample motion features are obtained. Based on a preset loss function, the initial model is trained using the sample appearance features and sample motion features to obtain a preset feature processing model that meets the requirements.

[0154] It should be noted that the units, devices, or modules described in the above embodiments can be implemented by computer chips or physical entities, or by products with certain functions. For ease of description, the above devices are described by dividing them into various modules according to their functions. Of course, in implementing this specification, the functions of each module can be implemented in one or more software and / or hardware, or the module that implements the same function can be implemented by a combination of multiple sub-modules or sub-units, etc. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection between the devices or units shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0155] As can be seen from the above, based on the video anomaly detection device provided in the embodiments of this specification, before specific implementation, an initial model can be constructed that includes at least an initial cross-modal attention mechanism module and an initial fully connected neural network; and the initial model can be trained using positive and negative sample packets and a preset feature extraction network to obtain a preset feature processing model that includes at least the cross-modal attention mechanism module. In specific implementation, after acquiring the target video to be processed, the acquired target video can first be divided into multiple target image segments and multiple target optical flow segments; then, the preset feature extraction network is used to process the multiple target image segments and multiple target optical flow segments to obtain video features of two different dimensions: target appearance features and target motion features; next, the preset feature processing model is used to remove redundant information of the target appearance features and target motion features through the cross-modal attention mechanism module to obtain corresponding optimized target appearance features and optimized target motion features for video anomaly detection; then, the above two features are fused to obtain fused target features that are relatively rich and comprehensive for video anomaly detection; finally, the preset feature processing model is used to process the fused target features through a fully connected neural network to obtain the corresponding target processing result, so as to determine whether there is an abnormal target in the target video based on the target processing result. This allows for the accurate and efficient detection and identification of abnormal targets in target videos, effectively improving the detection efficiency and accuracy of video anomaly detection.

[0156] While this specification provides the steps of operation for the methods described in the embodiments or flowcharts, more or fewer steps may be included based on conventional or non-inventive means. The order of steps listed in the embodiments is merely one possible order of execution among many steps and does not represent the only possible order. In actual device or client product execution, the methods shown in the embodiments or drawings may be executed sequentially or in parallel (e.g., in a parallel processor or multi-threaded processing environment, or even a distributed data processing environment). The terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, product, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, product, or apparatus. Without further limitations, the presence of other identical or equivalent elements in a process, method, product, or apparatus that includes said elements is not excluded. The terms "first," "second," etc., are used to denote names and do not indicate any particular order.

[0157] Those skilled in the art will also know that, besides implementing the controller using purely computer-readable program code, the same functions can be achieved by logically programming the method steps, making the controller function as logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers (PLCs), and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the devices within it used to implement various functions can also be considered structures within that hardware component. Alternatively, the devices used to implement various functions can be considered as both software modules implementing the method and structures within a hardware component.

[0158] This specification can be described in the general context of computer-executable instructions that are executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, classes, etc., that perform a specific task or implement a specific abstract data type. This specification can also be practiced in distributed computing environments, where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer-readable storage media, including storage devices.

[0159] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this specification can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions of this specification can essentially be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, mobile terminal, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments of this specification.

[0160] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on its differences from other embodiments. This specification can be used in numerous general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable electronic devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices, etc.

[0161] Although this specification has been described by way of examples, those skilled in the art will recognize that many variations and modifications are possible without departing from the spirit of this specification, and it is intended that the appended claims cover such variations and modifications without departing from the spirit of this specification.

Claims

1. A video anomaly detection method, characterized in that, include: Acquire the target video; Based on the target video, multiple target image segments and multiple target optical flow segments are obtained; By using a pre-defined feature extraction network to process multiple target image segments and multiple target optical flow segments, the corresponding target appearance features and target motion features are obtained. The target appearance features and target motion features are processed using a preset feature processing model to obtain target processing results for the target video; wherein, the preset feature processing model includes at least a cross-modal attention mechanism module; the cross-modal attention mechanism module is used to remove redundant information from the target appearance features and target motion features to obtain corresponding optimized target appearance features and optimized target motion features; the cross-modal attention mechanism module is also used to fuse the optimized target appearance features and optimized target motion features to obtain fused target features; Based on the target processing results, determine whether there are any abnormal targets in the target video; The cross-modal attention mechanism module removes redundant information from the target appearance features in the following manner to obtain corresponding optimized target appearance features: the target appearance features are determined as first primary modal features, and the target motion features are determined as first auxiliary modal features; based on the first primary modal features, first global information is obtained, and based on the first auxiliary modal features, first local information is obtained; based on the first global information and the first local information, redundant information from the target appearance features is detected and removed to obtain corresponding optimized target appearance features.

2. The method according to claim 1, characterized in that, The preset feature processing model further includes a fully connected neural network; wherein the fully connected neural network is used to process the fused target features and output the corresponding target processing result.

3. The method according to claim 1, characterized in that, The cross-modal attention mechanism module includes at least: an average pooling layer, a first convolutional layer, a second convolutional layer, and a sigmoid function.

4. The method according to claim 3, characterized in that, Based on the first primary modality features, first global information is obtained, including: The average pooling layer processes the first master modality features and outputs corresponding initial first global information; the first convolutional layer processes the initial first global information, obtains and utilizes the dependencies of the corresponding channels, and outputs corresponding first global awareness descriptors as the first global information.

5. The method according to claim 3, characterized in that, Based on the first auxiliary modal features, first local information is obtained, including: The second convolutional layer processes the first auxiliary modality features and outputs the corresponding first local perceptual descriptor as the first local information.

6. The method according to claim 3, characterized in that, Based on the first global information and the first local information, redundant information of the target appearance features is detected and removed to obtain corresponding optimized target appearance features, including: Multiply the first global information and the first local information to obtain the first channel descriptor; By using the first optimization weights at the channel level generated based on the Sigmoid function and the first channel descriptor, and processing the first master mode features, the corresponding optimized target appearance features are obtained.

7. The method according to claim 1, characterized in that, The method further includes: Construct an initial model; wherein the initial model includes at least an initial cross-modal attention mechanism module and an initial fully connected neural network; Obtain and construct the positive and negative packets of the samples based on the sample videos; Based on the positive and negative packets of the samples, and the preset feature extraction network, the corresponding sample appearance features and sample motion features are obtained. Based on a preset loss function, the initial model is trained using the appearance features and motion features of the samples to obtain a preset feature processing model that meets the requirements.

8. A video anomaly detection device, characterized in that, include: The acquisition module is used to acquire the target video; The segmentation module is used to segment the target video into multiple target image segments and multiple target optical flow segments; The extraction module is used to obtain the corresponding target appearance features and target motion features by processing multiple target image segments and multiple target optical flow segments through a preset feature extraction network; A processing module is used to process the target appearance features and target motion features using a preset feature processing model to obtain a target processing result for the target video; wherein, the preset feature processing model includes at least a cross-modal attention mechanism module; the cross-modal attention mechanism module is used to remove redundant information from the target appearance features and target motion features to obtain corresponding optimized target appearance features and optimized target motion features; the cross-modal attention mechanism module is also used to fuse the optimized target appearance features and optimized target motion features to obtain fused target features; The determination module is used to determine whether there are abnormal targets in the target video based on the target processing results; The cross-modal attention mechanism module removes redundant information from the target appearance features in the following manner to obtain corresponding optimized target appearance features: the target appearance features are determined as first primary modal features, and the target motion features are determined as first auxiliary modal features; based on the first primary modal features, first global information is obtained, and based on the first auxiliary modal features, first local information is obtained; based on the first global information and the first local information, redundant information from the target appearance features is detected and removed to obtain corresponding optimized target appearance features.

9. A computer-readable storage medium, characterized in that, It stores computer instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 7.

Images