Abnormal video detection method and system based on artificial intelligence, and terminal

Abstract

The invention discloses an abnormal video detection method, system and terminal based on artificial intelligence, and relates to the technical field of electronic information. The detection method comprises the following steps: converting an original video into a frame sequence, and obtaining a high-dimensional feature vector of each frame of image; extracting text features of the original video to obtain text feature vectors; performing feature fusion on the high-dimensional feature vector and the text feature vector by adopting a Late Fusion method and a cross-modal attention mechanism to obtain a multi-modal feature sequence; performing feature processing on the multi-modal feature sequence by adopting a multi-layer Transform encoder to obtain a processed feature sequence; performing post-processing on the processed feature sequence to obtain an abnormal information value; configuring an abnormal information threshold value, and when the abnormal information value is greater than the abnormal information threshold value, judging that the video is an abnormal video; and when the abnormal information value is smaller than or equal to the abnormal information threshold value, judging that the video is a non-abnormal video. The multi-modal information can be fully utilized, the feature fusion result is optimized, and the detection performance is improved.

Description

Technical Field

[0001] The present invention relates to the field of electronic information technology, and in particular to an abnormal video detection method, system and terminal based on artificial intelligence. Background Art

[0002] At present, online video platforms have become an important channel for people to obtain information. However, with the increasing richness and diversity of online video content, many bad videos, such as violent behavior videos, have appeared on streaming platforms. These video contents will not only have a negative impact on social development, but may also cause serious harm to the physical and mental health of young people. Therefore, how to efficiently and accurately detect and identify abnormal videos containing information such as violent behavior has become a technical problem that needs to be solved urgently.

[0003] At present, the detection of abnormal videos is mostly focused on artificial intelligence technology assisted by manual recognition. These methods usually rely on convolutional neural networks (CNN) and recurrent neural networks (RNN). For example, based on models such as long short-term memory (LSTM) networks, feature extraction and sequence modeling of video frames are performed. However, these models have some significant limitations: first, when processing complex video scenes, traditional convolutional neural networks cannot fully capture the detailed features of violent behavior in the video, which leads to a decrease in recognition accuracy; second, recurrent neural networks such as LSTM have the defects of low efficiency and high computational cost in temporal modeling of long sequence videos; third, videos usually contain multiple modal information such as images, audio and text, and the existing methods cannot effectively integrate multimodal information.

[0004] In summary, the existing technology has problems such as poor video feature extraction capability, insufficient time series modeling, and difficulty in multimodal information fusion. It exhibits defects such as low false positive rejection rate, high false positive rate, and high computational time cost on existing data sets. Summary of the invention

[0005] The purpose of the present invention is to provide an abnormal video detection method, system and terminal based on artificial intelligence to solve the problems of poor video feature extraction capability, insufficient time series modeling, difficulty in multimodal information fusion, etc. in the prior art, so as to make full use of multimodal information, optimize feature fusion results and improve detection performance.

[0006] In order to achieve the above object, the present invention provides the following technical solutions:

[0007] In a first aspect, the present invention provides an abnormal video detection method based on artificial intelligence, comprising the following steps:

[0008] S1. Configure the original video, convert the original video into a frame sequence, calculate the inter-frame parameters, and obtain the high-dimensional feature vector of each frame image based on the frame sequence and the inter-frame parameters; the inter-frame parameters include light flux and continuous optical flow difference;

[0009] S2. extract text features of the original video to obtain a text feature vector;

[0010] S3. Late Fusion method and cross-modal attention mechanism are used to fuse high-dimensional feature vectors and text feature vectors to obtain multimodal feature sequences; feature fusion specifically includes:

[0011] S31. Assigning an initial weight to each modal feature in the Late Fusion method;

[0012] S32. Align and map features from different modalities;

[0013] S33. A cross-modal attention mechanism is used to perform weighted fusion of different modal features, and the weights are automatically adjusted based on the contribution of different modal features to anomaly detection;

[0014] S34. Optimize fusion results;

[0015] S4. Use a multi-layer Transformer encoder to perform feature processing on the multimodal feature sequence to obtain a processed feature sequence;

[0016] S5. post-processing the processed feature sequence to obtain an abnormal information value;

[0017] S6. Configure an abnormal information threshold. When the abnormal information value is greater than the abnormal information threshold, the original video is judged to be an abnormal video; when the abnormal information value is less than or equal to the abnormal information threshold, the original video is judged to be a non-abnormal video.

[0018] As a possible implementation method, a high-dimensional feature vector of each frame image is obtained based on the frame sequence and inter-frame parameters, including:

[0019] S10. Preprocessing the frame sequence and pre-training the AlexNet model;

[0020] S11. Based on the pre-trained AlexNet model, forward propagation calculation is performed on the pre-processed frame sequence and inter-frame parameters to obtain the high-dimensional feature vector of each frame image.

[0021] As a possible implementation method, preprocessing the frame sequence includes: scaling the size of each frame image to the input size required by the AlexNet model; subtracting the average value of the data set used when pre-training the AlexNet model from the pixel value of each frame image to obtain the pixel difference value of each frame image.

[0022] As a possible implementation method, the pre-trained AlexNet model is specifically: pre-training the AlexNet model based on the ImageNet database.

[0023] As a possible implementation method, text features of the original video are extracted to obtain a text feature vector, specifically: the text features of the original video are extracted using a TextCNN model, and the text feature vector is obtained based on the Keras open source artificial neural network library.

[0024] As a possible implementation method, S33 specifically includes:

[0025] S331. Construct an attention network that can learn the relationship between different modal features and assign an attention weight to each modality;

[0026] S332. Use the attention network to process the high-dimensional feature vector and the text feature vector to obtain the attention weight of each modality;

[0027] S333. Use the Softmax function to adjust the attention weight so that the sum of the attention weights of all modalities is 1;

[0028] S334. weighting the high-dimensional feature vector and the text feature vector based on the adjusted attention weight;

[0029] S335. Merging the weighted high-dimensional feature vector and the text feature vector to form a fused feature representation;

[0030] S336. Normalize the fused feature representation to eliminate the scale difference between features of different modalities.

[0031] As a possible implementation method, the optimization fusion result is realized by optimizing the weights using the gradient descent method, which includes the following sub-steps:

[0032] S341. Use the RMSprop algorithm to dynamically adjust the learning rate and use the learning rate decay strategy to speed up the convergence;

[0033] S342. Use Gating Mechanism to control information flow and automatically adjust the contribution of different features;

[0034] S343. Design a weighted cross entropy loss function to automatically adjust the weights of different categories;

[0035] S344. Use Hyperopt tool to optimize weights to optimize fusion results.

[0036] As a possible implementation, the number of layers of the multi-layer Transformer encoder is 10 to 15 layers.

[0037] In a second aspect, the present invention provides an abnormal video detection system based on artificial intelligence, comprising:

[0038] A high-dimensional feature vector acquisition unit is configured to configure the original video, convert the original video into a frame sequence, calculate the inter-frame parameters, and obtain the high-dimensional feature vector of each frame image based on the frame sequence and the inter-frame parameters; the inter-frame parameters include light flux and continuous optical flow difference;

[0039] A text feature vector acquisition unit extracts text features of the original video to obtain a text feature vector;

[0040] The multimodal feature sequence acquisition unit uses the late fusion method and the cross-modal attention mechanism to fuse the high-dimensional feature vector and the text feature vector to obtain a multimodal feature sequence;

[0041] The abnormal information value acquisition unit uses a multi-layer Transformer encoder to perform feature processing on the multimodal feature sequence to obtain a processed feature sequence, and performs post-processing on the processed feature sequence to obtain an abnormal information value;

[0042] The abnormal video judgment unit is configured with an abnormal information threshold. When the abnormal information value is greater than the abnormal information threshold, the original video is judged to be an abnormal video; when the abnormal information value is less than or equal to the abnormal information threshold, the original video is judged to be a non-abnormal video.

[0043] In a third aspect, the present invention provides a terminal comprising a processor and a communication interface coupled to the processor, wherein the processor is used to run a computer program or instruction to implement the artificial intelligence-based abnormal video detection method provided in the first aspect.

[0044] Compared with the prior art, the beneficial effects of the present invention are:

[0045] 1. The artificial intelligence-based abnormal video detection method provided by the present invention uses the pre-trained AlexNet model to extract efficient image features from video frames, thereby improving the accuracy of image feature extraction, and uses the TextCNN model to extract text features from video texts, thereby accurately capturing key information in the text. Afterwards, the image features and text features are creatively deeply integrated, thereby making full use of the effective information in the video and effectively improving the detection accuracy.

[0046] 2. The artificial intelligence-based abnormal video detection method provided by the present invention adopts a multi-layer Transformer model, has powerful time series modeling capabilities and parallel computing capabilities, can effectively process long sequence videos, and improve the effect and efficiency of time series feature modeling.

[0047] 3. The artificial intelligence-based abnormal video detection method provided by the present invention innovatively uses the Late Fusion method to postpone the fusion time of the multimodal sequence to a later time. On the basis of reducing the training cost of the weighted value in the attention mechanism, it helps to more accurately select information that is useful for the current task from a large amount of information, thereby creatively optimizing the model effect and realizing the deep fusion of cross-modal features.

[0048] 4. The artificial intelligence-based abnormal video detection method provided by the present invention adds a fully connected feedforward neural network FFN for post-processing after the Transformer model, which can further optimize and fuse the features of the model output and improve the final prediction accuracy.

[0049] 5. The artificial intelligence-based abnormal video detection method provided by the present invention has low dependence on data sets, can accept large differences in training resources, has strong adaptability, can show performance exceeding the baseline level on different data sets, has strong detection capabilities, and can quickly detect a randomly crawled video in a short time, meeting the real-time requirements in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0051] Figure 1 Flow chart of an abnormal video detection method based on artificial intelligence in an embodiment of the present invention;

[0052] Figure 2 It is an image of luminous flux along with frame number after extracting a video frame at 0.1s in an embodiment of the present invention;

[0053] Figure 3 It is a continuous optical flow difference image obtained by calculating and extracting a video frame at 0.1s in an embodiment of the present invention. DETAILED DESCRIPTION

[0054] In order to clearly describe the technical solutions of the embodiments of the present invention, in the embodiments of the present invention, words such as "first" and "second" are used to distinguish the same items or similar items with basically the same functions and effects. For example, the first threshold and the second threshold are only used to distinguish different thresholds, and their order is not limited. Those skilled in the art can understand that words such as "first" and "second" do not limit the quantity and execution order, and words such as "first" and "second" do not necessarily limit them to be different.

[0055] It should be noted that, in the present invention, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0056] In the present invention, "at least one" means one or more, and "plurality" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. The following at least one item (items) or similar expressions thereof refer to any combination of these items, including any combination of single items (items) or plural items (items). For example, at least one item (items) of a, b or c can mean: a, b, c, the combination of a and b, the combination of a and c, the combination of b and c, or the combination of a, b and c, where a, b, c can be single or multiple.

[0057] With the increasing richness and diversity of online video content, many bad videos, such as violent behavior videos, have appeared on streaming media platforms. These video contents will not only have a negative impact on social development, but may also cause serious harm to the physical and mental health of young people. Therefore, how to efficiently and accurately detect and identify abnormal videos containing information such as violent behavior has become a technical problem that needs to be solved urgently. The current detection model and detection technology have problems such as poor video feature extraction ability, insufficient temporal modeling, and difficulty in multimodal information fusion. It has shown defects such as low rejection rate, high false rate, and large calculation time cost on existing data sets. To solve the above problems, the present invention provides an abnormal video detection method, system and terminal based on artificial intelligence, which extracts bad factors in the video, such as violent image features, violent text features, etc., and uses artificial intelligence models for detection and identification, which can solve the problems of poor video feature extraction ability, insufficient temporal modeling, and difficulty in multimodal information fusion in the prior art, and can make full use of multimodal information, optimize feature fusion results, and improve detection performance.

[0058] In a first aspect, the present invention provides an abnormal video detection method based on artificial intelligence, see Figure 1 , including the following steps:

[0059] S1. Configure the original video, convert the original video into a frame sequence, calculate the inter-frame parameters, and obtain the high-dimensional feature vector of each frame image based on the frame sequence and the inter-frame parameters; the inter-frame parameters include light flux and continuous optical flow difference;

[0060] Because there are bad behaviors in abnormal videos, such as violent beatings, violent destruction, etc., violent behaviors will cause large brightness differences between each frame of the video. In specific implementation, according to the actual accuracy requirements, the original video is divided into a series of frames, and frames can be extracted at a fixed sampling rate or intervals. At the same time, the original frames are sliced ​​and extracted to calculate the light flux and continuous optical flow difference. For example, the Farneback optical flow algorithm can be used for calculation.

[0061] As an example, see Figure 2 , is the luminous flux image with frame number after extracting a video frame at 0.1s. The continuous optical flow difference obtained by further calculation is shown in Figure 3 ,visible Figure 3 If the continuous optical flow difference in changes greatly, it means that there is a high probability that violent behavior has occurred in the video.

[0062] As a possible implementation method, obtaining a high-dimensional feature vector of each frame image based on the frame sequence and the inter-frame parameters includes:

[0063] S10. Preprocess frame sequence and pre-train AlexNet model;

[0064] As a possible implementation method, preprocessing the frame sequence includes: scaling the size of each frame image to the input size required by the AlexNet model; subtracting the average value of the data set used when pre-training the AlexNet model from the pixel value of each frame image to obtain the pixel difference value of each frame image.

[0065] As a possible implementation method, the pre-trained AlexNet model is specifically: pre-training the AlexNet model based on the ImageNet database.

[0066] S11. Based on the pre-trained AlexNet model, forward propagation calculation is performed on the pre-processed frame sequence and inter-frame parameters to obtain the high-dimensional feature vector of each frame image.

[0067] AlexNet is a classic convolutional neural network model that can quickly identify image features in a short time, effectively improving the implementation efficiency of this method and ensuring the accuracy of feature extraction.

[0068] S2. extract text features of the original video to obtain a text feature vector;

[0069] As a possible implementation method, the text features of the original video are extracted to obtain a text feature vector. Specifically, the text features of the original video are extracted using a TextCNN model, and the text feature vector is obtained based on the Keras open source artificial neural network library.

[0070] As an example, video text can be extracted and generated based on video tags, video introduction, video subtitles, etc. SpaCy is used to segment the video text into word sequences, and then the Lancaster Stemmer algorithm is used to remove common but meaningless words in the text and restore the words to their stems and basic forms. The TextCNN model is used to extract features from the video text, which can accurately capture key information in the video text and obtain a structured text feature vector. For example, video text extraction is performed on a certain video to obtain the following video keywords (Tag): Cam, Police, and its title is: "Body-cam video shows wedding party brawl", and the video introduction is "Newport police said they arrested six people from a wedding party which led to an officer being assaulted". After segmentation into word sequences, the keywords are extracted to obtain: Body-cam, Polic, wedding party, brawl, arrested, assaulted. By comparing these keywords with the sensitive word cloud, the text feature vector is obtained.

[0071] When the existing technology detects videos, it usually uses a single feature in the image, audio or text, which cannot achieve effective fusion of multimodal information. The present invention fuses image features and text features of different modalities, which can make full use of the effective information in the video and effectively improve the detection accuracy.

[0072] S3. Late Fusion method and cross-modal attention mechanism are used to fuse high-dimensional feature vectors and text feature vectors to obtain multimodal feature sequences; feature fusion specifically includes:

[0073] S31. Assign initial weights to each modal feature in the Late Fusion method

[0074] As an example, the initial weights are pre-set based on the correlation between each modality feature and the abnormal video, wherein the correlation is obtained by pre-training on a labeled dataset.

[0075] S32. Align and map features from different modalities;

[0076] As an example, feature normalization is performed on features from different modalities, and the feature mapping technique t-SNE is used to map features from different modalities into a common feature space to ensure that they are semantically comparable and facilitate feature fusion.

[0077] S33. A cross-modal attention mechanism is used to perform weighted fusion of different modal features, and the weights are automatically adjusted based on the contribution of different modal features to anomaly detection;

[0078] As a possible implementation method, S33 specifically includes:

[0079] S331. Construct an attention network that can learn the relationship between different modal features and assign an attention weight to each modality;

[0080] As an example, the goal of the attention network is to learn the relationship between features of different modalities and assign attention weights to each modality as follows:

[0081] A m =f att (F image , F text ), where m∈{image,text}

[0082] Among them, A m is the attention weight of modality m, f att is the attention mechanism function, F image and F text High-dimensional feature vectors representing images and texts respectively.

[0083] The design of the attention mechanism function is usually based on relevance scoring. Here we use the form of "dot-product attention":

[0084]

[0085] Among them, F image and F text are image and text feature vectors respectively, and d is the dimension of the feature for scaling.

[0086] S332. Use the attention network to process the high-dimensional feature vector and the text feature vector to obtain the attention weight of each modality;

[0087] As an example, based on the attention network constructed in the previous step, the high-dimensional features are processed to obtain the attention weights of each modality:

[0088] W image , W text =Attention(F image , F text )

[0089] Among them, W image and W text are the attention weights of image and text.

[0090] S333. Use the Softmax function to adjust the attention weight so that the sum of the attention weights of all modalities is 1;

[0091] As an example, Softmax is a function that transforms a vector into a probability distribution, and its formula is:

[0092]

[0093] The core of the Attention mechanism is to perform weighted summation of input features. For each modality, the Softmax function is used to ensure that the sum of the weights of all modalities is 1:

[0094]

[0095] Among them, W′ m is the adjusted attention weight.

[0096] S334. weighting the high-dimensional feature vector and the text feature vector based on the adjusted attention weight;

[0097] As an example, the weighted fusion of high-dimensional feature vectors based on the adjusted weights is as follows:

[0098] F fused =W′ image ·F image +W′ text ·F text

[0099] Among them, F fused It is the weighted fusion feature representation.

[0100] S335. Merging the weighted high-dimensional feature vector and the text feature vector to form a fused feature representation;

[0101] As an example, the weighted features of image and text modalities are combined as follows:

[0102] F combined =[W′ image ·F image , W′ text ·F text ]

[0103] Among them, the symbol “[,]” represents the concatenation of features.

[0104] Use additive attention mechanism for further fusion:

[0105]

[0106] Among them, W fuse is the fusion weight matrix, b fuse is the bias term, F combined is the combined modal eigenvector, W m is the attention weight of the modality.

[0107] S336. Normalize the fused feature representation to eliminate the scale differences between features of different modalities.

[0108] As an example, normalization is used to eliminate the scale differences of different modal features:

[0109]

[0110] Here, μ is the mean of the feature and σ is the standard deviation.

[0111] The cross-modal attention mechanism is adopted to enable image features to focus on text features, improve the effect of cross-modal information fusion, and make full use of multimodal information.

[0112] S34. Optimize fusion results;

[0113] As a possible implementation method, the optimization fusion result is realized by optimizing the weights using the gradient descent method, which includes the following sub-steps:

[0114] S341. Use the RMSprop algorithm to dynamically adjust the learning rate, and use the learning rate decay strategy to accelerate the convergence speed; as an example, use the RMSprop algorithm to dynamically adjust the learning rate, and use the learning rate decay strategy, specifically:

[0115]

[0116] Among them, η t is the learning rate at step t, and λ is the decay parameter.

[0117] S342. Use Gating Mechanism to control information flow and automatically adjust the contribution of different features;

[0118] As an example, the Gating Mechanism automatically adjusts the contribution of different features. The calculation method can be expressed as follows:

[0119] G=σ(W g ·F fused +b g )

[0120] Among them, W g is the Gating weight matrix, b g is the bias term, σ is the Sigmoid function, defined as:

[0121]

[0122] Gating Mechanism dynamically adjusts the contribution of different modal features by controlling the transmission of information flow.

[0123] S343. Design a weighted cross entropy loss function to automatically adjust the weights of different categories;

[0124] As an example, the optimization using the weighted cross entropy loss function is as follows:

[0125]

[0126] Among them, w i is the weight of the category, y i is the true label, p i is the probability predicted by the model. This loss function can be weighted for imbalanced datasets.

[0127] S344. Use Hyperopt tool to optimize weights to optimize fusion results.

[0128] As an example, use the Hyperopt tool to perform hyperparameter optimization and find the optimal weights: Among them, L(W) is the loss function and W is the weight to be optimized.

[0129] The conventional cross-modal attention mechanism revolves around the calculation, normalization and weighted sum of attention scores. The present invention innovatively uses the Late Fusion method to postpone the fusion time of multimodal sequences to a later time. On the basis of reducing the training cost of the weighted value in the attention mechanism, it helps to more accurately select information that is useful for the current task from a large amount of information, thereby creatively optimizing the model effect and realizing the deep fusion of cross-modal features.

[0130] S4. Use a multi-layer Transformer encoder to perform feature processing on the multimodal feature sequence to obtain a processed feature sequence;

[0131] As a possible implementation manner, the number of layers of the multi-layer Transformer encoder is 10 to 15 layers, for example, 10 layers, 11 layers, 12 layers, 13 layers, 14 layers, and 15 layers.

[0132] Through the self-attention mechanism, deep feature processing of multimodal feature sequences is performed to remove more worthless information, so that the output feature sequence contains more valuable information, which significantly improves the temporal modeling capability of long sequence videos.

[0133] S5. post-processing the processed feature sequence to obtain an abnormal information value;

[0134] As an example, a fully connected feedforward neural network FFN is used to post-process the processed feature sequence to further optimize and fuse the model output features to obtain the abnormal information value;

[0135] S6. Configure an abnormal information threshold. When the abnormal information value is greater than the abnormal information threshold, the original video is judged to be an abnormal video; when the abnormal information value is less than or equal to the abnormal information threshold, the original video is judged to be a non-abnormal video.

[0136] The present invention has low dependence on data sets, can accept large differences in training resources, has strong adaptability, can show performance exceeding the baseline level on different data sets, has strong detection capabilities, and can quickly detect a randomly crawled video in a short time, thus meeting the real-time requirements in practical applications.

[0137] Tables 1 to 4 compare the advantages and disadvantages of the method of the present invention with two traditional methods, CNN and LSTM, on four data sets, namely, RWF-2000, YTB-Fights, UBI-Fights, and ViolentFlow, on different evaluation indicators. Among them, the RWF-2000 dataset comes from: Cheng M, Cai K, Li M. RWF-2000: an open large scale video database for violence detection [C] / / 2020 25th International Conference on Pattern Recognition (ICPR). IEEE, 2021: 4183-4190. https: / / ieeexplore.ieee.org / abstract / document / 9412502 / ; the YTB-Fights dataset comes from: Soliman MM, Kamal MH, Nashed MA EM, et al. Violence recognition from videos using deep learning techniques [C] / / 2019 ninth international conference on intelligent computing and information systems (ICICIS). IEEE, 2019: 80-85. https: / / ieeexplore.ieee.org / document / 9014714; the UBI-Fights dataset comes from: Degardin B, H.Human activity analysis: Iterativeweak / self-supervised learning frameworks for detecting abnormal events[C] / / 2020IEEE International Joint Conference on Biometrics(IJCB).IEEE,2020:1-7.https: / / ieeexplore.ieee.org / document / 9304905; ViolentFlow data set comes from: Hassner T, Itcher Y, Kliper-Gross O.Violent flows:Real-time detection ofviolent crowd behavior[C] / / 2012IEEE computer society conference on computervision and pattern recognition workshops.IEEE,2012:1-6.https: / / ieeexplore.ieee.org / abstract / document / 6239348.

[0138] The evaluation indicators are accuracy, F1-Score, area under the receiver operating curve (AUC) and logarithmic loss (LogLoss):

[0139] Table 1 Performance of the present invention and two traditional methods, CNN and LSTM, on the RWF-2000 dataset

[0140]

[0141] Table 2 Performance of the present invention and two traditional methods, CNN and LSTM, on the YTB-Fights dataset

[0142]

[0143] Table 3 Performance of the present invention and two traditional methods, CNN and LSTM, on the UBI-Fights dataset

[0144]

[0145] Table 4 Performance of the present invention and two traditional methods, CNN and LSTM, on the ViolentFlow dataset

[0146]

[0147] From the above comparison, it can be found that the abnormal video detection method based on artificial intelligence provided by the present invention has shown advantages in various evaluation indicators and can improve video detection performance.

[0148] In a second aspect, the present invention provides an abnormal video detection system based on artificial intelligence, comprising:

[0149] A high-dimensional feature vector acquisition unit is configured to configure the original video, convert the original video into a frame sequence, calculate the inter-frame parameters, and obtain the high-dimensional feature vector of each frame image based on the frame sequence and the inter-frame parameters; the inter-frame parameters include light flux and continuous optical flow difference;

[0150] A text feature vector acquisition unit extracts text features of the original video to obtain a text feature vector;

[0151] The multimodal feature sequence acquisition unit uses the late fusion method and the cross-modal attention mechanism to fuse the high-dimensional feature vector and the text feature vector to obtain a multimodal feature sequence;

[0152] The abnormal information value acquisition unit uses a multi-layer Transformer encoder to perform feature processing on the multimodal feature sequence to obtain a processed feature sequence, and performs post-processing on the processed feature sequence to obtain an abnormal information value;

[0153] The abnormal video judgment unit is configured with an abnormal information threshold. When the abnormal information value is greater than the abnormal information threshold, the original video is judged to be an abnormal video; when the abnormal information value is less than or equal to the abnormal information threshold, the original video is judged to be a non-abnormal video.

[0154] In a third aspect, the present invention provides a terminal comprising a processor and a communication interface coupled to the processor, wherein the processor is used to run a computer program or instruction to implement the artificial intelligence-based abnormal video detection method provided in the first aspect.

[0155] Although the present invention is described herein in conjunction with various embodiments, in the process of implementing the claimed invention, those skilled in the art may understand and implement other variations of the disclosed embodiments by viewing the drawings, the disclosure, and the drawings, etc. In the specification, the word "comprising" does not exclude other components or steps, and "one" or "an" does not exclude multiple situations. A single processor or other unit can implement several functions listed in the specification. Certain measures are recorded in different embodiments, but this does not mean that these measures cannot be combined to produce good results.

[0156] Although the present invention has been described in conjunction with specific features and embodiments thereof, it is apparent that various modifications and combinations may be made thereto without departing from the spirit and scope of the present invention. Accordingly, this specification and the accompanying drawings are merely exemplary illustrations of the present invention and are deemed to cover any and all modifications, variations, combinations or equivalents within the scope of the present invention. Obviously, those skilled in the art may make various modifications and variations to the present invention without departing from the spirit and scope of the present invention. Thus, the present invention is intended to include such modifications and variations if they fall within the scope of the present invention and its equivalents.

Claims

1. An abnormal video detection method based on artificial intelligence, characterized in that: The steps include: S1. Configuring an original video, converting the original video into a frame sequence, calculating inter-frame parameters, and obtaining a high-dimensional feature vector of each frame image based on the frame sequence and the inter-frame parameters; the inter-frame parameters include light flux and continuous optical flow difference; S2. Extracting text features of the original video to obtain a text feature vector; S3. Using the Late Fusion method and the cross-modal attention mechanism to perform feature fusion on the high-dimensional feature vector and the text feature vector to obtain a multimodal feature sequence; The feature fusion specifically includes: S31. Assigning an initial weight to each modal feature in the Late Fusion method; S32. Align and map features from different modalities; S33. Using the cross-modal attention mechanism to perform weighted fusion of different modal features, the weights are automatically adjusted based on the contribution of different modal features to anomaly detection; S34. Optimize fusion results; S4. Using a multi-layer Transformer encoder to perform feature processing on the multimodal feature sequence to obtain a processed feature sequence; S5. post-processing the processed feature sequence to obtain an abnormal information value; S6. Configure an abnormal information threshold. When the abnormal information value is greater than the abnormal information threshold, the original video is judged to be an abnormal video; when the abnormal information value is less than or equal to the abnormal information threshold, the original video is judged to be a non-abnormal video.

2. The abnormal video detection method based on artificial intelligence according to claim 1 is characterized in that: Obtaining a high-dimensional feature vector of each frame image based on the frame sequence and the inter-frame parameters includes: S10. Preprocessing the frame sequence and pre-training the AlexNet model; S11. Based on the pre-trained AlexNet model, forward propagation calculation is performed on the pre-processed frame sequence and inter-frame parameters to obtain the high-dimensional feature vector of each frame image.

3. The abnormal video detection method based on artificial intelligence according to claim 2 is characterized in that: Preprocessing the frame sequence includes: scaling the size of each frame image to the input size required by the AlexNet model; subtracting the average value of the data set used when pre-training the AlexNet model from the pixel value of each frame image to obtain the pixel difference value of each frame image.

4. The abnormal video detection method based on artificial intelligence according to claim 2 is characterized in that: The pre-trained AlexNet model is specifically: pre-trained AlexNet model based on ImageNet database.

5. The abnormal video detection method based on artificial intelligence according to claim 1 is characterized in that: The extracting of text features of the original video to obtain a text feature vector specifically includes: extracting the text features of the original video using a TextCNN model, and obtaining a text feature vector based on a Keras open source artificial neural network library.

6. The abnormal video detection method based on artificial intelligence according to claim 1 is characterized in that: The S33 specifically includes: S331. Construct an attention network, wherein the attention network can learn the relationship between different modal features and assign an attention weight to each modality; S332. Processing the high-dimensional feature vector and the text feature vector using the attention network to obtain the attention weight of each modality; S333. Using the Softmax function to adjust the attention weight so that the sum of the attention weights of all modalities is 1; S334. Weighting the high-dimensional feature vector and the text feature vector based on the adjusted attention weight; S335. Merging the weighted high-dimensional feature vector and the text feature vector to form a fused feature representation; S336. Normalize the fused feature representation to eliminate the scale difference between features of different modalities.

7. The abnormal video detection method based on artificial intelligence according to claim 1 is characterized in that: The optimization fusion result is achieved by optimizing weights using the gradient descent method, including the following sub-steps: S341. Use the RMSprop algorithm to dynamically adjust the learning rate and use the learning rate decay strategy to speed up the convergence; S342. Use Gating Mechanism to control information flow and automatically adjust the contribution of different features; S343. Design a weighted cross entropy loss function to automatically adjust the weights of different categories; S344. Use Hyperopt tool to optimize weights to optimize fusion results.

8. The abnormal video detection method based on artificial intelligence according to claim 1 is characterized in that: The number of layers of the multi-layer Transformer encoder is 10 to 15 layers.

9. An abnormal video detection system based on artificial intelligence, characterized in that: include: A high-dimensional feature vector acquisition unit is configured to configure an original video, convert the original video into a frame sequence, calculate inter-frame parameters, and obtain a high-dimensional feature vector of each frame image based on the frame sequence and the inter-frame parameters; the inter-frame parameters include light flux and continuous optical flow difference; A text feature vector acquisition unit extracts text features of the original video to obtain a text feature vector; A multimodal feature sequence acquisition unit adopts a late fusion method and a cross-modal attention mechanism to perform feature fusion on the high-dimensional feature vector and the text feature vector to obtain a multimodal feature sequence; An abnormal information value acquisition unit, which uses a multi-layer Transformer encoder to perform feature processing on the multimodal feature sequence to obtain a processed feature sequence, and performs post-processing on the processed feature sequence to obtain an abnormal information value; An abnormal video judging unit is configured with an abnormal information threshold value, and when the abnormal information value is greater than the abnormal information threshold value, the original video is judged to be an abnormal video; When the abnormal information value is less than or equal to the abnormal information threshold, the original video is determined to be a non-abnormal video.

10. A terminal, characterized in that: It includes a processor and a communication interface coupled to the processor, and the processor is used to run a computer program or instruction to implement the abnormal video detection method based on artificial intelligence as described in any one of claims 1 to 8.

Images