An abnormal video detection method, system and terminal based on artificial intelligence

Abstract

The present invention discloses an abnormal video detection method, system and terminal based on artificial intelligence, which relates to the field of electronic information technology. The detection method includes: converting the original video into a frame sequence to obtain a high-dimensional feature vector of each frame image; extracting the text features of the original video to obtain a text feature vector; using the late fusion method and the cross-modal attention mechanism to perform feature fusion on the high-dimensional feature vector and the text feature vector to obtain a multimodal feature sequence; using a multi-layer Transformer encoder to perform feature processing on the multimodal feature sequence to obtain a processed feature sequence; post-processing the processed feature sequence to obtain an abnormal information value; configuring an abnormal information threshold, when the abnormal information value is greater than the abnormal information threshold, it is judged as an abnormal video; when the abnormal information value is less than or equal to the abnormal information threshold, it is judged as a non-abnormal video. The present invention can make full use of multimodal information, optimize feature fusion results, and improve detection performance.

Description

Technical Field

[0001] The present invention relates to the field of electronic information technology, and in particular to an abnormal video detection method, system and terminal based on artificial intelligence. Background Art

[0002] Online video platforms have become an important channel for people to obtain information. However, with the increasing richness and diversity of online video content, many undesirable videos, such as those depicting violent acts, have appeared on streaming platforms. These videos not only have a negative impact on social development but can also pose serious risks to the physical and mental health of young people. Therefore, efficiently and accurately detecting and identifying anomalous videos containing content such as violent acts has become a pressing technical challenge.

[0003] Currently, the detection of abnormal videos mostly focuses on artificial intelligence technology assisted by manual recognition. These methods typically rely on convolutional neural networks (CNNs) and recurrent neural networks (RNNs). For example, models such as long short-term memory (LSTM) networks are used to extract features from video frames and perform sequence modeling. However, these models have some significant limitations: First, traditional convolutional neural networks cannot fully capture the detailed features of violent behavior in complex video scenes, resulting in a decrease in recognition accuracy; second, recurrent neural networks such as LSTMs have the disadvantages of low efficiency and high computational cost in temporal modeling of long-sequence videos; third, videos usually contain multiple modal information such as images, audio, and text, and current methods are unable to effectively integrate multimodal information.

[0004] In summary, the existing technology has problems such as poor video feature extraction ability, insufficient time series modeling, and difficulty in multimodal information fusion. It has shown defects such as low false rejection rate, high false positive rate, and high computational time cost on existing datasets. Summary of the Invention

[0005] The purpose of the present invention is to provide an abnormal video detection method, system and terminal based on artificial intelligence to solve the problems of poor video feature extraction ability, insufficient time series modeling, and difficulty in multimodal information fusion in the existing technology. It can make full use of multimodal information, optimize feature fusion results, and improve detection performance.

[0006] In order to achieve the above object, the present invention provides the following technical solutions:

[0007] In a first aspect, the present invention provides an abnormal video detection method based on artificial intelligence, comprising the following steps:

[0008] S1. Configure the original video, convert it into a frame sequence, calculate inter-frame parameters, and obtain a high-dimensional feature vector for each frame based on the frame sequence and inter-frame parameters; the inter-frame parameters include light flux and continuous optical flow difference;

[0009] S2. Extract text features from the original video to obtain a text feature vector;

[0010] S3. Late Fusion method and cross-modal attention mechanism are used to fuse high-dimensional feature vectors and text feature vectors to obtain multimodal feature sequences. Feature fusion specifically includes:

[0011] S31. Assigning initial weights to each modal feature in the Late Fusion method;

[0012] S32. Align and map features from different modalities;

[0013] S33. A cross-modal attention mechanism is used to perform weighted fusion of features from different modalities, with weights automatically adjusted based on the contribution of different modal features to anomaly detection.

[0014] S34. Optimize fusion results;

[0015] S4. Use a multi-layer Transformer encoder to perform feature processing on the multimodal feature sequence to obtain a processed feature sequence;

[0016] S5. post-processing the processed feature sequence to obtain an abnormal information value;

[0017] S6. Configure an abnormal information threshold. When the abnormal information value is greater than the abnormal information threshold, the original video is judged to be an abnormal video; when the abnormal information value is less than or equal to the abnormal information threshold, the original video is judged to be a non-abnormal video.

[0018] As a possible implementation method, a high-dimensional feature vector of each frame image is obtained based on the frame sequence and inter-frame parameters, including:

[0019] S10. Preprocessing the frame sequence and pretraining the AlexNet model;

[0020] S11. Perform forward propagation calculations on the preprocessed frame sequence and inter-frame parameters based on the pre-trained AlexNet model to obtain a high-dimensional feature vector for each frame image.

[0021] As a possible implementation method, preprocessing the frame sequence includes: scaling the size of each frame image to the input size required by the AlexNet model; subtracting the average value of the dataset used when pre-training the AlexNet model from the pixel value of each frame image to obtain the pixel difference value of each frame image.

[0022] As a possible implementation method, the pre-trained AlexNet model is specifically: pre-training the AlexNet model based on the ImageNet database.

[0023] As a possible implementation method, text features of the original video are extracted to obtain a text feature vector. Specifically, the text features of the original video are extracted using a TextCNN model, and the text feature vector is obtained based on the Keras open source artificial neural network library.

[0024] As a possible implementation method, S33 specifically includes:

[0025] S331. Construct an attention network that can learn the relationship between different modal features and assign an attention weight to each modality;

[0026] S332. Use the attention network to process the high-dimensional feature vector and the text feature vector to obtain the attention weight of each modality;

[0027] S333. Use the Softmax function to adjust the attention weights so that the sum of the attention weights of all modalities is 1;

[0028] S334. Weighting the high-dimensional feature vector and the text feature vector based on the adjusted attention weight;

[0029] S335. Merging the weighted high-dimensional feature vector and the text feature vector to form a fused feature representation;

[0030] S336. Normalize the fused feature representation to eliminate scale differences between features of different modalities.

[0031] As a possible implementation method, the optimized fusion result is achieved by optimizing the weights using the gradient descent method, which includes the following sub-steps:

[0032] S341. Use the RMSprop algorithm to dynamically adjust the learning rate and use the learning rate decay strategy to accelerate convergence;

[0033] S342. Use Gating Mechanism to control information flow and automatically adjust the contribution of different features;

[0034] S343. Design a weighted cross entropy loss function to automatically adjust the weights of different categories;

[0035] S344. Use Hyperopt tool to optimize weights to optimize fusion results.

[0036] As a possible implementation, the number of layers of the multi-layer Transformer encoder is 10 to 15 layers.

[0037] In a second aspect, the present invention provides an abnormal video detection system based on artificial intelligence, comprising:

[0038] The high-dimensional feature vector acquisition unit is configured to configure the original video, convert the original video into a frame sequence, calculate the inter-frame parameters, and obtain the high-dimensional feature vector of each frame image based on the frame sequence and the inter-frame parameters; the inter-frame parameters include light flux and continuous optical flow difference;

[0039] A text feature vector acquisition unit extracts text features of the original video and obtains a text feature vector;

[0040] The multimodal feature sequence acquisition unit uses the late fusion method and cross-modal attention mechanism to fuse high-dimensional feature vectors and text feature vectors to obtain a multimodal feature sequence;

[0041] The abnormal information value acquisition unit uses a multi-layer Transformer encoder to perform feature processing on the multimodal feature sequence to obtain a processed feature sequence, and then post-processes the processed feature sequence to obtain the abnormal information value;

[0042] The abnormal video judgment unit is configured with an abnormal information threshold. When the abnormal information value is greater than the abnormal information threshold, the original video is judged to be an abnormal video; when the abnormal information value is less than or equal to the abnormal information threshold, the original video is judged to be a non-abnormal video.

[0043] In a third aspect, the present invention provides a terminal comprising a processor and a communication interface coupled to the processor, wherein the processor is configured to run a computer program or instruction to implement the artificial intelligence-based abnormal video detection method provided in the first aspect.

[0044] Compared with the prior art, the present invention has the following beneficial effects:

[0045] 1. The artificial intelligence-based abnormal video detection method provided by the present invention uses a pre-trained AlexNet model to extract efficient image features from video frames, thereby improving the accuracy of image feature extraction. It also uses a TextCNN model to extract text features from video text, which can accurately capture the key information in the text. Subsequently, it creatively integrates image features and text features in depth, making full use of the effective information in the video and effectively improving detection accuracy.

[0046] 2. The artificial intelligence-based abnormal video detection method provided by the present invention adopts a multi-layer Transformer model, has powerful time series modeling capabilities and parallel computing capabilities, can effectively process long sequence videos, and improve the effect and efficiency of time series feature modeling.

[0047] 3. The artificial intelligence-based abnormal video detection method provided by the present invention innovatively uses the late fusion method to postpone the fusion time of multimodal sequences to a later time. On the basis of reducing the training cost of the weighted value in the attention mechanism, it helps to more accurately select information that is useful for the current task from a large amount of information, thereby creatively optimizing the model effect and realizing the deep fusion of cross-modal features.

[0048] 4. The artificial intelligence-based abnormal video detection method provided by the present invention adds a fully connected feedforward neural network (FFN) for post-processing after the Transformer model, which can further optimize and integrate the features of the model output and improve the final prediction accuracy.

[0049] 5. The artificial intelligence-based abnormal video detection method provided by the present invention has low dependence on data sets, can accept large differences in training resources, has strong adaptability, can show performance exceeding the baseline level on different data sets, has strong detection capabilities, and can quickly detect a randomly crawled video in a relatively short period of time, meeting the real-time requirements in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The accompanying drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the accompanying drawings:

[0051] Figure 1 Flowchart of an abnormal video detection method based on artificial intelligence in an embodiment of the present invention;

[0052] Figure 2 This is an image of the luminous flux along with the frame number after extracting a video frame at 0.1s in an embodiment of the present invention;

[0053] Figure 3 It is a continuous optical flow difference image obtained by calculating and extracting a video frame at 0.1s in an embodiment of the present invention. DETAILED DESCRIPTION

[0054] To facilitate a clear description of the technical solutions of the embodiments of the present invention, the words "first" and "second" are used in the embodiments of the present invention to distinguish between identical or similar items with substantially the same functions and effects. For example, the first threshold and the second threshold are merely used to distinguish between different thresholds and do not limit their order. Those skilled in the art will understand that the words "first" and "second" do not limit the quantity or execution order, and the words "first" and "second" do not necessarily mean different.

[0055] It should be noted that, in the present invention, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the present invention should not be construed as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0056] In the present invention, "at least one" refers to one or more, and "more" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: the existence of A alone, the existence of A and B at the same time, and the existence of B alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. The following at least one item (item) or similar expressions thereof refer to any combination of these items, including any combination of single items (items) or plural items (items). For example, at least one item (item) of a, b or c can represent: a, b, c, the combination of a and b, the combination of a and c, the combination of b and c, or the combination of a, b and c, where a, b, c can be single or multiple.

[0057] With the increasing richness and diversity of online video content, many bad videos have appeared on streaming media platforms, such as videos of violent behavior. These video contents will not only have a negative impact on social development, but may also cause serious harm to the physical and mental health of young people. Therefore, how to efficiently and accurately detect and identify abnormal videos containing information such as violent behavior has become a technical problem that needs to be solved urgently. The current detection models and detection technologies have problems such as poor video feature extraction capabilities, insufficient temporal modeling, and difficulty in multimodal information fusion. They have shown defects such as low false rejection rate, high false positive rate, and high computing time cost on existing data sets. In order to solve the above problems, the present invention provides an abnormal video detection method, system and terminal based on artificial intelligence, which extracts adverse factors in the video, such as violent image features, violent text features, etc., and uses artificial intelligence models for detection and identification. It can solve the problems of poor video feature extraction capabilities, insufficient temporal modeling, and difficulty in multimodal information fusion in the existing technology, and can make full use of multimodal information, optimize feature fusion results, and improve detection performance.

[0058] In the first aspect, the present invention provides an abnormal video detection method based on artificial intelligence, see Figure 1 , including the following steps:

[0059] S1. Configure the original video, convert it into a frame sequence, calculate inter-frame parameters, and obtain a high-dimensional feature vector for each frame based on the frame sequence and inter-frame parameters; the inter-frame parameters include light flux and continuous optical flow difference;

[0060] Abnormal videos often contain undesirable behaviors, such as violent assaults and vandalism. These behaviors can cause significant brightness differences between frames. In practice, depending on the accuracy requirements, the original video is segmented into a series of frames. This can be done by sampling at a fixed rate or at intervals. The original frames are then sliced ​​and extracted, and the luminous flux and continuous optical flow differences are calculated. For example, the Farneback optical flow algorithm can be used for this calculation.

[0061] As an example, see Figure 2 , is the luminous flux along with the frame number image after extracting a video frame at 0.1s, and the continuous optical flow difference obtained by further calculation is referred to Figure 3 ,visible Figure 3 If the continuous optical flow difference in changes greatly, it means that there is a high probability that violent behavior has occurred in the video.

[0062] As a possible implementation method, obtaining a high-dimensional feature vector of each frame image based on the frame sequence and inter-frame parameters includes:

[0063] S10. Preprocess frame sequences and pretrain the AlexNet model.

[0064] As a possible implementation method, preprocessing the frame sequence includes: scaling the size of each frame image to the input size required by the AlexNet model; subtracting the average value of the dataset used when pre-training the AlexNet model from the pixel value of each frame image to obtain the pixel difference value of each frame image.

[0065] As a possible implementation method, the pre-trained AlexNet model is specifically: pre-training the AlexNet model based on the ImageNet database.

[0066] S11. Perform forward propagation calculations on the preprocessed frame sequence and inter-frame parameters based on the pre-trained AlexNet model to obtain a high-dimensional feature vector for each frame image.

[0067] AlexNet is a classic convolutional neural network model that can quickly identify image features in a short time, effectively improving the implementation efficiency of this method and ensuring the accuracy of feature extraction.

[0068] S2. Extract text features from the original video to obtain a text feature vector;

[0069] As a possible implementation method, the text features of the original video are extracted to obtain a text feature vector. Specifically, the TextCNN model is used to extract the text features of the original video, and the text feature vector is obtained based on the Keras open source artificial neural network library.

[0070] As an example, video text can be extracted and generated based on video tags, video introduction, video subtitles, etc. SpaCy is used to segment the video text into word sequences, and then the Lancaster Stemmer algorithm is used to remove common but meaningless words in the text and restore the words to their stems and basic forms. The TextCNN model is used to extract features from the video text, which can accurately capture the key information in the video text and obtain a structured text feature vector. For example, video text extraction is performed on a certain video to obtain the following video keywords (Tag): Cam, Police, and its title is: "Body-cam video shows wedding party brawl", and the video introduction is "Newport police said they arrested six people from a wedding party which led to an officer being assaulted". After segmentation into word sequences, the keywords are extracted to obtain: Body-cam, Policy, wedding party, brawl, arrested, assaulted. By comparing these keywords with the sensitive word cloud, a text feature vector is obtained.

[0071] When detecting videos, existing technologies usually use a single feature in images, audio or text, which cannot achieve effective fusion of multimodal information. The present invention fuses image features and text features of different modalities, which can fully utilize the effective information in the video and effectively improve detection accuracy.

[0072] S3. Late Fusion method and cross-modal attention mechanism are used to fuse high-dimensional feature vectors and text feature vectors to obtain multimodal feature sequences. Feature fusion specifically includes:

[0073] S31. Assign initial weights to each modal feature in the Late Fusion method

[0074] As an example, the initial weights are pre-set based on the correlation between each modality feature and the abnormal video, where the correlation is obtained by pre-training on a labeled dataset.

[0075] S32. Align and map features from different modalities;

[0076] As an example, feature normalization is performed on features from different modalities, and the feature mapping technique t-SNE is used to map features from different modalities into a common feature space to ensure that they are semantically comparable and facilitate feature fusion.

[0077] S33. A cross-modal attention mechanism is used to perform weighted fusion of features from different modalities, with weights automatically adjusted based on the contribution of different modal features to anomaly detection.

[0078] As a possible implementation method, S33 specifically includes:

[0079] S331. Construct an attention network that can learn the relationship between different modal features and assign an attention weight to each modality;

[0080] As an example, the goal of the attention network is to learn the relationship between features of different modalities and assign attention weights to each modality as follows:

[0081] A m =f att (F image , F text ), where m∈{image,text}

[0082] Among them, A m is the attention weight of modality m, f att is the attention mechanism function, F image and F text High-dimensional feature vectors representing images and text respectively.

[0083] The design of the attention mechanism function is usually based on relevance scoring. Here we use the form of "dot-product attention":

[0084]

[0085] Among them, F image and F text are the image and text feature vectors respectively, and d is the dimension of the feature for scaling.

[0086] S332. Use the attention network to process the high-dimensional feature vector and the text feature vector to obtain the attention weight of each modality;

[0087] As an example, based on the attention network constructed in the previous step, the high-dimensional features are processed to obtain the attention weights of each modality:

[0088] W image , W text =Attention(F image , F text )

[0089] Among them, W image and W text are the attention weights of image and text.

[0090] S333. Use the Softmax function to adjust the attention weights so that the sum of the attention weights of all modalities is 1;

[0091] As an example, Softmax is a function that converts a vector into a probability distribution, and its formula is:

[0092]

[0093] The core of the Attention mechanism is to perform weighted summation of input features. For each modality, a Softmax function is used to ensure that the sum of the weights of all modalities is 1:

[0094]

[0095] Among them, W′ m is the adjusted attention weight.

[0096] S334. Weighting the high-dimensional feature vector and the text feature vector based on the adjusted attention weight;

[0097] As an example, the weighted fusion of high-dimensional feature vectors based on the adjusted weights is as follows:

[0098] F fused =W′ image ·F image +W′ text ·F text

[0099] Among them, F fused It is the weighted fusion feature representation.

[0100] S335. Merging the weighted high-dimensional feature vector and the text feature vector to form a fused feature representation;

[0101] As an example, the weighted features of image and text modalities are combined as follows:

[0102] F combined =[W′ image ·F image , W′ text ·F text ]

[0103] Among them, the symbols “[,]” represent the splicing of features.

[0104] Additive attention is used for further fusion:

[0105]

[0106] Among them, W fuse is the fusion weight matrix, b fuse is the bias term, F combined is the combined modal eigenvector, W m is the modality attention weight.

[0107] S336. Normalize the fused feature representation to eliminate the scale differences between different modal features.

[0108] As an example, normalization is used to eliminate the scale differences of different modal features as follows:

[0109]

[0110] Here, μ is the mean of the feature and σ is the standard deviation.

[0111] The cross-modal attention mechanism is adopted to enable image features to focus on text features, improve the effect of cross-modal information fusion, and enable multimodal information to be fully utilized.

[0112] S34. Optimize fusion results;

[0113] As a possible implementation method, the optimized fusion result is achieved by optimizing the weights using the gradient descent method, which includes the following sub-steps:

[0114] S341. Use the RMSprop algorithm to dynamically adjust the learning rate and use the learning rate decay strategy to accelerate convergence. As an example, the RMSprop algorithm is used to dynamically adjust the learning rate and use the learning rate decay strategy, specifically:

[0115]

[0116] Among them, η t is the learning rate at step t, and λ is the decay parameter.

[0117] S342. Use Gating Mechanism to control information flow and automatically adjust the contribution of different features;

[0118] As an example, the Gating Mechanism automatically adjusts the contribution of different features. The calculation method can be expressed as follows:

[0119] G=σ(W g ·F fused +b g )

[0120] Among them, W g is the Gating weight matrix, b g is the bias term, σ is the Sigmoid function, defined as:

[0121]

[0122] Gating Mechanism dynamically adjusts the contribution of different modal features by controlling the transmission of information flow.

[0123] S343. Design a weighted cross entropy loss function to automatically adjust the weights of different categories;

[0124] As an example, the weighted cross entropy loss function is used for optimization:

[0125]

[0126] Among them, w i is the weight of the category, y i is the true label, p i is the probability predicted by the model. This loss function can be weighted for imbalanced datasets.

[0127] S344. Use Hyperopt tool to optimize weights to optimize fusion results.

[0128] As an example, we use the Hyperopt tool to perform hyperparameter optimization and find the optimal weights: Among them, L(W) is the loss function and W is the weight to be optimized.

[0129] The conventional cross-modal attention mechanism revolves around the calculation, normalization and weighted sum of attention scores. The present invention innovatively uses the Late Fusion method to postpone the fusion time of multimodal sequences to a later time. On the basis of reducing the training cost of the weighted values ​​in the attention mechanism, it helps to more accurately select information that is useful for the current task from a large amount of information, thereby creatively optimizing the model effect and realizing the deep fusion of cross-modal features.

[0130] S4. Use a multi-layer Transformer encoder to perform feature processing on the multimodal feature sequence to obtain a processed feature sequence;

[0131] As a possible implementation, the number of layers of the multi-layer Transformer encoder is 10 to 15 layers, for example, 10 layers, 11 layers, 12 layers, 13 layers, 14 layers, and 15 layers.

[0132] Through the self-attention mechanism, deep feature processing of multimodal feature sequences is performed, which removes more worthless information, making the output feature sequence contain more valuable information, and significantly improving the temporal modeling capability of long-sequence videos.

[0133] S5. post-processing the processed feature sequence to obtain an abnormal information value;

[0134] As an example, a fully connected feedforward neural network (FFN) is used to post-process the processed feature sequence to further optimize and fuse the model output features to obtain the abnormal information value;

[0135] S6. Configure an abnormal information threshold. When the abnormal information value is greater than the abnormal information threshold, the original video is judged to be an abnormal video; when the abnormal information value is less than or equal to the abnormal information threshold, the original video is judged to be a non-abnormal video.

[0136] The present invention has low dependence on data sets, can accept large differences in training resources, has strong adaptability, can show performance exceeding the baseline level on different data sets, has strong detection capabilities, and can quickly detect a randomly crawled video in a relatively short time, meeting the real-time requirements in practical applications.

[0137] Tables 1 to 4 compare the advantages and disadvantages of the method of the present invention with two traditional methods, CNN and LSTM, on different evaluation indicators of four datasets, including RWF-2000, YTB-Fights, UBI-Fights, and ViolentFlow. The RWF-2000 dataset is from Cheng M, Cai K, Li M. RWF-2000: an open large-scale video database for violence detection [C] / / 2020 25th International Conference on Pattern Recognition (ICPR). IEEE, 2021: 4183-4190. https: / / ieeexplore.ieee.org / abstract / document / 9412502 / ; the YTB-Fights dataset is from Soliman MM, Kamal MH, Nashed MA EM, et al. Violence recognition from videos using deep learning techniques [C] / / 2019 ninth international conference on intelligent computing and information systems (ICICIS). IEEE, 2019: 80-85. https: / / ieeexplore.ieee.org / document / 9014714; the UBI-Fights dataset is from Degardin B, H.Human activity analysis: Iterativeweak / self-supervised learning frameworks for detecting abnormal events[C] / / 2020IEEE International Joint Conference on Biometrics(IJCB).IEEE,2020:1-7.https: / / ieeexplore.ieee.org / document / 9304905; ViolentFlow data set comes from: Hassner T, Itcher Y, Kliper-Gross O.Violent flows:Real-time detection ofviolent crowd behavior[C] / / 2012IEEE computer society conference on computervision and pattern recognition workshops.IEEE,2012:1-6.https: / / ieeexplore.ieee.org / abstract / document / 6239348.

[0138] The evaluation indicators are accuracy, F1-Score, area under the receiver operating curve (AUC) and logarithmic loss (LogLoss):

[0139] Table 1 Performance of the present invention and two traditional methods, CNN and LSTM, on the RWF-2000 dataset

[0140]

[0141] Table 2 Performance of the present invention and two traditional methods, CNN and LSTM, on the YTB-Fights dataset

[0142]

[0143] Table 3 Performance of the present invention and two traditional methods, CNN and LSTM, on the UBI-Fights dataset

[0144]

[0145] Table 4 Performance of the present invention and two traditional methods, CNN and LSTM, on the ViolentFlow dataset

[0146]

[0147] From the above comparison, it can be found that the abnormal video detection method based on artificial intelligence provided by the present invention has shown advantages in various evaluation indicators and can improve video detection performance.

[0148] In a second aspect, the present invention provides an abnormal video detection system based on artificial intelligence, comprising:

[0149] The high-dimensional feature vector acquisition unit is configured to configure the original video, convert the original video into a frame sequence, calculate the inter-frame parameters, and obtain the high-dimensional feature vector of each frame image based on the frame sequence and the inter-frame parameters; the inter-frame parameters include light flux and continuous optical flow difference;

[0150] A text feature vector acquisition unit extracts text features of the original video and obtains a text feature vector;

[0151] The multimodal feature sequence acquisition unit uses the late fusion method and cross-modal attention mechanism to fuse high-dimensional feature vectors and text feature vectors to obtain a multimodal feature sequence;

[0152] The abnormal information value acquisition unit uses a multi-layer Transformer encoder to perform feature processing on the multimodal feature sequence to obtain a processed feature sequence, and then post-processes the processed feature sequence to obtain the abnormal information value;

[0153] The abnormal video judgment unit is configured with an abnormal information threshold. When the abnormal information value is greater than the abnormal information threshold, the original video is judged to be an abnormal video; when the abnormal information value is less than or equal to the abnormal information threshold, the original video is judged to be a non-abnormal video.

[0154] In a third aspect, the present invention provides a terminal comprising a processor and a communication interface coupled to the processor, wherein the processor is configured to run a computer program or instruction to implement the artificial intelligence-based abnormal video detection method provided in the first aspect.

[0155] Although the present invention is described herein in conjunction with various embodiments, in the process of implementing the claimed invention, those skilled in the art can understand and implement other variations of the disclosed embodiments by viewing the drawings, the disclosure, and the drawings, etc. In the specification, the word "comprising" does not exclude other components or steps, and "one" or "an" does not exclude multiple situations. A single processor or other unit can implement several functions listed in the specification. Certain measures are recorded in different embodiments, but this does not mean that these measures cannot be combined to produce good results.

[0156] Although the present invention has been described with reference to specific features and embodiments thereof, it will be apparent that various modifications and combinations thereof may be made without departing from the spirit and scope of the invention. Accordingly, this specification and drawings are merely illustrative of the present invention and are deemed to cover any and all modifications, variations, combinations or equivalents within the scope of the invention. It will be apparent that various modifications and variations of the present invention may be made by those skilled in the art without departing from the spirit and scope of the invention. Thus, the present invention is intended to include such modifications and variations as fall within the scope of the invention and its equivalents.

Claims

1. An abnormal video detection method based on artificial intelligence, characterized in that: The steps include: S1. Configuring the original video, converting the original video into a frame sequence, calculating inter-frame parameters, and obtaining a high-dimensional feature vector for each frame based on the frame sequence and inter-frame parameters; the inter-frame parameters include light flux and continuous optical flow difference; the light flux is the brightness difference between frames; S2 extracts text features of the original video to obtain a text feature vector; S3. Using a late fusion method and a cross-modal attention mechanism to fuse the high-dimensional feature vector and the text feature vector to obtain a multimodal feature sequence; The feature fusion specifically includes: S31 assigning an initial weight to each modal feature in the Late Fusion method; S32. Align and map features from different modalities; S33. Using the cross-modal attention mechanism to perform weighted fusion of features from different modalities, the weights are automatically adjusted based on the contribution of the features from different modalities to anomaly detection; S34. Optimize fusion results; S4. Performing feature processing on the multimodal feature sequence using a multi-layer Transformer encoder to obtain a processed feature sequence; the multi-layer Transformer encoder has 10 to 15 layers; S5. Post-processing the processed feature sequence to obtain an abnormal information value; S6. Configure an abnormal information threshold. When the abnormal information value is greater than the abnormal information threshold, the original video is determined to be an abnormal video; when the abnormal information value is less than or equal to the abnormal information threshold, the original video is determined to be a non-abnormal video.

2. The abnormal video detection method based on artificial intelligence according to claim 1 is characterized in that: Obtaining a high-dimensional feature vector of each frame image based on the frame sequence and inter-frame parameters, including: S10. Preprocessing the frame sequence and pre-training the AlexNet model; S11. Perform forward propagation calculations on the preprocessed frame sequence and inter-frame parameters based on the pretrained AlexNet model to obtain a high-dimensional feature vector for each frame image.

3. The abnormal video detection method based on artificial intelligence according to claim 2 is characterized in that: Preprocessing the frame sequence includes: scaling the size of each frame image to the input size required by the AlexNet model; and subtracting the average value of the data set used when pre-training the AlexNet model from the pixel value of each frame image to obtain the pixel difference value of each frame image.

4. The abnormal video detection method based on artificial intelligence according to claim 2 is characterized in that: The pre-trained AlexNet model is specifically: pre-trained AlexNet model based on the ImageNet database.

5. The abnormal video detection method based on artificial intelligence according to claim 1 is characterized in that: The extracting of text features of the original video to obtain a text feature vector specifically comprises: extracting the text features of the original video using a TextCNN model, and obtaining a text feature vector based on the Keras open source artificial neural network library.

6. The abnormal video detection method based on artificial intelligence according to claim 1 is characterized in that: The S33 specifically includes: S331. Constructing an attention network that can learn the relationship between features of different modalities and assign an attention weight to each modality; S332. Processing the high-dimensional feature vector and the text feature vector using the attention network to obtain an attention weight for each modality; S333. Using the Softmax function to adjust the attention weights so that the sum of the attention weights of all modalities is 1; S334. Weighting the high-dimensional feature vector and the text feature vector based on the adjusted attention weight; S335. Merging the weighted high-dimensional feature vector and the text feature vector to form a fused feature representation; S336. Normalize the fused feature representation to eliminate scale differences between features of different modalities.

7. The abnormal video detection method based on artificial intelligence according to claim 1 is characterized in that: The optimization fusion result is achieved by optimizing weights using the gradient descent method, including the following sub-steps: S341. Use the RMSprop algorithm to dynamically adjust the learning rate and use the learning rate decay strategy to accelerate convergence. S342. Use Gating Mechanism to control information flow and automatically adjust the contribution of different features; S343. Design a weighted cross-entropy loss function to automatically adjust the weights of different categories; S344. Use Hyperopt to optimize weights and optimize fusion results.

8. An abnormal video detection system based on artificial intelligence, characterized in that: include: a high-dimensional feature vector acquisition unit configured to receive an original video, convert the original video into a frame sequence, calculate inter-frame parameters, and obtain a high-dimensional feature vector for each frame based on the frame sequence and the inter-frame parameters; the inter-frame parameters include luminance and continuous optical flow difference; the luminance is the luminance difference between frames; A text feature vector acquisition unit extracts text features of the original video to obtain a text feature vector; A multimodal feature sequence acquisition unit uses a late fusion method and a cross-modal attention mechanism to perform feature fusion on the high-dimensional feature vector and the text feature vector to obtain a multimodal feature sequence; An abnormal information value acquisition unit is configured to perform feature processing on the multimodal feature sequence using a multi-layer Transformer encoder to obtain a processed feature sequence, where the multi-layer Transformer encoder has 10 to 15 layers; and perform post-processing on the processed feature sequence to obtain an abnormal information value. an abnormal video judging unit configured with an abnormal information threshold value, and judging the original video as an abnormal video when the abnormal information value is greater than the abnormal information threshold value; When the abnormal information value is less than or equal to the abnormal information threshold, the original video is determined to be a non-abnormal video.

9. A terminal for abnormal video detection method based on artificial intelligence, characterized in that: It includes a processor and a communication interface coupled to the processor, and the processor is used to run a computer program or instruction to implement the abnormal video detection method based on artificial intelligence as described in any one of claims 1 to 7.

Images