Method and device for identifying abnormal behavior in video, equipment and storage medium

Abstract

The embodiment of the invention provides a method and device for identifying an abnormal behavior in a video, equipment and a storage medium, and relates to the technical field of artificial intelligence, and the method comprises the steps: determining whether each image frame in a to-be-identified video has a first identification result of the abnormal behavior through an image identification model; determining a text description of each image frame in the to-be-identified video so as to obtain a text description sequence of the to-be-identified video; through a text recognition model, determining a second recognition result that an abnormal behavior exists in an image frame corresponding to the text description sequence; determining a comprehensive recognition result of the ith image frame based on the first recognition result corresponding to the ith image frame and the second recognition result corresponding to the ith image frame; the ith image frame is any image frame in the video to be identified; and when the number of the image frames of which the comprehensive identification result meets the first set condition meets a second set condition, determining that the abnormal behavior exists in the to-be-identified video.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method, device, equipment and storage medium for identifying abnormal behavior in a video. Background Art

[0002] At present, the main purpose of monitoring some scenes in life is to observe in real time whether there are abnormal phenomena in the monitored scenes, so that emergency measures can be taken quickly. When judging whether there are abnormal events or target objects in the video, a small model is usually used to classify the single-source video.

[0003] However, due to the uneven distribution of data, videos from different sources often have different characteristics. Therefore, it is difficult to align and train a small model with only a small number of labeled samples. It is impossible to identify abnormal events in different scenarios with the same model, and therefore it is impossible to apply the same model to different scenarios. Summary of the invention

[0004] The embodiments of the present application provide a method, apparatus, device and storage medium for identifying abnormal behavior in a video, which are used to identify abnormal behavior in videos in different scenarios.

[0005] In a first aspect, an embodiment of the present application provides a method for identifying abnormal behavior in a video, comprising:

[0006] Determine, by means of an image recognition model, a first recognition result of whether each image frame in the video to be recognized has abnormal behavior;

[0007] Determining a text description of each image frame in the to-be-recognized video, thereby obtaining a text description sequence of the to-be-recognized video;

[0008] Determining, by a text recognition model, a second recognition result that an abnormal behavior exists in the image frame corresponding to the text description sequence;

[0009] Determine a comprehensive recognition result of the i-th image frame based on a first recognition result corresponding to the i-th image frame and a second recognition result corresponding to the i-th image frame; the i-th image frame is any image frame in the video to be recognized;

[0010] When the number of image frames whose comprehensive recognition results satisfy the first set condition meets the second set condition, it is determined that abnormal behavior exists in the video to be recognized.

[0011] In the embodiment of the present application, the image recognition model and the text recognition model are used to identify whether there is abnormal behavior in the video to be identified, which increases the reliability of abnormal behavior identification and avoids the problem of inaccurate recognition results of the video to be identified due to weak generalization ability of the image recognition model or the text recognition model, or poor training effect. Based on the first recognition result of the image recognition model and the second recognition result of the text recognition model, the comprehensive recognition result of the video to be identified is determined, and the comprehensive recognition result includes the common recognition results of the image recognition model and the text recognition model, so that the recognition accuracy of the abnormal behavior of the video to be identified is higher.

[0012] Optionally, before determining the comprehensive recognition result of the i-th image frame, the method further includes:

[0013] The first recognition result sequence and the second recognition result sequence are respectively input into a sequence value estimation model representing the continuity of the behavior to obtain a first sequence estimation result and a second sequence estimation result; the first recognition result sequence is obtained in sequence based on the first recognition result of each image frame in the video to be recognized; the second recognition result sequence is obtained in sequence based on the second recognition result of each image frame in the video to be recognized;

[0014] Obtaining a first deviation result sequence according to the first recognition result sequence and the first sequence estimation result;

[0015] Obtaining a second deviation result sequence according to the second recognition result sequence and the second sequence estimation result;

[0016] Determining a comprehensive recognition result of the i-th image frame based on the first recognition result of the i-th image frame and the second recognition result of the i-th image frame includes:

[0017] Determine, based on a first deviation result of the i-th image frame in the first deviation result sequence and a second deviation result of the i-th image frame in the second deviation result sequence, a first weight corresponding to the first recognition result of the i-th image frame and a second weight corresponding to the second recognition result corresponding to the i-th image frame;

[0018] A comprehensive recognition result of the i-th image frame is determined based on the first weight of the i-th image frame, the second weight of the i-th image frame, the first recognition result of the i-th image frame, and the second recognition result of the i-th image frame.

[0019] In an embodiment of the present application, the recognition result is input into a sequence value estimation model to obtain a sequence estimation result, which improves the continuity of the recognition result compared to the recognition result sequence; based on the difference between the recognition result sequence and the sequence estimation result, the better model of the image recognition model and the text recognition model is determined, thereby giving the better model a greater weight, thereby improving the accuracy of the final comprehensive recognition result.

[0020] Optionally, the sequence value estimation model is obtained by training a sequence of sample recognition results output by the image recognition model through sample videos with abnormal behavior; the sequence value estimation model is used to predict the recognition results of subsequent frames based on the continuity of behavior through the recognition results of historical frames.

[0021] In an embodiment of the present application, the output of the image recognition model is used as a sequence of sample recognition results of the sequence value estimation model to train the sequence value estimation model, so that the sequence value estimation model has better recognition results. The recognition results of subsequent frames are predicted by the recognition results of historical frames, thereby enhancing the consistency of the recognition results.

[0022] Optionally, the image recognition model is a cascade model with generalized recognition capability obtained by training multiple single abnormal behavior recognition models; wherein any single abnormal behavior recognition model is obtained by performing supervised learning training on samples with the same type of abnormal behavior.

[0023] In the embodiment of the present application, the image recognition model is obtained by training multiple single abnormal behavior recognition models, so that the image recognition model aggregates the recognition results of multiple single abnormal behavior recognition models, thereby improving the generalization ability of the image recognition model.

[0024] Optionally, the image recognition model is trained in the following manner, including:

[0025] Input the same sample into each single abnormal behavior recognition model respectively to obtain a single recognition result output by each single abnormal behavior recognition model; determine the loss value of each single abnormal behavior recognition model according to the real recognition result of the sample video and the single recognition result of each single abnormal behavior recognition model;

[0026] The sample is input into the image recognition model, and the image recognition model is trained based on the loss value of each single abnormal behavior recognition model until a training termination condition is met.

[0027] In an embodiment of the present application, a single abnormal behavior recognition model is trained by using the same sample to obtain a single recognition result, and then the sample is input into an image recognition model to train the image recognition model. The image recognition model is trained using the loss value of the single abnormal behavior recognition model, so that the image recognition model covers the recognition capability of the single abnormal behavior recognition model.

[0028] Optionally, the loss value of each single anomaly recognition model is determined by the following formula:

[0029]

[0030] Optionally, the samples of the multiple single abnormal behavior recognition models and the samples of the image recognition model are obtained respectively in the following ways:

[0031] Obtain a normal sample set and an abnormal sample set;

[0032] Clustering the abnormal sample set to obtain abnormal subsets of multiple categories; each category corresponds to a single abnormal recognition model;

[0033] Constructing a sample set for each single anomaly recognition model; wherein the normal samples are the first part of the normal subset randomly determined from the normal sample set, and the abnormal samples are the first part of the corresponding abnormal subset;

[0034] Construct a sample set of the image recognition model; wherein the normal sample is the second part of each normal subset, and the abnormal sample is the second part of each abnormal subset; the second part in the normal subset is different from the first part in the normal subset, and the second part in the abnormal subset is different from the first part in the abnormal subset.

[0035] In the embodiment of the present application, when constructing samples, first clustering is performed based on abnormal samples to obtain multiple abnormal samples of different categories, and then normal samples are randomly added to the abnormal samples of each category to obtain samples of normal samples and abnormal samples, and then a part of the normal samples and abnormal samples are used as the sample set of a single abnormal recognition model, and another part of the normal samples and abnormal samples are used as the sample set of the image recognition model. By constructing different sample sets to train the single abnormal recognition model and the image recognition model respectively, the generalization ability of the model is improved.

[0036] Optionally, the text recognition model is a large language model;

[0037] Determining, by means of a text recognition model, whether there is a second recognition result of abnormal behavior in the image frame corresponding to the text description sequence, including:

[0038] The text description sequence and the abnormal behavior prompt word are input into the large language model to determine whether a second recognition result of abnormal behavior exists in the image frame corresponding to the text description sequence.

[0039] In the embodiment of the present application, by using a large language model as a text recognition model, the accuracy of text recognition can be improved. By inputting a text description sequence and abnormal behavior prompt words into the large language model, a second recognition result of whether an image frame has abnormal behavior can be more accurately determined.

[0040] Optionally, the step of inputting the text description sequence and the abnormal behavior prompt word into the large language model to determine whether a second recognition result of abnormal behavior exists in the image frame corresponding to the text description sequence includes:

[0041] Determine a text description subsequence of the i-th image frame from the text description sequence;

[0042] Inputting the text description subsequence and the scene confirmation prompt word of the i-th image frame into the large language model to determine the scene text description of the i-th image frame;

[0043] The scene text description and abnormal behavior prompt words of the i-th image frame are input into the large language model to obtain a second recognition result of whether there is abnormal behavior in the i-th image frame.

[0044] In an embodiment of the present application, a subsequence is determined in a text description sequence, a scene confirmation prompt word is determined according to the subsequence, the scene text description of the image frame is determined through a large language model, and then the scene text description and the abnormal behavior prompt word are input into the large language model, thereby making the recognition result of abnormal behavior more accurate.

[0045] Optionally, determining a text description of each image frame in the to-be-identified video includes:

[0046] Determining an initial text description of each image frame in the video to be identified;

[0047] Determining image features of the i-th image frame, and determining text features of the i-th image frame from an initial text description of the i-th image frame;

[0048] From the text features of the i-th image frame, text features that meet similarity requirements with the image features of the i-th image frame are determined as text descriptions of the i-th image frame.

[0049] In an embodiment of the present application, by first determining an initial text description, then determining text features based on the initial text description, and by using text features that meet similarity requirements as text descriptions of image frames, the problem of possible errors in the text description of a single image and the generation of different descriptions for images with similar content is avoided.

[0050] In a second aspect, an embodiment of the present application provides a code change analysis device, including:

[0051] A recognition result module, used to determine, through an image recognition model, a first recognition result of whether each image frame in the video to be recognized has abnormal behavior;

[0052] The determination module is further used to determine the text description of each image frame in the video to be identified, so as to obtain a text description sequence of the video to be identified;

[0053] The recognition result module is further used to determine, through a text recognition model, a second recognition result that an abnormal behavior exists in the image frame corresponding to the text description sequence;

[0054] A comprehensive recognition module, used to determine a comprehensive recognition result of an i-th image frame based on a first recognition result corresponding to an i-th image frame and a second recognition result corresponding to the i-th image frame; the i-th image frame is any image frame in the video to be recognized;

[0055] The abnormal behavior module is used to determine that abnormal behavior exists in the video to be identified when the number of image frames whose comprehensive identification results meet the first set condition meets the second set condition.

[0056] Optionally, the comprehensive identification module is further used for:

[0057] The first recognition result sequence and the second recognition result sequence are respectively input into a sequence value estimation model representing the continuity of the behavior to obtain a first sequence estimation result and a second sequence estimation result; the first recognition result sequence is obtained in sequence based on the first recognition result of each image frame in the video to be recognized; the second recognition result sequence is obtained in sequence based on the second recognition result of each image frame in the video to be recognized;

[0058] Obtaining a first deviation result sequence according to the first recognition result sequence and the first sequence estimation result;

[0059] Obtaining a second deviation result sequence according to the second recognition result sequence and the second sequence estimation result;

[0060] Determining a comprehensive recognition result of the i-th image frame based on the first recognition result of the i-th image frame and the second recognition result of the i-th image frame includes:

[0061] Determine, based on a first deviation result of the i-th image frame in the first deviation result sequence and a second deviation result of the i-th image frame in the second deviation result sequence, a first weight corresponding to the first recognition result of the i-th image frame and a second weight corresponding to the second recognition result corresponding to the i-th image frame;

[0062] A comprehensive recognition result of the i-th image frame is determined based on the first weight of the i-th image frame, the second weight of the i-th image frame, the first recognition result of the i-th image frame, and the second recognition result of the i-th image frame.

[0063] Optionally, the sequence value estimation model is obtained by training a sequence of sample recognition results output by the image recognition model through sample videos with abnormal behavior; the sequence value estimation model is used to predict the recognition results of subsequent frames based on the continuity of behavior through the recognition results of historical frames.

[0064] Optionally, the image recognition model is a cascade model with generalized recognition capability obtained by training multiple single abnormal behavior recognition models; wherein any single abnormal behavior recognition model is obtained by performing supervised learning training on samples with the same type of abnormal behavior.

[0065] Optionally, the recognition result module is specifically used to:

[0066] Input the same sample into each single abnormal behavior recognition model respectively to obtain a single recognition result output by each single abnormal behavior recognition model; determine the loss value of each single abnormal behavior recognition model according to the real recognition result of the sample and the single recognition result of each single abnormal behavior recognition model;

[0067] The sample is input into the image recognition model, and the image recognition model is trained based on the loss value of each single abnormal behavior recognition model until a training termination condition is met.

[0068] Optionally, the loss value of each single anomaly recognition model is determined by the following formula:

[0069]

[0070] Optionally, the recognition result module is specifically used to:

[0071] Obtain a normal sample set and an abnormal sample set;

[0072] Clustering the abnormal sample set to obtain abnormal subsets of multiple categories; each category corresponds to a single abnormal recognition model;

[0073] Constructing a sample set for each single anomaly recognition model; wherein the normal samples are the first part of the normal subset randomly determined from the normal sample set, and the abnormal samples are the first part of the corresponding abnormal subset;

[0074] Construct a sample set of the image recognition model; wherein the normal sample is the second part of each normal subset, and the abnormal sample is the second part of each abnormal subset; the second part in the normal subset is different from the first part in the normal subset, and the second part in the abnormal subset is different from the first part in the abnormal subset.

[0075] Optionally, the recognition result module is specifically used to:

[0076] Determining, by means of a text recognition model, whether there is a second recognition result of abnormal behavior in the image frame corresponding to the text description sequence, including:

[0077] The text description sequence and the abnormal behavior prompt word are input into the large language model to determine whether a second recognition result of abnormal behavior exists in the image frame corresponding to the text description sequence.

[0078] Optionally, the recognition result module is specifically used to:

[0079] Determine a text description subsequence of the i-th image frame from the text description sequence;

[0080] Inputting the text description subsequence and the scene confirmation prompt word of the i-th image frame into the large language model to determine the scene text description of the i-th image frame;

[0081] The scene text description and abnormal behavior prompt words of the i-th image frame are input into the large language model to obtain a second recognition result of whether there is abnormal behavior in the i-th image frame.

[0082] Optionally, the recognition result module is specifically used to:

[0083] Determining an initial text description of each image frame in the video to be identified;

[0084] Determining image features of the i-th image frame, and determining text features of the i-th image frame from an initial text description of the i-th image frame;

[0085] From the text features of the i-th image frame, text features that meet similarity requirements with the image features of the i-th image frame are determined as text descriptions of the i-th image frame.

[0086] In a third aspect, an embodiment of the present application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any of the above-mentioned methods when executing the program.

[0087] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium storing a computer program executable by a computer device, wherein when the program is run on the computer device, the computer device executes the steps of any of the above-described methods.

[0088] In a fifth aspect, the present application provides a computer program product, which, when executed on a computer, enables the computer to execute the methods of the various embodiments of the first aspect described above.

[0089] In the embodiment of the present application, the image recognition model and the text recognition model are used to identify whether there is abnormal behavior in the video to be identified, which increases the reliability of abnormal behavior identification and avoids the problem of inaccurate recognition results of the video to be identified due to weak generalization ability of the image recognition model or the text recognition model, or poor training effect. Based on the first recognition result of the image recognition model and the second recognition result of the text recognition model, the comprehensive recognition result of the video to be identified is determined, and the comprehensive recognition result includes the common recognition results of the image recognition model and the text recognition model, so that the recognition accuracy of the abnormal behavior of the video to be identified is higher. BRIEF DESCRIPTION OF THE DRAWINGS

[0090] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0091] Figure 1 A system architecture diagram provided for an embodiment of the present application;

[0092] Figure 2 A flowchart of a method for identifying abnormal behavior in a video provided by an embodiment of the present application;

[0093] Figure 3 A schematic diagram of a process for determining a comprehensive recognition result of an image frame provided in an embodiment of the present application (I);

[0094] Figure 4 A schematic diagram of a process for determining a comprehensive recognition result of an image frame provided in an embodiment of the present application (II);

[0095] Figure 5 A structural diagram of a training method for a single abnormal behavior recognition model provided in an embodiment of the present application;

[0096] Figure 6 A structural diagram of a method for training an image recognition model provided in an embodiment of the present application;

[0097] Figure 7 A schematic diagram of a process for obtaining samples of multiple single abnormal behavior recognition models and samples of image recognition models provided in an embodiment of the present application;

[0098] Figure 8 A schematic diagram of a second recognition result flow of abnormal behavior in an image frame corresponding to a text description sequence provided in an embodiment of the present application;

[0099] Fig. 9A schematic diagram of a process for determining a text description of each image frame in a video to be identified provided in an embodiment of the present application;

[0100] Fig.10 A schematic diagram of the structure of a device for identifying abnormal behavior in a video provided in an embodiment of the present application;

[0101] Fig.11 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0102] In order to make the purpose, technical solution and beneficial effects of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0103] To facilitate understanding of this solution, the application scenario of this solution is introduced below.

[0104] At present, visual anomaly detection algorithms are usually used to supervise and detect abnormal scenes. Visual anomaly detection algorithms often need to collect the data to be detected, divide the detected data into normal samples and abnormal samples, and then train the model, so that the trained model can be used to achieve the purpose of anomaly detection for a specific scene. For example, if you need to detect whether someone is smoking in a scene (smoking is an abnormal scene), you need to collect the corresponding smoking abnormal samples for model training and detection. However, the trained model cannot detect other abnormal scenes. For example, the model for detecting whether there is smoking cannot detect whether there is a scene of alcoholism. Text is naturally universal. Recently, some studies have used weakly supervised video anomaly detection methods under text prompts. It uses text features and video features to participate in the training of deep networks. In essence, it uses text features to strengthen visual features to achieve anomaly detection. However, it is still a single-scene supervised learning method, which cannot be generalized to other scenes outside the training set, and has not fundamentally improved the generalization ability of anomaly detection.

[0105] Anomaly detection models for different scenarios cannot achieve universal effects, and models trained in one scenario often cannot be directly applied to other scenarios. To achieve cross-domain video anomaly detection, there are still some challenges and limitations: 1. Unbalanced pairing of datasets: Anomaly instances that may be easily identified in one dataset may be difficult to be correctly classified as anomalies in another dataset. This may cause the model to favor a certain field and ignore information in other fields, thereby affecting its generalization ability. 2. Inconsistency in feature extraction: When using features from different data sources, the features may be inconsistent due to different representations, measurements, or preprocessing methods of the data. This may make it difficult for the model to learn meaningful connections and reduce its performance. 3. Lack of common information: If objects from different datasets do not have common semantic information as a basis, cross-domain modeling may not be able to fully utilize their intrinsic relationships. 4. Inefficient computation: Cross-domain video anomaly detection usually requires a lot of computing resources to train the model and perform inference. Especially when large neural networks are involved, this may become a performance bottleneck. 5. Explainability issues: Current cross-domain video anomaly detection methods often lack explainability, which means that it is impossible to understand why the model makes a specific prediction result. This brings challenges to the deployment and application of the model. The following are the specific implementation steps of this application:

[0106] First, the present application collects multiple normal samples and abnormal samples, wherein the number of normal samples is much larger than the number of abnormal samples. The abnormal samples are divided into multiple categories by clustering algorithm, and the normal samples are randomly added to the abnormal samples to obtain a training sample set distinguished by category, and each class of training sample set is randomly divided into a basic training set and a query training set. Then, the basic training set of each category is input into the basic model, and the loss function is adjusted to obtain a trained basic model of each category. The query training sets of all categories are respectively input into the trained multiple basic models and a cascade model, and the cascade model is adjusted by the loss function of the basic model to obtain a trained cascade model. After the training sample set is input into the trained cascade model, the first predicted abnormal score value of the training sample is obtained, and the abnormal score value is input into the score sequence model to obtain the estimated abnormal score value. The score sequence model is adjusted by the first predicted abnormal score value and the estimated abnormal score value to obtain a trained score training model. Next, all training sample sets are input into the large model, and the second predicted abnormal score value of the training sample set is determined by designing prompt words. Finally, the abnormal score values ​​output by the cascade model and the large model are input into the score sequence model. The weight values ​​of each model are determined by the difference between the abnormal score value output by the model itself and the score value output by the score sequence model, thereby completing the final model training. When using the trained model, the video to be identified is input into the cascade model and the large model, and the output abnormal score values ​​are input into the score sequence model respectively. By comparing with the set threshold, it is determined whether the video to be identified is an abnormal video.

[0107] See also Figure 1 , is a system architecture diagram provided in an embodiment of the present application, the system architecture includes a terminal device 101 and a server 102.

[0108] The terminal device 101 is pre-installed with a service application for text matching, wherein the service application is a client application, a web application, a small program application, etc. The terminal device 101 can be a smart phone, a POS machine, a desktop notebook, a computer, etc., but is not limited thereto.

[0109] Server 102 is the backend server for business applications. Server 102 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), as well as big data and artificial intelligence platforms.

[0110] refer to Figure 2, a method for identifying abnormal behavior in a video provided in an embodiment of the present application, comprising the following steps:

[0111] Step 201: Determine, by using an image recognition model, whether each image frame in the video to be recognized has a first recognition result of abnormal behavior.

[0112] Specifically, the image recognition model is used to identify whether there is abnormal behavior in the image. Each image frame in the video to be identified is input into the image recognition model, and a first recognition result of the video to be identified is output. The first recognition result represents the possibility of abnormal behavior in the video to be identified obtained through the image recognition model.

[0113] The image recognition model can be a visual anomaly detection model, which is divided into a supervised model, a semi-supervised learning model and an unsupervised model. The embodiment of the present application uses a supervised model to train the visual anomaly detection model, and can also use a semi-supervised learning model or an unsupervised model.

[0114] Step 202: Determine the text description of each image frame in the video to be identified, thereby obtaining a text description sequence of the video to be identified.

[0115] Specifically, the text description of each image frame in the video to be identified reflects what happens in the scene where each frame of the image is located. Based on the text description of each image frame, a text description sequence of the video to be identified is obtained, so that what happens in the scene where the video to be identified is summarized.

[0116] Step 203: Determine, by using the text recognition model, a second recognition result that indicates that abnormal behavior exists in the image frame corresponding to the text description sequence.

[0117] Specifically, by inputting the text description sequence of the video to be identified into the text recognition model, a second recognition result indicating the presence of abnormal behavior in the image frame corresponding to the text description sequence can be obtained. The second recognition result indicates the possibility of the presence of abnormal behavior in the video to be identified obtained through the text recognition model.

[0118] Step 204 , determining a comprehensive recognition result of the ith image frame based on the first recognition result corresponding to the ith image frame and the second recognition result corresponding to the ith image frame; the ith image frame is any image frame in the video to be recognized.

[0119] Specifically, the first recognition result and the second recognition result are combined to obtain a comprehensive recognition result of each image frame, and the comprehensive recognition result is used to characterize the possibility of abnormal behavior in each image frame.

[0120] Step 205: When the number of image frames whose comprehensive recognition results satisfy the first set condition meets the second set condition, it is determined that abnormal behavior exists in the video to be recognized.

[0121] Specifically, it is determined whether the comprehensive recognition result of the image frame meets the first set condition. If not, the image frame is not processed in the next step; if it is satisfied, it is further determined whether the number of image frames meets the second condition. If the number of image frames meets the second condition, it is considered that the video to be recognized has abnormal behavior; if the number of image frames does not meet the second condition, it is considered that the video to be recognized does not have abnormal behavior.

[0122] In the embodiment of the present application, the first recognition result and the second recognition result obtained by the image recognition model and the text recognition model of the video to be recognized can be the probability that the video to be recognized has abnormal behavior, or can be the score value of the video to be recognized having abnormal behavior. The larger the score value, the greater the possibility of abnormal behavior. This article takes the output of the image recognition model and the text recognition model as the score value as an example for explanation.

[0123] In some embodiments, before determining the comprehensive recognition result of the i-th image frame, Figure 3 As shown, the following steps are also included:

[0124] Step 301, input the first recognition result sequence and the second recognition result sequence into the sequence value estimation model representing the continuity of the behavior respectively, to obtain the first sequence estimation result and the second sequence estimation result; the first recognition result sequence is obtained in sequence based on the first recognition result of each image frame in the video to be recognized; the second recognition result sequence is obtained in sequence based on the second recognition result of each image frame in the video to be recognized.

[0125] Specifically, a first recognition result sequence of the video to be recognized is obtained according to the first recognition result of each image frame, and the first recognition result sequence is obtained by arranging the image frames of the video to be recognized in sequence. Assuming that there are 10 frames of the video to be recognized, the first recognition result sequence is: in is the recognition result of the first image frame, is the recognition result of the second image frame, ..., is the recognition result of the tenth image frame. A second recognition result sequence of the video to be recognized is obtained according to the second recognition result of each image frame. The second recognition result sequence is also obtained by arranging the image frames of the video to be recognized in order. The second recognition result sequence is: in, is the recognition result of the first image frame, is the recognition result of the second image frame, ..., is the recognition result of the tenth image frame. The first recognition result sequence and the second recognition result sequence are respectively input into the sequence value estimation model to obtain the first sequence estimation result and the second sequence estimation result, as shown in formula (1) and formula (2):

[0126]

[0127] Among them, ξ is the sequence value estimation model, k is the number of image frames of the video to be identified, is the first recognition result sequence, is the second recognition result sequence, is the first sequence estimation result, This is the second sequence estimation result.

[0128] The first sequence of estimation results and the second sequence of estimation results have stronger coherence than the first recognition result sequence and the second recognition result sequence.

[0129] The sequence value estimation model is a method for processing and predicting important events in time series with very long intervals and delays. Through the sequence value estimation model, data with time delays can be corrected.

[0130] Step 302: Obtain a first deviation result sequence according to the first recognition result sequence and the first sequence estimation result.

[0131] Step 303: Obtain a second deviation result sequence according to the second recognition result sequence and the second sequence estimation result.

[0132] Specifically, as shown in the figure, the difference between the first recognition result sequence and the first sequence estimation result is calculated to obtain the first deviation result, and the second recognition result sequence and the second sequence estimation result are calculated to obtain the second deviation result. The first deviation result and the second deviation result are shown in formula (3) and formula (4) respectively:

[0133]

[0134] Among them, △V k is the first deviation result, △T k is the second deviation result, is the first sequence estimation result, is the first recognition result sequence, is the second sequence estimation result, is the second recognition result sequence.

[0135] Step 304, determining a comprehensive recognition result of the i-th image frame based on the first recognition result of the i-th image frame and the second recognition result of the i-th image frame, includes:

[0136] Determine, based on a first deviation result of the i-th image frame in the first deviation result sequence and a second deviation result of the i-th image frame in the second deviation result sequence, a first weight corresponding to the first recognition result of the i-th image frame and a second weight corresponding to the second recognition result corresponding to the i-th image frame;

[0137] A comprehensive recognition result of the i-th image frame is determined based on the first weight of the i-th image frame, the second weight of the i-th image frame, the first recognition result of the i-th image frame, and the second recognition result of the i-th image frame.

[0138] Specifically, the first weight of the image recognition model is determined according to the first deviation result, and the second weight of the text recognition model is determined according to the second deviation result. Generally speaking, the model with a larger deviation result has a smaller weight. If the first deviation result is larger, it means that the image recognition model result is less accurate than the text recognition model result, and the first weight is less than the second weight; if the second deviation result is larger, it means that the image recognition model result is better than the text recognition model result. The first weight and the second weight are calculated as shown in formula (5) and formula (6), respectively:

[0139]

[0140] Among them, w V is the first weight, w T is the second weight.

[0141] According to the first recognition result corresponding to the image recognition model, the second recognition result corresponding to the text recognition model, and the first weight and the second weight, a comprehensive recognition result is determined, as shown in formula (7):

[0142]

[0143] Among them, SCORE i The comprehensive recognition results.

[0144] Figure 4 The process of determining the comprehensive recognition result of abnormal behavior in the video to be recognized is shown.

[0145] For example, assume that an abnormal video A has a total of 10 frames, and 5 frames (the 4th to the 8th frames) have abnormal behaviors.

[0146] a) Image recognition model detection:

[0147] By inputting abnormal video A into the image recognition model, the first recognition result of each frame of abnormal video A can be obtained:

[0148] b) Sequence value estimation model detection:

[0149] The abnormal score of each frame of abnormal video A Input the sequence value estimation model to obtain the first recognition result sequence of each frame of abnormal video A

[0150] c) Text recognition model detection:

[0151] By inputting abnormal video A into the text recognition model, the second recognition result of each frame of abnormal video A can be obtained:

[0152] d) Sequence value estimation model detection:

[0153] The abnormal score of each frame of abnormal video A Input the sequence value estimation model to obtain the second recognition result sequence of each frame of abnormal video A

[0154]

[0155] e) Deviation result calculation:

[0156] First deviation results based on image recognition model:

[0157]

[0158] Second deviation results based on text recognition model:

[0159]

[0160] First weight based on image recognition model:

[0161]

[0162] Second weight based on text recognition model:

[0163]

[0164] The comprehensive recognition results are:

[0165]

[0166] In some embodiments, the sequence value estimation model is obtained by training a sequence of sample recognition results output by an image recognition model through sample videos with abnormal behavior; the sequence value estimation model is used to predict the recognition results of subsequent frames based on the continuity of behavior through the recognition results of historical frames.

[0167] Specifically, when training the sequence value estimation model, firstly, the abnormal score value of the historical image frame of the sample video is input into the sequence value estimation model, and the estimated abnormal score value of the historical image frame of the sample video is output, and then the abnormal score value of the sample video is used as a label to guide the training of the sequence value estimation model. As the basic network, three layers of fully connected layers are connected as the final model output to form a sequence value estimation model. Among them, when training the sequence value estimation model, the loss function is shown in formula (8):

[0168]

[0169] Among them, L mse is the minimum square error, i is the serial number of each image frame, e i is the anomaly score value output by the image recognition model, The estimated anomaly score value output by the model for the sequence value estimation. As training progresses, e i As it gets closer to the true value, It will also get closer and closer to the true value, thus realizing the direct prediction of the anomaly score value of the current frame from the anomaly score value of the historical image frame.

[0170] In some embodiments, the image recognition model is a cascade model with generalized recognition capability obtained by training multiple single abnormal behavior recognition models; wherein any single abnormal behavior recognition model is obtained by supervised learning training for each sample with the same type of abnormal behavior.

[0171] Specifically, the training of the image recognition model first trains multiple single abnormal behavior recognition models, and then cascades the trained multiple single abnormal behavior recognition models, and uniformly trains the multiple single abnormal behavior recognition models to obtain a cascade model with generalized recognition capabilities, that is, an image recognition model. Among them, the single abnormal behavior recognition model is a model that has the ability to recognize a certain abnormal behavior. For example, there are three single abnormal behavior recognition models, namely abnormal behavior recognition model A, abnormal behavior recognition model B, and abnormal behavior recognition model C. Abnormal behavior recognition model A is a model that has the ability to recognize abnormal behavior in the presence of smoking scenes, abnormal behavior recognition model B is a model that has the ability to recognize abnormal behavior in the presence of fighting scenes, and abnormal behavior recognition model C is a model that has the ability to recognize abnormal behavior in the presence of drinking scenes. The training of the single abnormal behavior recognition model is supervised training.

[0172] In some embodiments, the image recognition model is trained by:

[0173] Input the same sample into each single abnormal behavior recognition model respectively to obtain a single recognition result output by each single abnormal behavior recognition model; determine the loss value of each single abnormal behavior recognition model according to the real recognition result of the sample video and the single recognition result of each single abnormal behavior recognition model;

[0174] The samples are input into the image recognition model, and the image recognition model is trained based on the loss value of each single abnormal behavior recognition model until the training termination condition is met.

[0175] Specifically, when training an image recognition model, such as Figure 5 and Figure 6 As shown, Figure 5First, a single abnormal behavior recognition model is trained. A single abnormal behavior recognition model can only recognize one abnormal behavior. For example, abnormal behavior recognition model A can only recognize abnormal behaviors such as smoking scenes. For other abnormal behaviors such as fighting or drinking, abnormal behavior recognition model A has no recognition capability. The loss value of each single abnormal behavior recognition model is determined based on the real recognition result of the sample video and the single recognition result of the single abnormal behavior recognition model.

[0176] After the training of the single abnormal behavior recognition model is completed, the samples are input into the image recognition model, i.e., the cascade model, and the image recognition model is trained according to the loss value of the single abnormal behavior recognition model. Figure 6 shown.

[0177] In some embodiments, the loss value of each single anomaly recognition model is determined by formula (9):

[0178]

[0179] Among them, loss is the loss value of the single anomaly recognition model, y i is the real recognition result of the sample video. is a single recognition result of a single anomaly recognition model, a is the confidence level, which is used to increase the distance between normal sample videos and abnormal sample videos, and a takes the value of (0,1).

[0180] In some embodiments, Figure 7 As shown, samples of multiple single abnormal behavior recognition models and samples of image recognition models are obtained in the following ways:

[0181] Step 701: Obtain a normal sample set and an abnormal sample set.

[0182] Specifically, sample videos without abnormal behaviors are taken as normal sample sets, and sample videos with abnormal behaviors are taken as abnormal sample sets.

[0183] Step 702: cluster the abnormal sample set to obtain abnormal subsets of multiple categories; each category corresponds to a single abnormal recognition model.

[0184] Specifically, the abnormal sample set is clustered by a clustering algorithm, and the abnormal sample set is divided into multiple abnormal subsets. If the abnormal sample set contains abnormal behaviors such as smoking, drinking, fighting, etc., the abnormal sample set is divided into a smoking abnormal subset, a drinking abnormal subset, and a fighting abnormal subset. Each abnormal subset corresponds to a single abnormal recognition model, such as the smoking abnormal subset corresponds to the smoking abnormal recognition model.

[0185] Step 703: construct a sample set for each single anomaly recognition model; wherein the normal sample is the first part of the normal subset randomly determined from the normal sample set, and the abnormal sample is the first part of the corresponding abnormal subset.

[0186] Specifically, when training a single anomaly recognition model, a sample set of the single anomaly recognition model is constructed, and the normal subset is randomly determined in the normal sample set. The normal subset is divided into two parts, and the first part is used as the sample set of the single anomaly recognition model. The abnormal subset is also divided into two parts, and the first part is also used as the sample set of the single anomaly recognition model. Therefore, the sample set of the single anomaly recognition model contains two parts: the first part of the normal subset and the first part of the abnormal subset.

[0187] Step 704, construct a sample set of the image recognition model; wherein the normal sample is the second part of each normal subset, and the abnormal sample is the second part of each abnormal subset; the second part in the normal subset is different from the first part in the normal subset, and the second part in the abnormal subset is different from the first part in the abnormal subset.

[0188] Specifically, when training the image recognition model, a sample set of the image recognition model is constructed. The second part of the normal subset and the second part of the abnormal subset are used as the sample set of the image recognition model. Similarly, the sample set of the image recognition model also includes two parts, and the sample set of the image recognition model does not overlap with the sample set of the single abnormal recognition model.

[0189] For example, obtain normal sample sets and abnormal sample sets, such as obtaining 100,000 normal sample sets and 1,000 abnormal sample sets of smoking, drinking, and fighting, where the number of normal sample sets is much larger than the number of abnormal sample sets. Cluster the 1,000 abnormal samples, such as using the K-means clustering method to divide them into three categories of abnormal subsets, namely, the smoking abnormal subset, the drinking abnormal subset, and the fighting abnormal subset. The smoking abnormal subset contains 500 samples, the drinking abnormal subset contains 300 samples, and the fighting abnormal subset contains 200 samples. Randomly select a certain number of normal samples from the 100,000 normal samples and add them to the abnormal subsets respectively, such as adding 30,000 normal subsets to the smoking abnormal subset, adding 30,000 normal subsets to the drinking abnormal subset, and adding 40,000 normal subsets to the fighting abnormal subset, then you can get 3 sample sets. These 3 data sets have different data distributions due to different data structures (both normal samples and abnormal samples are different). The three sample sets are further randomly divided into two parts. The first part of the normal subset and the first part of the abnormal subset are used as the basic training set B, and the second part of the normal subset and the second part of the abnormal subset are used as the query training set Q. Then each basic training set is B1, B2, B3, and the query set is Q1, Q2, Q3, as shown in Table 1:

[0190] Dataset Normal sample number Number of abnormal samples Sample set 1 (B1+Q1) 30000 500 Sample set 2 (B2+Q2) 30000 300 Sample set 3 (B3+Q3) 40000 200

[0191] Table 1

[0192] When training a single anomaly recognition model, the basic training sets B1, B2, and B3 are used. When training an image recognition model, the query set training sets Q1, Q2, and Q3 are used. Figure 5 and Figure 6 As shown in the figure, each single anomaly model only captures part of the anomaly, so this solution proposes a unified image recognition model capture, namely a cascade model, which fits various anomaly distributions through the cascade model, thereby reducing overfitting in the specified anomaly type.

[0193] Each single anomaly model belongs to the category of supervised learning, and each single anomaly model often focuses on detecting abnormal samples in the sample set. In actual use, it is necessary to consider abnormal samples that are not covered in the sample set. Therefore, this solution uses the query set training set to train the cascade model and update the weights, so that the entire image recognition model can effectively simulate abnormal samples that have not appeared in the previous training, thereby enhancing the generalization of the model.

[0194] For the cascade model, its network architecture is consistent with the single anomaly recognition model. The losses of multiple single anomaly recognition models are used as supervision input into the cascade model, and its loss function is the same as formula (9).

[0195] During the training process, the weights of the single anomaly recognition model use the trained weights and are not updated. Only the weights of the cascade model are trained. Since the single anomaly recognition model is trained with the basic training set, and the cascade model is trained with the query training set. The role of the cascade model is to learn the distribution of cross-domain data through the single anomaly recognition model, so that the anomaly detection model can detect anomalies in cross-domain data. By building Figure 6 The model architecture can jointly train multiple single anomaly recognition models into a cascade model, thereby realizing video anomaly detection with a single model.

[0196] In the process of training the cascade model, assuming that n rounds of training are required, for each round of training, a single anomaly recognition model is first trained on the basic training set B. After the training is completed, the cascade model is trained on the query training set Q. Then, the weights of the cascade model need to be updated to the single anomaly recognition model to start a new round of training.

[0197] The cascade model is trained by using the loss of multiple single anomaly recognition models, so that the cascade model can improve the performance of multiple models on cross-domain data, making the performance optimal on the query training set Q. And by updating the weights to the single anomaly recognition model and optimizing on the basic training set B, the cascade model can not only fit the basic training set B, but also be optimal on the query training set Q, thus obtaining a cascade model with strong generalization ability.

[0198] like Figure 6 As shown in the figure, when training the cascade model, the comprehensive value of the loss of the three single anomaly recognition models is calculated, and the cascade model is adjusted with the comprehensive value of the loss until the cascade model meets the training requirements. The comprehensive value of the loss can be the average value of the loss of the three single anomaly recognition models, or it can be the sum of the losses of the three single anomaly recognition models.

[0199] In some embodiments, the text recognition model is a large language model; determining, through the text recognition model, whether there is a second recognition result of abnormal behavior in the image frame corresponding to the text description sequence, including: inputting the text description sequence and the abnormal behavior prompt word into the large language model, thereby determining whether there is a second recognition result of abnormal behavior in the image frame corresponding to the text description sequence.

[0200] Specifically, the embodiment of the present application further determines whether there is abnormal behavior in the video to be identified through a text recognition model. The text recognition model uses a large language model, inputs the text description sequence and the abnormal behavior prompt word into the large language model, and obtains whether there is abnormal behavior in the image frame corresponding to the text description sequence. The abnormal behavior prompt word is used to prompt the large language model to obtain the abnormal behavior score value.

[0201] In some embodiments, the text description sequence and the abnormal behavior prompt word are input into the large language model to determine whether there is a second recognition result of abnormal behavior in the image frame corresponding to the text description sequence, such as Figure 8 As shown, the following steps are included:

[0202] Step 801: Determine a text description subsequence of the i-th image frame from a text description sequence.

[0203] Step 802: Input the text description subsequence and scene confirmation prompt words of the i-th image frame into the large language model to determine the scene text description of the i-th image frame.

[0204] Step 803: Input the scene text description and abnormal behavior prompt words of the i-th image frame into the large language model to obtain a second recognition result of whether abnormal behavior exists in the i-th image frame.

[0205] Specifically, for each frame in the sample video, a large language model is used to generate a text description to obtain a text description subsequence, such as setting C = [C1, C2, ..., C N Since the text description of a single frame is not sufficient to describe the entire scene, the description of a single frame image C = [C1, C2, ..., C N ], for the large model, design prompt words, assuming C = [C1, C2, C3, C4, C5], each frame is determined by the previous and next frames to determine the scene text description, such as the scene description of the third needle C3 can be composed of C2 and C4. The designed prompt words can be: Based on the descriptions of the following 2nd, 3rd, and 4th frames, summarize what happened in the scene. And input the descriptions of C2, C3, and C4, and you can get the summarized scene description. And so on, each frame is determined by the previous and next frames, so you can get the scene text description of each frame.

[0206] After obtaining the scene text description corresponding to each frame, further design abnormal behavior prompt words to obtain the abnormality score, where the designed abnormal behavior prompt words must contain the abnormal type to be detected. If you need to detect a smoking scene, the designed prompt words are: Based on the following scene description, use the output value of 0-1 to predict the probability that the scene is a smoking scene. In this way, the second recognition result of whether there is abnormal behavior in the i-th image frame can be directly obtained.

[0207] In some embodiments, a text description of each image frame in the video to be identified is determined, such as Fig. 9 As shown, the following steps are included:

[0208] Step 901: Determine the initial text description of each image frame in the video to be identified;

[0209] Step 902: determining image features of the i-th image frame, and determining text features of the i-th image frame from an initial text description of the i-th image frame;

[0210] Step 903: Determine, from the text features of the i-th image frame, text features that meet similarity requirements with the image features of the i-th image frame as the text description of the i-th image frame.

[0211] Specifically, for each image frame in the video to be identified, the BLIP2 model is used to generate an initial text description C = [C1, C2, ..., C N ]. Since BLIP2 has a certain error in describing a single image, it may produce different descriptions for images with similar content. Therefore, the feature similarity between the text encoder and the image encoder can be used for text filtering. Assume that a video has 10 frames of images. The 10 frames of images are passed through the image encoder to obtain the image features F1 = [F1, F2, ..., F 10], and pass the corresponding text description through the text encoder to obtain the text feature T i =[T1,T2,……,T 10 ], F1 and T i A cosine similarity comparison is performed, and the description corresponding to the most similar text feature is used as the text description of the frame, thereby obtaining the filtered text feature.

[0212] Based on the same technical concept, the embodiment of the present application provides a device for identifying abnormal behavior in a video, such as Fig.10 As shown, the device 1000 includes:

[0213] The recognition result module 1001 is used to determine, through an image recognition model, whether each image frame in the video to be recognized has a first recognition result of abnormal behavior;

[0214] The determination module 1002 is further configured to determine a text description of each image frame in the video to be identified, thereby obtaining a text description sequence of the video to be identified;

[0215] The recognition result module 1001 is further used to determine, through a text recognition model, a second recognition result that an abnormal behavior exists in the image frame corresponding to the text description sequence;

[0216] The comprehensive recognition module 1003 is used to determine the comprehensive recognition result of the i-th image frame based on the first recognition result corresponding to the i-th image frame and the second recognition result corresponding to the i-th image frame; the i-th image frame is any image frame in the video to be recognized;

[0217] The abnormal behavior module 1004 is used to determine that abnormal behavior exists in the video to be identified when the number of image frames whose comprehensive identification results meet the first set condition meets the second set condition.

[0218] Optionally, the comprehensive identification module 1003 is further used for:

[0219] The first recognition result sequence and the second recognition result sequence are respectively input into a sequence value estimation model representing the continuity of the behavior to obtain a first sequence estimation result and a second sequence estimation result; the first recognition result sequence is obtained in sequence based on the first recognition result of each image frame in the video to be recognized; the second recognition result sequence is obtained in sequence based on the second recognition result of each image frame in the video to be recognized;

[0220] Obtaining a first deviation result sequence according to the first recognition result sequence and the first sequence estimation result;

[0221] Obtaining a second deviation result sequence according to the second recognition result sequence and the second sequence estimation result;

[0222] Determining a comprehensive recognition result of the i-th image frame based on the first recognition result of the i-th image frame and the second recognition result of the i-th image frame includes:

[0223] Determine, based on a first deviation result of the i-th image frame in the first deviation result sequence and a second deviation result of the i-th image frame in the second deviation result sequence, a first weight corresponding to the first recognition result of the i-th image frame and a second weight corresponding to the second recognition result corresponding to the i-th image frame;

[0224] A comprehensive recognition result of the i-th image frame is determined based on the first weight of the i-th image frame, the second weight of the i-th image frame, the first recognition result of the i-th image frame, and the second recognition result of the i-th image frame.

[0225] Optionally, the sequence value estimation model is obtained by training a sequence of sample recognition results output by the image recognition model through sample videos with abnormal behavior; the sequence value estimation model is used to predict the recognition results of subsequent frames based on the continuity of behavior through the recognition results of historical frames.

[0226] Optionally, the image recognition model is a cascade model with generalized recognition capability obtained by training multiple single abnormal behavior recognition models; wherein any single abnormal behavior recognition model is obtained by performing supervised learning training on samples with the same type of abnormal behavior.

[0227] Optionally, the recognition result module 1001 is specifically used for:

[0228] Input the same sample into each single abnormal behavior recognition model respectively to obtain a single recognition result output by each single abnormal behavior recognition model; determine the loss value of each single abnormal behavior recognition model according to the real recognition result of the sample video and the single recognition result of each single abnormal behavior recognition model;

[0229] The sample is input into the image recognition model, and the image recognition model is trained based on the loss value of each single abnormal behavior recognition model until a training termination condition is met.

[0230] Optionally, the loss value of each single anomaly recognition model is determined by the following formula:

[0231]

[0232] Optionally, the recognition result module 1001 is specifically used for:

[0233] Obtain a normal sample set and an abnormal sample set;

[0234] Clustering the abnormal sample set to obtain abnormal subsets of multiple categories; each category corresponds to a single abnormal recognition model;

[0235] Constructing a sample set for each single anomaly recognition model; wherein the normal samples are the first part of the normal subset randomly determined from the normal sample set, and the abnormal samples are the first part of the corresponding abnormal subset;

[0236] Construct a sample set of the image recognition model; wherein the normal sample is the second part of each normal subset, and the abnormal sample is the second part of each abnormal subset; the second part in the normal subset is different from the first part in the normal subset, and the second part in the abnormal subset is different from the first part in the abnormal subset.

[0237] Optionally, the recognition result module 1001 is specifically used for:

[0238] Determining, by means of a text recognition model, whether there is a second recognition result of abnormal behavior in the image frame corresponding to the text description sequence, including:

[0239] The text description sequence and the abnormal behavior prompt word are input into the large language model to determine whether a second recognition result of abnormal behavior exists in the image frame corresponding to the text description sequence.

[0240] Optionally, the recognition result module 1001 is specifically used for:

[0241] Determine a text description subsequence of the i-th image frame from the text description sequence;

[0242] Inputting the text description subsequence and the scene confirmation prompt word of the i-th image frame into the large language model to determine the scene text description of the i-th image frame;

[0243] The scene text description and abnormal behavior prompt words of the i-th image frame are input into the large language model to obtain a second recognition result of whether there is abnormal behavior in the i-th image frame.

[0244] Optionally, the recognition result module 1001 is specifically used for:

[0245] Determining an initial text description of each image frame in the video to be identified;

[0246] Determining image features of the i-th image frame, and determining text features of the i-th image frame from an initial text description of the i-th image frame;

[0247] From the text features of the i-th image frame, text features that meet similarity requirements with the image features of the i-th image frame are determined as text descriptions of the i-th image frame.

[0248] Based on the same technical concept, the embodiment of the present application provides a computer device, which may be a terminal or a server, such as Fig.11 As shown, it includes at least one processor 1101 and a memory 1102 connected to the at least one processor. The specific connection medium between the processor 1101 and the memory 1102 is not limited in the embodiment of the present application. Fig.11 For example, the processor 1101 and the memory 1102 are connected via a bus. The bus can be divided into an address bus, a data bus, a control bus, and the like.

[0249] In an embodiment of the present application, the memory 1102 stores instructions that can be executed by at least one processor 1101. The at least one processor 1101 can execute the steps included in the above-mentioned method for identifying abnormal behavior in a video by executing the instructions stored in the memory 1102.

[0250] Among them, the processor 1101 is the control center of the computer device, and can use various interfaces and lines to connect various parts of the computer device, by running or executing instructions stored in the memory 1102 and calling data stored in the memory 1102. Optionally, the processor 1101 may include one or more processing units, and the processor 1101 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 1101. In some embodiments, the processor 1101 and the memory 1102 may be implemented on the same chip, and in some embodiments, they may also be implemented separately on independent chips.

[0251] Processor 1101 can be a general-purpose processor, such as a central processing unit (CPU), a digital signal processor, an application-specific integrated circuit (ASIC), a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, and can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware processor for execution, or can be executed by a combination of hardware and software modules in the processor.

[0252] Memory 1102, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs and modules. Memory 1102 may include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory, random access memory (Random Access Memory, RAM), static random access memory (Static Random Access Memory, SRAM), programmable read-only memory (Programmable Read Only Memory, PROM), read-only memory (Read Only Memory, ROM), electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), magnetic memory, disk, optical disk, etc. Memory 1102 is any other medium that can be used to carry or store a desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. The memory 1102 in the embodiment of the present application can also be a circuit or any other device that can realize a storage function, for storing program instructions and / or data.

[0253] Based on the same inventive concept, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program that can be executed by a computer device. When the program runs on the computer device, the computer device executes the steps of the above-mentioned method for identifying abnormal behavior in a video.

[0254] Based on the same inventive concept, an embodiment of the present application provides a computer program product, characterized in that the computer program product includes a computer program stored on a computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer device, the computer device executes the steps of the above-mentioned method for identifying abnormal behavior in a video.

[0255] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0256] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0257] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0258] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0259] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for identifying abnormal behavior in a video, characterized in that: include: Determine, by means of an image recognition model, a first recognition result of whether each image frame in the video to be recognized has abnormal behavior; Determining a text description of each image frame in the to-be-recognized video, thereby obtaining a text description sequence of the to-be-recognized video; Determining, by a text recognition model, a second recognition result that an abnormal behavior exists in the image frame corresponding to the text description sequence; Determine a comprehensive recognition result of the i-th image frame based on a first recognition result corresponding to the i-th image frame and a second recognition result corresponding to the i-th image frame; The i-th image frame is any image frame in the video to be identified; When the number of image frames whose comprehensive recognition results satisfy the first set condition meets the second set condition, it is determined that abnormal behavior exists in the video to be recognized.

2. The method according to claim 1, characterized in that Before determining the comprehensive recognition result of the i-th image frame, the method further includes: The first recognition result sequence and the second recognition result sequence are respectively input into a sequence value estimation model representing the continuity of the behavior to obtain a first sequence estimation result and a second sequence estimation result; the first recognition result sequence is obtained in sequence based on the first recognition result of each image frame in the video to be recognized; the second recognition result sequence is obtained in sequence based on the second recognition result of each image frame in the video to be recognized; Obtaining a first deviation result sequence according to the first recognition result sequence and the first sequence estimation result; Obtaining a second deviation result sequence according to the second recognition result sequence and the second sequence estimation result; Determining a comprehensive recognition result of the i-th image frame based on the first recognition result of the i-th image frame and the second recognition result of the i-th image frame includes: Determine, based on a first deviation result of the i-th image frame in the first deviation result sequence and a second deviation result of the i-th image frame in the second deviation result sequence, a first weight corresponding to the first recognition result of the i-th image frame and a second weight corresponding to the second recognition result corresponding to the i-th image frame; A comprehensive recognition result of the i-th image frame is determined based on the first weight of the i-th image frame, the second weight of the i-th image frame, the first recognition result of the i-th image frame, and the second recognition result of the i-th image frame.

3. The method according to claim 2, characterized in that The sequence value estimation model is obtained by training a sample video with abnormal behavior using a sequence of sample recognition results output by the image recognition model; the sequence value estimation model is used to predict the recognition results of subsequent frames based on the continuity of the behavior using the recognition results of historical frames.

4. The method according to claim 1, characterized in that The image recognition model is a cascade model with generalized recognition capability obtained by training multiple single abnormal behavior recognition models; wherein any single abnormal behavior recognition model is obtained by supervised learning training for each sample with the same type of abnormal behavior.

5. The method according to claim 4, characterized in that The image recognition model is trained in the following manner, including: Input the same sample into each single abnormal behavior recognition model respectively to obtain a single recognition result output by each single abnormal behavior recognition model; determine the loss value of each single abnormal behavior recognition model according to the real recognition result of the sample video and the single recognition result of each single abnormal behavior recognition model; The sample is input into the image recognition model, and the image recognition model is trained based on the loss value of each single abnormal behavior recognition model until a training termination condition is met.

6. The method according to claim 5, characterized in that The loss value of each single anomaly recognition model is determined by the following formula:

7. The method according to claim 4, characterized in that The samples of the plurality of single abnormal behavior recognition models and the samples of the image recognition model are obtained respectively in the following manners: Obtain a normal sample set and an abnormal sample set; Clustering the abnormal sample set to obtain abnormal subsets of multiple categories; each category corresponds to a single abnormal recognition model; Constructing a sample set for each single anomaly recognition model; wherein the normal samples are the first part of the normal subset randomly determined from the normal sample set, and the abnormal samples are the first part of the corresponding abnormal subset; Construct a sample set of the image recognition model; wherein the normal sample is the second part of each normal subset, and the abnormal sample is the second part of each abnormal subset; the second part in the normal subset is different from the first part in the normal subset, and the second part in the abnormal subset is different from the first part in the abnormal subset.

8. The method according to any one of claims 1 to 3, characterized in that The text recognition model is a large language model; Determining, by means of a text recognition model, whether there is a second recognition result of abnormal behavior in the image frame corresponding to the text description sequence, including: The text description sequence and the abnormal behavior prompt word are input into the large language model to determine whether a second recognition result of abnormal behavior exists in the image frame corresponding to the text description sequence.

9. The method according to claim 8, characterized in that The step of inputting the text description sequence and the abnormal behavior prompt word into the large language model to determine whether there is a second recognition result of abnormal behavior in the image frame corresponding to the text description sequence includes: Determine a text description subsequence of the i-th image frame from the text description sequence; Inputting the text description subsequence and the scene confirmation prompt word of the i-th image frame into the large language model to determine the scene text description of the i-th image frame; The scene text description and abnormal behavior prompt words of the i-th image frame are input into the large language model to obtain a second recognition result of whether there is abnormal behavior in the i-th image frame.

10. The method according to claim 8, characterized in that The step of determining a text description of each image frame in the video to be identified includes: Determining an initial text description of each image frame in the video to be identified; Determining image features of the i-th image frame, and determining text features of the i-th image frame from an initial text description of the i-th image frame; From the text features of the i-th image frame, text features that meet similarity requirements with the image features of the i-th image frame are determined as text descriptions of the i-th image frame.

11. A code change analysis device, characterized in that: include: A recognition result module, used to determine, through an image recognition model, a first recognition result of whether each image frame in the video to be recognized has abnormal behavior; The determination module is further used to determine the text description of each image frame in the video to be identified, so as to obtain a text description sequence of the video to be identified; The recognition result module is further used to determine, through a text recognition model, a second recognition result that an abnormal behavior exists in the image frame corresponding to the text description sequence; A comprehensive recognition module, used to determine a comprehensive recognition result of the ith image frame based on a first recognition result corresponding to the ith image frame and a second recognition result corresponding to the ith image frame; The i-th image frame is any image frame in the video to be identified; The abnormal behavior module is used to determine that abnormal behavior exists in the video to be identified when the number of image frames whose comprehensive identification results meet the first set condition meets the second set condition.

12. A computer device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the steps of the method according to any one of claims 1 to 10 are implemented when the processor executes the program.

13. A computer-readable storage medium, characterized in that: It stores a computer program executable by a computer device, and when the program is run on the computer device, the computer device executes the steps of any method described in claims 1 to 10.

Images