Method, device, equipment and storage medium for identifying abnormal behavior in video

Abstract

An embodiment of the present application provides a method, apparatus, device and storage medium for identifying abnormal behavior in a video, relating to the field of artificial intelligence technology. The method comprises: determining, through an image recognition model, a first recognition result of whether each image frame in a video to be identified has abnormal behavior; determining a text description of each image frame in the video to be identified, thereby obtaining a text description sequence of the video to be identified; determining, through a text recognition model, a second recognition result of whether abnormal behavior exists in the image frames corresponding to the text description sequence; determining a comprehensive recognition result of the i-th image frame based on the first recognition result corresponding to the i-th image frame and the second recognition result corresponding to the i-th image frame; the i-th image frame is any image frame in the video to be identified; and determining that abnormal behavior exists in the video to be identified when the number of image frames whose comprehensive recognition results satisfy the first set condition meets the second set condition.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device, and storage medium for identifying abnormal behavior in a video. Background Art

[0002] Currently, the main purpose of monitoring some scenes in daily life is to observe whether there are any abnormal phenomena in the monitored scene in real time so that emergency measures can be taken quickly. When determining whether a video contains an abnormal event or target object, a small model is usually used to classify a single-source video.

[0003] However, due to uneven data distribution, videos from different sources often have different characteristics. Therefore, it is difficult to align and train a small model using only a small number of labeled samples. It is impossible to use the same model to identify abnormal events in different scenarios, and therefore it is impossible to apply the same model to different scenarios. Summary of the Invention

[0004] Embodiments of the present application provide a method, apparatus, device, and storage medium for identifying abnormal behavior in videos, which are used to identify abnormal behavior in videos in different scenarios.

[0005] In a first aspect, embodiments of the present application provide a method for identifying abnormal behavior in a video, comprising:

[0006] Determine, by an image recognition model, a first recognition result of whether there is abnormal behavior in each image frame of the video to be recognized;

[0007] Determining a text description of each image frame in the video to be identified, thereby obtaining a text description sequence of the video to be identified;

[0008] Determining, by a text recognition model, a second recognition result that an abnormal behavior exists in the image frame corresponding to the text description sequence;

[0009] Determine a comprehensive recognition result of the i-th image frame based on the first recognition result corresponding to the i-th image frame and the second recognition result corresponding to the i-th image frame; the i-th image frame is any image frame in the video to be recognized;

[0010] When the number of image frames whose comprehensive recognition results satisfy the first set condition meets the second set condition, it is determined that abnormal behavior exists in the video to be recognized.

[0011] In the embodiments of the present application, the presence of abnormal behavior in the video to be identified is identified using an image recognition model and a text recognition model, thereby increasing the reliability of abnormal behavior identification and avoiding the problem of inaccurate recognition results for the video to be identified due to weak generalization capabilities or poor training results of the image recognition model or the text recognition model. Based on the first recognition result of the image recognition model and the second recognition result of the text recognition model, a comprehensive recognition result of the video to be identified is determined. The comprehensive recognition result includes the joint recognition results of the image recognition model and the text recognition model, thereby achieving higher accuracy in identifying abnormal behavior in the video to be identified.

[0012] Optionally, before determining the comprehensive recognition result of the i-th image frame, the method further includes:

[0013] Inputting the first recognition result sequence and the second recognition result sequence into a sequence value estimation model representing behavior continuity, respectively, to obtain a first sequence estimation result and a second sequence estimation result; the first recognition result sequence is obtained in sequence based on the first recognition result of each image frame in the video to be recognized; and the second recognition result sequence is obtained in sequence based on the second recognition result of each image frame in the video to be recognized;

[0014] Obtaining a first deviation result sequence according to the first recognition result sequence and the first sequence estimation result;

[0015] Obtaining a second deviation result sequence according to the second recognition result sequence and the second sequence estimation result;

[0016] Determining a comprehensive recognition result of the i-th image frame based on the first recognition result of the i-th image frame and the second recognition result of the i-th image frame includes:

[0017] Determining, based on a first deviation result of the i-th image frame in the first deviation result sequence and a second deviation result of the i-th image frame in the second deviation result sequence, a first weight corresponding to the first recognition result of the i-th image frame and a second weight corresponding to the second recognition result corresponding to the i-th image frame;

[0018] A comprehensive recognition result of the i-th image frame is determined based on the first weight of the i-th image frame, the second weight of the i-th image frame, the first recognition result of the i-th image frame, and the second recognition result of the i-th image frame.

[0019] In an embodiment of the present application, the recognition result is input into a sequence value estimation model to obtain a sequence estimation result, which improves the continuity of the recognition result compared to the recognition result sequence; based on the difference between the recognition result sequence and the sequence estimation result, the better model of the image recognition model and the text recognition model is determined, thereby giving the better model a greater weight, thereby improving the accuracy of the final comprehensive recognition result.

[0020] Optionally, the sequence value estimation model is obtained by training a sequence of sample recognition results output by the image recognition model through sample videos containing abnormal behavior; the sequence value estimation model is used to predict the recognition results of subsequent frames based on the continuity of behavior through the recognition results of historical frames.

[0021] In an embodiment of the present application, the output of the image recognition model is used as a sequence of sample recognition results of the sequence value estimation model to train the sequence value estimation model, so that the sequence value estimation model has better recognition results. The recognition results of subsequent frames are predicted by the recognition results of historical frames, thereby enhancing the consistency of the recognition results.

[0022] Optionally, the image recognition model is a cascade model with generalized recognition capability obtained by training multiple single abnormal behavior recognition models; wherein any single abnormal behavior recognition model is obtained by supervised learning training for each sample with the same type of abnormal behavior.

[0023] In the embodiment of the present application, the image recognition model is obtained by training multiple single abnormal behavior recognition models, so that the image recognition model aggregates the recognition results of multiple single abnormal behavior recognition models, thereby improving the generalization ability of the image recognition model.

[0024] Optionally, the image recognition model is trained in the following manner, including:

[0025] Input the same sample into each single abnormal behavior recognition model to obtain a single recognition result output by each single abnormal behavior recognition model; determine the loss value of each single abnormal behavior recognition model based on the actual recognition result of the sample video and the single recognition result of each single abnormal behavior recognition model;

[0026] The sample is input into the image recognition model, and the image recognition model is trained based on the loss value of each single abnormal behavior recognition model until a training termination condition is met.

[0027] In an embodiment of the present application, a single abnormal behavior recognition model is trained using the same sample to obtain a single recognition result, and then the sample is input into an image recognition model to train the image recognition model. The image recognition model is trained using the loss value of the single abnormal behavior recognition model, so that the image recognition model covers the recognition capabilities of the single abnormal behavior recognition model.

[0028] Optionally, the loss value of each single anomaly recognition model is determined by the following formula:

[0029]

[0030] Optionally, the samples of the multiple single abnormal behavior recognition models and the samples of the image recognition model are respectively obtained by:

[0031] Obtain normal sample sets and abnormal sample sets;

[0032] Clustering the abnormal sample set to obtain abnormal subsets of multiple categories; each category corresponds to a single abnormality recognition model;

[0033] Constructing a sample set for each single anomaly recognition model; wherein the normal samples are the first part of the normal subset randomly determined from the normal sample set, and the abnormal samples are the first part of the corresponding abnormal subset;

[0034] Construct a sample set of the image recognition model; wherein the normal samples are the second part of each normal subset, and the abnormal samples are the second part of each abnormal subset; the second part in the normal subset is different from the first part in the normal subset, and the second part in the abnormal subset is different from the first part in the abnormal subset.

[0035] In the embodiments of the present application, when constructing samples, the abnormal samples are first clustered to obtain multiple abnormal samples of different categories. Normal samples are then randomly added to the abnormal samples of each category to obtain a sample of normal and abnormal samples. A portion of the normal and abnormal samples is then used as the sample set for the single abnormality recognition model, and another portion of the normal and abnormal samples is used as the sample set for the image recognition model. By constructing different sample sets to train the single abnormality recognition model and the image recognition model respectively, the generalization ability of the model is improved.

[0036] Optionally, the text recognition model is a large language model;

[0037] Determining, by a text recognition model, whether a second recognition result of abnormal behavior exists in the image frame corresponding to the text description sequence, includes:

[0038] The text description sequence and the abnormal behavior prompt word are input into the large language model to determine whether a second recognition result of abnormal behavior exists in the image frame corresponding to the text description sequence.

[0039] In an embodiment of the present application, by using a large language model as a text recognition model, the accuracy of text recognition can be improved. By inputting a text description sequence and abnormal behavior prompt words into the large language model, a second recognition result of whether an image frame has abnormal behavior can be more accurately determined.

[0040] Optionally, the step of inputting the text description sequence and the abnormal behavior prompt word into the large language model to determine whether a second recognition result of abnormal behavior exists in the image frame corresponding to the text description sequence includes:

[0041] Determining a text description subsequence of the i-th image frame from the text description sequence;

[0042] Inputting the text description subsequence and scene confirmation prompt words of the i-th image frame into the large language model to determine the scene text description of the i-th image frame;

[0043] The scene text description and abnormal behavior prompt words of the i-th image frame are input into the large language model to obtain a second recognition result of whether there is abnormal behavior in the i-th image frame.

[0044] In an embodiment of the present application, by determining a subsequence in a text description sequence, confirming a prompt word according to the scene corresponding to the subsequence, determining the scene text description of the image frame through a large language model, and then inputting the scene text description and abnormal behavior prompt word into the large language model, the recognition result of abnormal behavior is made more accurate.

[0045] Optionally, determining a text description of each image frame in the to-be-identified video includes:

[0046] Determining an initial text description of each image frame in the video to be identified;

[0047] determining image features of the i-th image frame, and determining text features of the i-th image frame from an initial text description of the i-th image frame;

[0048] From the text features of the i-th image frame, text features that meet similarity requirements with the image features of the i-th image frame are determined as the text description of the i-th image frame.

[0049] In an embodiment of the present application, by first determining an initial text description, then determining text features based on the initial text description, and by using text features that meet similarity requirements as text descriptions of image frames, the problem of possible errors in the text description of a single image and the generation of different descriptions for images with similar content is avoided.

[0050] In a second aspect, an embodiment of the present application provides a device for identifying abnormal behavior in a video, comprising:

[0051] A recognition result module is used to determine, through an image recognition model, a first recognition result of whether each image frame in the video to be recognized has abnormal behavior;

[0052] The determination module is further configured to determine a text description of each image frame in the video to be identified, thereby obtaining a text description sequence of the video to be identified;

[0053] The recognition result module is further configured to determine, through a text recognition model, a second recognition result indicating that abnormal behavior exists in the image frame corresponding to the text description sequence;

[0054] a comprehensive recognition module, configured to determine a comprehensive recognition result of an i-th image frame based on a first recognition result corresponding to the i-th image frame and a second recognition result corresponding to the i-th image frame; the i-th image frame being any image frame in the video to be recognized;

[0055] The abnormal behavior module is used to determine that abnormal behavior exists in the video to be identified when the number of image frames whose comprehensive recognition results meet the first set condition meets the second set condition.

[0056] Optionally, the comprehensive identification module is further configured to:

[0057] Inputting the first recognition result sequence and the second recognition result sequence into a sequence value estimation model representing behavior continuity, respectively, to obtain a first sequence estimation result and a second sequence estimation result; the first recognition result sequence is obtained in sequence based on the first recognition result of each image frame in the video to be recognized; and the second recognition result sequence is obtained in sequence based on the second recognition result of each image frame in the video to be recognized;

[0058] Obtaining a first deviation result sequence according to the first recognition result sequence and the first sequence estimation result;

[0059] Obtaining a second deviation result sequence according to the second recognition result sequence and the second sequence estimation result;

[0060] Determining a comprehensive recognition result of the i-th image frame based on the first recognition result of the i-th image frame and the second recognition result of the i-th image frame includes:

[0061] Determining, based on a first deviation result of the i-th image frame in the first deviation result sequence and a second deviation result of the i-th image frame in the second deviation result sequence, a first weight corresponding to the first recognition result of the i-th image frame and a second weight corresponding to the second recognition result corresponding to the i-th image frame;

[0062] A comprehensive recognition result of the i-th image frame is determined based on the first weight of the i-th image frame, the second weight of the i-th image frame, the first recognition result of the i-th image frame, and the second recognition result of the i-th image frame.

[0063] Optionally, the sequence value estimation model is obtained by training a sequence of sample recognition results output by the image recognition model through sample videos containing abnormal behavior; the sequence value estimation model is used to predict the recognition results of subsequent frames based on the continuity of behavior through the recognition results of historical frames.

[0064] Optionally, the image recognition model is a cascade model with generalized recognition capability obtained by training multiple single abnormal behavior recognition models; wherein any single abnormal behavior recognition model is obtained by supervised learning training for each sample with the same type of abnormal behavior.

[0065] Optionally, the recognition result module is specifically configured to:

[0066] Inputting the same sample into each single abnormal behavior recognition model to obtain a single recognition result output by each single abnormal behavior recognition model; determining a loss value of each single abnormal behavior recognition model based on the actual recognition result of the sample and the single recognition result of each single abnormal behavior recognition model;

[0067] The sample is input into the image recognition model, and the image recognition model is trained based on the loss value of each single abnormal behavior recognition model until a training termination condition is met.

[0068] Optionally, the loss value of each single anomaly recognition model is determined by the following formula:

[0069]

[0070] Optionally, the recognition result module is specifically configured to:

[0071] Obtain normal sample sets and abnormal sample sets;

[0072] Clustering the abnormal sample set to obtain abnormal subsets of multiple categories; each category corresponds to a single abnormality recognition model;

[0073] Constructing a sample set for each single anomaly recognition model; wherein the normal samples are the first part of the normal subset randomly determined from the normal sample set, and the abnormal samples are the first part of the corresponding abnormal subset;

[0074] Construct a sample set of the image recognition model; wherein the normal samples are the second part of each normal subset, and the abnormal samples are the second part of each abnormal subset; the second part in the normal subset is different from the first part in the normal subset, and the second part in the abnormal subset is different from the first part in the abnormal subset.

[0075] Optionally, the recognition result module is specifically configured to:

[0076] Determining, by a text recognition model, whether a second recognition result of abnormal behavior exists in the image frame corresponding to the text description sequence, includes:

[0077] The text description sequence and the abnormal behavior prompt word are input into the large language model to determine whether a second recognition result of abnormal behavior exists in the image frame corresponding to the text description sequence.

[0078] Optionally, the recognition result module is specifically configured to:

[0079] Determining a text description subsequence of the i-th image frame from the text description sequence;

[0080] Inputting the text description subsequence and scene confirmation prompt words of the i-th image frame into the large language model to determine the scene text description of the i-th image frame;

[0081] The scene text description and abnormal behavior prompt words of the i-th image frame are input into the large language model to obtain a second recognition result of whether there is abnormal behavior in the i-th image frame.

[0082] Optionally, the recognition result module is specifically configured to:

[0083] Determining an initial text description of each image frame in the video to be identified;

[0084] determining image features of the i-th image frame, and determining text features of the i-th image frame from an initial text description of the i-th image frame;

[0085] From the text features of the i-th image frame, text features that meet similarity requirements with the image features of the i-th image frame are determined as the text description of the i-th image frame.

[0086] In a third aspect, an embodiment of the present application provides a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any of the above-described methods when executing the program.

[0087] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium storing a computer program executable by a computer device, which enables the computer device to perform the steps of any of the above-mentioned methods when the program is run on the computer device.

[0088] In a fifth aspect, the present application provides a computer program product, which, when executed on a computer, enables the computer to execute the methods of the various embodiments of the first aspect above.

[0089] In the embodiments of the present application, the presence of abnormal behavior in the video to be identified is identified using an image recognition model and a text recognition model, thereby increasing the reliability of abnormal behavior identification and avoiding the problem of inaccurate recognition results for the video to be identified due to weak generalization capabilities or poor training results of the image recognition model or the text recognition model. Based on the first recognition result of the image recognition model and the second recognition result of the text recognition model, a comprehensive recognition result of the video to be identified is determined. The comprehensive recognition result includes the joint recognition results of the image recognition model and the text recognition model, thereby achieving higher accuracy in identifying abnormal behavior in the video to be identified. BRIEF DESCRIPTION OF THE DRAWINGS

[0090] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0091] Figure 1 A system architecture diagram provided for an embodiment of the present application;

[0092] Figure 2 A flowchart of a method for identifying abnormal behavior in a video provided in an embodiment of the present application;

[0093] Figure 3 A schematic diagram of a process for determining a comprehensive recognition result of an image frame provided in an embodiment of the present application (I);

[0094] Figure 4 A schematic diagram (II) of a process for determining a comprehensive recognition result of an image frame provided in an embodiment of the present application;

[0095] Figure 5 A schematic diagram of the structure of a training method for a single abnormal behavior recognition model provided in an embodiment of the present application;

[0096] Figure 6 A schematic diagram of the structure of a training method for an image recognition model provided in an embodiment of the present application;

[0097] Figure 7 A schematic diagram of a process for obtaining samples of multiple single abnormal behavior recognition models and samples of image recognition models provided in an embodiment of the present application;

[0098] Figure 8 A schematic diagram of a second recognition result flow of abnormal behavior in an image frame corresponding to a text description sequence provided in an embodiment of the present application;

[0099] Figure 9A schematic diagram of a process for determining a text description of each image frame in a video to be identified provided in an embodiment of the present application;

[0100] Figure 10 A schematic diagram of the structure of a device for identifying abnormal behavior in a video provided in an embodiment of the present application;

[0101] Figure 11 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0102] In order to make the purpose, technical solutions and beneficial effects of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0103] To facilitate understanding of this solution, the following describes its application scenarios.

[0104] Currently, monitoring and detecting abnormal scenes are typically achieved using visual anomaly detection algorithms. These algorithms often require collecting the data to be tested, classifying it into normal and abnormal samples, and then training a model. This allows the trained model to be used to detect anomalies in a specific scenario. For example, if one needs to detect smoking in a specific scene (smoking is considered an abnormal scene), corresponding abnormal smoking samples must be collected for model training and detection. However, a trained model cannot detect other abnormal scenes. For example, a model designed to detect smoking cannot detect alcohol consumption. Text, however, is inherently versatile. Recent research has explored weakly supervised video anomaly detection methods using text cues. These methods use both text and video features to train a deep network. Essentially, they augment visual features with text features to achieve anomaly detection. However, these methods remain single-scenario supervised learning methods and are unable to generalize to scenes outside the training set, failing to substantially improve the generalization capabilities of anomaly detection.

[0105] Anomaly detection models for different scenarios cannot achieve universal results. Models trained in one scenario often cannot be directly applied to other scenarios. Cross-domain video anomaly detection still faces several challenges and limitations: 1. Imbalanced dataset pairing: Anomalies that may be easily identified in one dataset may be difficult to correctly classify as anomalies in another. This can cause the model to favor one domain and ignore information from other domains, thus affecting its generalization ability. 2. Inconsistent feature extraction: When using features from different data sources, differences in data representation, metrics, or preprocessing can lead to inconsistent features. This can make it difficult for the model to learn meaningful connections and reduce its performance. 3. Lack of common information: If objects from different datasets do not share common semantic information as a foundation, cross-domain modeling may not fully utilize their inherent relationships. 4. Inefficient computation: Cross-domain video anomaly detection typically requires significant computing resources to train the model and perform inference. This can become a performance bottleneck, especially when large neural networks are involved. 5. Explainability: Current cross-domain video anomaly detection methods often lack explainability, meaning it is difficult to understand why the model made a specific prediction. This brings challenges to the deployment and application of the model. The following are the specific implementation steps of this application:

[0106] First, the present application collects multiple normal samples and abnormal samples, wherein the number of normal samples is much greater than the number of abnormal samples. The abnormal samples are divided into multiple categories by a clustering algorithm, and the normal samples are randomly added to the abnormal samples to obtain a training sample set distinguished by category. Each category of training sample set is then randomly divided into a basic training set and a query training set. Then, the basic training set of each category is input into the basic model, and the loss function is adjusted to obtain a trained basic model for each category. The query training sets of all categories are respectively input into the trained multiple basic models and a cascade model, and the cascade model is adjusted by the loss function of the basic model to obtain a trained cascade model. After the training sample set is input into the trained cascade model, the first predicted abnormal score value of the training sample is obtained, the abnormal score value is input into the score sequence model to obtain an estimated abnormal score value, and the score sequence model is adjusted by the first predicted abnormal score value and the estimated abnormal score value to obtain a trained score training model. Next, all the training sample sets are input into the large model, and the second predicted abnormal score value of the training sample set is determined by designing prompt words. Finally, the anomaly scores output by the cascade model and the large model are fed into the score sequence model. The difference between the model's own anomaly scores and the scores output by the score sequence model is used to determine the weights of each model, completing the final model training. When using the trained models, the video to be identified is fed into the cascade model and the large model, and the output anomaly scores are fed back into the score sequence model. The scores are then compared with the set threshold to determine whether the video to be identified is an anomaly.

[0107] See also Figure 1 , is a system architecture diagram provided in an embodiment of the present application, wherein the system architecture includes a terminal device 101 and a server 102.

[0108] The terminal device 101 is pre-installed with a business application for text matching, wherein the business application is a client application, a web application, a mini-program application, etc. The terminal device 101 can be a smart phone, a POS machine, a desktop notebook, a computer, etc., but is not limited thereto.

[0109] Server 102 is the backend server for business applications. Server 102 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0110] refer to Figure 2, a method for identifying abnormal behavior in a video provided in an embodiment of the present application, comprising the following steps:

[0111] Step 201: Determine, by using an image recognition model, whether each image frame in the video to be recognized has a first recognition result of abnormal behavior.

[0112] Specifically, the image recognition model is used to identify whether there is abnormal behavior in the image. Each image frame in the video to be identified is input into the image recognition model, and a first recognition result of the video to be identified is output. The first recognition result represents the possibility of abnormal behavior in the video to be identified obtained by the image recognition model.

[0113] The image recognition model can be a visual anomaly detection model, which is divided into supervised models, semi-supervised learning models and unsupervised models. The embodiment of the present application uses a supervised model to train the visual anomaly detection model, and can also use a semi-supervised learning model or an unsupervised model.

[0114] Step 202: Determine the text description of each image frame in the video to be identified, thereby obtaining a text description sequence of the video to be identified.

[0115] Specifically, the text description of each image frame in the video to be identified reflects what is happening in the scene where each frame of the image is located. Based on the text description of each image frame, a text description sequence of the video to be identified is obtained, so that what is happening in the scene where the video to be identified is located can be summarized.

[0116] Step 203: Determine, by using the text recognition model, a second recognition result indicating that abnormal behavior exists in the image frame corresponding to the text description sequence.

[0117] Specifically, by inputting the text description sequence of the video to be identified into the text recognition model, a second recognition result can be obtained, indicating that abnormal behavior exists in the image frame corresponding to the text description sequence. The second recognition result represents the possibility of abnormal behavior existing in the video to be identified through the text recognition model.

[0118] Step 204 : Determine a comprehensive recognition result of the i-th image frame based on the first recognition result corresponding to the i-th image frame and the second recognition result corresponding to the i-th image frame; the i-th image frame is any image frame in the video to be recognized.

[0119] Specifically, the first recognition result and the second recognition result are combined to obtain a comprehensive recognition result of each image frame, and the comprehensive recognition result is used to characterize the possibility of abnormal behavior in each image frame.

[0120] Step 205: When the number of image frames whose comprehensive recognition results satisfy the first set condition meets the second set condition, it is determined that abnormal behavior exists in the video to be recognized.

[0121] Specifically, the system determines whether the comprehensive recognition result of the image frame meets the first set condition. If not, the image frame is not processed further. If it does, the system then determines whether the number of image frames meets the second condition. If the number of image frames meets the second condition, the video to be recognized is considered to have abnormal behavior. If the number of image frames does not meet the second condition, the video to be recognized is considered to have no abnormal behavior.

[0122] In the embodiments of the present application, the first and second recognition results obtained by the image recognition model and text recognition model for the video to be recognized can be either the probability of abnormal behavior in the video to be recognized or a score indicating the presence of abnormal behavior in the video to be recognized. A higher score indicates a greater likelihood of abnormal behavior. This article uses the output of the image recognition model and text recognition model as an example to illustrate the score.

[0123] In some embodiments, before determining the comprehensive recognition result of the i-th image frame, Figure 3 As shown, the following steps are also included:

[0124] Step 301: Input the first recognition result sequence and the second recognition result sequence into a sequence value estimation model representing behavior continuity, respectively, to obtain a first sequence estimation result and a second sequence estimation result; the first recognition result sequence is obtained in sequence based on the first recognition result of each image frame in the video to be identified; the second recognition result sequence is obtained in sequence based on the second recognition result of each image frame in the video to be identified.

[0125] Specifically, a first recognition result sequence of the video to be recognized is obtained based on the first recognition result of each image frame. The first recognition result sequence is obtained by arranging the image frames of the video to be recognized in sequence. Assuming that there are 10 frames of the video to be recognized, the first recognition result sequence is: ,in is the recognition result of the first image frame, is the recognition result of the second image frame, ..., is the recognition result of the tenth image frame. A second recognition result sequence of the video to be recognized is obtained based on the second recognition result of each image frame. The second recognition result sequence is also obtained by arranging the image frames of the video to be recognized in order. The second recognition result sequence is: ,in, is the recognition result of the first image frame, is the recognition result of the second image frame, ..., is the recognition result of the tenth image frame. The first recognition result sequence and the second recognition result sequence are respectively input into the sequence value estimation model to obtain the first sequence estimation result and the second sequence estimation result, as shown in formula (1) and formula (2):

[0126]

[0127]

[0128] in, is the sequence value estimation model, k is the number of image frames of the video to be identified, is the first recognition result sequence, is the second recognition result sequence, is the first sequence estimation result, This is the second sequence estimation result.

[0129] The first sequence of estimation results and the second sequence of estimation results have stronger coherence than the first sequence of recognition results and the second sequence of recognition results.

[0130] Sequential value estimation models are a way to process and predict important events in time series with very long intervals and delays. Sequential value estimation models can be used to correct data with time delays.

[0131] Step 302: Obtain a first deviation result sequence according to the first recognition result sequence and the first sequence estimation result.

[0132] Step 303: Obtain a second deviation result sequence according to the second recognition result sequence and the second sequence estimation result.

[0133] Specifically, as shown in the figure, the difference between the first recognition result sequence and the first sequence estimation result is calculated to obtain the first deviation result, and the second recognition result sequence and the second sequence estimation result are calculated to obtain the second deviation result. The first deviation result and the second deviation result are shown in formula (3) and formula (4) respectively:

[0134]

[0135]

[0136] in, is the first deviation result, is the second deviation result, is the first sequence estimation result, is the first recognition result sequence, is the second sequence estimation result, is the second recognition result sequence.

[0137] Step 304, determining a comprehensive recognition result of the i-th image frame based on the first recognition result of the i-th image frame and the second recognition result of the i-th image frame, includes:

[0138] Determining, based on a first deviation result of the i-th image frame in the first deviation result sequence and a second deviation result of the i-th image frame in the second deviation result sequence, a first weight corresponding to the first recognition result of the i-th image frame and a second weight corresponding to the second recognition result corresponding to the i-th image frame;

[0139] A comprehensive recognition result of the i-th image frame is determined based on the first weight of the i-th image frame, the second weight of the i-th image frame, the first recognition result of the i-th image frame, and the second recognition result of the i-th image frame.

[0140] Specifically, the first weight of the image recognition model is determined based on the first deviation result, and the second weight of the text recognition model is determined based on the second deviation result. Generally speaking, the model with a larger deviation result has a smaller weight. If the first deviation result is larger, it means that the image recognition model result is less accurate than the text recognition model result, and the first weight is smaller than the second weight; if the second deviation result is larger, it means that the image recognition model result is better than the text recognition model result. The first weight and the second weight are calculated as shown in formula (5) and formula (6), respectively:

[0141]

[0142]

[0143] in, is the first weight, is the second weight.

[0144] According to the first recognition result corresponding to the image recognition model, the second recognition result corresponding to the text recognition model, and the first weight and the second weight, a comprehensive recognition result is determined, as shown in formula (7):

[0145]

[0146] in, The comprehensive recognition results.

[0147] Figure 4 The process of determining the comprehensive recognition result of abnormal behavior in the video to be recognized is shown.

[0148] For example, assume that an abnormal video A has a total of 10 frames, and 5 of the frames (frames 4 to 8) contain abnormal behavior.

[0149] a) Image recognition model detection:

[0150] By inputting abnormal video A into the image recognition model, we can obtain the first recognition result of each frame of abnormal video A: ;

[0151] b) Sequence value estimation model detection:

[0152] The abnormal score of each frame of abnormal video A Input the sequence value estimation model to obtain the first recognition result sequence of each frame image of abnormal video A = ;

[0153] c) Text recognition model detection:

[0154] By inputting abnormal video A into the text recognition model, we can obtain the second recognition result of each frame image of abnormal video A: ;

[0155] d) Sequence value estimation model detection:

[0156] The abnormal score of each frame of abnormal video A Input the sequence value estimation model to obtain the second recognition result sequence of each frame image of abnormal video A = ;

[0157] e) Deviation result calculation:

[0158] First deviation results based on image recognition model:

[0159] = ;

[0160] Second deviation results based on text recognition model:

[0161] = ;

[0162] First weight based on image recognition model:

[0163]

[0164] Second weight based on text recognition model:

[0165]

[0166] The comprehensive recognition results are:

[0167] =

[0168] In some embodiments, the sequence value estimation model is obtained by training a sequence of sample recognition results output by an image recognition model using sample videos of abnormal behavior; the sequence value estimation model is used to predict the recognition results of subsequent frames based on the continuity of behavior using the recognition results of historical frames.

[0169] Specifically, when training the sequence value estimation model, the anomaly score values ​​of the historical image frames of the sample video are first input into the sequence value estimation model, and the estimated anomaly score values ​​of the historical image frames of the sample video are output. Then, the anomaly score values ​​of the sample video are used as labels to guide the training of the sequence value estimation model. As the basic network, three fully connected layers are connected as the final model output to form the sequence value estimation model. Among them, when training the sequence value estimation model, the loss function is shown in Formula (8):

[0170]

[0171] in, is the minimum square error, is the sequence number of each image frame, is the anomaly score value output by the image recognition model, The estimated anomaly score value output by the model for the sequence value estimation. As training progresses, is getting closer and closer to the true value, then It will also get closer and closer to the true value, thus achieving the direct prediction of the anomaly score value of the current frame from the anomaly score value of the historical image frame.

[0172] In some embodiments, the image recognition model is a cascade model with generalized recognition capabilities obtained by training multiple single abnormal behavior recognition models; wherein any single abnormal behavior recognition model is obtained by supervised learning training for each sample with the same type of abnormal behavior.

[0173] Specifically, the image recognition model is trained by first training multiple single abnormal behavior recognition models. These trained models are then cascaded and trained uniformly to produce a cascaded model with generalized recognition capabilities, namely, the image recognition model. A single abnormal behavior recognition model is a model that recognizes a specific abnormal behavior. For example, there are three single abnormal behavior recognition models: abnormal behavior recognition model A, abnormal behavior recognition model B, and abnormal behavior recognition model C. Abnormal behavior recognition model A recognizes abnormal behavior such as smoking, abnormal behavior recognition model B recognizes abnormal behavior such as fighting, and abnormal behavior recognition model C recognizes abnormal behavior such as drinking. The training of the single abnormal behavior recognition models is supervised.

[0174] In some embodiments, the image recognition model is trained by:

[0175] Input the same sample into each single abnormal behavior recognition model to obtain a single recognition result output by each single abnormal behavior recognition model; determine the loss value of each single abnormal behavior recognition model based on the actual recognition result of the sample video and the single recognition result of each single abnormal behavior recognition model;

[0176] The samples are input into the image recognition model, and the image recognition model is trained based on the loss value of each single abnormal behavior recognition model until the training termination condition is met.

[0177] Specifically, when training an image recognition model, such as Figure 5 and Figure 6 As shown, Figure 5 First, a single abnormal behavior recognition model is trained. A single abnormal behavior recognition model can only identify one type of abnormal behavior. For example, abnormal behavior recognition model A can only identify abnormal behavior in smoking scenes. It cannot recognize other abnormal behaviors such as fighting or drinking. The loss value of each single abnormal behavior recognition model is determined based on the actual recognition results of the sample video and the single recognition result of the single abnormal behavior recognition model.

[0178] After the training of a single abnormal behavior recognition model is completed, the samples are input into the image recognition model, i.e., the cascade model, and the image recognition model is trained based on the loss value of the single abnormal behavior recognition model. Figure 6 shown.

[0179] In some embodiments, the loss value of each single anomaly recognition model is determined by formula (9):

[0180] in, is the loss value of the single anomaly recognition model, is the real recognition result of the sample video, is a single recognition result of a single anomaly recognition model, a is the confidence level, which is used to increase the distance between normal sample videos and abnormal sample videos. The value of a is (0, 1).

[0181] In some embodiments, as Figure 7 As shown in the figure, samples of multiple single abnormal behavior recognition models and samples of image recognition models are obtained in the following ways:

[0182] Step 701: Obtain a normal sample set and an abnormal sample set.

[0183] Specifically, sample videos without abnormal behaviors are taken as normal sample sets, and sample videos with abnormal behaviors are taken as abnormal sample sets.

[0184] Step 702: Cluster the abnormal sample set to obtain abnormal subsets of multiple categories; each category corresponds to a single abnormality recognition model.

[0185] Specifically, a clustering algorithm is used to cluster the abnormal sample set, dividing it into multiple abnormal subsets. For example, if the abnormal sample set includes abnormal behaviors such as smoking, drinking, and fighting, the abnormal sample set will be divided into a smoking abnormal subset, a drinking abnormal subset, and a fighting abnormal subset. Each abnormal subset corresponds to a single abnormality recognition model, for example, the smoking abnormal subset corresponds to the smoking abnormality recognition model.

[0186] Step 703: Construct a sample set for each single anomaly recognition model; wherein the normal samples are the first part of the normal subset randomly determined from the normal sample set, and the abnormal samples are the first part of the corresponding abnormal subset.

[0187] Specifically, when training a single anomaly recognition model, a sample set for the single anomaly recognition model is constructed, and a normal subset is randomly determined from the normal sample set. The normal subset is divided into two parts, the first of which serves as the sample set for the single anomaly recognition model. The abnormal subset is also divided into two parts, the first of which also serves as the sample set for the single anomaly recognition model. Therefore, the sample set for the single anomaly recognition model contains two parts: the first part of the normal subset and the first part of the abnormal subset.

[0188] Step 704: Construct a sample set for the image recognition model; wherein the normal samples are the second part of each normal subset, and the abnormal samples are the second part of each abnormal subset; the second part in the normal subset is different from the first part in the normal subset, and the second part in the abnormal subset is different from the first part in the abnormal subset.

[0189] Specifically, when training the image recognition model, a sample set for the image recognition model is constructed. The second portion of the normal subset and the second portion of the abnormal subset are used as the sample set for the image recognition model. Similarly, the sample set for the image recognition model also contains two portions, and the sample set for the image recognition model does not overlap with the sample set for the single abnormality recognition model.

[0190] For example, consider a set of normal and abnormal samples. For example, consider a set of 100,000 normal samples and 1,000 abnormal sample sets, each of which represents smoking, drinking, and fighting. The number of normal samples is significantly greater than the number of abnormal samples. Cluster the 1,000 abnormal samples, for example using K-means clustering, into three abnormal subsets: a smoking abnormal subset, a drinking abnormal subset, and a fighting abnormal subset. The smoking abnormal subset contains 500 samples, the drinking abnormal subset contains 300 samples, and the fighting abnormal subset contains 200 samples. Randomly select a certain number of normal samples from the 100,000 normal samples and add them to the abnormal subsets. For example, add 30,000 normal samples to the smoking abnormal subset, 30,000 normal samples to the drinking abnormal subset, and 40,000 normal samples to the fighting abnormal subset. This yields three sample sets. These three datasets have different data structures (both normal and abnormal samples) and therefore have different data distributions. These three sample sets are further randomly divided into two parts. The first part of the normal subset and the first part of the abnormal subset are used as the basic training set B, and the second part of the normal subset and the second part of the abnormal subset are used as the query training set Q. Then each basic training set is B1, B2, and B3, and the query set is Q1, Q2, and Q3, as shown in Table 1:

[0191]

[0192] Table 1

[0193] When training a single anomaly recognition model, the basic training sets B1, B2, and B3 are used. When training an image recognition model, the query set training sets Q1, Q2, and Q3 are used. Figure 5 and Figure 6 Each single anomaly model only captures part of the anomaly. Therefore, this solution proposes a unified image recognition model capture, namely a cascade model. The cascade model is used to fit various anomaly distributions, thereby reducing overfitting in specific anomaly types.

[0194] Each single anomaly model falls under the category of supervised learning and often focuses on detecting anomalies within a sample set. However, in actual use, it's necessary to consider anomalies not covered in the sample set. Therefore, this solution uses the query set training set to train the cascade model and update the weights, enabling the entire image recognition model to effectively simulate anomalies not previously encountered in training, enhancing the model's generalization.

[0195] For the cascade model, its network architecture is consistent with the single anomaly recognition model, combining multiple single anomaly recognition models As the supervised input in the cascade model, its loss function is the same as formula (9).

[0196] During the training process, the weights of the single anomaly recognition model use the trained weights and are not updated. Only the weights of the cascade model are trained. Since the single anomaly recognition model is trained with the basic training set, and the cascade model is trained with the query training set, the role of the cascade model is to learn the distribution of cross-domain data through the single anomaly recognition model, so that the anomaly detection model can detect anomalies in cross-domain data. By building Figure 6 The model architecture can jointly train multiple single anomaly recognition models into a cascade model, thereby realizing video anomaly detection with a single model.

[0197] In the process of training the cascade model, assuming that n rounds of training are required, for each round of training, a single anomaly recognition model is first trained on the basic training set B. After the training is completed, the cascade model is trained on the query training set Q. Then, the weights of the cascade model need to be updated to the single anomaly recognition model to start a new round of training.

[0198] By training the cascade model with the losses of multiple single anomaly recognition models, the cascade model can improve the performance of multiple models on cross-domain data, achieving optimal performance on the query training set Q. By updating the weights of the single anomaly recognition models and optimizing them on the base training set B, the cascade model not only fits the base training set B but also achieves optimal performance on the query training set Q, resulting in a cascade model with strong generalization capabilities.

[0199] like Figure 6 As shown in the figure, when training the cascade model, the combined loss value of the three single anomaly recognition models is calculated and used to adjust the cascade model until the cascade model meets the training requirements. The combined loss value can be the average loss value of the three single anomaly recognition models or the sum of the losses of the three single anomaly recognition models.

[0200] In some embodiments, the text recognition model is a large language model; determining, through the text recognition model, whether there is a second recognition result of abnormal behavior in the image frame corresponding to the text description sequence, includes: inputting the text description sequence and the abnormal behavior prompt word into the large language model, thereby determining whether there is a second recognition result of abnormal behavior in the image frame corresponding to the text description sequence.

[0201] Specifically, embodiments of the present application further utilize a text recognition model to determine whether abnormal behavior exists in the video being recognized. This model utilizes a large language model, which inputs a text description sequence and abnormal behavior prompts into the large language model to determine whether abnormal behavior exists in the image frame corresponding to the text description sequence. The abnormal behavior prompts the large language model to determine an abnormal behavior score.

[0202] In some embodiments, the text description sequence and the abnormal behavior prompt word are input into the large language model to determine whether there is a second recognition result of abnormal behavior in the image frame corresponding to the text description sequence, such as Figure 8 As shown, the following steps are included:

[0203] Step 801: Determine a text description subsequence of the i-th image frame from a text description sequence.

[0204] Step 802: Input the text description subsequence and scene confirmation prompt words of the i-th image frame into the large language model to determine the scene text description of the i-th image frame.

[0205] Step 803: Input the scene text description and abnormal behavior prompt words of the i-th image frame into the large language model to obtain a second recognition result of whether abnormal behavior exists in the i-th image frame.

[0206] Specifically, for each frame in the sample video, a large language model is used to generate a text description to obtain a text description subsequence, such as Since the text description of a single frame is not sufficient to describe the entire scene, the description of the single frame image is used to describe the scene. , for large model design prompt words, assuming , each frame is determined by the scene text description of the previous and next two frames, such as the third stitch The scene description can be represented by and The prompt words you design can be: Based on the descriptions of the following 2nd, 3rd, and 4th frames, summarize what happened in the scene. And enter the description The description of the scene can be summarized, and so on. Each frame is determined by the previous and next frames, so the scene text description of each frame can be obtained.

[0207] After obtaining the scene text description for each frame, we further design abnormal behavior prompt words and obtain an anomaly score. The designed abnormal behavior prompt words must include the type of anomaly to be detected. For example, if we want to detect smoking scenes, the designed prompt words might be: Based on the following scene description, use a 0-1 output value to predict the probability that this scene is a smoking scene. This directly obtains the second recognition result of whether abnormal behavior exists in the i-th image frame.

[0208] In some embodiments, a text description of each image frame in the video to be identified is determined, such as Figure 9 As shown, the following steps are included:

[0209] Step 901: Determine an initial text description of each image frame in the video to be identified;

[0210] Step 902: determining image features of the i-th image frame, and determining text features of the i-th image frame from an initial text description of the i-th image frame;

[0211] Step 903 : Determine, from the text features of the i-th image frame, text features that meet similarity requirements with the image features of the i-th image frame as the text description of the i-th image frame.

[0212] Specifically, for each image frame in the video to be identified, the BLIP2 model is used to generate an initial text description Since BLIP2 has certain errors in describing a single image, it may produce different descriptions for images with similar content. Therefore, the feature similarity between the text encoder and the image encoder can be used for text filtering. Assume that a video has 10 frames of images. The 10 frames of images are passed through the image encoder to obtain the image features. , the corresponding text description is passed through the text encoder to obtain the text feature: ,Will and Perform cosine similarity comparison and use the description corresponding to the most similar text feature as the text description of the frame to obtain the filtered text feature.

[0213] Based on the same technical concept, the embodiment of the present application provides a device for identifying abnormal behavior in a video, such as Figure 10 As shown, the device 1000 includes:

[0214] The recognition result module 1001 is used to determine, by using an image recognition model, whether each image frame in the video to be recognized has a first recognition result of abnormal behavior;

[0215] The determination module 1002 is further configured to determine a text description of each image frame in the video to be identified, thereby obtaining a text description sequence of the video to be identified;

[0216] The recognition result module 1001 is further configured to determine, through a text recognition model, a second recognition result indicating that abnormal behavior exists in the image frame corresponding to the text description sequence;

[0217] The comprehensive recognition module 1003 is configured to determine a comprehensive recognition result of the i-th image frame based on the first recognition result corresponding to the i-th image frame and the second recognition result corresponding to the i-th image frame; the i-th image frame is any image frame in the video to be recognized;

[0218] The abnormal behavior module 1004 is configured to determine that abnormal behavior exists in the video to be identified when the number of image frames whose comprehensive identification results meet the first set condition meets the second set condition.

[0219] Optionally, the comprehensive identification module 1003 is further configured to:

[0220] Inputting the first recognition result sequence and the second recognition result sequence into a sequence value estimation model representing behavior continuity, respectively, to obtain a first sequence estimation result and a second sequence estimation result; the first recognition result sequence is obtained in sequence based on the first recognition result of each image frame in the video to be recognized; and the second recognition result sequence is obtained in sequence based on the second recognition result of each image frame in the video to be recognized;

[0221] Obtaining a first deviation result sequence according to the first recognition result sequence and the first sequence estimation result;

[0222] Obtaining a second deviation result sequence according to the second recognition result sequence and the second sequence estimation result;

[0223] Determining a comprehensive recognition result of the i-th image frame based on the first recognition result of the i-th image frame and the second recognition result of the i-th image frame includes:

[0224] Determining, based on a first deviation result of the i-th image frame in the first deviation result sequence and a second deviation result of the i-th image frame in the second deviation result sequence, a first weight corresponding to the first recognition result of the i-th image frame and a second weight corresponding to the second recognition result corresponding to the i-th image frame;

[0225] A comprehensive recognition result of the i-th image frame is determined based on the first weight of the i-th image frame, the second weight of the i-th image frame, the first recognition result of the i-th image frame, and the second recognition result of the i-th image frame.

[0226] Optionally, the sequence value estimation model is obtained by training a sequence of sample recognition results output by the image recognition model through sample videos containing abnormal behavior; the sequence value estimation model is used to predict the recognition results of subsequent frames based on the continuity of behavior through the recognition results of historical frames.

[0227] Optionally, the image recognition model is a cascade model with generalized recognition capability obtained by training multiple single abnormal behavior recognition models; wherein any single abnormal behavior recognition model is obtained by supervised learning training for each sample with the same type of abnormal behavior.

[0228] Optionally, the recognition result module 1001 is specifically configured to:

[0229] Input the same sample into each single abnormal behavior recognition model to obtain a single recognition result output by each single abnormal behavior recognition model; determine the loss value of each single abnormal behavior recognition model based on the actual recognition result of the sample video and the single recognition result of each single abnormal behavior recognition model;

[0230] The sample is input into the image recognition model, and the image recognition model is trained based on the loss value of each single abnormal behavior recognition model until a training termination condition is met.

[0231] Optionally, the loss value of each single anomaly recognition model is determined by the following formula:

[0232]

[0233] Optionally, the recognition result module 1001 is specifically configured to:

[0234] Obtain normal sample sets and abnormal sample sets;

[0235] Clustering the abnormal sample set to obtain abnormal subsets of multiple categories; each category corresponds to a single abnormality recognition model;

[0236] Constructing a sample set for each single anomaly recognition model; wherein the normal samples are the first part of the normal subset randomly determined from the normal sample set, and the abnormal samples are the first part of the corresponding abnormal subset;

[0237] Construct a sample set of the image recognition model; wherein the normal samples are the second part of each normal subset, and the abnormal samples are the second part of each abnormal subset; the second part in the normal subset is different from the first part in the normal subset, and the second part in the abnormal subset is different from the first part in the abnormal subset.

[0238] Optionally, the recognition result module 1001 is specifically configured to:

[0239] Determining, by a text recognition model, whether a second recognition result of abnormal behavior exists in the image frame corresponding to the text description sequence, includes:

[0240] The text description sequence and the abnormal behavior prompt word are input into the large language model to determine whether a second recognition result of abnormal behavior exists in the image frame corresponding to the text description sequence.

[0241] Optionally, the recognition result module 1001 is specifically configured to:

[0242] Determining a text description subsequence of the i-th image frame from the text description sequence;

[0243] Inputting the text description subsequence and scene confirmation prompt words of the i-th image frame into the large language model to determine the scene text description of the i-th image frame;

[0244] The scene text description and abnormal behavior prompt words of the i-th image frame are input into the large language model to obtain a second recognition result of whether there is abnormal behavior in the i-th image frame.

[0245] Optionally, the recognition result module 1001 is specifically configured to:

[0246] Determining an initial text description of each image frame in the video to be identified;

[0247] determining image features of the i-th image frame, and determining text features of the i-th image frame from an initial text description of the i-th image frame;

[0248] From the text features of the i-th image frame, text features that meet similarity requirements with the image features of the i-th image frame are determined as the text description of the i-th image frame.

[0249] Based on the same technical concept, the embodiment of the present application provides a computer device, which can be a terminal or a server, such as Figure 11 As shown, it includes at least one processor 1101 and a memory 1102 connected to the at least one processor. The specific connection medium between the processor 1101 and the memory 1102 is not limited in the embodiment of the present application. Figure 11 For example, the processor 1101 and the memory 1102 are connected via a bus, which can be divided into an address bus, a data bus, a control bus, and the like.

[0250] In an embodiment of the present application, the memory 1102 stores instructions that can be executed by at least one processor 1101. By executing the instructions stored in the memory 1102, the at least one processor 1101 can execute the steps included in the above-mentioned method for identifying abnormal behavior in a video.

[0251] The processor 1101 is the control center of the computer device. It can connect various parts of the computer device using various interfaces and lines, and run or execute instructions stored in the memory 1102 and call data stored in the memory 1102. Optionally, the processor 1101 may include one or more processing units. The processor 1101 may integrate an application processor and a modem processor. The application processor mainly processes the operating system, user interface, and application programs, while the modem processor mainly handles wireless communications. It is understood that the modem processor may not be integrated into the processor 1101. In some embodiments, the processor 1101 and the memory 1102 may be implemented on the same chip. In some embodiments, they may also be implemented on separate chips.

[0252] The processor 1101 may be a general-purpose processor, such as a central processing unit (CPU), a digital signal processor, an application-specific integrated circuit (ASIC), a field-programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component, and may implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of this application. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application may be directly implemented as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0253] Memory 1102 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs, and modules. Memory 1102 may include at least one type of storage medium, such as flash memory, a hard disk, a multimedia card, a card-type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic storage, a magnetic disk, an optical disk, and the like. Memory 1102 is any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to such. Memory 1102 in the embodiments of the present application may also be a circuit or any other device capable of performing a storage function, used to store program instructions and / or data.

[0254] Based on the same inventive concept, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program that can be executed by a computer device. When the program runs on the computer device, the computer device executes the steps of the above-mentioned method for identifying abnormal behavior in a video.

[0255] Based on the same inventive concept, an embodiment of the present application provides a computer program product, characterized in that the computer program product includes a computer program stored on a computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer device, the computer device executes the steps of the above-mentioned method for identifying abnormal behavior in a video.

[0256] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0257] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0258] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0259] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0260] Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.

Claims

1. A method for identifying abnormal behavior in a video, characterized in that: include: Determine, by an image recognition model, a first recognition result of whether there is abnormal behavior in each image frame of the video to be recognized; Determining a text description of each image frame in the video to be identified, thereby obtaining a text description sequence of the video to be identified; Determining, by a text recognition model, a second recognition result that an abnormal behavior exists in the image frame corresponding to the text description sequence; Determining a comprehensive recognition result of the i-th image frame based on the first recognition result corresponding to the i-th image frame and the second recognition result corresponding to the i-th image frame; The i-th image frame is any image frame in the video to be identified; When the number of image frames satisfying the first set condition in the comprehensive recognition result meets the second set condition, determining that abnormal behavior exists in the video to be recognized; Before determining the comprehensive recognition result of the i-th image frame, the method further includes: Inputting the first recognition result sequence and the second recognition result sequence into a sequence value estimation model representing behavior continuity, respectively, to obtain a first sequence estimation result and a second sequence estimation result; the first recognition result sequence is obtained in sequence based on the first recognition result of each image frame in the video to be recognized; and the second recognition result sequence is obtained in sequence based on the second recognition result of each image frame in the video to be recognized; Obtaining a first deviation result sequence according to the first recognition result sequence and the first sequence estimation result; Obtaining a second deviation result sequence according to the second recognition result sequence and the second sequence estimation result; Determining a comprehensive recognition result of the i-th image frame based on the first recognition result of the i-th image frame and the second recognition result of the i-th image frame includes: Determining, based on a first deviation result of the i-th image frame in the first deviation result sequence and a second deviation result of the i-th image frame in the second deviation result sequence, a first weight corresponding to the first recognition result of the i-th image frame and a second weight corresponding to the second recognition result corresponding to the i-th image frame; A comprehensive recognition result of the i-th image frame is determined based on the first weight of the i-th image frame, the second weight of the i-th image frame, the first recognition result of the i-th image frame, and the second recognition result of the i-th image frame.

2. The method according to claim 1, wherein The sequence value estimation model is obtained by training the sample recognition result sequence output by the image recognition model through sample videos with abnormal behavior; the sequence value estimation model is used to predict the recognition results of subsequent frames based on the recognition results of historical frames based on the continuity of behavior.

3. The method according to claim 1, wherein The image recognition model is a cascade model with generalized recognition capability obtained by training multiple single abnormal behavior recognition models; wherein any single abnormal behavior recognition model is obtained by supervised learning training for each sample with the same type of abnormal behavior.

4. The method according to claim 3, wherein The image recognition model is trained in the following manner, including: Input the same sample into each single abnormal behavior recognition model to obtain a single recognition result output by each single abnormal behavior recognition model; determine the loss value of each single abnormal behavior recognition model based on the actual recognition result of the sample video and the single recognition result of each single abnormal behavior recognition model; The sample is input into the image recognition model, and the image recognition model is trained based on the loss value of each single abnormal behavior recognition model until a training termination condition is met.

5. The method according to claim 3, wherein The samples of the multiple single abnormal behavior recognition models and the samples of the image recognition model are respectively obtained by the following methods: Obtain normal sample sets and abnormal sample sets; Clustering the abnormal sample set to obtain abnormal subsets of multiple categories; each category corresponds to a single abnormality recognition model; Constructing a sample set for each single anomaly recognition model; wherein the normal samples are the first part of the normal subset randomly determined from the normal sample set, and the abnormal samples are the first part of the corresponding abnormal subset; Construct a sample set of the image recognition model; wherein the normal samples are the second part of each normal subset, and the abnormal samples are the second part of each abnormal subset; the second part in the normal subset is different from the first part in the normal subset, and the second part in the abnormal subset is different from the first part in the abnormal subset.

6. The method according to claim 1, wherein The text recognition model is a large language model; Determining, by a text recognition model, whether a second recognition result of abnormal behavior exists in the image frame corresponding to the text description sequence, includes: The text description sequence and the abnormal behavior prompt word are input into the large language model to determine whether a second recognition result of abnormal behavior exists in the image frame corresponding to the text description sequence.

7. The method according to claim 6, wherein The step of inputting the text description sequence and the abnormal behavior prompt word into the large language model to determine whether the image frame corresponding to the text description sequence contains a second recognition result of abnormal behavior, including: Determining a text description subsequence of the i-th image frame from the text description sequence; Inputting the text description subsequence and scene confirmation prompt words of the i-th image frame into the large language model to determine the scene text description of the i-th image frame; The scene text description and abnormal behavior prompt words of the i-th image frame are input into the large language model to obtain a second recognition result of whether there is abnormal behavior in the i-th image frame.

8. The method according to claim 1, wherein Determining a text description of each image frame in the to-be-identified video includes: Determining an initial text description of each image frame in the video to be identified; determining image features of the i-th image frame, and determining text features of the i-th image frame from an initial text description of the i-th image frame; From the text features of the i-th image frame, text features that meet similarity requirements with the image features of the i-th image frame are determined as the text description of the i-th image frame.

9. A device for identifying abnormal behavior in a video, characterized in that: include: A recognition result module is used to determine, through an image recognition model, a first recognition result of whether each image frame in the video to be recognized has abnormal behavior; The determination module is further configured to determine a text description of each image frame in the video to be identified, thereby obtaining a text description sequence of the video to be identified; The recognition result module is further configured to determine, through a text recognition model, a second recognition result indicating that abnormal behavior exists in the image frame corresponding to the text description sequence; a comprehensive recognition module, configured to determine a comprehensive recognition result of the i-th image frame based on the first recognition result corresponding to the i-th image frame and the second recognition result corresponding to the i-th image frame; The i-th image frame is any image frame in the video to be identified; An abnormal behavior module, configured to determine that abnormal behavior exists in the video to be identified when the number of image frames whose comprehensive recognition results meet the first set condition meets the second set condition; Before determining the comprehensive recognition result of the i-th image frame, the method further includes: Inputting the first recognition result sequence and the second recognition result sequence into a sequence value estimation model representing behavior continuity, respectively, to obtain a first sequence estimation result and a second sequence estimation result; the first recognition result sequence is obtained in sequence based on the first recognition result of each image frame in the video to be recognized; and the second recognition result sequence is obtained in sequence based on the second recognition result of each image frame in the video to be recognized; Obtaining a first deviation result sequence according to the first recognition result sequence and the first sequence estimation result; Obtaining a second deviation result sequence according to the second recognition result sequence and the second sequence estimation result; Determining a comprehensive recognition result of the i-th image frame based on the first recognition result of the i-th image frame and the second recognition result of the i-th image frame includes: Determining, based on a first deviation result of the i-th image frame in the first deviation result sequence and a second deviation result of the i-th image frame in the second deviation result sequence, a first weight corresponding to the first recognition result of the i-th image frame and a second weight corresponding to the second recognition result corresponding to the i-th image frame; A comprehensive recognition result of the i-th image frame is determined based on the first weight of the i-th image frame, the second weight of the i-th image frame, the first recognition result of the i-th image frame, and the second recognition result of the i-th image frame.

10. A computer device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the steps of the method according to any one of claims 1 to 8 are implemented when the processor executes the program.

11. A computer-readable storage medium, characterized in that It stores a computer program that can be executed by a computer device. When the program is run on the computer device, the computer device is enabled to perform the steps of any one of the methods according to claims 1 to 8.

Images