Abnormal behavior detection method, system, device and medium

Abstract

The present invention relates to a method, system, device and medium for detecting abnormal behavior. The method comprises: obtaining a video to be identified, and extracting a plurality of video frames from the video in sequence; wherein each video frame carries a corresponding time mark; inputting each video frame and a preset behavior description set into a multimodal large language model, performing multimodal fusion on the input video frames and the behavior description set, and identifying abnormal behaviors in each video frame; wherein the behavior description set is used to indicate abnormal behaviors to be detected; arranging the abnormal behaviors in each video frame according to the generation order, and inputting them into a behavior analysis model together with corresponding time marks, and detecting abnormal behaviors in the video; wherein the behavior analysis model is a time series model based on Transformer. The present invention improves the problem that the prior art only uses a single modality analysis method, resulting in a low abnormal behavior recognition rate, and realizes accurate recognition of abnormal behaviors.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal target detection, and in particular to a method, system, device and medium for detecting abnormal behavior. Background Art

[0002] In traditional physics, chemistry and biology experiment assessments, students' experimental processes rely on manual observation to maintain the fairness of the exam. However, this method has significant problems, mainly because it is difficult to effectively monitor cheating and irregular behaviors. For example: manual observation is inefficient and it is difficult to fully cover the behavior of all candidates, which can easily lead to monitoring blind spots; insufficient monitoring efforts may not stop cheating in time; manual observation is prone to fatigue during a long assessment process; the recording and feedback process is cumbersome and delayed, etc.

[0003] At present, although there are some video monitoring systems on the market, most of them only provide video recording and playback functions, lacking in-depth analysis of examinees' behavior, especially accurate identification of cheating and irregular behaviors. These systems cannot detect cheating and irregular behaviors of examinees during the experiment in real time, such as plagiarism, looking at mobile phones, using cheating tools, etc., resulting in the inability to effectively ensure the fairness of the examination. Therefore, it is necessary to provide a detection method, system, equipment and medium for abnormal behavior. Summary of the invention

[0004] In view of the above-mentioned shortcomings of the prior art, an object of the present invention is to provide a method, system, device and medium for detecting abnormal behavior, which improves the problem that the prior art only uses a single modal analysis method, resulting in a low abnormal behavior recognition rate.

[0005] To achieve the above-mentioned purpose and other related purposes, the present invention provides a method for detecting abnormal behavior, comprising: obtaining a video to be identified, and extracting multiple video frames from the video in sequence; wherein each video frame carries a corresponding time mark; inputting each video frame and a preset behavior description set into a multimodal large language model, performing multimodal fusion on the input video frames and the behavior description set, and identifying abnormal behaviors existing in each video frame; wherein the behavior description set is used to indicate abnormal behaviors to be detected; arranging the abnormal behaviors in each video frame according to a generation order, and inputting them together with the corresponding time mark into a behavior analysis model to detect abnormal behaviors existing in the video; wherein the behavior analysis model is a Transformer-based timing model.

[0006] In one embodiment of the present invention, the multimodal large language model is a MiniCPM-V model, and for each video frame: the video frame and a preset behavior description set are input into the large language model, the input video frame and the behavior description set are multimodally fused, and abnormal behaviors in the video frame are identified, including: extracting visual features from the video frame; performing word segmentation and word embedding processing on each behavior description in the behavior description set to obtain a vector sequence of the behavior description set; encoding the vector sequence to generate a text feature sequence; performing multimodal fusion on the visual features and the text feature sequence to obtain fused features, and identifying abnormal behaviors in the video frame based on the fused features.

[0007] In one embodiment of the present invention, the behavior analysis model is a Transformer-based model, and the abnormal behaviors in each video frame are arranged according to the generation order, and are input into the behavior analysis model together with the corresponding time mark to detect the abnormal behaviors in the video, including: arranging the abnormal behaviors in each video frame according to the generation order, for the abnormal behaviors in each video frame: searching for the index value corresponding to the abnormal behavior in the video frame, and encoding the corresponding index value to generate a behavior embedding vector; position encoding the time mark corresponding to each video frame respectively, and obtaining the time embedding vector corresponding to each video frame;

[0008] The time embedding vector is fused with the corresponding behavior embedding vector, and the fused vectors are arranged in sequence to obtain a fused vector sequence; based on the multi-head self-attention mechanism, the fused vector sequence is encoded to obtain an encoding result; the encoding result is mapped to a preset behavior category dimension to detect abnormal behavior in the video.

[0009] In one embodiment of the present invention, the fused vector sequence is encoded based on the multi-head self-attention mechanism to obtain an encoding result, including: for each attention head: based on a preset query weight matrix, key weight matrix and value weight matrix, the fused vector sequence is linearly transformed to generate a corresponding query matrix, key matrix and value matrix; a global attention matrix is ​​generated according to the query matrix and the key matrix; a global feature is generated according to the global attention matrix and the value matrix; a global attention matrix corresponding to at least one attention head is sparsely processed to obtain a sparse attention matrix; a sparse feature is generated according to the sparse attention matrix and the corresponding value matrix; each global feature is spliced, and the spliced ​​global features and sparse features are weighted to obtain an encoding result.

[0010] In an embodiment of the present invention, after detecting the abnormal behavior in the video, the method further includes: mapping the abnormal behavior to a corresponding video frame and presenting it in the video.

[0011] In one embodiment of the present invention, the method further includes: saving the extracted multiple video frames to a buffer, and upon receiving a completion signal sent by the multimodal large language model, inputting the video frames in the buffer into the multimodal large language model; wherein the multimodal large language model generates a completion signal after completing the recognition of the input video frames.

[0012] In an embodiment of the present invention, the method further includes: detecting whether the video picture is abnormal from the extracted video frame, and sending an alarm message when an abnormality exists.

[0013] In one embodiment of the present invention, a system for detecting abnormal behavior is also provided, the system comprising: a video acquisition module, used to acquire a video to be identified, and extract multiple video frames from the video in sequence; wherein each video frame carries a corresponding time stamp; an abnormal behavior initial inspection module, used to input each video frame and a preset behavior description set into a multimodal large language model, perform multimodal fusion on the input video frames and the behavior description set, and identify abnormal behaviors in each video frame; wherein the behavior description set is used to indicate abnormal behaviors to be detected; an abnormal behavior determination module, used to arrange the abnormal behaviors in each video frame according to a generation order, and input them together with the corresponding time stamps into a behavior analysis model to detect abnormal behaviors in the video; wherein the behavior analysis model is a Transformer-based timing model.

[0014] In one embodiment of the present invention, an electronic device is also provided, comprising: one or more processors; a storage device for storing one or more programs, and when the one or more programs are executed by the one or more processors, the electronic device implements any of the above-mentioned abnormal behavior detection methods.

[0015] In one embodiment of the present invention, a computer-readable storage medium is further provided, on which a computer program is stored. When the computer program is executed by a processor of a computer, the computer executes any of the above-mentioned abnormal behavior detection methods.

[0016] As described above, an abnormal behavior detection method, system, device and medium of the present invention have the following beneficial effects: when it is necessary to detect abnormal behavior in a video, multiple video frames are first extracted from the video, and a time stamp is attached to each video frame. The video frames and the behavior description set are input into a multimodal large language model, and the visual features and the semantic information of the behavior description are multimodally fused to identify abnormal behaviors in the video frames. Each abnormal behavior detected in the video and the corresponding time stamp are input into the behavior analysis model, and all abnormal behaviors present in the video are identified by capturing the time-dependent information between frames. The present invention adopts a joint processing method of multimodal fusion and timing analysis, which improves the problem of low abnormal behavior recognition caused by single-modal recognition in the prior art, realizes accurate recognition of abnormal behaviors, and can adapt to dynamic monitoring needs in different scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A schematic diagram of a flow chart of a method for detecting abnormal behavior provided by an embodiment of the present invention;

[0018] Figure 2 Shown is a structural block diagram of an abnormal behavior detection system provided by an embodiment of the present invention;

[0019] Figure 3 Shown is a structural schematic diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0020] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0021] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and thus the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0022] In the following description, numerous details are discussed to provide a more thorough explanation of the embodiments of the present invention. However, it is obvious to those skilled in the art that the embodiments of the present invention can be implemented without these specific details. In other embodiments, well-known structures and devices are shown in the form of block diagrams rather than in detail to avoid making the embodiments of the present invention difficult to understand.

[0023] The present invention provides a method for detecting abnormal behavior. When it is necessary to detect abnormal behavior in a video, multiple video frames are first extracted from the video, and a time mark is attached to each video frame. The video frame and the behavior description set are input into a multimodal large language model, and the visual features and the semantic information of the behavior description are multimodally fused to identify the abnormal behavior in the video frame. Each abnormal behavior detected in the video and the corresponding time mark are input into a behavior analysis model, and all abnormal behaviors existing in the video are identified by capturing the time dependency information between frames. The present invention adopts a joint processing method of multimodal fusion and timing analysis, improves the problem that the recognition degree of abnormal behavior is not high due to the single modality recognition of the prior art, realizes the accurate recognition of abnormal behavior, and can adapt to the dynamic monitoring needs in different scenarios. In this way, it is possible to monitor the behavior of examinees in physical, chemical and biological experiments in real time, automatically identify and record possible irregular behaviors, such as checking mobile phones, reading books, using watches, etc. Through the deep understanding of the video images and related text descriptions by the multimodal large model, it is possible to accurately judge whether the examinees have irregular behaviors, and provide timely warnings and feedback to teachers to ensure the fairness and standardization of the examination. In addition, the abnormal behavior detection method of the present invention can also greatly improve the intelligence and automation level of examination monitoring, which not only reduces the workload of teachers, but also improves the fairness and efficiency of examinations, helps to timely discover and stop the irregular behavior of candidates, and ensures the transparency and fairness of the examination process.

[0024] See also Figure 1 ,The abnormal behavior detection method includes the following steps:

[0025] S1. Obtain a video to be identified, and extract multiple video frames from the video in sequence; wherein each video frame carries a corresponding time mark.

[0026] After obtaining the video of the student's experimental operation, each video frame in the video can be extracted frame by frame or according to a preset interval sampling method, and each extracted video frame is preprocessed to form a video frame sequence V = {v1, v2, v3…v i}, where v iis the i-th video frame in the sequence. The preprocessing process includes but is not limited to denoising, color space conversion, adjusting the resolution to a preset threshold, etc., and those skilled in the art can adaptively select based on the needs of actual model processing. Each video frame has a corresponding time marker, which is used to characterize the specific time point of the video frame in the video. It can be understood that the time marker can be a specific time point or a frame number, as long as the specific time point of the frame in the video can be accurately located, and no limitation is made here. It should be noted that the video to be identified can be a complete video, such as a complete monitoring video of a student's experimental test, or a video clip of a complete video. After analyzing it, continue to extract the next video clip of the complete video for analysis until the entire video analysis is completed.

[0027] In one embodiment of the present invention, obtaining a video to be identified and sequentially extracting a plurality of video frames from the video includes:

[0028] Get the video to be identified;

[0029] Extract individual video frames from the video frame by frame;

[0030] Based on the optical flow method, the extracted video frames are processed and multiple key frames are selected.

[0031] Considering that there are a large number of video frames in a video, if frame-by-frame processing is performed, the processing rate will be greatly reduced. If frames are extracted at a preset fixed interval, some frames with obvious changes in action may be missed, affecting the accuracy of subsequent behavior detection. To improve the above problems, the optical flow method can be used to extract key frames from video frames for subsequent processing. Specifically, after extracting M video frames from the video frame by frame, for the frame pairs consisting of two adjacent video frames: the motion vector of each pixel between the frames is calculated by the optical flow algorithm to generate the optical flow field between the frames. The average motion amplitude of the two adjacent frames is calculated based on the optical flow field, and it is determined whether the amplitude is greater than the preset threshold; the frame pairs greater than the preset threshold are selected, and the latter video frame in the frame pair is used as the key frame. In this way, N key frames (M is much greater than N) are selected from the M video frames of the video, so that the key information in the video can be retained while reducing the computational cost.

[0032] S2. Input each video frame and a preset behavior description set into a multimodal large language model, perform multimodal fusion on the input video frames and the behavior description set, and identify abnormal behaviors in each video frame; wherein the behavior description set is used to indicate abnormal behaviors to be detected.

[0033] For each video frame: the video frame and the behavior description set are input into the multimodal large language model to extract the visual features of the video frame and the semantic features of each behavior description. The visual features and semantic features are multimodally fused to capture the features associated with the behavior description in the video frame, so as to determine whether there is a preset abnormal behavior in the video frame, and output the detection result, which includes the category of the abnormal behavior and the corresponding confidence score. Among them, the behavior description set is a set of texts expressed in natural language, which defines the target behavior or abnormal behavior in the video frame to be detected, and each behavior description provides the multimodal large language model with semantic information for identifying specific behaviors. Exemplarily, the behavior description is "Is the person in the picture looking at the phone?" "Has anyone left the experimental table?" etc. It can be understood that the behavior description set is not fixed, but has scene adaptability. For different experimental scenes, the behavior description set can be adaptively modified to cover the target behavior specifications of specific scenes. The specific description content is not limited here. In the above manner, each frame can be analyzed to identify the abnormal behavior in the frame. For example, when the examinee stops operating for a long time and there is no operator at the lab table, it is considered as an absence. When the examinee looks at the phone or reads a book, the details in the image are analyzed to determine whether the examinee frequently lowers his head, picks up the phone, or flips through a book.

[0034] In one embodiment of the present invention, the multimodal large language model is a MiniCPM-V model. For each video frame: the video frame and a preset behavior description set are input into the large language model, multimodal fusion is performed on the input video frame and the behavior description set, and abnormal behavior in the video frame is identified, including:

[0035] Extract visual features from video frames;

[0036] Performing word segmentation and word embedding processing on each behavior description of the behavior description set to obtain a vector sequence of the behavior description set;

[0037] Encoding the vector sequence to generate a text feature sequence;

[0038] The visual features and the text feature sequence are multimodally fused to obtain fused features, and abnormal behaviors in the video frames are identified based on the fused features.

[0039] The multimodal large language model can be any model with a visual encoder and a text encoder, including but not limited to CLIP, BEiT-3, MiniCPM-V, etc. In order to more accurately capture the global and local information in the video frame and facilitate the identification of abnormal behavior, the multimodal large language model in this implementation is the MiniCPM-V model. For example, in order to achieve the best performance of the model, the MiniCPM-V-2_6 version can be selected. Extract each video frame v in the video frame sequence V frame by frame i , for the extracted video frame v i Perform the following processing: convert the video frame v i The visual encoder input to the multimodal large language model captures the global scene information and local detail information in the video frame and combines the two types of information as the visual features V in the video frame. i At the same time, each behavior description in the behavior description set is segmented, a complete text sentence is split into several subwords, and the text after segmentation is embedded, and the corresponding behavior description is converted into a text embedding vector. All text embedding vectors are arranged according to the order of the corresponding behavior descriptions to form a text embedding vector sequence. The text embedding vector sequence corresponding to the behavior description set is input into the text encoder of the multimodal large language model, and the semantic features of each vector are extracted to generate a text feature sequence T. Through the cross-attention mechanism, the text feature sequence T and the visual features V of the input video frame are combined. i Perform multimodal fusion, pay more attention to the areas in the video frame related to the behavior description during fusion, and generate fusion features. Map the fusion features to the preset behavior category space, identify whether there is abnormal behavior in the current video frame, and output the detected abnormal behavior and the corresponding confidence. It is understandable that each video frame may not have abnormal behavior, may only have one abnormal behavior, or may have multiple different abnormal behaviors. For each video frame, in this way, the pre-processed multimodal large language model is used to extract the abnormal behavior label to obtain the abnormal behavior classification result at the frame level. By using the multimodal large model, it is possible to deeply understand the operation behavior in the video, reduce the workload and reduce manual errors, and improve the accuracy of abnormal behavior identification.

[0040] It should be noted that the multimodal large language model can directly call the interface and use the open source large language model, or it can use its own data set to train the large language model to make it more robust and more adaptable to specific scenarios. The specific method is not limited here.

[0041] S3. Arrange the abnormal behaviors in each video frame according to the generation order, and input them together with the corresponding time stamps into the behavior analysis model to detect the abnormal behaviors in the video; wherein the behavior analysis model is a Transformer-based timing model.

[0042] Specifically, in one embodiment of the present invention, the behavior analysis model is a Transformer-based model, and the abnormal behaviors in each video frame are arranged according to the generation order and input into the behavior analysis model together with the corresponding time stamp to detect the abnormal behaviors in the video, including:

[0043] Arrange the abnormal behaviors in each video frame according to the generation order. For each abnormal behavior in the video frame: find the index value corresponding to the abnormal behavior in the video frame, encode the corresponding index value, and generate a behavior embedding vector;

[0044] The time tags corresponding to each video frame are respectively position-encoded to obtain the time embedding vector corresponding to each video frame;

[0045] Fuse the time embedding vector with the corresponding behavior embedding vector, and arrange the fused vectors in order to obtain a fused vector sequence;

[0046] Based on a multi-head self-attention mechanism, encoding the fused vector sequence to obtain an encoding result;

[0047] The encoding result is mapped to a preset behavior category dimension to detect abnormal behaviors in the video.

[0048] In order to efficiently model the temporal dependency in the video frame sequence, the behavior analysis model in this embodiment is a Transformer-based model. For the i-th video frame processed by the multimodal large language model: the corresponding abnormal behavior is mapped to the index value of the abnormal behavior, and the index value is mapped to a high-dimensional vector through the embedding layer of the behavior analysis model to generate a behavior embedding vector e ai After all the behavior embedding vectors of the video frames are generated, the behavior embedding vector sequence E of the video frame sequence is obtained. A ={e a1 ,e a2 ,e a3 …e ai In addition, according to the time stamp of each video frame, a time stamp sequence T = {t1, t2, t3…t i}, input each time mark into the position encoder of the behavior analysis model to generate the time embedding vector e ti After all time tags are encoded, the time embedding vector sequence E of the video frame sequence is obtained. T={e t1 ,e t2 ,e t3 …e ti For each video frame: fuse the behavior embedding vector and the corresponding time embedding vector to obtain the fusion vector F i =e ai +e ti The fusion vectors of all video frames are arranged in the order of the corresponding video frames to form a fusion vector sequence F = {F1, F2, F3…F i}. The fusion vector sequence is input into the encoder of the Transformer-based model. Based on the multi-head self-attention mechanism, the dependency between the abnormal behavior sequence and the time mark is captured. The fusion vector sequence is gradually encoded through the multi-layer Transformer encoder to generate the global features of each fusion vector, which is the encoding result of the fusion vector. The encoding result is mapped to the preset behavior category dimension to obtain the abnormal behavior category and its probability of the frame. After each fusion vector sequence is detected, the abnormal behavior in the video is determined.

[0049] In one embodiment of the present invention, the encoding process of the fused vector sequence based on the multi-head self-attention mechanism to obtain the encoding result includes:

[0050] For each attention head:

[0051] Based on a preset query weight matrix, a key weight matrix and a value weight matrix, a linear transformation is performed on the fused vector sequence to generate a corresponding query matrix, a key matrix and a value matrix;

[0052] Generate a global attention matrix based on the query matrix and the key matrix;

[0053] Generate global features based on the global attention matrix and value matrix;

[0054] Performing sparse processing on the global attention matrix corresponding to at least one attention head to obtain a sparse attention matrix;

[0055] Generate sparse features based on the sparse attention matrix and the corresponding value matrix;

[0056] The global features are concatenated, and the concatenated global features and sparse features are weighted to obtain the encoding result.

[0057] The encoder of the Transformer-based model consists of multiple cascaded attention heads. The following processing is performed on the i-th attention head: Since the attention head has an independent query weight matrix W Qi , key weight matrix W Ki Sum value weight matrix WVi Perform a linear transformation on the input fusion vector sequence F and replace F with the query weight matrix W of the attention head. Qi , key weight matrix W Ki Sum value weight matrix W Vi Multiply them together to get the query vector matrix Q, key matrix K and value matrix V. Among them, the query matrix Q is used to represent the query information between frames, the key matrix K is used to calculate the correlation with the query matrix, and the value matrix V is used to characterize the actual eigenvalues ​​of each frame. According to the query matrix Q and the key matrix K, the attention score corresponding to the attention head is calculated to generate the global attention matrix A. Multiply the global attention matrix A and the value matrix V to obtain the global features. Considering that the global attention matrix has many parameters, in order to reduce the computational complexity and enhance the model's ability to capture key information, the corresponding global attention matrix of at least one attention head is sparsely processed to obtain the sparse attention matrix A of the attention head. s It should be noted that the sparse processing methods include but are not limited to Threshold sparse processing, Top-k sparse processing, etc. Those skilled in the art can set it adaptively based on the actual model needs, and it is not limited here. s Multiply the sum matrix V to obtain the sparse features. After all the attention heads that need to be sparsely processed are processed in the above manner, the global features of each attention head are spliced ​​to form a spliced ​​global feature. The global features that are sparsely processed and the corresponding sparse features in the spliced ​​global features are weightedly fused to obtain the encoding result for input to the next attention head or classifier.

[0058] In one embodiment of the present invention, after the abnormal behavior existing in the video is detected, it also includes: mapping the abnormal behavior to the corresponding video frame and presenting it in the video. The detected abnormal behavior is highlighted in the target area of ​​the corresponding video frame (such as a rectangular frame), and the specific abnormal behavior (such as "looking at the phone") is marked, so as to display the specific location where the abnormal behavior occurs in an intuitive way. Furthermore, if a video frame includes multiple abnormal behaviors, multiple target areas can be marked in the frame for display. After each video frame of the video is annotated and updated frame by frame, the annotation information is superimposed on the original video to generate an output video containing abnormal behavior annotations, wherein the annotation information includes the target area where the abnormal behavior occurs and the specific abnormal behavior.

[0059] In one embodiment of the present invention, the method further includes: saving the extracted multiple video frames to a buffer, and inputting the video frames in the buffer to the multimodal large language model when receiving a completion signal sent by the multimodal large language model; wherein the multimodal large language model generates a completion signal after the input video frames are recognized. The extracted video frames are saved to the buffer in chronological order, and the buffer is used to temporarily store unprocessed video frame data for subsequent batch processing. When the video frames stored in the buffer meet preset conditions (such as a fixed number of frames or time period) or receive a completion signal from the multimodal large language model, the video frames in the buffer are uniformly input to the multimodal large language model as a batch for analysis and processing. The multimodal large language model recognizes the input video frames in turn, extracts abnormal behavior information of each frame and generates corresponding detection results. When all video frames are processed, the model generates a completion signal, indicating that the video frame analysis task of the current batch has ended. The completion signal not only ensures the coordination and consistency of the buffer and model input processes, but also prepares for the processing of the next batch, ensuring efficient operation and real-time response of the entire processing flow.

[0060] In one embodiment of the present invention, the method further includes: detecting whether the video screen is abnormal from the extracted video frame, and sending an alarm message when an abnormality exists. By detecting the video screen, it is determined whether the screen is abnormal, such as a black screen, a green screen, a flowery screen or an obstruction. When an abnormality is detected in the camera, an alarm will be automatically issued to remind the invigilator to handle it. This function ensures the stability of the experimental video monitoring and the reliability of the data, and avoids distortion of the monitoring results due to equipment problems. In addition, all monitoring data and examinee behavior records will be stored in the cloud, which is convenient for teachers to conduct subsequent analysis, review and evaluation, and provide data support for teaching improvements. Exemplarily, the data and behavior records can be saved in MinIO or Database databases for subsequent calls at any time.

[0061] See also Figure 2, the abnormal behavior detection system 100 includes: a video acquisition module 110, an abnormal behavior initial inspection module 120 and an abnormal behavior determination module 130. The above-mentioned video acquisition module 110 is used to acquire the video to be identified, and extract multiple video frames from the video in sequence; wherein each video frame carries a corresponding time stamp. The abnormal behavior initial inspection module 120 is used to input each video frame and a preset behavior description set into a multimodal large language model, perform multimodal fusion on the input video frames and the behavior description set, and identify the abnormal behavior in each video frame; wherein the behavior description set is used to indicate the abnormal behavior to be detected. The abnormal behavior determination module 130 is used to arrange the abnormal behaviors in each video frame according to the generation order, and input them together with the corresponding time stamps into the behavior analysis model to detect the abnormal behavior in the video; wherein the behavior analysis model is a Transformer-based timing model.

[0062] For the specific definition of the abnormal behavior detection system, please refer to the definition of the abnormal behavior detection method above, which will not be repeated here. Each module in the above abnormal behavior detection system can be implemented in whole or in part by software, hardware and a combination thereof. The above modules can be embedded in or independent of the processor in the computer device in hardware format, or stored in the memory in the computer device in software format, so that the processor can call the operations corresponding to the above modules.

[0063] It should be noted that, in order to highlight the innovative part of the present invention, the present embodiment does not introduce modules that are not closely related to solving the technical problem proposed by the present invention, but this does not mean that there are no other modules in the present embodiment.

[0064] See also Figure 3 The electronic device 1 may include a memory 12, a processor 13 and a bus, and may also include a computer program stored in the memory 12 and executable on the processor 13, such as an abnormal behavior detection program.

[0065] Among them, the memory 12 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, mobile hard disk, multimedia card, card-type memory (for example: SD or DX memory, etc.), magnetic memory, disk, optical disk, etc. The memory 12 can be an internal storage unit of the electronic device 1 in some embodiments, such as a mobile hard disk of the electronic device 1. The memory 12 can also be an external storage device of the electronic device 1 in other embodiments, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device 1. Further, the memory 12 can also include both an internal storage unit of the electronic device 1 and an external storage device. The memory 12 can not only be used to store application software and various types of data installed in the electronic device 1, such as the code for detecting abnormal behavior, etc., but also can be used to temporarily store data that has been output or is to be output.

[0066] In some embodiments, the processor 13 may be composed of an integrated circuit, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and combinations of various control chips. The processor 13 is the control core (Control Unit) of the electronic device 1, and uses various interfaces and lines to connect various components of the entire electronic device 1, and executes or executes programs or modules (such as abnormal behavior detection programs, etc.) stored in the memory 12, and calls data stored in the memory 12 to execute various functions of the electronic device 1 and process data.

[0067] The processor 13 executes the operating system and various installed applications of the electronic device 1. The processor 13 executes the applications to implement the steps in the above abnormal behavior detection method.

[0068] Exemplarily, the computer program may be divided into one or more modules, which are stored in the memory 12 and executed by the processor 13 to complete the present application. The one or more modules may be a series of computer program instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program in the electronic device 1. For example, the computer program may be divided into a video acquisition module 110, an abnormal behavior initial detection module 120, and an abnormal behavior determination module 130.

[0069] The above-mentioned integrated unit implemented in the form of a software function module can be stored in a computer-readable storage medium, and the computer-readable storage medium can be non-volatile or volatile. The above-mentioned software function module is stored in a storage medium, including several instructions for enabling a computer device (which can be a personal computer, a computer device, or a network device, etc.) or a processor to perform part of the functions of the abnormal behavior detection method described in each embodiment of the present application.

[0070] In summary, the present invention discloses an abnormal behavior detection method, system, device and medium. When it is necessary to identify abnormal behavior in a video, multiple video frames are extracted from the video and a time stamp is attached to each video frame, so as to accurately associate the video frames with the behavior detection results, so as to provide basic support for subsequent abnormal behavior identification and time positioning. The extracted video frames and the preset behavior description set are input into a multimodal large language model, and the visual features are multimodally fused with the semantic information, so that the abnormal behavior category in the video frame can be accurately identified. Furthermore, by inputting the identified abnormal behavior and time stamp into the behavior analysis model, the multi-head self-attention mechanism is used to capture the time dependency in the frame sequence, and generate accurate abnormal behavior categories and their occurrence sequence. Compared with the traditional behavior detection method that relies on a single modality or static analysis, the present invention adopts a joint processing method of multimodal fusion and time series analysis, which not only improves the accuracy of abnormal behavior detection, but also fully explores the behavioral correlation between frames through time series modeling. The implementation process of the entire solution is efficient and flexible, and can adapt to the dynamic monitoring needs in different scenarios. At the same time, it has a low implementation complexity and can run smoothly in different hardware environments, providing an efficient and robust solution for abnormal behavior detection. Therefore, the present invention effectively overcomes the various shortcomings of the prior art and has a high industrial utilization value.

[0071] The above embodiments are merely illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Anyone familiar with the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by a person of ordinary skill in the art without departing from the spirit and technical concept disclosed by the present invention shall still be covered by the claims of the present invention.

Claims

1. A method for detecting abnormal behavior, characterized in that: The method comprises: Obtain a video to be identified, and extract multiple video frames from the video in sequence; wherein each video frame carries a corresponding time mark; Inputting each video frame and a preset behavior description set into a multimodal large language model, performing multimodal fusion on the input video frame and the behavior description set, and identifying abnormal behaviors existing in each video frame; wherein the behavior description set is used to indicate abnormal behaviors to be detected; The abnormal behaviors in each video frame are arranged according to the generation order, and are input into the behavior analysis model together with the corresponding time stamp to detect the abnormal behaviors in the video; wherein the behavior analysis model is a Transformer-based timing model.

2. The abnormal behavior detection method according to claim 1, characterized in that: The multimodal large language model is a MiniCPM-V model. For each video frame: the video frame and a preset behavior description set are input into the large language model, multimodal fusion is performed on the input video frame and the behavior description set, and abnormal behaviors in the video frame are identified, including: Extract visual features from video frames; Performing word segmentation and word embedding processing on each behavior description of the behavior description set to obtain a vector sequence of the behavior description set; Encoding the vector sequence to generate a text feature sequence; The visual features and the text feature sequence are multimodally fused to obtain fused features, and abnormal behaviors in the video frames are identified based on the fused features.

3. The abnormal behavior detection method according to claim 1, characterized in that: The behavior analysis model is a Transformer-based model. The abnormal behaviors in each video frame are arranged according to the generation order and input into the behavior analysis model together with the corresponding time mark to detect the abnormal behaviors in the video, including: Arrange the abnormal behaviors in each video frame according to the generation order. For each abnormal behavior in the video frame: find the index value corresponding to the abnormal behavior in the video frame, encode the corresponding index value, and generate a behavior embedding vector; The time tags corresponding to each video frame are respectively position-encoded to obtain the time embedding vector corresponding to each video frame; Fuse the time embedding vector with the corresponding behavior embedding vector, and arrange the fused vectors in order to obtain a fused vector sequence; Based on a multi-head self-attention mechanism, encoding the fused vector sequence to obtain an encoding result; The encoding result is mapped to a preset behavior category dimension to detect abnormal behaviors in the video.

4. The abnormal behavior detection method according to claim 3, characterized in that: The encoding process is performed on the fused vector sequence based on the multi-head self-attention mechanism to obtain an encoding result, including: For each attention head: Based on a preset query weight matrix, a key weight matrix and a value weight matrix, a linear transformation is performed on the fused vector sequence to generate a corresponding query matrix, a key matrix and a value matrix; Generate a global attention matrix based on the query matrix and the key matrix; Generate global features based on the global attention matrix and value matrix; Performing sparse processing on the global attention matrix corresponding to at least one attention head to obtain a sparse attention matrix; Generate sparse features based on the sparse attention matrix and the corresponding value matrix; The global features are concatenated, and the concatenated global features and sparse features are weighted to obtain the encoding result.

5. The abnormal behavior detection method according to claim 1, characterized in that: After detecting the abnormal behavior in the video, the method further includes: mapping the abnormal behavior to a corresponding video frame and presenting it in the video.

6. The abnormal behavior detection method according to claim 1, characterized in that: The method also includes: saving the extracted multiple video frames to a buffer, and upon receiving a completion signal sent by the multimodal large language model, inputting the video frames in the buffer to the multimodal large language model; wherein the multimodal large language model generates a completion signal after completing recognition of the input video frames.

7. The abnormal behavior detection method according to claim 1, characterized in that: The method further includes: detecting whether the video picture is abnormal from the extracted video frame, and sending an alarm message when an abnormality exists.

8. A system for detecting abnormal behavior, characterized in that: The system comprises: A video acquisition module, used to acquire a video to be identified, and sequentially extract multiple video frames from the video; wherein each video frame carries a corresponding time mark; The abnormal behavior initial detection module is used to input each video frame and a preset behavior description set into a multimodal large language model, perform multimodal fusion on the input video frame and the behavior description set, and identify the abnormal behavior in each video frame; wherein the behavior description set is used to indicate the abnormal behavior to be detected; The abnormal behavior determination module is used to arrange the abnormal behaviors in each video frame according to the generation order, and input them together with the corresponding time stamps into the behavior analysis model to detect the abnormal behaviors in the video; wherein the behavior analysis model is a Transformer-based timing model.

9. An electronic device, characterized in that: The electronic device comprises: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the electronic device to implement the abnormal behavior detection method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor of a computer, the computer is caused to execute the abnormal behavior detection method according to any one of claims 1 to 7.

Images