Abnormal behavior determination method, device, equipment, storage medium and program product

Abstract

The application provides an abnormal behavior judgment method and device, equipment, storage medium and program product, and relates to the field of artificial intelligence. The method comprises the following steps: calling a target detection model to perform target detection processing on each video frame in a video stream, so as to identify a detection target in the video frame and obtain position information of the detection target in the video frame; judging whether the detection target triggers a suspicious event based on the position information and a preset preliminary screening rule; if yes, generating a to-be-verified event package comprising a key image frame, position information and trigger context information, the key image frame being a video frame corresponding to a time when the detection target triggers the suspicious event, and the trigger context information being used for indicating the preliminary screening rule triggered by the detection target; calling a visual language model to output a verification result based on the to-be-verified event package, the verification result being used for indicating whether the detection target has abnormal behavior. The method of the application can reduce the false positive rate of abnormal behavior recognition and accurately identify the behavior intention.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to a method, apparatus, device, storage medium, and program product for determining abnormal behavior. Background Technology

[0002] With the widespread application of video surveillance systems in key security areas such as bank branches, rail transit, and commercial complexes, the automatic, accurate, and real-time detection of abnormal behavior in surveillance videos has become a crucial requirement in the public safety sector. Abnormal behavior typically refers to events that may threaten safety or disrupt order, such as people entering sensitive areas, lingering beyond their permitted time, damaging facilities, falling to the ground, or gathering unusually. Due to the large volume of surveillance video data, the complexity of the scenarios, and the diversity of behaviors, efficiently identifying truly threatening abnormal behavior from massive video streams while minimizing false alarms is the core challenge currently facing abnormal behavior detection technology.

[0003] In existing technologies, abnormal behavior detection mainly adopts the following schemes: First, schemes based on traditional video motion detection (VMD). These schemes detect moving objects by analyzing pixel changes between video frames, and trigger an alarm when the detected pixel changes exceed a preset threshold. Second, schemes based on single-target detection models. These schemes introduce target detection models such as the YOLO (You Only Look Once) series, first identifying and locating specific targets such as "people" in video frames in real time to obtain target bounding boxes, then assigning a unique tracking ID to each target using a target tracking algorithm, and relying on a simple rule engine (such as area intrusion, timeout, abnormal clustering, etc.) for anomaly judgment and alarm.

[0004] However, while traditional video motion detection solutions are simple to implement and computationally inexpensive, they are highly susceptible to environmental factors such as changes in lighting, shadows, swaying branches, and small animals crossing the area, easily generating a large number of invalid alarms. Solutions based on single-target detection models can identify target categories, but their judgment logic is entirely based on low-level rules such as geometric position and dwell time, lacking a deep semantic understanding of scene context and behavioral intent. For example, they cannot distinguish between "a person using a crowbar to damage an ATM" and "a bank employee using professional tools to repair an ATM," as both behave identically in rules such as "area intrusion" and "excessive stay," leading to a large number of normal operational behaviors being falsely reported as abnormal events. It is clear that existing abnormal behavior detection technologies rely solely on target detection and simple rule-based judgment, failing to understand the contextual information and behavioral semantics within the monitored scene, resulting in a high false alarm rate and an inability to accurately identify behavioral intent. Summary of the Invention

[0005] This application provides a method, apparatus, device, storage medium, and program product for determining abnormal behavior, in order to solve the technical problem that existing abnormal behavior detection technologies rely solely on target detection and simple rule determination, which cannot understand the contextual information and behavioral semantics in the monitoring scenario, resulting in a high false alarm rate for abnormal behavior recognition and an inability to accurately identify behavioral intent.

[0006] Firstly, this application provides a method for determining abnormal behavior, the method comprising:

[0007] The target detection model is invoked to perform target detection processing on each video frame in the video stream in order to identify the detected targets in the video frames and obtain the position information of the detected targets in the video frames;

[0008] Based on the location information and the preset initial screening rules, it is determined whether the detected target has triggered a suspicious event;

[0009] If so, a verification event packet is generated, including a key image frame, the location information, and triggering context information. The key image frame is the video frame corresponding to when the detected target triggers the suspicious event, and the triggering context information is used to indicate the initial screening rule triggered by the detected target.

[0010] The visual language model is invoked to output a verification result based on the event package to be verified. The verification result is used to indicate whether the detected target has abnormal behavior.

[0011] Secondly, this application provides an abnormal behavior determination device, the device comprising:

[0012] The initial screening module is used to call the target detection model to perform target detection processing on each video frame in the video stream, so as to identify the detected targets in the video frames and obtain the position information of the detected targets in the video frames;

[0013] Based on the location information and the preset initial screening rules, determine whether the detected target has triggered a suspicious event;

[0014] If so, a verification event packet is generated, including a key image frame, the location information, and triggering context information. The key image frame is the video frame corresponding to when the detected target triggers the suspicious event, and the triggering context information is used to indicate the initial screening rule triggered by the detected target.

[0015] The semantic verification module is used to call the visual language model to output a verification result based on the event package to be verified. The verification result is used to indicate whether the detected target has abnormal behavior.

[0016] Thirdly, this application provides an electronic device, including: a processor, and a memory communicatively connected to the processor;

[0017] The memory stores computer-executed instructions;

[0018] The processor executes computer execution instructions stored in the memory to implement the method as described in the first aspect.

[0019] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the method described in the first aspect.

[0020] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the method described in the first aspect.

[0021] The abnormal behavior determination method, apparatus, equipment, storage medium, and program product provided in this application have the following technical effects:

[0022] First, in the initial screening stage, this method only uses location information and preset screening rules to determine whether a suspicious event has been triggered. This process has low computational overhead and is fast, enabling rapid screening of massive video streams. Then, an event packet to be verified is generated only when a suspicious event is triggered, and a visual language model is invoked for secondary verification. This "rule-based initial screening followed by model verification" process ensures that the final abnormal behavior determination no longer relies solely on low-level rules such as geometric location and dwell time, but introduces a visual language model to perform semantic-level understanding of key image frames and triggering context information.

[0023] Specifically, the event package to be verified contains key image frames, location information, and triggering context information indicating the initial screening rule that has been triggered. The visual language model can understand both image content and text semantics simultaneously. Therefore, it can combine the context, such as the initial screening rule triggered by the detected target, to perform deep reasoning on the actual behavior of the detected target in the key image frame. For example, if the detected target triggers the initial screening rule "person stays in front of ATM for too long," the visual language model can distinguish between a suspicious person committing vandalism and a worker performing maintenance work based on the person's clothing, held items, and actions in the image. This overcomes the deficiency of existing technologies that "can only determine what happened, but cannot understand the intent and nature of the event." Thus, semantic verification of the event package to be verified using the visual language model can accurately identify normal behaviors that are misjudged by the initial screening rule, significantly reducing the false alarm rate of abnormal behavior detection. Furthermore, since the visual language model is only invoked after a rule is triggered, it avoids performing high-cost reasoning on a massive number of normal video frames, ensuring both detection accuracy and processing efficiency. Attached Figure Description

[0024] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0025] Figure 1 Flowchart of the abnormal behavior determination method provided in the embodiments of this application Figure 1 ;

[0026] Figure 2 Flowchart of the abnormal behavior determination method provided in the embodiments of this application Figure 2 ;

[0027] Figure 3 A schematic diagram of the abnormal behavior determination device provided in the embodiments of this application;

[0028] Figure 4 This is a schematic diagram of the electronic device structure provided in an embodiment of this application.

[0029] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0030] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0031] It should be noted that the abnormal behavior determination method, device, equipment, storage medium and program product provided in this application can be used in the field of artificial intelligence, or in any field other than artificial intelligence. The application field of the abnormal behavior determination method, device, equipment, storage medium and program product in this application is not limited.

[0032] To address the technical problem that existing abnormal behavior detection technologies rely solely on target detection and simple rule-based judgment, failing to understand the contextual information and behavioral semantics within the monitoring scenario, resulting in a high false alarm rate and an inability to accurately identify behavioral intent, the technical concept of this application is as follows:

[0033] First, a target detection model is used to identify and locate targets in the video stream in real time. Combined with location information and preset initial screening rules, it quickly determines whether a suspicious event has been triggered. Once triggered, a verification event packet is generated, containing key image frames, target location information, and trigger context information. The trigger context information indicates the type of initial screening rule triggered. Then, a visual language model is invoked to perform deep semantic understanding and reasoning on the event packet, ultimately outputting a verification result indicating whether the detected target exhibits abnormal behavior. By introducing a visual language model to perform secondary semantic verification of the initial screening result, the system can distinguish between real threats and normal interference by combining scene context and behavioral intent, thereby significantly reducing the false alarm rate while ensuring real-time detection.

[0034] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0035] Example 1

[0036] This embodiment provides a method for determining abnormal behavior, which can be executed by an edge computing device, a central server, or a combination of both. One possible deployment configuration is that a camera is responsible for capturing video streams, target detection and initial screening are performed by an edge computing device deployed close to the camera, and semantic verification of the visual language model is performed by a server in the cloud or a central data center. The edge computing device and the server can communicate via wired or wireless networks. When sufficient computing power is available, all functions can also be integrated into a single server or a dedicated abnormal behavior analysis device.

[0037] Figure 1 Flowchart of the abnormal behavior determination method provided in the embodiments of this application Figure 1 ,like Figure 1 As shown, the method includes:

[0038] S101. Call the target detection model to perform target detection processing on each video frame in the video stream, so as to identify the detected targets in the video frames and obtain the position information of the detected targets in the video frames;

[0039] In this step, the object detection model can adopt a two-stage object detection architecture based on deep learning. Specifically, the two-stage object detection architecture can include a region proposal network and a classification regression network. The region proposal network is used to generate candidate regions that may contain objects, and the classification regression network is used to perform category identification and accurate bounding box regression on the candidate regions.

[0040] The object detection model is pre-trained offline. The training data used in the pre-training can cover common security target categories such as human bodies, tools, safety helmets, and vehicles. In this embodiment, the input of the object detection model is video frame images, and the output is several detection results. Each detection result includes the target category, confidence score, and bounding box coordinates in normalized or pixel coordinate systems. When the object detection model is actually running, it can perform frame-by-frame detection on the video stream at a set frame rate (e.g., 5 frames per second) or perform full detection on consecutive video frames, which can be dynamically adjusted according to the computing power of the edge device.

[0041] Optionally, the video frames can be preprocessed before calling the object detection model. Preprocessing includes, but is not limited to, size normalization, noise reduction, and brightness equalization. The size of the preprocessed video frames is consistent with the input size during the training of the object detection model. For example, the original video frames can be scaled proportionally to a shorter side of 608 pixels and a longer side of no more than 1024 pixels.

[0042] To improve the robustness of the detection, this embodiment can also set a confidence threshold (e.g., 0.5) to exclude detection results with a confidence score below 0.5.

[0043] Furthermore, since object detection models typically generate multiple candidate boxes when predicting the same object, and these candidate boxes highly overlap and each has a confidence score, non-maximum suppression (NMS) can be used to select the most representative box from this set of candidate boxes and suppress other redundant boxes. The specific steps are as follows:

[0044] The first step is to sort all candidate detection boxes in descending order of confidence score;

[0045] The second step is to select the candidate detection box with the highest confidence as the retained detection box and add it to the final output set.

[0046] The third step is to calculate the intersection-union ratio (IUR) of the remaining candidate detection boxes with the retained detection box one by one. The IUR is equal to the ratio of the intersection area of ​​the two detection boxes to the union area, which is used to measure the degree of overlap between the two detection boxes.

[0047] Fourth, all candidate detection boxes with an intersection-union ratio greater than the preset suppression threshold are considered as duplicate detections of the same target and are suppressed and deleted.

[0048] Fifth, repeat steps two through four for the remaining unsuppressed candidate detection boxes until all candidate detection boxes have been processed.

[0049] In the final set of retained detection boxes, each target corresponds to only one high-confidence detection box, and the overlap between the detection boxes is controlled within a threshold, thereby effectively eliminating the interference caused by repeated detection to subsequent target association and rule determination.

[0050] In this step, the final output set of detected targets is denoted as . Each detection target Includes target category, confidence score, and bounding box coordinates in normalized or pixel coordinate system. ,in, This is the x-coordinate of the top-left corner of the bounding box in the image coordinate system. If pixels are used, it represents the number of pixels that the point is from the left edge of the image; if normalized coordinates are used, it is a ratio (0 to 1) relative to the image width. This is the ordinate of the top-left corner of the bounding box in the image coordinate system. If pixels are used, it represents the number of pixels from the top edge of the image; if normalized coordinates are used, it is the ratio (0-1) to the image height. This represents the width of the bounding box, i.e., the horizontal dimension of the bounding box. This represents the height of the bounding box, i.e., the vertical dimension of the bounding box.

[0051] S102. Based on location information and preset screening rules, determine whether the target has triggered a suspicious event;

[0052] In this step, to accurately determine the behavior of the detected target in continuous time and space, this embodiment first uses the intersection-over-union (IoU) ratio of adjacent detection boxes to associate the detected target, and then applies several initial screening rules based on this association result. Specifically:

[0053] For two adjacent frames in the video stream and Calculate the IoU between all detected targets in the previous frame and all detected targets in the current frame. If a certain pair of detected targets... (belong (frame) and (belong If frames belong to the same category and have an IoU greater than a preset threshold (e.g., 0.35), they are considered the same target, and the current frame's IoU is set to the same target. The new boundary frame received After an existing trajectory, the trajectory length is increased by one frame; if a target in the current frame does not match the same target in the previous frame, a new trajectory is generated with the bounding box of the target in the current frame as the starting point of the trajectory; if a target in the previous frame is not matched in the current frame, the trajectory is temporarily retained and matching attempts are continued in subsequent frames.

[0054] The trajectory data structure can be represented as:

[0055]

[0056] in, , This refers to the timestamps of the video frames corresponding to each bounding box in the trajectory.

[0057] In this embodiment, one or more sensitive areas can be pre-configured in the monitoring screen. Each sensitive area is defined by a polygonal outline or a combination of multiple rectangles, such as the ATM equipment area, the vault door passage area, and the computer room cabinet area. For the bounding box of any detected target, whether the target is "located" in that area can be determined by whether the center point of the bounding box falls within the sensitive area.

[0058] Regarding the initial screening rules, this embodiment can set the following three initial screening rules, which can be freely combined. Triggering any one of these rules is considered to have triggered a suspicious event:

[0059] Rule 1: Time Proportion Rule

[0060] This rule calculates the percentage of time a detected target's trajectory falls within a sensitive area, relative to the total trajectory duration. If this percentage exceeds a preset percentage threshold, and the total trajectory duration also exceeds a preset duration threshold, a suspicious event is triggered. For example, the preset percentage threshold could be 0.8, and the preset duration threshold could be 30 seconds. For instance, if a pedestrian appears in camera A's view for 40 seconds, and the cumulative time spent within the sensitive area is 34 seconds (85%), then this time percentage rule is triggered.

[0061] Rule 2: Repeated Approach Rule

[0062] This rule is used to identify probing behaviors targeting sensitive areas. Specifically, it counts the number of times within a preset time window (e.g., 5 minutes), the minimum distance between the edge of the target's bounding box and the boundary of the sensitive area is less than a preset pixel distance (e.g., 10 pixels). If this number exceeds a preset threshold (e.g., 5 times), a suspicious event is triggered.

[0063] For example, if someone loiters near an ATM, approaches it multiple times but stays for a short time each time, and their bounding box approaches the ATM area boundary a total of 8 times within 5 minutes, exceeding the threshold of 5 times, then the repeated approach rule is triggered.

[0064] Rule 3: The rule of simultaneous appearance of person and object:

[0065] This rule targets the unusual behavior of a target and a tool-like object appearing simultaneously. If a human target and a tool-like object are detected, and their IoU (Intersection over Union) in the same frame is greater than 0.2 or the distance between their bounding box center points is less than a preset distance (e.g., 50 pixels), and this co-occurrence relationship is maintained for at least 3 consecutive frames, then this human-object co-occurrence rule is triggered. This rule can capture the process of "a person holding a tool approaching a sensitive area".

[0066] When any of the above rules are triggered, proceed to step S103.

[0067] S103. Generate a verification event packet including key image frames, location information and triggering context information. The key image frames are the video frames corresponding to when the detected target triggers a suspicious event. The triggering context information is used to indicate the initial screening rules triggered by the detected target.

[0068] In this step, the key image frames specifically include the following three types:

[0069] Trigger frame: The video frame that corresponds to when the current rule is determined to be triggered.

[0070] Context frames: Several frames before and after the trigger frame, used to show the continuity before and after the event occurs. For example, you can take 10 frames before the trigger frame and 10 frames after the trigger frame, for a total of 21 frames.

[0071] Event stitching diagram: All the above frames are stitched together horizontally in chronological order to form a long image, which can be used as an optional input to allow the visual language model to view the whole picture of the event at once.

[0072] In this step, the location information includes the bounding box coordinates of the detected target in the current trigger frame, as well as the trajectory coordinate sequence of the detected target in the most recent preset time period. The trajectory coordinate sequence is a sequence of the bounding box coordinates of the detected target arranged in chronological order.

[0073] Trigger context information may include the following fields:

[0074]

[0075] Furthermore, to assist the visual language model in understanding the target behavior, this embodiment can also generate the following auxiliary images:

[0076] Target cropping image: The bounding box region of each relevant detected target is cropped from the key image frame and scaled to a fixed size (e.g., 224×224) to preserve the appearance details of the detected target;

[0077] Target mask image: Generate a binary mask on the key image frame that corresponds to the bounding box of the detected target, gray out non-target areas, and highlight the detected target itself;

[0078] Sensitive region annotation image: The sensitive regions involved in the triggering event are filled with a semi-transparent color on the key image frame, so that the visual language model can intuitively see the relative position of the detection target and the sensitive region.

[0079] Next, the aforementioned image data, along with location information and trigger context information, are encapsulated into a verification event packet. This verification event packet can be encoded using either protocol buffer encoding or JSON encoding. Protocol buffers are language-independent and platform-independent binary serialization formats. Their basic characteristic is the predefined data structure rules, such as "a certain field is an integer" or "a certain field is a byte array," and then the structured data is compactly encoded into a binary stream according to these rules. The advantages are small size after encoding, fast parsing speed, and low resource consumption, making it suitable for transmitting large data such as image frames or data with high real-time requirements. JSON is a text-based data exchange format that uses key-value pairs and nested structures to describe data. For example, an event packet can be represented as text containing fields such as "image frame Base64 encoding," "rule type," and "timestamp" for transmission. The advantages are clear structure, high readability, and high versatility; any system that supports text processing can easily generate or parse it. In practical applications, one can choose pure JSON, pure protocol buffers, or a combination of both: image frames are stored in the protocol buffer as large binary fields, and text-based metadata is represented using JSON-readable fields.

[0080] S104. Call the visual language model based on the event package to be verified, and output the verification result. The verification result is used to indicate whether there is abnormal behavior in the detected target.

[0081] It should be noted that a visual language model is a multimodal model that can simultaneously understand image content and natural language text. Visual language models can perform reasoning and decision-making based on input image-text pairs.

[0082] In this step, key image frames are first extracted from the event packet to be verified. Bounding boxes for all relevant targets are drawn on the image, and a unique numerical label (e.g., label 1, 2, 3) is added to each bounding box. Simultaneously, a semi-transparent filled outline of the sensitive region is overlaid on the image to indicate its location. This preprocessed image serves as the image input for the visual language model.

[0083] At the same time, the fields in the trigger context information will be converted into a natural language description, for example:

[0084] "The surveillance footage comes from a camera in the ATM area of ​​a bank branch. Time range: 10:30:20 to 10:31:20 on August 1, 2025. Target 1 is classified as a person, and its bounding box center point falls within the sensitive area (ATM operation area) for 85% of the time. The trigger rule type is a time ratio rule, the ratio threshold is 80%, and the total duration is 60 seconds. The target is marked with a red box and a number label in the figure."

[0085] Then, the above natural language description and the preprocessed image are input into the visual language model. Based on the joint image-text features, the visual language model outputs formalized verification results, including:

[0086] Behavior category determination: one of three categories, namely "normal behavior", "suspicious behavior" or "cannot be determined".

[0087] Confidence score: for example, 0.92.

[0088] Explanatory description: The model uses natural language to explain the basis for judgment, such as "The person is wearing a uniform and holding maintenance tools, and the maintenance tools correspond to the position of the ATM maintenance port. Based on this, it is judged to be normal maintenance behavior."

[0089] Figure 1 The method shown has the following technical effects:

[0090] First, in the initial screening stage, this method only uses location information and preset screening rules to determine whether a suspicious event has been triggered. This process has low computational overhead and is fast, enabling rapid screening of massive video streams. Then, an event packet to be verified is generated only when a suspicious event is triggered, and a visual language model is invoked for secondary verification. This "rule-based initial screening followed by model verification" process ensures that the final abnormal behavior determination no longer relies solely on low-level rules such as geometric location and dwell time, but introduces a visual language model to perform semantic-level understanding of key image frames and triggering context information.

[0091] Specifically, the event package to be verified contains key image frames, location information, and triggering context information indicating the initial screening rule that has been triggered. The visual language model can understand both image content and text semantics simultaneously. Therefore, it can combine the context, such as the initial screening rule triggered by the detected target, to perform deep reasoning on the actual behavior of the detected target in the key image frame. For example, if the detected target triggers the initial screening rule "person stays in front of ATM for too long," the visual language model can distinguish between a suspicious person committing vandalism and a worker performing maintenance work based on the person's clothing, held items, and actions in the image. This overcomes the deficiency of existing technologies that "can only determine what happened, but cannot understand the intent and nature of the event." Thus, semantic verification of the event package to be verified using the visual language model can accurately identify normal behaviors that are misjudged by the initial screening rule, significantly reducing the false alarm rate of abnormal behavior detection. Furthermore, since the visual language model is only invoked after a rule is triggered, it avoids performing high-cost reasoning on a massive number of normal video frames, ensuring both detection accuracy and processing efficiency.

[0092] Example 2

[0093] This embodiment provides another method for determining abnormal behavior. The execution subject of this method can be referred to in Embodiment 1. Figure 2 Flowchart of the abnormal behavior determination method provided in the embodiments of this application Figure 2 ,like Figure 2 As shown, the method includes:

[0094] S201. Call the target detection model to perform target detection processing on each video frame in the video stream, so as to identify the detected targets in the video frames and obtain the position information of the detected targets in the video frames;

[0095] In this step, at least one camera deployed within the monitoring area acquires a video stream in real time, which consists of several consecutive video frames. The acquired video stream is then decoded and preprocessed, including but not limited to noise reduction, dehazing, contrast enhancement, resolution normalization, and frame extraction.

[0096] After decoding and preprocessing the video stream, a pre-trained object detection model is invoked to perform object detection on each frame of the preprocessed video image. In this embodiment, the object detection model is a deep neural network-based visual detection model that can identify targets belonging to preset categories from the image, such as "people," "vehicles," and "bags," and output the category label of each target and the target's location information in the image. The location information can be represented in the form of bounding boxes, such as the center point coordinates, width, and height of the target bounding box, or the coordinates of the upper left and lower right corners of the target bounding box.

[0097] The target detection model can adopt a single-stage detection method or a two-stage detection method. This embodiment does not limit its specific network structure, as long as it can achieve real-time target detection.

[0098] S202. Perform target tracking processing on the detected target to obtain the dynamic state information of the detected target in multiple video frames;

[0099] Specifically, based on target detection, this step uses a target tracking algorithm to associate the detected targets in adjacent video frames, assigns a unique tracking identifier to each independent target, and maintains the tracking identifier for the duration of the target's continuous appearance, thereby forming the target's motion trajectory.

[0100] In this embodiment, the target tracking algorithm can be implemented based on deep feature matching or motion model prediction. For example, Kalman filtering can be used to predict the target's position in the next frame, and then the Hungarian matching algorithm can be used to correlate it with the detection result of the current frame. The target tracking algorithm can obtain the dynamic state information of each detected target.

[0101] In this embodiment, dynamic state information includes, but is not limited to: the target's motion trajectory in consecutive video frames; the target's motion speed, motion acceleration, and motion direction; whether the target enters, leaves, or crosses the boundary of a preset area; and the target's historical position sequence.

[0102] Optionally, the temporal information of the target's behavior can be acquired as dynamic state information. Specifically, during the target tracking process, in addition to acquiring the spatial dynamic state information of the target, such as its trajectory, speed, and direction, the temporal information of the target's behavior is also acquired and incorporated as an important component of the dynamic state information.

[0103] Specifically, the system assigns a unique tracking identifier to each detected target and records all temporal characteristics of the target from its first appearance in a video frame to its last disappearance. The temporal information of the target's behavior includes, but is not limited to, the following types:

[0104] Target appearance time information: the moment when the target is first identified by the target detection model and appears in the monitoring screen, and the total duration for which it appears in the monitoring screen;

[0105] Area dwell time information: the time when the bounding box of the detected target overlaps with the preset sensitive area, the time when it leaves the area, and the continuous dwell time in the area;

[0106] Behavior duration information: detect the start and end times of a target's specific behavior, such as the duration of a falling action from the start to the return to a normal posture, or the duration of a fast running action;

[0107] Time period attribute information: The time period attribute during which the target's current behavior occurs, such as daytime business hours, nighttime non-business hours, early morning hours, etc.

[0108] The system integrates the aforementioned time information with the spatial location and velocity information of the detected target to form complete dynamic state information of the target, which is then encapsulated as a target state vector. This target state vector is continuously updated as video frames are updated, ensuring the real-time nature and accuracy of the time information.

[0109] For example, in a bank ATM monitoring scenario, the system assigns a tracking identifier (1001) to a person entering the ATM maintenance area and records their entry time as 10:14:30 AM on the same day. When the person's cumulative stay in the area reaches 60 seconds, the initial screening rule engine determines that the "overstaying" rule is triggered based on this time information. Simultaneously, the system includes this time information as part of the trigger context information, encapsulating it along with key image frames, location information, and monitoring scene context information into a verification event packet. In the subsequent semantic verification stage, the visual language model can combine this time information with the time period attribute in the monitoring scene context to determine whether the person is performing suspicious operations outside of business hours or conducting normal equipment maintenance during business hours, thus outputting a more accurate verification result.

[0110] Furthermore, time information can also be used to generate analysis prompts. For example, when the trigger rule is "area intrusion" and the time is 2:00 AM, the system will specifically emphasize in the analysis prompt, "This is a non-business period; please focus on determining whether the person has engaged in illegal intrusion," to guide the visual language model to pay attention to the impact of the time dimension on the determination of the nature of the behavior.

[0111] It should be noted that since time information reflects the temporal characteristics and time context of the target behavior, it is passed on as part of the dynamic state information to the visual language model. This enables the visual language model to judge the nature of the behavior by integrating multi-dimensional information of time and space, as well as visual and semantic information. For example, the same maintenance behavior is usually considered normal work when it occurs during business hours, but the degree of suspicion increases significantly when it occurs outside business hours. Therefore, introducing time information can improve the accuracy of the visual language model in judging abnormal behavior.

[0112] S203. Based on location information and dynamic status information, determine whether the detection target triggers the initial screening rule;

[0113] This step can be performed by the initial screening rule engine. Specifically, the initial screening rule engine is pre-configured with one or more initial screening rules. The initial screening rule engine receives the location information output by the target detection module and the dynamic status information obtained in the aforementioned steps in real time, and performs logical judgments according to the rule definitions.

[0114] For example, the initial screening rules include at least one or more of the following rules:

[0115] Area intrusion rule: One or more sensitive areas are pre-defined in the monitoring screen, such as the inside area of ​​the bank counter, the ATM equipment maintenance area, the vault entrance area, etc. When the overlap area between the bounding box of the detected target and the sensitive area exceeds the preset threshold or the center point of the target falls into the sensitive area, the area intrusion rule is triggered.

[0116] Timeout rule: When the continuous dwell time of the detected target in a specific area exceeds a preset time threshold, such as 60 seconds, 90 seconds or 120 seconds, the timeout rule is triggered.

[0117] Abnormal posture / speed rule: When the movement speed of the detected target is abnormally higher or lower than the normal movement speed range of pedestrians, or when the detected target exhibits posture characteristics such as falling, climbing, or crouching, the abnormal posture / speed rule is determined to be triggered.

[0118] Clustering rule: When the number of detected targets in a specific area exceeds a preset threshold, the clustering rule is triggered.

[0119] If the detected target triggers any of the above rules, it is determined that the detected target has triggered a suspicious event, and S204 continues to be executed. It should be noted that the initial screening rule determination is based on this dynamic state information and location information. The dynamic state information can provide richer contextual information, thus allowing for the setting of more refined initial screening rules. For example, the timeout rule must be based on the continuous dwell time of the target in a specific area, and the area intrusion rule needs to combine the target's movement trajectory to distinguish whether the target is passing through briefly or continuously intruding. By introducing dynamic state information, the screening accuracy in the initial screening stage can be improved.

[0120] Dynamic status information can provide temporal context for determining the initial screening rules.

[0121] S204. Obtain the monitoring scene context information when the detected target triggers a suspicious event, and generate a verification event packet including key image frames, location information, trigger context information and monitoring scene context information;

[0122] In this step, the system obtains the following four types of information to generate the event packet to be verified:

[0123] Key image frames: These can be one or more frames extracted from the video stream at the moment the suspicious event is triggered. Key image frames contain bounding box annotations of the detected target, or they can be video clips containing the detected target, which are used for visual analysis by the subsequent visual language model.

[0124] Location information: refers to the bounding box coordinates of the detected target in the key image frame, as well as information such as the camera number and the position of the target in the frame;

[0125] Trigger context information: This refers to information directly related to the triggering event, such as the rule type (e.g., "timeout" or "area intrusion"), trigger time, target tracking identifier, and target category.

[0126] Monitoring scene context information: refers to the semantic information of the monitoring scene where the target is located, including but not limited to camera identification, timestamp, scene type (e.g., ATM area, counter area, vault area), location of preset sensitive areas within the monitoring scene, historical event statistics of the monitoring scene, and current traffic flow.

[0127] The system encodes and encapsulates the above four types of information according to a preset data structure, generates an event packet to be verified, and transmits the data packet to subsequent steps for analysis.

[0128] In this step, the introduction of contextual information about the monitoring scene enables subsequent semantic analysis to fully incorporate the environmental background in which the event occurs. For example, the same behavior occurring in an ATM repair area may have drastically different security implications than occurring in a sales counter area. Contextual information such as pedestrian traffic in different scenes during the same time period and historical event statistics can also help the visual language model to more accurately determine the nature of the event, thereby improving the accuracy of the visual language model's judgment.

[0129] S205. Generate analysis prompts based on the trigger context information and the monitoring scenario context information;

[0130] In one possible implementation, a large language model can be used to generate analytical prompts. Leveraging the powerful semantic understanding and natural language generation capabilities of the large language model, analytical prompts highly relevant to the currently triggered suspicious event can be automatically generated based on the triggering context information in the event package to be verified and the monitoring scene context information. This further improves the accuracy and adaptability of the visual language model verification. It should be noted that the "large language model" in this embodiment refers to a pre-trained natural language processing model that takes plain text as input and output. Its core function lies in semantic modeling and generation at the text level, which differs from the aforementioned "visual language model" responsible for joint image and text reasoning in terms of model architecture and input modality. Here, the large language model is specifically used to transform structured contextual information into high-quality, natural language analytical prompts.

[0131] Specifically, this implementation includes the following sub-steps:

[0132] 1. Parse and extract trigger context information and monitoring scene context information from the event package to be verified, and perform structured encoding on the discrete fields to form input data that can be processed by a large language model. The trigger context information includes at least: trigger rule type, trigger time, target identifier, target category, target bounding box coordinates, etc.; the monitoring scene context information includes at least: scene type, camera identifier, description of preset sensitive areas, time period attributes, and regional historical event statistics, etc.

[0133] Next, the above fields are organized according to a preset format, such as converting them into JSON objects or key-value pair sequences, and the format of time expression is standardized. For time information that requires semantic understanding, it can be mapped to natural language descriptions, such as "morning business hours" and "non-business hours", to facilitate understanding by large language models.

[0134] Second, to ensure that the prompt words output by the large language model meet business requirements, this embodiment uses a prompt word generation instruction template to embed structured information into a natural language instruction, and explicitly defines the output format, content elements, and style constraints within the instruction. In this embodiment, the instruction template includes at least the following information:

[0135] Background description: Describe the scene type and camera location where the event occurred, for example, "in the bank ATM area, the camera number is CAM_ATM_01";

[0136] Event Description: Describe the event triggered during the initial screening stage, such as "a person stays in this area for more than 60 seconds";

[0137] Analysis requirements: The generated analysis prompts must guide the visual language model to focus on specific visual cues, such as "wearing work clothes," "holding tools," or "current operation."

[0138] Output format constraints: The generated analysis prompts must include three elements: "scenario description", "event description", and "analysis task", and be listed in bullet points.

[0139] Style and length constraints: The language should be formal and concise, and the word count should be kept within the preset range.

[0140] For example, the following instructions can be generated:

[0141] "Based on the following monitoring event information, generate an analysis prompt for the visual analysis model. Monitoring scenario: Bank ATM area, camera number CAM_ATM_01. Trigger rule: Timeout. Event description: A person stayed in the ATM maintenance area for more than 60 seconds. Generation requirements: 1. The prompt should include three parts: scenario, event, and task; 2. The task part should guide the analysis to determine whether the person is wearing work clothes, holding tools, and performing what operation, and to determine whether it is normal maintenance or suspicious behavior; 3. The language should be concise, directly outputting the prompt content without any explanation."

[0142] Third, input the above prompt word generation instructions into a pre-trained large language model. The large language model will then infer based on the context information provided in the instructions to generate analytical prompt words in natural language form. During the inference process, the generation quality can be improved through the following methods:

[0143] Few-sample guidance: Include a few examples in the instructions, such as providing reference prompts for the two rules of "timeout" and "regional intrusion", to guide the large language model to learn the desired output format and level of detail;

[0144] Decoding strategy control: Top-p sampling, temperature adjustment and other decoding strategies are adopted to ensure the diversity of output while controlling the stability and readability of the generated text;

[0145] Candidate generation and selection: Multiple candidate suggestions are generated using beam search, and the optimal output is selected by a scoring function (such as text length, keyword coverage, etc.).

[0146] For example, the analysis prompts output by the large language model are:

[0147] "[Scenario] Bank ATM area, camera number CAM_ATM_01; "

[0148] [Event] The person marked with a box in the image has stayed in the ATM maintenance area for more than 60 seconds, triggering the timeout rule;

[0149] [Task] Please analyze the person's behavior, focusing on: 1. Whether they are wearing work clothes; 2. Whether they are carrying special tools; 3. What operation are they performing on the ATM? 4. Determine whether this behavior constitutes 'normal maintenance' or 'suspicious behavior,' and provide your reasoning.

[0150] IV. To ensure the usability of the generated analysis prompts, a verification step can be included in this embodiment. Specifically, the system can perform text parsing on the generated analysis prompts to check whether they contain necessary elements, such as whether they simultaneously contain the three semantic blocks of "scenario," "event," and "task," and whether they contain key discriminative words such as "normal maintenance" and "suspicious behavior." If the verification passes, the analysis prompt is input as the final result to step S206; if the verification fails, a rollback operation is performed, that is, the large language model is called again to generate, or a preset template can be used to quickly generate backup prompts.

[0151] In another possible implementation, analysis prompts are generated based on the trigger context information and the monitoring scene context information, including:

[0152] Generate a monitoring scene description text based on the monitoring scene type indicated by the monitoring scene context information;

[0153] Based on the triggering context information, generate event description text and analysis task text corresponding to the initial screening rule triggered by the detection target;

[0154] Combine the monitoring scene description text, event description text, and analysis task text into analysis prompt words.

[0155] In this implementation, the monitoring scene context information includes a monitoring scene type field. The monitoring scene type indicates the semantic category of the monitored area where the detection target is located, such as "ATM area," "bank lobby," "vault entrance area," "self-service equipment area," and "counter area." The system can retrieve or dynamically generate corresponding monitoring scene description text from a preset scene description knowledge base based on the monitoring scene type. The monitoring scene description text concisely summarizes the environmental background and functional attributes of the scene in natural language.

[0156] For example:

[0157] When the monitoring scene type is "ATM area", the generated monitoring scene description text is: "Bank ATM self-service area";

[0158] When the monitoring scene type is "vault entrance area", the generated monitoring scene description text is: "bank vault entrance and exit area";

[0159] When the monitoring scene type is "business counter area", the generated monitoring scene description text is: "bank business hall counter service area".

[0160] Furthermore, the monitoring scene context information can also include additional attribute information about the scene, such as the current time period (business hours / non-business hours), and descriptions of the location of sensitive areas within the scene. The system can integrate this additional attribute information into the monitoring scene description text, for example, generating: "Bank ATM self-service area, non-business hours". In this way, the visual language model can obtain a richer scene background, which can help the visual language model judge the rationality of the target behavior.

[0161] In this implementation, the triggering context information must include at least the type of the initial screening rule. Different initial screening rule types correspond to different event description methods and analysis task focuses. In this embodiment, the system can preset description templates and task templates corresponding to each initial screening rule type, thereby generating event description text based on the description templates and analysis task text based on the task templates.

[0162] Specifically, the generation methods for event description text and analysis task text are as follows, depending on the type of initial screening rule:

[0163] When the initial screening rule is a regional intrusion rule, the triggering context information can also include the sensitive area identifier of the target intrusion. The generated event description text can be, for example, "The person in the picture has entered the preset sensitive area"; the generated analysis task text can be, for example, "Please determine the purpose of the person entering the area, whether they are performing unauthorized operations, whether there is a security threat, and give a comprehensive judgment conclusion."

[0164] When the initial screening rule is the timeout rule, the trigger context information can also include the target's dwell time. The generated event description text could be, for example, "The person in the picture has stayed in the current area for more than the preset time (e.g., 60 seconds)". The generated analysis task text could be, for example, "Please pay close attention to: 1. Whether the person is wearing work clothes; 2. Whether the person is holding tools; 3. What kind of operation is the person performing on the on-site equipment; 4. Make a comprehensive judgment on whether the behavior is normal work behavior or suspicious behavior."

[0165] When the initial screening rule is an abnormal posture / velocity rule, the triggering context information can also include a description of the target posture category or velocity characteristics. The generated event description text could be, for example, "An abnormal posture such as falling / running / violent movement was detected in the figure"; the generated analysis task text could be, for example, "Please analyze whether the person's current state is in a dangerous situation, determine whether emergency action is required, and explain the basis for the judgment."

[0166] When the initial screening rule is an aggregation rule, the triggering context information can also include the location information of the aggregation area and the number of aggregation targets. The generated event description text can be, for example, "Multiple people gathered in the area shown in the figure in a short period of time"; the generated analysis task text can be, for example, "Please analyze whether there are disputes, fights, looting or other abnormal group events at the scene, and give a comprehensive judgment conclusion."

[0167] Furthermore, the event description text can carry location indication information of the detected target. For example, the bounding box information of the detected target in the key image frame can be converted into a text description, forming text fragments such as "the person marked by the red box in the picture". This makes it easier for the visual language model to focus the analysis on the correct target.

[0168] Finally, the system combines the monitoring scene description text, event description text, and analysis task text into a complete and structured analysis prompt text according to the preset assembly logic. There are multiple ways to combine them. For example, they can be spliced ​​in order and different parts can be distinguished by using segmentation marks. For example, the monitoring scene description text can be guided by "Scenario:", the event description text can be guided by "Event:", and the analysis task text can be guided by "Task:". Each part is separated by a newline or other separator.

[0169] Alternatively, the three parts can be combined into a single coherent natural language text, such as: "In the bank ATM area, the person marked by the box in the image has been there for more than 60 seconds. Please determine whether their behavior is normal maintenance, focusing on whether they are wearing work clothes, whether they are holding tools, and what kind of operation they are performing."

[0170] The generated analysis prompts are as follows:

[0171] Scenario: Bank ATM self-service area, outside of business hours;

[0172] Event: The person marked in red in the image stayed in the area for more than 60 seconds, triggering the timeout rule;

[0173] Task: Please analyze this person's behavior. Focus on: 1. Are they wearing work clothes? 2. Are they carrying tools? 3. What operation are they performing on the ATM? 4. Determine whether this behavior is 'normal maintenance' or 'suspicious behavior,' and provide your reasoning.

[0174] The generated analysis prompts, along with key image frames from the event package to be verified, are input into the visual language model to complete subsequent semantic reasoning.

[0175] It should be noted that in the above solution, by combining the monitoring scene description text, event description text, and analysis task text and inputting them into the visual language model, the visual language model can clearly understand the scene background, specific content of the event, and the key points of analysis that need to be focused on, thus avoiding irrelevant image areas from distracting the model's attention and improving the accuracy of judging the target behavior.

[0176] S206. Input the key image frames and analysis prompts into the visual language model so that the visual language model can perform semantic reasoning on the behavior of the detected target based on the key image frames and analysis prompts, and output the verification results.

[0177] In this step, the key image frames generated in S204 and the analysis prompts generated in S205 are jointly input into a pre-defined visual language model. The visual language model is a multimodal deep learning model capable of simultaneously understanding image content and natural language text, and performing joint reasoning based on both. Specifically, in this step, the visual language model performs semantic analysis on the input data, including but not limited to:

[0178] Identify the identity attributes of targets in key image frames, such as whether the person is wearing specific work clothes, carrying tools, or is a resident staff member in the scene;

[0179] Identify the target's ongoing actions and the objects being manipulated, such as repairing equipment, withdrawing cash, or committing sabotage.

[0180] By combining the task instructions in the analysis prompts, a comprehensive judgment is made on the behavioral intent and the nature of the event;

[0181] Output a verification result in natural language text format, including analysis and description of key visual features and final judgment conclusion.

[0182] In this embodiment, for example, the verification result may take the form of: "The person in the picture is wearing a 'XX Bank Maintenance' uniform, holding a special tool, and the ATM machine casing is open, indicating that equipment maintenance is being performed. Judgment: Normal maintenance." or "The person in the picture is using an iron tool to pry open the ATM machine's cash dispenser, demonstrating a clear intent to sabotage. Judgment: Suspicious behavior."

[0183] After the visual language model outputs the verification results, the system parses the verification results and extracts key semantic information, such as whether it contains judgment words such as "normal", "abnormal", "suspicious", and "dangerous", as well as the corresponding confidence scores.

[0184] It's important to note that by constructing analytical prompts, the reasoning process of the visual language model gains clear scene-specificity and task orientation. Specifically, the analytical prompts incorporate contextual information such as trigger rule type, scene type, and event description, guiding the visual language model to focus on the most critical visual discriminative elements of the current event. For example, for an overstaying event, the prompts guide the model to focus on whether the target is wearing work clothes, carrying tools, and performing what operation; for a gathering event, the prompts guide the model to focus on whether there are signs of conflict such as disputes or fights at the scene. This task-oriented prompting method can significantly improve the accuracy and efficiency of visual language model reasoning.

[0185] S207. If the verification result indicates that the detected target has abnormal behavior, an alarm message will be output.

[0186] In this step, the system performs decision processing on the verification results output by the visual language model in S206. Specifically, if the verification results include abnormal behavior judgments, such as "suspicious behavior," "destructive behavior," or "dangerous behavior," an alarm process is immediately triggered to generate alarm information. In this embodiment, the alarm information may include the following:

[0187] The camera identifier and location information that triggered the alarm;

[0188] Alarm time;

[0189] Key image frames or video clips;

[0190] Semantic analysis text generated by a visual language model, such as "VLM analysis: Suspicious individuals are using iron tools to sabotage an ATM";

[0191] Auxiliary information such as the rule types triggered during the initial screening stage.

[0192] The system further pushes alarm information to the security center, relevant management personnel, or mobile terminals, and can trigger on-site warning operations such as audible and visual alarms, so that security personnel can quickly understand the situation and respond based on the alarm information.

[0193] Furthermore, if the verification result indicates that the behavior of the detected target is normal, such as "under repair", "under cleaning" or "normal withdrawal", the system will mark the event as "verified - normal" and archive the event along with the verification result without outputting alarm information.

[0194] Figure 3 This is a schematic diagram of the abnormal behavior determination device provided in the embodiments of this application, such as... Figure 3 As shown, the device 30 includes:

[0195] The initial screening module 301 is used to call the target detection model to perform target detection processing on each video frame in the video stream, so as to identify the detected targets in the video frames and obtain the position information of the detected targets in the video frames;

[0196] Based on location information and preset screening rules, determine whether the target has triggered a suspicious event;

[0197] If so, a verification event packet is generated, which includes key image frames, location information, and triggering context information. The key image frames are the video frames corresponding to when the detected target triggers a suspicious event, and the triggering context information is used to indicate the initial screening rules triggered by the detected target.

[0198] The semantic verification module 302 is used to call the visual language model based on the event package to be verified and output the verification result. The verification result is used to indicate whether the detected target has abnormal behavior.

[0199] Figure 4 This is a schematic diagram of the electronic device structure provided in the embodiments of this application, such as... Figure 4 As shown, the device 40 includes at least one processor 401 and a memory 402. Optionally, the device 40 also includes a communication component 403. The processor 401, memory 402, and communication component 403 are connected via a bus 404.

[0200] In a specific implementation, at least one processor 401 executes computer execution instructions stored in memory 402, causing at least one processor 401 to perform the above-described method.

[0201] The specific implementation process of processor 401 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0202] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0203] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0204] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this application.

[0205] It should be further noted that although the steps in the flowchart are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0206] It should be understood that the above-described device embodiments are merely illustrative, and the device of this application can also be implemented in other ways. For example, the division of units / modules in the above embodiments is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units, modules, or components may be combined, or integrated into another system, or some features may be ignored or not executed.

[0207] Furthermore, unless otherwise specified, the functional units / modules in the various embodiments of this application can be integrated into one unit / module, or each unit / module can exist physically separately, or two or more units / modules can be integrated together. The integrated units / modules described above can be implemented in hardware or as software program modules.

[0208] When an integrated unit / module is implemented in hardware, the hardware can be digital circuits, analog circuits, etc. The physical implementation of the hardware structure includes, but is not limited to, transistors, memristors, etc. Unless otherwise specified, the processor can be any suitable hardware processor, such as a CPU, GPU, FPGA, DSP, and ASIC, etc. Unless otherwise specified, the storage unit can be any suitable magnetic or magneto-optical storage medium, such as Resistive Random Access Memory (RRAM), Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), Enhanced Dynamic Random Access Memory (EDRAM), High-Bandwidth Memory (HBM), Hybrid Memory Cube (HMC), etc.

[0209] If the integrated unit / module is implemented as a software program module and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0210] In the above embodiments, the descriptions of each embodiment have their own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

[0211] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0212] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A method for determining abnormal behavior, characterized in that, The method includes: The target detection model is invoked to perform target detection processing on each video frame in the video stream in order to identify the detected targets in the video frames and obtain the position information of the detected targets in the video frames; Based on the location information and the preset initial screening rules, determine whether the detected target has triggered a suspicious event; If so, a verification event packet is generated, including a key image frame, the location information, and triggering context information. The key image frame is the video frame corresponding to when the detected target triggers the suspicious event, and the triggering context information is used to indicate the initial screening rule triggered by the detected target. The visual language model is invoked to output a verification result based on the event package to be verified. The verification result is used to indicate whether the detected target has abnormal behavior.

2. The method according to claim 1, characterized in that, The step of determining whether the detected target triggers a suspicious event based on the location information and preset initial screening rules includes: The detected target is subjected to target tracking processing to obtain the dynamic state information of the detected target in multiple video frames; Based on the location information and the dynamic status information, it is determined whether the detected target triggers the initial screening rule; If the target being detected triggers the initial screening rule, then it is determined that the target being detected has triggered the suspicious event.

3. The method according to claim 2, characterized in that, The step of obtaining the dynamic state information of the detected target in multiple video frames includes: The time information of the target behavior of the detected target is obtained as the dynamic state information.

4. The method according to claim 1, characterized in that, The generation of the event packet to be verified, which includes key image frames, location information, and triggering context information, includes: Obtain the monitoring scene context information when the detected target triggers the suspicious event; Generate the event packet to be verified, which includes the key image frame, the location information, the trigger context information, and the monitoring scene context information.

5. The method according to claim 4, characterized in that, The invocation of the visual language model outputs verification results based on the event packet to be verified, including: Based on the trigger context information and the monitoring scene context information, generate analysis prompt words; The key image frame and the analysis prompt are input into the visual language model, so that the visual language model can perform semantic reasoning on the behavior of the detected target based on the key image frame and the analysis prompt, and output the verification result.

6. The method according to claim 5, characterized in that, The step of generating analysis prompt words based on the trigger context information and the monitoring scene context information includes: Based on the monitoring scenario type indicated by the monitoring scenario context information, generate monitoring scenario description text; Based on the triggering context information, generate event description text and analysis task text corresponding to the initial screening rule triggered by the detection target; The monitoring scene description text, the event description text, and the analysis task text are combined to form the analysis prompt words.

7. The method according to claim 1, characterized in that, After outputting the verification result, the method further includes: If the verification result indicates that the detected target exhibits abnormal behavior, an alarm message will be output.

8. An abnormal behavior determination device, characterized in that, The device includes: The initial screening module is used to call the target detection model to perform target detection processing on each video frame in the video stream, so as to identify the detected targets in the video frames and obtain the position information of the detected targets in the video frames; Based on the location information and the preset initial screening rules, determine whether the detected target has triggered a suspicious event; If so, a verification event packet is generated, including a key image frame, the location information, and triggering context information. The key image frame is the video frame corresponding to when the detected target triggers the suspicious event, and the triggering context information is used to indicate the initial screening rule triggered by the detected target. The semantic verification module is used to call the visual language model to output a verification result based on the event package to be verified. The verification result is used to indicate whether the detected target has abnormal behavior.

9. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 7.

11. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method of any one of claims 1 to 7.

Images