Abnormal behavior detection method and device, equipment and storage medium

Abstract

The invention belongs to the field of big data processing, and particularly relates to an abnormal behavior detection method and device, equipment and a storage medium, and the method comprises the steps: obtaining an image sequence of a target region, extracting feature regions of multiple objects in multiple frames of continuous images contained in the image sequence, determining the target feature trajectory data of each object, obtaining the spatial motion data of the target region, and obtaining the spatial motion data of the target region; the spatial motion data is sensing data generated when an object performs physical motion in a target area, aligning the target feature trajectory data and the spatial motion data according to a timestamp, generating time-synchronized multi-modal time sequence data, inputting the multi-modal time sequence data into a pre-trained abnormal behavior detection model, and outputting a detection result, the detection result is whether an abnormal behavior exists or not and the target object with the abnormal behavior; according to the method, accurate and synchronous abnormal behavior and corresponding target object detection is realized by fusing the image and sensing multi-modal data, and the detection accuracy and reliability are improved.

Description

Technical Field

[0001] This application relates to the field of big data processing, and in particular to an abnormal behavior detection method, apparatus, device, and storage medium. Background Technology

[0002] In today's society, elevators are indispensable vertical transportation tools in densely populated places such as residential buildings, office buildings, and shopping malls. Their safety performance is closely linked to the safety of people's lives and property. Analysis and statistics of numerous elevator accidents have revealed a close and direct correlation between improper elevator riding behavior and elevator accidents. Among these, the act of passengers jumping inside elevators is extremely dangerous. If this behavior is not detected and stopped in time, it can lead to elevator damage, triggering safety devices and trapping people, or even causing the elevator to plummet, resulting in a catastrophic accident and irreparable loss of life and property.

[0003] Currently, the identification of passengers jumping behavior in elevators mainly relies on video surveillance systems. By analyzing the image sequences captured by cameras, the posture and spatial depth information of key human points are extracted, and combined with a preset human motion model, it is determined whether there are abnormal jumping actions.

[0004] However, existing technologies consume a lot of resources due to direct real-time processing of video images, and relying solely on image analysis results in a high false negative rate and low accuracy in jumping behavior detection, leading to low accuracy in abnormal behavior recognition results. Summary of the Invention

[0005] This application provides an abnormal behavior detection method, apparatus, device, and storage medium to solve the problem of low accuracy in abnormal behavior identification results.

[0006] In a first aspect, embodiments of this application provide an abnormal behavior detection method, including:

[0007] Obtain an image sequence of the target region, extract feature regions of multiple objects from multiple consecutive images in the image sequence, and determine the target feature trajectory data of each object;

[0008] Acquire spatial motion data of the target area, wherein the spatial motion data is sensor data generated when an object performs physical actions within the target area;

[0009] Align the target feature trajectory data and the spatial action data according to the timestamp to generate time-synchronized multimodal time-series data;

[0010] The multimodal time-series data is input into a pre-trained abnormal behavior detection model, and the detection results are output, which indicate whether abnormal behavior exists and the target object where the abnormal behavior occurs.

[0011] As an optional implementation, the step of extracting feature regions of multiple objects from multiple consecutive frames of images contained in the image sequence and determining the target feature trajectory data of each object includes:

[0012] Detect the feature regions of each object in any frame image, determine the coordinate information of the feature regions in the current frame image, and define the feature search range based on the coordinate information in the current frame image;

[0013] Detect the feature regions of each object in the previous frame of the current frame image, and determine the coordinate information of the feature regions of each object in the previous frame image;

[0014] Based on the coordinate information of the feature region in the previous frame image and the coordinate information of the feature region in the current frame image, the matching cost of two adjacent frames is calculated. Based on the matching cost, the feature regions of each object in the two adjacent frames are matched to determine the target feature trajectory data of each object in multiple consecutive frames.

[0015] As an optional implementation, defining the feature search range based on coordinate information in the current frame image includes:

[0016] Extract the first horizontal coordinate, the second horizontal coordinate, the first vertical coordinate, and the second vertical coordinate from the coordinate information of the feature region in the current frame image, wherein the first horizontal coordinate is less than the second horizontal coordinate, and the first vertical coordinate is greater than the second vertical coordinate;

[0017] Based on the first and second ordinates, calculate half the height of the feature region to determine the search range for the ordinates;

[0018] Determine the reference range of the horizontal coordinate based on the first horizontal coordinate and the second horizontal coordinate;

[0019] The feature search range is defined based on the horizontal coordinate reference range and the vertical coordinate search range.

[0020] As an optional implementation, the step of calculating the matching cost between two adjacent frames based on the coordinate information of the feature regions in the previous frame and the coordinate information of the feature regions in the current frame, and matching the feature regions of each object in the two adjacent frames based on the matching cost, includes:

[0021] Extract the third horizontal coordinate, fourth horizontal coordinate, third vertical coordinate, and fourth vertical coordinate from the coordinate information of the feature region in the previous frame image, wherein the third horizontal coordinate is less than the fourth horizontal coordinate, and the third vertical coordinate is greater than the fourth vertical coordinate;

[0022] If the third ordinate is within the search range of the ordinate, and either the third or fourth abscissa is within the reference range of the abscissa, then the matching cost is calculated and the minimum matching cost is determined.

[0023] The feature region in the previous frame image corresponding to the minimum matching cost is determined as the successfully matched feature region, forming continuous target feature trajectory data.

[0024] As an optional implementation, aligning the target feature trajectory data and the spatial motion data according to timestamps to generate time-synchronized multimodal time-series data includes:

[0025] The target feature trajectory data is processed to calculate the change in the ordinate value at adjacent preset time points, forming a change sequence;

[0026] The change sequence is symbolically aggregated, and consecutive change values ​​with the same symbol are merged to form a characteristic change pattern;

[0027] Based on the suspected abnormal time periods in the aforementioned feature change patterns, spatial action data with corresponding timestamps are extracted.

[0028] The target feature trajectory data and spatial motion data are length aligned, and interpolation is used to make the data volume of the two consistent.

[0029] The aligned data are combined into multimodal time series data, which includes spatial motion data and relative coordinate changes of feature regions.

[0030] As an optional implementation, the step of inputting the multimodal time-series data into a pre-trained abnormal behavior detection model and outputting detection results includes:

[0031] The multimodal time series data is length-normalized, and sequences shorter than the preset length are padded with zeros and a corresponding length mask is generated to obtain the normalized multimodal time series data.

[0032] The standardized multimodal time-series data is input into a pre-trained abnormal behavior detection model, which outputs the probability of abnormal behavior.

[0033] When the probability of abnormal behavior exceeds a preset threshold, it is determined that abnormal behavior exists and the target object of the abnormal behavior is identified.

[0034] As an optional implementation, the method for constructing the abnormal behavior detection model includes:

[0035] Construct a training dataset, which includes multimodal time-series sample data labeled with normal and abnormal behaviors;

[0036] Construct a model architecture, which includes a feature extraction layer, a modality fusion layer, and a classification layer;

[0037] The model architecture is trained using the training dataset, and the prediction loss is calculated using the cross-entropy loss function to construct an abnormal behavior detection model.

[0038] Secondly, embodiments of this application provide an abnormal behavior detection device, comprising:

[0039] The determination module is used to acquire an image sequence of the target region, extract feature regions of multiple objects from multiple consecutive images in the image sequence, and determine the target feature trajectory data of each object;

[0040] The acquisition module is used to acquire spatial motion data of the target area, wherein the spatial motion data is the sensor data generated when the object performs physical actions in the target area;

[0041] The generation module is used to align the target feature trajectory data and the spatial action data according to the timestamp to generate time-synchronized multimodal time-series data;

[0042] The determining module is further configured to input the multimodal time series data into a pre-trained abnormal behavior detection model and output detection results, wherein the detection results are whether abnormal behavior exists and the target object in which the abnormal behavior occurs.

[0043] As an optional implementation, the determining module is further configured to detect the feature regions of each of the objects in any frame image, determine the coordinate information of the feature regions in the current frame image, and delineate the feature search range based on the coordinate information in the current frame image;

[0044] The determining module is further configured to detect the feature regions of each of the objects in the previous frame of the current frame image, and determine the coordinate information of the feature regions of each of the objects in the previous frame image;

[0045] The determining module is further configured to calculate the matching cost of two adjacent frames based on the coordinate information of the feature region in the previous frame image and the coordinate information of the feature region in the current frame image, and to match the feature regions of each object in the two adjacent frames based on the matching cost, thereby determining the target feature trajectory data of each object in multiple consecutive frames of images.

[0046] As an optional implementation, the abnormal behavior detection device further includes: an extraction module and a delineation module;

[0047] The extraction module is further configured to extract the first horizontal coordinate, the second horizontal coordinate, the first vertical coordinate, and the second vertical coordinate from the coordinate information of the feature region in the current frame image, wherein the first horizontal coordinate is less than the second horizontal coordinate and the first vertical coordinate is greater than the second vertical coordinate;

[0048] The determining module is further configured to calculate half of the height of the feature region based on the first ordinate and the second ordinate, thereby determining the ordinate search range;

[0049] The determining module is further configured to determine a reference range for the horizontal coordinate based on the first horizontal coordinate and the second horizontal coordinate;

[0050] The delineation module is also used to delineate the feature search range based on the horizontal coordinate reference range and the vertical coordinate search range.

[0051] As an optional implementation, the extraction module is further configured to extract the third abscissa, the fourth abscissa, the third ordinate, and the fourth ordinate from the coordinate information of the feature region in the previous frame image, wherein the third abscissa is less than the fourth abscissa and the third ordinate is greater than the fourth ordinate.

[0052] The determining module is further configured to calculate the matching cost and determine the minimum matching cost if the third ordinate is within the ordinate search range and either the third or fourth abscissa is within the abscissa reference range.

[0053] The determining module is further configured to determine the feature region in the previous frame image corresponding to the minimum matching cost as the successfully matched feature region, thereby forming continuous target feature trajectory data.

[0054] As an optional implementation, the abnormal behavior detection device further includes: a processing module;

[0055] The processing module is also used to process the target feature trajectory data, calculate the change value of the ordinate at adjacent preset time points, and form a change sequence;

[0056] The processing module is also used to perform symbolic aggregation on the change sequence, merging consecutive change values ​​with the same symbol to form a feature change pattern;

[0057] The processing module is also used to extract spatial action data with corresponding timestamps based on the suspected abnormal time periods in the feature change pattern.

[0058] The processing module is also used to perform length alignment on the target feature trajectory data and spatial motion data, and to make the data volume of the two consistent through interpolation.

[0059] The processing module is also used to combine the aligned data into multimodal time series data, which includes spatial motion data and relative coordinate changes of feature regions.

[0060] As an optional implementation, the generation module is further configured to perform length standardization processing on the multimodal time series data, fill zero values ​​into sequences that are less than a preset length, and generate a corresponding length mask to obtain standardized multimodal time series data;

[0061] The determining module is further configured to input the standardized multimodal time series data into a pre-trained abnormal behavior detection model and output the abnormal behavior probability.

[0062] The determining module is further configured to determine the existence of abnormal behavior and identify the target object where the abnormal behavior occurs when the probability of the abnormal behavior exceeds a preset threshold.

[0063] As an optional implementation, the abnormal behavior detection device further includes: a construction module;

[0064] The construction module is used to construct a training dataset, which includes multimodal time-series sample data labeled with normal and abnormal behaviors;

[0065] The building module is also used to build a model architecture, which includes a feature extraction layer, a modality fusion layer, and a classification layer.

[0066] The building module is also used to train the model architecture using the training dataset, calculate the prediction loss using the cross-entropy loss function, and build an abnormal behavior detection model.

[0067] Thirdly, embodiments of this application provide an abnormal behavior detection device, including: a receiver, a transmitter, a memory, and a processor;

[0068] Receiver, used to receive instructions and data;

[0069] A transmitter is used to send commands and data;

[0070] The memory stores computer-executed instructions;

[0071] The processor executes computer execution instructions stored in the memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.

[0072] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect and / or various possible implementations of the first aspect.

[0073] The abnormal behavior detection method provided in this application firstly extracts feature regions of objects from multiple consecutive images and generates target feature trajectory data to accurately capture the visual motion patterns of objects. Secondly, it combines spatiotemporal motion data from sensors that reflect the physical actions of objects and aligns the data according to timestamps to construct time-synchronized multimodal temporal data, effectively overcoming problems such as noise interference, occlusion, or signal loss that exist in single-modal data. Finally, the fused multimodal data is input into a pre-trained abnormal behavior detection model, which not only improves the model's ability to discriminate complex behaviors but also enhances the robustness and interpretability of the detection results, enabling accurate identification of the existence of abnormal behaviors and their corresponding objects. This method utilizes the complementarity of visual and sensor data to improve the accuracy, reliability, and practicality of abnormal behavior detection. Attached Figure Description

[0074] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0075] Figure 1 Flowchart of the abnormal behavior detection method provided in this application Figure 1 ;

[0076] Figure 2 Flowchart of the abnormal behavior detection method provided in this application Figure 2 ;

[0077] Figure 3 A schematic diagram of the feature search range provided for this application;

[0078] Figure 4 Feature trajectory data provided for this application;

[0079] Figure 5 A flowchart illustrating the construction method of the abnormal behavior detection model provided in this application;

[0080] Figure 6 A schematic diagram of the abnormal behavior detection device provided in this application;

[0081] Figure 7 A schematic diagram of the abnormal behavior detection device provided in this application.

[0082] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0083] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0084] In today's society, elevators are a core vertical transportation tool in densely populated areas such as residential buildings, office buildings, and shopping malls. Their operational safety is directly related to the lives and property of the general public. Analysis of numerous elevator accident cases shows that improper elevator behavior is a significant contributing factor to accidents, with intentional or unintentional jumping inside the elevator car being particularly dangerous. Such behavior instantly alters the load state of the elevator car, impacting the guide rails and suspension system, easily triggering the elevator's safety protection mechanism, leading to an emergency stop or passenger entrapment. In extreme cases, it may even damage the elevator's balance system or safety brake device, causing the car to fall uncontrollably, resulting in a major safety accident with mass casualties and unimaginable consequences.

[0085] Currently, the identification of jumping behavior in elevators mainly relies on video surveillance technology. Cameras deployed inside the elevator car collect continuous video streams, and computer vision algorithms are used to analyze human posture. The system typically extracts key points on the human body (such as joint positions) and spatial depth information, combining this with a pre-set human motion model to determine whether there are abnormal movements such as jumping or violent vertical movement, thereby monitoring dangerous behaviors.

[0086] However, visual image-based analysis methods face numerous interferences in complex elevator scenarios, such as multiple people obstructing the view, changes in lighting, and passengers performing similar actions like walking or bending over to pick up items, which can easily lead to misjudgments or missed detections. Especially when passengers' jumps are small or the duration of the action is short, the system struggles to accurately capture dynamic features, resulting in a low accuracy rate in detecting jumping behavior.

[0087] To address the aforementioned issues, the abnormal behavior detection method provided in this application first acquires an image sequence within a target area, extracts feature regions of multiple objects from multiple consecutive frames of images, and generates target feature trajectory data for each object accordingly. Simultaneously, it collects spatial motion sensing data generated when the object performs physical actions within the area. Data from different modalities (visual trajectory and sensor signals) are precisely aligned based on their timestamps to construct time-synchronized multimodal temporal data. Subsequently, this fused data is input into a pre-trained abnormal behavior detection model, which comprehensively analyzes the spatiotemporal dynamic features and ultimately outputs whether abnormal behavior exists and the corresponding target object. This method improves detection accuracy and reliability.

[0088] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings.

[0089] Figure 1 Flowchart of the abnormal behavior detection method provided in this application Figure 1 The execution entity in this embodiment is, for example, an abnormal behavior detection system. Figure 1 As shown, the method includes:

[0090] S101: Obtain the image sequence of the target region, extract the feature regions of multiple objects in the multiple consecutive images contained in the image sequence, and determine the target feature trajectory data of each object.

[0091] The target area refers to the specific spatial range where image acquisition and object monitoring are required. Its boundaries are clearly defined by the monitoring needs and application scenarios, and it must ensure that the activity space of the object to be analyzed is covered. For example, in the elevator jump detection scenario, the target area is the interior space of the elevator car, which must completely include the area where passengers may move inside the car (from the floor to the top, from the door to the back wall) to ensure that the camera can capture the movements of all passengers.

[0092] An image sequence refers to a continuous collection of images arranged in chronological order, formed by continuously capturing images of a target area at fixed time intervals using image acquisition devices (such as cameras or industrial cameras). For example, in elevator detection, a smart camera captures images of the elevator car at 25 frames per second, and each frame is sorted by timestamp to form a complete image sequence containing the passenger's journey from entering the car to completing the jumping action.

[0093] A feature region refers to a key area related to the monitoring target extracted from each frame of an image sequence. For example, in elevator passenger monitoring, the head area can be used as a feature region (located using a rectangular bounding box, with coordinates such as...). .

[0094] Target feature trajectory data refers to a structured data set that reflects the dynamic position and morphological information of each object over time by continuously tracking the feature regions of multiple objects in an image sequence of the target area.

[0095] Specifically, by continuously capturing images of a specific spatial range (i.e., the target area) using image acquisition devices (such as surveillance cameras, industrial cameras, etc.), a continuous set of images arranged in chronological order is generated. Multiple object feature regions are extracted from multiple consecutive images in the image sequence. This involves accurately locating and separating objects from each frame of the image, and then extracting the identifiable feature regions of these objects.

[0096] For example, taking elevator bounce detection as an example, in elevator safety monitoring scenarios, it is necessary to use smart cameras to capture video streams inside the elevator car in real time, forming an image sequence containing the dynamic changes of passengers. Target detection algorithms (such as YOLOv8) are used to locate the head position in each frame, obtaining the coordinates of the head rectangle as the feature region. At the same time, it is necessary to distinguish the feature regions of different people in the same frame to ensure the correlation of the feature regions of the same person in adjacent frames. To determine the target feature trajectory data of each object, a trajectory tracking algorithm (such as a matching method based on the upper border of the head frame) is used to concatenate the feature regions of the same object in consecutive frames in chronological order, recording the coordinates, timestamps, and other information of the feature regions to form the motion path data of the object changing over time. For example, in an elevator scenario, a unique ID is assigned to each passenger, and the coordinate changes of their head feature region are tracked to generate corresponding head trajectory data.

[0097] S102: Obtain spatial motion data of the target area. Spatial motion data is the sensor data generated when the object performs physical actions within the target area.

[0098] Spatial motion data refers to quantifiable sensor data (such as gyroscope angle change data, pressure sensor pressure fluctuation data, etc.) that are collected by sensors and reflect the physical actions of objects within the target area.

[0099] Specifically, quantifiable sensor data generated by the physical movements of an object is collected by sensors deployed within the target area. Changes in spatial motion data must have a direct causal relationship with the object's physical movements and reflect key characteristics of the movements (such as amplitude, frequency, and direction).

[0100] For example, inside an elevator car, vibration data (such as pitch, yaw, and roll angle changes) can be collected using a gyroscope integrated into a smart camera, or pressure fluctuation data of the car floor can be collected using a pressure sensor. This data is directly related to the physical actions of the object—for example, a passenger jumping inside the elevator will cause instantaneous fluctuations in the gyroscope data, and human walking will cause periodic changes in the radar detection distance, thus transforming the abstract "action" into a concrete numerical signal. In elevator jump detection, the three-dimensional angle change data collected by the gyroscope must be able to distinguish between "high-frequency vibrations caused by jumping" and "low-frequency fluctuations caused by normal walking."

[0101] S103: Align the target feature trajectory data and spatial motion data according to the timestamp to generate time-synchronized multimodal time series data.

[0102] Multimodal time-series data refers to a structured data set containing multi-dimensional information, formed by aligning data from different sources (such as vision, sensors, etc.) that reflect different dimensions of the same object or scene according to a unified timestamp and arranged in chronological order.

[0103] Specifically, by establishing a unified time reference, a spatiotemporal correlation is established between two types of data from different sources. Target feature trajectory data (such as the coordinate changes of a person's head in an image) comes from image sequences, while spatial motion data (such as angular fluctuations from a gyroscope) comes from sensors. The acquisition frequencies of the two may differ (e.g., 25 frames / second for images, 50Hz for sensors). Through timestamp matching, visual trajectory information and sensor motion information at the same moment can be correlated.

[0104] For example, in the detection of jumping on a straight ladder, the synchronous data at a certain moment includes both the upward change of the Y-axis coordinate of the human head (visual dimension) and the instantaneous fluctuation of the Z-axis angle of the gyroscope (sensing dimension). The two types of information corroborate each other, which not only avoids the defect of single visual data being affected by occlusion, but also makes up for the problem that single sensor data is prone to misjudging the source of vibration.

[0105] S104: Input multimodal time series data into a pre-trained abnormal behavior detection model and output the detection results, which indicate whether abnormal behavior exists and the target object that exhibits abnormal behavior.

[0106] Specifically, after inputting multimodal time-series data, the model first extracts visual trajectory features (such as coordinate change amplitude and frequency) and sensor action features (such as angle fluctuation peak and period) from the multimodal time-series data. Then, it compares the differences between the features and the training samples through built-in classification logic, and finally outputs two types of detection results: whether there is abnormal behavior (such as jumping or no abnormality); and the identifier of the target object where the abnormal behavior occurred (such as the unique ID of the corresponding elevator passenger), ensuring that the detection results have both behavioral nature judgment and specific object location.

[0107] For example, in a multi-person elevator scenario, relying solely on visual data may result in missed detections of jumping due to people obstructing the view, while relying solely on sensor data may misjudge the vibration of moving heavy objects as jumping. However, by fusing synchronized head trajectories and gyroscope data, the model can accurately identify whether the trajectory and vibration of a person with a certain ID match jumping characteristics, thus reducing missed and false detections and clearly identifying abnormal objects.

[0108] The abnormal behavior detection method provided in this application acquires an image sequence of a target region, extracts feature regions of multiple objects from multiple consecutive images in the image sequence, determines the target feature trajectory data of each object, acquires spatial motion data of the target region (sensor data generated when an object performs physical actions within the target region), aligns the target feature trajectory data and spatial motion data according to timestamps to generate time-synchronized multimodal time-series data, inputs the multimodal time-series data into a pre-trained abnormal behavior detection model, and outputs detection results, which indicate whether abnormal behavior exists and the target object that exhibits abnormal behavior. This method integrates image and sensor multimodal data to achieve accurate and synchronized detection of abnormal behavior and corresponding target objects, improving detection accuracy and reliability.

[0109] Figure 2 Flowchart of the abnormal behavior detection method provided in this application Figure 2 ,like Figure 2 As shown, in this embodiment... Figure 1 Based on the examples, the abnormal behavior detection method is described in detail, which includes:

[0110] S201: Obtain the image sequence of the target region.

[0111] Step S201 is similar to step S101, and will not be described again here.

[0112] S202: Detect the feature regions of each object in any frame image, determine the coordinate information of the feature regions in the current frame image, and define the feature search range based on the coordinate information in the current frame image.

[0113] Specifically, the current frame image is scanned using an object detection algorithm (such as YOLOv8) to accurately locate all objects to be monitored (such as passengers in an elevator), extract feature regions that can identify objects (such as rectangular boxes containing human heads), and calculate the specific coordinate information of these regions in the image coordinate system (such as the upper left corner). and bottom right corner These coordinates clearly define the location of the feature region in the current frame. Based on this, and considering the motion characteristics of the object within the target area (e.g., the limited movement range of people in an elevator and mostly small-scale activities), the possible range of the feature region in the next frame image is delineated with the coordinates of the feature region in the current frame as the center (i.e., the feature search range).

[0114] For example, when a person jumps in a vertical elevator, the vertical movement is relatively large, so the Y-axis search range can be appropriately expanded, while when walking normally, the range can be reduced to balance tracking speed and accuracy.

[0115] Optionally, specific implementation methods for defining the feature search range based on coordinate information in the current frame image include:

[0116] Extract the first x-coordinate, second x-coordinate, first y-coordinate, and second y-coordinate from the coordinate information of the feature region in the current frame image. The first x-coordinate is less than the second x-coordinate, and the first y-coordinate is greater than the second y-coordinate.

[0117] Based on the first and second ordinates, calculate half of the feature region height to determine the search range for the ordinates.

[0118] Determine the reference range of the horizontal coordinates based on the first and second horizontal coordinates;

[0119] The feature search range is defined based on the reference range of the horizontal axis and the search range of the vertical axis.

[0120] Specifically, such as Figure 3 As shown, from the coordinate information of the current frame feature region (taking the human head rectangle as an example), four key parameters are accurately obtained: the first horizontal coordinate (denoted as...). The first is the X-axis coordinate corresponding to the top left corner of the rectangle, and the second is the x-axis coordinate (denoted as ). The X-axis coordinate of the lower right corner of the rectangle is given, and it satisfies the following conditions. ), the first ordinate (denoted as The first is the Y-axis coordinate corresponding to the top left corner of the rectangle, and the second is the ordinate (denoted as...). The Y-axis coordinate corresponding to the bottom right corner of the rectangle, and satisfying These four coordinates together form the rectangular outline of the feature region. For example, the coordinates of the head bounding box in the current frame are (…). =100, =250, =160, =210), which clarifies the horizontal and vertical boundaries of the human head in the image.

[0121] Next, the search range for the vertical coordinate is calculated, and the vertical search boundary is dynamically determined based on the height of the feature region. First, the extracted first vertical coordinate... and the second ordinate Calculate the height of the feature region (i.e. For example, in the above example, the height is 250-210=40, then take half of the height ( (i.e., 20); then, using the upper border y1 of the current feature region as a reference, expand upwards and downwards by half the height each time, finally forming the search range of the y-coordinate, expressed as: Based on the example above, the search range for the vertical coordinate is [250-20, 250+20], which is [230, 270]. This range fully covers the possible vertical displacement of the object (such as the up-and-down movement of a person's head due to jumping or shaking).

[0122] In determining the horizontal coordinate reference range, the horizontal boundary of the current frame's feature region is directly used as the basic reference. That is, the first extracted horizontal coordinate is taken. and the second horizontal axis ,Will As a reference range for the horizontal coordinate, for example, the horizontal coordinate reference range is [100, 160]. In confined scenes such as elevators, the horizontal movement of objects is usually small. Using the horizontal boundary of the current frame as a reference can cover possible displacements while avoiding interference caused by an excessively large horizontal search range.

[0123] Finally, the horizontal coordinate reference range and the vertical coordinate search range are combined, along with constraints on matching validity. Looking vertically, the feature region to be matched in the next frame (such as the head bounding box in the previous frame, denoted as...) is... The top border y-coordinate It must fall within the search range of the vertical coordinate calculated in the current frame. Internally, from a horizontal perspective, the horizontal coordinate ranges of the current frame and the frame to be matched must overlap, i.e., this is determined by calculation. To determine, for example, the x-coordinate of the head frame of the frame to be matched is ( =110, =170), then (160, 170) = 160, max(100, 110) = 110, 160 > 110, indicating that the horizontal ranges overlap. Only when both conditions are met—that the vertical coordinate is within the search range and the horizontal ranges overlap—can the region to be matched be determined to be within the feature search range defined in the current frame.

[0124] S203: Detect the feature regions of each object in the previous frame of the current frame image, and determine the coordinate information of the feature regions of each object in the previous frame image.

[0125] Specifically, firstly, the same target detection algorithm as the current frame (such as the quantized YOLOv8 algorithm in the elevator scene) is used to perform a full scan of the previous frame image, accurately locating all objects to be monitored (such as the heads of passengers), and identifying the feature regions corresponding to each object (such as the head rectangle); then, according to the preset coordinate definition rules, the key coordinate parameters of each feature region are extracted—taking the head rectangle as an example, the first horizontal coordinate (the upper left X-axis, denoted as...) is determined. The second horizontal coordinate (the lower right corner of the X-axis, denoted as...) ,and The first ordinate (the upper left corner of the Y-axis, denoted as...) The second vertical coordinate (the lower right Y-axis, denoted as...) ,and ), forming complete coordinate information of each object feature region in the previous frame (such as ( =105, =252, =165, =212), while assigning a temporary identifier to each feature region.

[0126] S204: Based on the coordinate information of the feature region in the previous frame image and the coordinate information of the feature region in the current frame image, calculate the matching cost of the two adjacent frames. Based on the matching cost, match the feature regions of each object in the two adjacent frames to determine the target feature trajectory data of each object in the multi-frame continuous images.

[0127] First, for each feature region in the current frame (such as the head rectangle, coordinates) ), and each feature region (coordinates) of the previous frame Calculate the matching cost one by one: Taking the elevator scenario as an example, the cost calculation will combine the correlation in the horizontal and vertical directions—the horizontal direction through Measuring the horizontal overlap of feature regions between two frames, and the vertical overlap via The deviation of the top border's vertical axis is measured, and then weighted by a coefficient. , Combine the two (e.g.) If the feature region of the previous frame Not within the vertical coordinate search range defined in the current frame, or with no horizontal overlap. If the cost is set to infinity (the match is invalid), then the cost is set to infinity.

[0128] By comparing matching costs, the feature regions of the current frame are associated with the feature regions of the previous frame that have the lowest and most effective (not infinite) costs, and are determined to be the same object, inheriting the tracking ID of the feature regions of the previous frame. If a feature region in the current frame has no matching region in the previous frame, a new ID is assigned to it. If a certain ID in the previous frame has no matching region in the current frame, it is marked as temporarily stored (and deleted if there are no matches in multiple consecutive frames). After traversing all feature regions in this way, the coordinate information of the same ID in multiple consecutive frames can be concatenated by timestamp to form the target feature trajectory data of the object—for example, the trajectory data of an object with ID 1 will include the first frame. Frame 2 , The nth frame ( The image sequence contains the object's dynamic position changes and corresponding timestamps.

[0129] Optionally, based on the coordinate information of the feature regions in the previous frame image and the coordinate information of the feature regions in the current frame image, the matching cost between two adjacent frames is calculated. Specific implementation methods for matching the feature regions of each object in two adjacent frames based on the matching cost include:

[0130] Extract the third x-coordinate, fourth x-coordinate, third y-coordinate, and fourth y-coordinate from the coordinate information of the feature region in the previous frame image. The third x-coordinate is less than the fourth x-coordinate, and the third y-coordinate is greater than the fourth y-coordinate.

[0131] If the third ordinate is within the ordinate search range, and either the third or fourth abscissa is within the abscissa reference range, then calculate the matching cost and determine the minimum matching cost.

[0132] The feature region in the previous frame image corresponding to the minimum matching cost is determined as the successfully matched feature region, forming continuous target feature trajectory data.

[0133] To calculate the matching cost and complete the matching based on the coordinates of the feature regions in the previous and current frames, it is necessary to first determine the coordinate parameters of the feature regions in the previous frame, and then achieve accurate association through range verification and cost calculation. First, extract the third horizontal coordinate (denoted as ) from the coordinate information of the feature regions in the previous frame image. The corresponding top-left corner of the rectangular bounding box of the feature region in the previous frame on the X-axis. <Fourth horizontal axis ), fourth horizontal coordinate ( (bottom right X-axis), third ordinate ( Top left Y-axis The fourth vertical axis ), fourth ordinate ( These four parameters (including the bottom right Y-axis) constitute the positional reference of the feature region in the previous frame, which corresponds perfectly to the coordinate definition of the feature region in the current frame.

[0134] First, determine whether the feature region of the previous frame satisfies the range validity requirement—that is, the third ordinate v1 must fall within the predefined ordinate search range of the current frame (e.g., ), and the third horizontal coordinate Or the fourth horizontal coordinate There must be at least one horizontal coordinate reference range within the current frame. Only when both conditions are met simultaneously can a matching basis be established. Next, for the feature region of the previous frame that meets the conditions, the matching cost is calculated according to a preset formula (such as a cost function that combines horizontal overlap and vertical deviation), and the minimum matching cost with the smallest value is selected from all valid costs. The feature region of the previous frame corresponding to the minimum matching cost is determined as the object that successfully matches the feature region of the current frame. Then, the coordinate information of the same object is concatenated frame by frame to form continuous target feature trajectory data.

[0135] S205: Obtain spatial motion data of the target area. Spatial motion data is the sensor data generated when the object performs physical actions within the target area.

[0136] Step S205 is similar to step S102, and will not be described again here.

[0137] S206: Process the target feature trajectory data, calculate the change value of the ordinate at adjacent preset time points, and form a change sequence.

[0138] like Figure 4 As shown, firstly, the vertical coordinates (such as the Y-axis coordinates of the upper border of the human head feature region) of each frame image corresponding to a preset time point are selected from the target feature trajectory data. These time points need to be selected at fixed intervals (such as 40ms / frame synchronized with the image frame rate, or 20ms intervals set according to motion analysis requirements) to ensure the uniformity of the timing sequence; then, in chronological order, the ordinate of the previous preset time point is subtracted from the ordinate of the next preset time point to obtain the change value of the ordinate of adjacent time points—for example, the ordinate of the first frame. =250, Frame 2 =248, Frame 3 =255, then the adjacent change values ​​are 248-250=-2 and 255-248=+7 respectively; finally, arrange all the adjacent change values ​​in chronological order to form a change sequence that can intuitively reflect the vertical movement trend of the object (such as [-2, +7, -3, +1, ...]).

[0139] This processing method effectively removes static positional information from trajectory data, highlighting dynamic change characteristics, and is particularly suitable for scenarios that require the identification of vertical movements (such as elevator jump detection). For example, when a passenger jumps, the sequence of changes in the vertical coordinate will show a typical pattern of "negative-positive-negative" (the vertical coordinate decreases when squatting, corresponding to a negative change value; the vertical coordinate increases when rising, corresponding to a positive change value; and decreases again when descending), while the change sequence of normal walking is mostly small fluctuations of positive and negative values. By comparing the sequence patterns, the type of movement can be quickly distinguished.

[0140] S207: Perform symbolic aggregation on the change sequence, merge consecutive change values ​​with the same symbol to form a characteristic change pattern.

[0141] Positive changes (vertical axis increases) are typically marked as "+", negative changes (vertical axis decreases) are marked as "-", and changes approaching zero (a threshold can be set, such as ±0.5 pixels, considered as no significant displacement) are marked as "0". Then, the entire change sequence is traversed, and consecutively appearing identical symbols (such as 3 consecutive "+" or 2 consecutive "-") are merged into one symbol unit. The specific numerical differences within the symbol are eliminated, and only the symbol type and continuous duration (or corresponding frame number) are retained. For example, the original change sequence "-2, -1, +3, +5, +2, -4" will be transformed into a simplified sequence "- (2 frames), + (3 frames), - (1 frame)" after aggregation, thus forming the feature change pattern.

[0142] This aggregation process effectively filters out minute noise in trajectory data (such as ±1 pixel fluctuations caused by image jitter), highlighting the core motion trend of the action, and is especially suitable for scenarios that require rapid identification of action types. For example, in elevator jumping detection, the sequence of changes in a passenger's complete jumping action, after symbolic aggregation, will show a typical feature change pattern of "- (squatting) → + (rising) → - (falling)"; while the sequence of changes in normal walking, after aggregation, is mostly a pattern of "alternating + / - with few consecutive frames of a single symbol" (such as "+ (1 frame) → - (1 frame) → + (1 frame)"). By comparing different feature change patterns, normal actions and abnormal actions can be quickly distinguished.

[0143] S208: Extract spatial action data with corresponding timestamps based on suspected abnormal time periods in the feature change pattern.

[0144] First, it is necessary to identify time periods that match the characteristics of abnormal movements from the symbolically aggregated feature change patterns. For example, in the scenario of jumping on a straight ladder, if the feature change pattern presents "-(2 frames) → +(3 frames) → -(2 frames)" (corresponding to the typical jumping trend of squatting-rising-falling), then the time range covered by this pattern (such as the 5th to 11th frames, corresponding to the timestamps 160ms to 440ms) is marked as a suspected abnormal time period. Then, extract the data segments in the spatial motion data that completely match the timestamp range, such as the Z-axis angle change data recorded by the gyroscope within 160ms to 440ms and the pressure fluctuation data of the pressure sensor at the same time, to ensure that the extracted sensor data can completely reflect the physical motion characteristics within the suspected abnormal time period.

[0145] For example, the suspected time period of feature change patterns is calculated based on image frame timestamps (40ms / frame). Spatial motion data must also carry millisecond-level timestamps. Timestamp alignment ensures that the captured sensor data and visual feature trends correspond to the same motion process. For example, if the suspected abnormal time period is from 160ms to 440ms, all data points in the spatial motion data with timestamps between 160ms and 440ms must be accurately captured, ensuring that key data at the start and end of the motion are not missed, and that redundant data from irrelevant time periods are not included.

[0146] S209: Align the lengths of the target feature trajectory data and spatial motion data, and use interpolation to make the data volume of the two consistent.

[0147] The target feature trajectory data and spatial motion data are length aligned and their data volume is unified through interpolation. The target feature trajectory data is generated from image sequences, with an acquisition frequency consistent with the frame rate (e.g., 25 frames / second, generating one data point every 40ms). The spatial motion data is acquired by sensors, typically at a higher frequency (e.g., 50Hz gyroscope, generating one data point every 20ms), resulting in less data from the former than the latter within the same time frame. Length alignment requires first determining a unified time reference (e.g., using the start and end times of a suspected abnormal period as boundaries), statistically analyzing the original lengths of both types of data within this range, and then supplementing the shorter data with interpolation. If the target feature trajectory data length is N and the spatial motion data length is M (M>N), then the target feature trajectory data is interpolated, supplementing (M / N-1) virtual data points between adjacent data points; if N>M, then the spatial motion data is interpolated, ensuring that the final lengths of the two types of data are completely consistent.

[0148] For example, in target feature trajectory data, the ordinates of the first (t=0ms, y=250) and the second (t=40ms, y=248) trajectory show a decreasing trend. Linear interpolation can be used during interpolation, and a new data point (y=249) can be added at t=20ms to ensure a uniform distribution of data points and to match the original trend. If spatial motion data is missing (e.g., data at a certain time point is abnormal and needs to be removed), it can be supplemented by the mean or polynomial interpolation of adjacent data to ensure that the supplemented data still reflects the true fluctuations of physical motion (e.g., the continuity of gyroscope angle changes). After final alignment, both types of data have one corresponding data point within the same time interval, such as one target feature trajectory data point and one spatial motion data point every 20ms, satisfying the requirement for point-by-point fusion of multimodal data.

[0149] S210: Combine the aligned data into multimodal time series data, which includes spatial motion data and relative coordinate changes of feature regions.

[0150] The time-synchronized spatial motion data and the relative coordinate change data of the feature region are integrated into a structured unified data set in timestamp order, allowing the two dimensions of information to complement each other. The spatial motion data includes quantitative indicators of physical motion captured by sensors (such as the angle fluctuation value of the gyroscope and the pressure change value of the pressure sensor), while the relative coordinate change of the feature region is the core information after processing the target feature trajectory data (such as the adjacent change value of the vertical coordinate of the upper border of the human head feature region and the horizontal coordinate offset). The two have achieved consistency in data volume through prior length alignment. When combining them, only the timestamp is used as an index to associate the two types of data at the same time.

[0151] For example, in a straight ladder jump detection scenario, the multimodal time-series data at a certain moment can be represented as "timestamp 200ms: spatial motion data (gyroscope Z-axis angle + 3.2°), relative coordinate change of feature region (vertical coordinate change + 5 pixels)". The entire data sequence is arranged from earliest to latest according to the timestamp, forming a structured form of "timestamp 180ms - data group 1, timestamp 200ms - data group 2, timestamp 220ms - data group 3...". This combination method not only preserves the accurate recording of physical vibration by spatial motion data, but also includes the positional change trend of the object in the visual dimension. The two types of information corroborate each other—for example, the large fluctuation of the gyroscope angle and the significant increase of the vertical coordinate appear simultaneously, which can both point to the jumping action.

[0152] S211: Perform length standardization on the multimodal time series data, fill zero values ​​into sequences that are less than the preset length, and generate the corresponding length mask to obtain the standardized multimodal time series data.

[0153] To unify the length of different data sequences and eliminate inconsistencies in data dimensions caused by differences in action duration, a preset length must first be set (e.g., determined based on the longest duration of typical abnormal actions in the scene; in the ladder jumping detection, this could be set to 50 data points). Then, each multimodal time series data is traversed, and its actual length is compared with the preset length. If the actual length is insufficient (e.g., a sequence contains only 38 data points), zero values ​​are added to the end of the sequence until the length reaches the preset value. The added zero values ​​are only used as placeholders and do not represent actual action features. If the actual length exceeds the preset length, the data segment of the previous preset length is truncated to ensure that all sequence lengths are completely consistent.

[0154] A length mask is generated simultaneously with zero-value padding. The length mask is typically a binary sequence with the same length as the multimodal time series data. A 1 indicates that the data at the corresponding position is original and valid data, while a 0 indicates that the data at the corresponding position is zero-padded data. For example, if the original length of a sequence is 38 and the preset length is 50, then the first 38 bits of the mask are 1s, and the last 12 bits are 0s. The resulting standardized multimodal time series data achieves both uniform length (all are preset lengths) and preserves the valid length information of the original data through the length mask. This ensures that the model can accurately focus on the valid data segments corresponding to real actions during training and inference, improving the accuracy of abnormal behavior detection.

[0155] S212: Input the standardized multimodal time series data into the pre-trained abnormal behavior detection model and output the probability of abnormal behavior.

[0156] By leveraging the model's ability to learn multi-dimensional collaborative features, data is transformed into quantitative behavioral risk assessments. Standardized multimodal time-series data (including spatial motion data of uniform length, relative coordinate changes of feature regions, and length masks) first passes through the model's feature fusion module. This module extracts key features from both types of data simultaneously (such as high-frequency vibration features in spatial motion data and "descent-ascent-descent" jumping trend features in coordinate changes) through attention mechanisms or convolutional layers, and establishes correlations between features (such as the synchronicity between vibration peaks and coordinate rise amplitudes), avoiding the one-sidedness of single-dimensional features.

[0157] Based on the "normal / abnormal behavior feature library" formed during previous training, the currently input fused features are compared with samples in the library: if the feature has a high match with abnormal behavior samples (such as the typical multimodal feature of jumping on a straight ladder), a higher abnormal behavior probability is output (e.g., 0.92); if it has a high match with normal behavior samples (such as standing steadily or walking slowly), a lower probability is output (e.g., 0.08). Simultaneously, a length mask helps the model ignore invalid data padded with zero values, performing inference only based on truly valid data segments, ensuring the accuracy of probability calculations. The final output is the abnormal behavior probability.

[0158] S213: When the probability of abnormal behavior exceeds a preset threshold, it is determined that there is abnormal behavior and the target object of the abnormal behavior is identified.

[0159] First, a reasonable preset threshold needs to be set according to the scenario requirements (e.g., in elevator jumping detection, the threshold is set to 0.8 to balance the false positive rate and the false negative rate). If the probability of abnormal behavior corresponding to a certain multimodal time series data output by the model (e.g., 0.95) is higher than this threshold, then the action associated with that data is directly determined to be abnormal behavior. Since each standardized multimodal time series data is bound to the previous target feature trajectory data through a timestamp, and the target feature trajectory data corresponds to the tracking ID of a specific object (e.g., the unique identifier of an elevator passenger), the target object that produced the abnormal action can be located in reverse through the data association relationship, thus clarifying which object has exhibited abnormal behavior.

[0160] From the tracking ID of the target feature trajectory data, to the spatial motion data aligned by timestamps, and then to the probability output of the standardized multimodal time series data, each stage retains the object's identity association. For example, if a multimodal time series data corresponds to an object with tracking ID 3, and its anomaly probability is 0.92 (exceeding the 0.8 threshold), then it can be directly determined that the object with ID 3 exhibits abnormal behavior. This can be further analyzed by combining the object's feature region coordinates (such as those in the current frame). =150, (240) head frames, and can also mark the location of abnormal objects in the image.

[0161] Figure 5 A flowchart illustrating the method for constructing the abnormal behavior detection model provided in this application is shown below. Figure 5 As shown, in this embodiment... Figure 1 Examples and Figure 2 Based on the examples, the method for constructing the abnormal behavior detection model is described in detail, including:

[0162] S301: Construct a training dataset, which includes multimodal time-series sample data labeled with normal and abnormal behaviors.

[0163] First, data on two types of behavior need to be systematically collected in the target scenario (such as elevators or shopping mall passages): normal behavior data should cover common compliant actions in the scenario (such as passengers standing steadily, walking slowly, and entering and exiting normally in elevators), while abnormal behavior data should include risky actions that need to be identified (such as jumping, running, and leaning against the elevator door in elevators). For each type of behavior, corresponding target feature trajectory data (such as changes in head coordinates) and spatial motion data (such as gyroscope vibration and pressure fluctuations) should be collected simultaneously and processed into multimodal time series data according to the process.

[0164] The next step is the annotation phase, which requires adding clear behavioral category labels to each multimodal time-series sample: for normal behavior samples, label them "normal" (e.g., "standing," "walking," etc., to improve model classification accuracy); for abnormal behavior samples, label them "abnormal" and specify the abnormality type (e.g., "jumping," "running"), while ensuring that the annotations completely match the time range of the data (e.g., for a 5-second jumping sample, all behaviors within those 5 seconds must be labeled as "abnormal - jumping"). Furthermore, the dataset needs to be quality-checked, removing samples with missing data or incorrect labels, and expanding the sample size through data augmentation (e.g., time-scale stretching, adding slight noise), ultimately forming a training dataset that is sufficiently large, accurately labeled, and covers the main behavioral types of the scenario, ensuring that the model can learn the key features that distinguish between normal and abnormal behaviors from the samples.

[0165] S302: Construct the model architecture, which includes a feature extraction layer, a modality fusion layer, and a classification layer.

[0166] A model architecture comprising a feature extraction layer, a modality fusion layer, and a classification layer is constructed. The core principle is to achieve effective processing of multimodal time-series data and anomaly detection through layered collaboration. Each layer is functionally independent yet tightly integrated. Specifically, the feature extraction layer is responsible for mining key information within the dimensions of spatial action data and feature region coordinate change data: for spatial action data (such as gyroscope angle sequences), it captures the fluctuation features in the time dimension (such as high-frequency vibration patterns during jumping) through 1D convolutional layers or LSTM layers; for feature region coordinate change data (such as ordinate change sequences), it extracts trend features (such as the "descent-ascent-descent" jumping trajectory pattern) through sub-networks with the same structure, ensuring that data from both modalities can be transformed into feature vectors rich in behavioral information.

[0167] The modality fusion layer is the core of the architecture, aiming to break down the independence of two modalities and establish cross-dimensional correlation features. It typically employs attention mechanisms (such as Cross-Attention) or concatenation fusion (Concat) combined with batch normalization, allowing the model to automatically learn the weight ratios of the two features. For example, in ladder jumping detection, the fusion layer strengthens the collaborative features of "high-frequency gyroscope vibration" and "rapid coordinate rise," weakening noise interference from a single modality (such as coordinate deviation caused by visual occlusion), and outputs a fused global feature vector. The classification layer then makes the final probability judgment based on the fused features. Through a fully connected layer and a sigmoid activation function, it maps the global features to abnormal behavior probability values ​​in the 0-1 range, directly outputting the model's behavior classification result for the current input data. This forms a complete inference chain of "extraction-fusion-judgment," adapting to the anomaly detection needs of multimodal time-series data.

[0168] S303: Train the model architecture using the training dataset, calculate the prediction loss using the cross-entropy loss function, and build an abnormal behavior detection model.

[0169] When training the model architecture using a labeled training dataset, iterative optimization allows the model to gradually learn multimodal features that distinguish between normal and abnormal behavior, ultimately forming a stable abnormal behavior detection model. During training, standardized multimodal time-series sample data is input into the model architecture in batches: first, the feature extraction layer extracts local features for both modalities; then, the modality fusion layer generates global correlation features; finally, the classification layer outputs the predicted probability that a sample belongs to abnormal behavior. Subsequently, the cross-entropy loss function is used to calculate the loss value between the predicted result and the sample's true label (0 for normal, 1 for abnormal)—this function effectively measures the difference between the predicted probability distribution and the true label distribution. For example, when the model misclassifies an abnormal sample as normal (predicted probability 0.1, true label 1), a large loss value is generated, and vice versa.

[0170] During training, gradient descent optimization algorithms (such as Adam) are used to adjust the parameters of each layer of the model (such as the convolutional kernel weights of the feature extraction layer and the attention coefficients of the fusion layer) in reverse based on the cross-entropy loss value, continuously reducing the prediction loss. At the same time, early stopping mechanisms and regularization methods (such as Dropout) are introduced to avoid model overfitting (i.e., the model can only fit the training data and cannot generalize to new samples). Through multiple rounds of complete traversal of the training dataset, training stops when the model's loss value on the validation set tends to stabilize and no longer decreases, and the accuracy of abnormal behavior recognition reaches the preset standard. The model parameter combination at this point is the final abnormal behavior detection model, which has the ability to judge anomalies in newly input multimodal time-series data.

[0171] Figure 6 A schematic diagram of the abnormal behavior detection device provided in this application is shown below. Figure 6 As shown, the abnormal behavior detection device 400 provided in this embodiment includes:

[0172] The determination module 401 is used to acquire an image sequence of the target area, extract feature regions of multiple objects in multiple consecutive images contained in the image sequence, and determine the target feature trajectory data of each object;

[0173] The acquisition module 402 is used to acquire spatial motion data of the target area. The spatial motion data is the sensor data generated when the object performs physical actions in the target area.

[0174] The generation module 403 is used to align the target feature trajectory data and spatial action data according to the timestamp to generate time-synchronized multimodal time series data;

[0175] The determination module 401 is also used to input multimodal time series data into a pre-trained abnormal behavior detection model and output detection results, which are whether abnormal behavior exists and the target object where abnormal behavior occurs.

[0176] As an optional implementation, the determining module 401 is further configured to detect the feature regions of each object in any frame image, determine the coordinate information of the feature regions in the current frame image, and delineate the feature search range based on the coordinate information in the current frame image.

[0177] The determining module 401 is also used to detect the feature regions of each object in the previous frame image of the current frame image and determine the coordinate information of the feature regions of each object in the previous frame image;

[0178] The determination module 401 is also used to calculate the matching cost of two adjacent frames based on the coordinate information of the feature region in the previous frame image and the coordinate information of the feature region in the current frame image, and to match the feature regions of each object in the two adjacent frames based on the matching cost, so as to determine the target feature trajectory data of each object in multiple consecutive frames of images.

[0179] As an optional implementation, the abnormal behavior detection device further includes: an extraction module 404 and a delineation module 405;

[0180] The extraction module 404 is also used to extract the first horizontal coordinate, the second horizontal coordinate, the first vertical coordinate and the second vertical coordinate of the coordinate information of the feature region in the current frame image, wherein the first horizontal coordinate is less than the second horizontal coordinate and the first vertical coordinate is greater than the second vertical coordinate;

[0181] The determining module 401 is also used to calculate half of the feature region height based on the first ordinate and the second ordinate, thereby determining the ordinate search range;

[0182] The determining module 401 is also used to determine the reference range of the horizontal coordinate based on the first horizontal coordinate and the second horizontal coordinate;

[0183] The delineation module 405 is also used to delineate the feature search range based on the horizontal coordinate reference range and the vertical coordinate search range.

[0184] As an optional implementation, the extraction module 404 is also used to extract the third horizontal coordinate, the fourth horizontal coordinate, the third vertical coordinate, and the fourth vertical coordinate from the coordinate information of the feature region in the previous frame image, wherein the third horizontal coordinate is less than the fourth horizontal coordinate and the third vertical coordinate is greater than the fourth vertical coordinate.

[0185] The determination module 401 is also used to calculate the matching cost and determine the minimum matching cost if the third ordinate is within the ordinate search range and either the third or fourth abscissa is within the abscissa reference range.

[0186] The determination module 401 is also used to determine the feature region in the previous frame image corresponding to the minimum matching cost as the successfully matched feature region, forming continuous target feature trajectory data.

[0187] As an optional implementation, the abnormal behavior detection device further includes: a processing module 406;

[0188] The processing module 406 is also used to process the target feature trajectory data, calculate the change value of the ordinate at adjacent preset time points, and form a change sequence;

[0189] Processing module 406 is also used to perform symbolic aggregation on the change sequence, merging consecutive change values ​​with the same symbol to form a characteristic change pattern;

[0190] The processing module 406 is also used to extract spatial action data with corresponding timestamps based on suspected abnormal time periods in the feature change pattern;

[0191] The processing module 406 is also used to perform length alignment on the target feature trajectory data and spatial motion data, and to make the data volume of the two consistent through interpolation processing;

[0192] The processing module 406 is also used to combine the aligned data into multimodal time series data, which includes spatial motion data and relative coordinate changes of feature regions.

[0193] As an optional implementation, the generation module 403 is also used to perform length standardization processing on the multimodal time series data, fill zero values ​​for sequences that are less than the preset length, and generate a corresponding length mask to obtain the standardized multimodal time series data.

[0194] The determination module 401 is also used to input the standardized multimodal time series data into the pre-trained abnormal behavior detection model and output the abnormal behavior probability;

[0195] The determination module 401 is also used to determine the existence of abnormal behavior and identify the target object where the abnormal behavior occurs when the probability of abnormal behavior exceeds a preset threshold.

[0196] As an optional implementation, the abnormal behavior detection device further includes: a construction module 407;

[0197] Module 407 is used to build the training dataset, which includes multimodal time-series sample data labeled with normal and abnormal behaviors;

[0198] Module 407 is also used to build the model architecture, which includes a feature extraction layer, a modality fusion layer, and a classification layer.

[0199] Module 407 is also used to train the model architecture using the training dataset, calculate the prediction loss using the cross-entropy loss function, and build an abnormal behavior detection model.

[0200] Figure 7 This is a schematic diagram of the abnormal behavior detection device provided in this application. Figure 7 As shown, this application provides an abnormal behavior detection device 500, which includes: a receiver 501, a transmitter 502, a processor 503, and a memory 504.

[0201] Receiver 501 is used to receive instructions and data;

[0202] Transmitter 502 is used to send commands and data;

[0203] Memory 504 is used to store instructions executed by the computer;

[0204] The processor 503 is used to execute computer execution instructions stored in the memory 504 to implement the various steps of the abnormal behavior detection method in the above embodiments. For details, please refer to the relevant descriptions in the foregoing embodiments of the abnormal behavior detection method.

[0205] Alternatively, the memory 504 can be either standalone or integrated with the processor 503.

[0206] When the memory 504 is set up independently, the electronic device also includes a bus for connecting the memory 504 and the processor 503.

[0207] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the abnormal behavior detection method performed by the abnormal behavior detection device described above.

[0208] It will be understood by those skilled in the art that all or some of the steps, systems, or apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software may be distributed on a computer-readable medium, which may include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0209] The technical solutions of this application have been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it is readily understood by those skilled in the art that the scope of protection of this application is obviously not limited to these specific embodiments. The above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. These modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A method for detecting abnormal behavior, characterized in that, include: Obtain an image sequence of the target region, extract feature regions of multiple objects from multiple consecutive images in the image sequence, and determine the target feature trajectory data of each object; Acquire spatial motion data of the target area, wherein the spatial motion data is sensor data generated when an object performs physical actions within the target area; Align the target feature trajectory data and the spatial action data according to the timestamp to generate time-synchronized multimodal time-series data; The multimodal time-series data is input into a pre-trained abnormal behavior detection model, and the detection results are output, which indicate whether abnormal behavior exists and the target object where the abnormal behavior occurs.

2. The method according to claim 1, characterized in that, The step of extracting feature regions of multiple objects from multiple consecutive frames of images contained in the image sequence and determining the target feature trajectory data of each object includes: Detect the feature regions of each object in any frame image, determine the coordinate information of the feature regions in the current frame image, and define the feature search range based on the coordinate information in the current frame image; Detect the feature regions of each object in the previous frame of the current frame image, and determine the coordinate information of the feature regions of each object in the previous frame image; Based on the coordinate information of the feature region in the previous frame image and the coordinate information of the feature region in the current frame image, the matching cost of two adjacent frames is calculated. Based on the matching cost, the feature regions of each object in the two adjacent frames are matched to determine the target feature trajectory data of each object in multiple consecutive frames.

3. The method according to claim 2, characterized in that, The step of defining the feature search range based on coordinate information in the current frame image includes: Extract the first horizontal coordinate, the second horizontal coordinate, the first vertical coordinate, and the second vertical coordinate from the coordinate information of the feature region in the current frame image, wherein the first horizontal coordinate is less than the second horizontal coordinate, and the first vertical coordinate is greater than the second vertical coordinate; Based on the first and second ordinates, calculate half the height of the feature region to determine the search range for the ordinates; Determine the reference range of the horizontal coordinate based on the first horizontal coordinate and the second horizontal coordinate; The feature search range is defined based on the horizontal coordinate reference range and the vertical coordinate search range.

4. The method according to claim 3, characterized in that, The step of calculating the matching cost between two adjacent frames based on the coordinate information of the feature regions in the previous frame and the coordinate information of the feature regions in the current frame, and matching the feature regions of each object in the two adjacent frames based on the matching cost, includes: Extract the third horizontal coordinate, fourth horizontal coordinate, third vertical coordinate, and fourth vertical coordinate from the coordinate information of the feature region in the previous frame image, wherein the third horizontal coordinate is less than the fourth horizontal coordinate, and the third vertical coordinate is greater than the fourth vertical coordinate; If the third ordinate is within the search range of the ordinate, and either the third or fourth abscissa is within the reference range of the abscissa, then the matching cost is calculated and the minimum matching cost is determined. The feature region in the previous frame image corresponding to the minimum matching cost is determined as the successfully matched feature region, forming continuous target feature trajectory data.

5. The method according to claim 1, characterized in that, The step of aligning the target feature trajectory data and the spatial action data according to timestamps to generate time-synchronized multimodal time-series data includes: The target feature trajectory data is processed to calculate the change in the ordinate value at adjacent preset time points, forming a change sequence; The change sequence is symbolically aggregated, and consecutive change values ​​with the same symbol are merged to form a characteristic change pattern; Based on the suspected abnormal time periods in the aforementioned feature change patterns, spatial action data with corresponding timestamps are extracted. The target feature trajectory data and spatial motion data are length aligned, and interpolation is used to make the data volume of the two consistent. The aligned data are combined into multimodal time series data, which includes spatial motion data and relative coordinate changes of feature regions.

6. The method according to claim 5, characterized in that, The step of inputting the multimodal time-series data into a pre-trained abnormal behavior detection model and outputting detection results includes: The multimodal time series data is length-normalized, and sequences shorter than the preset length are padded with zeros and a corresponding length mask is generated to obtain the normalized multimodal time series data. The standardized multimodal time-series data is input into a pre-trained abnormal behavior detection model, which outputs the probability of abnormal behavior. When the probability of abnormal behavior exceeds a preset threshold, it is determined that abnormal behavior exists and the target object of the abnormal behavior is identified.

7. The method according to any one of claims 1-6, characterized in that, The method for constructing the abnormal behavior detection model includes: Construct a training dataset, which includes multimodal time-series sample data labeled with normal and abnormal behaviors; Construct a model architecture, which includes a feature extraction layer, a modality fusion layer, and a classification layer; The model architecture is trained using the training dataset, and the prediction loss is calculated using the cross-entropy loss function to construct an abnormal behavior detection model.

8. An abnormal behavior detection device, characterized in that, include: The determination module is used to acquire an image sequence of the target region, extract feature regions of multiple objects from multiple consecutive images in the image sequence, and determine the target feature trajectory data of each object; The acquisition module is used to acquire spatial motion data of the target area, wherein the spatial motion data is the sensor data generated when the object performs physical actions in the target area; The generation module is used to align the target feature trajectory data and the spatial action data according to the timestamp to generate time-synchronized multimodal time-series data; The determining module is further configured to input the multimodal time series data into a pre-trained abnormal behavior detection model and output detection results, wherein the detection results are whether abnormal behavior exists and the target object in which the abnormal behavior occurs.

9. An abnormal behavior detection device, characterized in that, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-7.

Images