AI-based abnormal behavior identification and monitoring method and system for factory areas

Abstract

The present invention provides an AI-based method and system for identifying and monitoring abnormal behavior in a factory area. The method includes: obtaining continuous video stream data collected in real time by multiple monitoring devices within a target factory area; segmenting the continuous video stream data in the time dimension and preprocessing the spatial dimension to obtain a multi-frame behavior feature set corresponding to a dynamic behavior sequence; inputting the multi-frame behavior feature set into a pre-trained abnormal behavior recognition model; performing cross-modal correlation fusion on skeletal key point features, environmental object contour features, and motion trajectory features to generate a target behavior feature vector; performing similarity matching with a preset abnormal behavior feature library to determine the abnormal behavior category; and when it is determined to belong to the abnormal behavior category, triggering a corresponding early warning signal based on the behavior type, and performing a real-time alarm display. The present invention can solve the problems of insufficient accuracy and real-time performance of existing monitoring systems.

Description

Technical Field

[0001] The present invention relates to the fields of data processing and artificial intelligence technology, and in particular to an AI-based abnormal behavior identification and monitoring method and system for a factory area. Background Art

[0002] In the area of ​​industrial plant safety management, traditional monitoring systems rely heavily on manual inspections or simple rules based on single sensors (such as infrared detection and area intrusion alarms) to identify abnormal behavior, which presents significant flaws. Single sensors or visual features are susceptible to interference from lighting changes and occlusion, resulting in high false alarm rates. Furthermore, static rule bases cannot adapt to dynamic plant environments (such as production line layout adjustments and the introduction of new equipment), requiring frequent manual threshold updates and high maintenance costs. Furthermore, they lack in-depth analysis of behavioral continuity and multimodal associations, making it difficult to identify complex risks that require cross-time and cross-space linkage (such as the temporal correlation between smoking and contact with flammable materials). Especially in high-risk work scenarios, existing systems have a significant rate of missed detection for complex behaviors such as illegal power connections and collaborative work in hazardous areas, which require object recognition and trajectory analysis. This makes them unable to meet the proactive safety control needs of intelligent manufacturing. Therefore, a method that can address the shortcomings of existing monitoring systems in accuracy and real-time performance is urgently needed. Summary of the Invention

[0003] In view of this, the embodiment of the present invention at least provides an AI-based method and system for identifying and monitoring abnormal behavior in a factory area. The technical solution of the embodiment of the present invention is implemented as follows:

[0004] In one aspect, an embodiment of the present invention provides an AI-based method for identifying and monitoring abnormal behavior in a factory area, the method comprising:

[0005] Acquire continuous video stream data collected in real time by multiple monitoring devices within a target factory area, wherein the continuous video stream data includes a dynamic behavior sequence of at least one target object;

[0006] The continuous video stream data is segmented in the time dimension and preprocessed in the space dimension to obtain a multi-frame behavior feature set corresponding to the dynamic behavior sequence, wherein the multi-frame behavior feature set includes skeleton key point features, environment-related object contour features, and motion trajectory features of the target object in each video frame;

[0007] Inputting the multi-frame behavior feature set into a pre-trained abnormal behavior recognition model, performing cross-modal correlation fusion on the skeleton key point features, environment-related object contour features, and motion trajectory features through a multimodal feature encoding layer in the abnormal behavior recognition model to generate a target behavior feature vector;

[0008] Based on the similarity matching between the target behavior feature vector and a preset abnormal behavior feature library, determining whether the behavior type of the target object belongs to a predefined abnormal behavior category;

[0009] If it is determined to belong to the abnormal behavior category, a corresponding warning signal is triggered according to the behavior type, and the warning signal and the associated video frame are sent to the target terminal device for real-time alarm display.

[0010] Furthermore, the continuous video stream data is segmented in the time dimension and preprocessed in the space dimension to obtain a multi-frame behavior feature set corresponding to the dynamic behavior sequence, including:

[0011] Dividing the continuous video stream data into a plurality of video segments according to a preset time interval, each video segment comprising a fixed number of continuous video frames;

[0012] For each video frame, a pre-trained skeleton key point detection network is called to extract the skeleton key point features of the target object. The skeleton key point features include the three-dimensional coordinates of each joint point and the motion speed parameters between adjacent joint points.

[0013] Synchronously calling an object contour segmentation network to perform contour recognition on objects that spatially interact with the target object in the video frame, and extracting contour features of the environment-related objects, wherein the contour features of the environment-related objects include object shape descriptors and relative distance parameters to the target object;

[0014] Based on the temporal relationship of the video clips, the position change of the target object in adjacent video frames is tracked to generate the motion trajectory feature, wherein the motion trajectory feature includes a displacement direction sequence and an acceleration change curve;

[0015] The skeleton key point features, environment-related object contour features and action trajectory features corresponding to the same video frame are combined to form the multi-frame behavior feature set.

[0016] Furthermore, before calling the pre-trained skeleton key point detection network to extract the skeleton key point features of the target object, the method further includes the step of training the skeleton key point detection network:

[0017] Obtain a historical surveillance video dataset in a factory scenario, and manually annotate each video frame in the historical surveillance video dataset, where the annotation content includes the joint position and joint motion label of the target object;

[0018] Constructing an initial key point detection network, wherein the initial key point detection network includes a feature pyramid module and a three-dimensional coordinate regression module;

[0019] Input the video frames in the historical surveillance video dataset into the initial key point detection network, extract multi-scale spatial features through the feature pyramid module, and predict the three-dimensional coordinates and motion speed parameters of each joint point through the three-dimensional coordinate regression module;

[0020] Calculating the Euclidean distance loss between the predicted coordinates and the labeled coordinates, and combining the difference loss between the motion speed parameter and the joint motion label to generate a total training loss;

[0021] The initial key point detection network is back-propagated and optimized based on the total training loss value until the total training loss value converges to a preset threshold, thereby obtaining the pre-trained skeleton key point detection network.

[0022] Furthermore, the multi-frame behavior feature set is input into a pre-trained abnormal behavior recognition model, and the skeleton key point features, environment-related object contour features, and motion trajectory features are cross-modally correlated and fused through a multimodal feature encoding layer in the abnormal behavior recognition model to generate a target behavior feature vector, including:

[0023] Inputting the skeleton key point features into the first feature encoding sub-network in the multimodal feature encoding layer to generate a first encoding feature vector, wherein the first feature encoding sub-network uses a temporal convolution structure to capture the joint point motion pattern;

[0024] Inputting the contour features of the environment-associated object into a second feature encoding subnetwork in the multimodal feature encoding layer to generate a second encoded feature vector, wherein the second feature encoding subnetwork uses an attention mechanism to focus on the contours of objects that interact most frequently with the target object;

[0025] Inputting the motion trajectory feature into a third feature encoding subnetwork in the multimodal feature encoding layer to generate a third encoding feature vector, wherein the third feature encoding subnetwork uses a recurrent neural network structure to analyze the continuity of the displacement direction sequence;

[0026] The first encoding feature vector, the second encoding feature vector and the third encoding feature vector are concatenated and fused through a fully connected layer for dimensionality reduction to obtain the target behavior feature vector.

[0027] Furthermore, the training process of the abnormal behavior recognition model includes:

[0028] Collect historical abnormal behavior video samples and normal behavior video samples in the factory area, extract the multi-frame behavior feature set for each video sample, and mark the corresponding abnormal behavior category label;

[0029] Constructing an initial abnormal behavior recognition model, wherein the initial abnormal behavior recognition model includes the multimodal feature encoding layer and the classifier layer;

[0030] Inputting the multi-frame behavior feature set into the initial abnormal behavior recognition model, outputting the target behavior feature vector through the multimodal feature encoding layer, and calculating the matching probability between the target behavior feature vector and the feature template corresponding to each abnormal behavior category through the classifier layer;

[0031] Based on the cross entropy loss value between the matching probability and the abnormal behavior category label, updating the parameters of the initial abnormal behavior recognition model;

[0032] An adversarial training strategy is introduced to add noise perturbation data to the multi-frame behavior feature set, and the classification consistency loss value of the perturbed feature set and the original feature set is calculated, and the cross entropy loss value is combined for joint optimization until the model converges.

[0033] Furthermore, the determining whether the target object's behavior type belongs to a predefined abnormal behavior category based on similarity matching between the target behavior feature vector and a preset abnormal behavior feature library includes:

[0034] Reading baseline feature vectors of all predefined abnormal behavior categories from the abnormal behavior feature library, wherein the baseline feature vectors are obtained by centralizing features extracted from historical abnormal behavior data using a clustering algorithm;

[0035] Calculate the cosine similarity between the target behavior feature vector and each benchmark feature vector, and select the abnormal behavior categories corresponding to the top K benchmark feature vectors with the highest similarity as candidate categories;

[0036] Performing a time dimension alignment check on the benchmark feature vector of the candidate category to determine whether the change trend of the target behavior feature vector within a continuous time window is consistent with the behavior pattern of the candidate category;

[0037] If the verification passes, the abnormal behavior category with the highest similarity among the candidate categories is determined as the behavior type of the target object; otherwise, the behavior type is marked as an unknown abnormal category and a manual review process is triggered.

[0038] Furthermore, the method for constructing the abnormal behavior feature library includes:

[0039] Collect videos of abnormal behavior cases that have occurred in the target factory area, including illegal smoking, illegal use of electronic devices, and trespassing in dangerous areas;

[0040] Each abnormal behavior case video is segmented, and a multi-frame behavioral feature set within each time period is extracted, and the corresponding abnormal behavior category and severity level are marked;

[0041] A hierarchical clustering algorithm is used to group the multi-frame behavior feature sets of the same abnormal behavior category, and the feature mean of each group is calculated as the benchmark feature vector;

[0042] A priority weight is assigned to each reference feature vector according to the severity level, and the reference feature vector and its priority weight are stored in the abnormal behavior feature library.

[0043] Furthermore, if it is determined that the behavior belongs to the abnormal behavior category, triggering a corresponding warning signal according to the behavior type includes:

[0044] When the behavior type is the first type of abnormal behavior, a first-level warning signal is generated and the sound and light alarms in the target factory area are activated. At the same time, an emergency warning message containing a screenshot of the behavior and location information is sent to the terminal device of the security manager;

[0045] When the behavior type is the second type of abnormal behavior, a second-level warning signal is generated and a video clip of the behavior and recommended handling measures are sent to the terminal device;

[0046] When the behavior type is an unknown abnormal category, a level 3 warning signal is generated and the associated video stream is uploaded to the cloud analysis platform;

[0047] The video stream data corresponding to the unknown anomaly category is divided into multiple tasks to be labeled, and assigned to multiple expert terminals for independent labeling;

[0048] Receive the annotation results returned by each expert terminal, wherein the annotation results include behavior type suggestions and feature correction parameters;

[0049] A majority voting algorithm is used to perform consistency check on the annotation results. If more than a preset proportion of experts suggest the same behavior type, the behavior type is added to the new category of the abnormal behavior feature library;

[0050] If the annotation results do not reach consistency, a secondary annotation process is initiated until an annotation result that meets the consistency condition is generated.

[0051] Furthermore, the method further includes the step of performing incremental learning on the abnormal behavior recognition model:

[0052] Regularly collect warning processing records and manual review results fed back by the target terminal devices, and filter out target case data with model recognition errors or confidence levels below a threshold;

[0053] Re-extracting features from the video stream in the target case data to generate an incremental training data set, and adding a revised behavior type label to each data sample;

[0054] Freezing the parameters of the multimodal feature encoding layer in the abnormal behavior recognition model and only fine-tuning the weights of the classifier layer;

[0055] During the fine-tuning training process, the elastic weight merging algorithm is used to retain the important parameters of the original classifier layer.

[0056] On the other hand, the present invention provides a computer system comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, and the processor implements the steps of the above method when executing the program.

[0057] The AI-based factory abnormal behavior identification and monitoring method provided by the present invention obtains real-time video stream data of the target factory area and divides it into a multi-frame behavior feature set. It combines the multimodal feature fusion of skeletal key points, environmental object contours and motion trajectories to generate a target behavior feature vector that can comprehensively characterize the behavior pattern. Then, through the matching and early warning mechanism of the abnormal behavior feature library, accurate and efficient abnormal behavior identification and response are achieved. The cross-modal fusion of these three features, capturing human posture details through skeletal keypoint features, locating dangerous object interaction scenarios through environmental object contour features, and analyzing displacement patterns, overcomes the limitations of single visual features and constructs a complete feature profile of abnormal behavior from three dimensions: spatial interaction, object association, and motion pattern. This significantly improves the accuracy of identifying complex abnormal behaviors such as illegal smoking and dangerous operations. Furthermore, temporal segmentation preserves behavioral continuity, while spatial preprocessing extracts key features with high information density. This avoids redundant data interference while ensuring that the dynamic evolution of behavioral sequences is fully captured. This enables the abnormal behavior recognition model to quickly identify risky time periods and spatial regions, improving real-time response to sudden abnormal behaviors. Furthermore, differentiated warning signals are dynamically triggered based on feature matching results, and the feature library and model parameters are continuously optimized based on feedback from terminal devices, forming a closed-loop control mechanism of "identification-warning-verification-iteration". This not only reduces the impact of false alarms on the production process, but also enables early intervention at the early stages of violations through a predictive mechanism, significantly reducing the incidence of safety accidents in the factory. Skeletal keypoint features accurately identify illegal postures (such as holding dangerous objects), environmental object contour features correlate specific equipment violations (such as identifying unauthorized power connections), and motion trajectory features detect abnormal movement paths (such as entering dangerous areas). These three features work together to ensure stable identification of various risky behaviors in complex scenarios such as changing lighting and occlusion. The dynamic update strategy for the baseline feature vectors in the abnormal behavior feature library can adapt to changes in production line layouts and equipment configurations, avoiding the lag inherent in traditional static rule libraries. This ensures that the recognition model continues to adapt to evolving factory environments, forming a long-term, reliable intelligent security system.

[0058] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] The accompanying drawings herein are incorporated into and constitute a part of the specification. These drawings illustrate embodiments consistent with the present invention and, together with the specification, are used to explain the technical solutions of the present invention.

[0060] Figure 1 A schematic diagram of the implementation flow of an AI-based method for identifying and monitoring abnormal behavior in a factory area provided in an embodiment of the present invention.

[0061] Figure 2 A schematic diagram of the structure of an abnormal behavior recognition and monitoring device provided in an embodiment of the present invention.

[0062] Figure 3 A schematic diagram of a hardware entity of a computer system provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0063] In order to make the purpose, technical solutions and advantages of the present invention clearer, the technical solutions of the present invention are further elaborated in detail below with reference to the accompanying drawings and embodiments. The described embodiments should not be regarded as limiting the present invention. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0064] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present invention pertains. The terms used herein are for the purpose of describing the present invention only and are not intended to limit the present invention.

[0065] An embodiment of the present invention provides an AI-based method for identifying and monitoring abnormal factory behavior. The method can be executed by a processor of a computer system. The computer system can refer to a server, laptop, tablet, desktop computer, mobile device (e.g., mobile phone, personal digital assistant, dedicated messaging device), or other device with data processing capabilities.

[0066] Figure 1 A schematic diagram of the implementation flow of an AI-based factory abnormal behavior recognition and monitoring method provided in an embodiment of the present invention is shown as follows: Figure 1 As shown, the method includes the following steps:

[0067] Step S110: Acquire continuous video stream data collected in real time by multiple monitoring devices in the target factory area, wherein the continuous video stream data includes a dynamic behavior sequence of at least one target object.

[0068] The target factory area is, for example, an industrial factory area where monitoring equipment is deployed and behavior monitoring is required. Its physical boundaries are determined by walls, access control systems or geographic coordinates, such as the final assembly workshop area or fermentation area of ​​a liquor production plant. Monitoring equipment covers hardware devices with video acquisition capabilities, including but not limited to fixed cameras, rotatable pan-tilt cameras, infrared thermal imagers and three-dimensional vision equipment with depth sensors. These devices are installed in key locations such as factory entrances and exits, production line workstations, high-risk work areas and logistics channels according to preset layout rules to cover all-round monitoring perspectives of the target factory area. Continuous video stream data is, for example, an uncompressed or compressed video signal sequence continuously output by the monitoring equipment at a constant frame rate. Its temporal continuity is manifested in that the time interval between adjacent video frames is fixed and uninterrupted, for example, real-time transmission using the H.264 encoding format at 25 frames per second. The target object is the person whose behavioral characteristics need to be identified in the video stream data. The dynamic behavior sequence is, for example, the spatial position change and motion state evolution process of the target object within a continuous time window, such as the motion sequence of an operator switching from walking to climbing a shelf. In specific implementation, the security system of the target factory area aggregates the video stream data of each monitoring device in real time through a wired or wireless network, and performs preliminary detection and tracking of the target object in the video stream, forming dynamic behavior sequence metadata containing timestamp, device number and target ID, providing structured input for subsequent processing.

[0069] Step S120: The continuous video stream data is segmented in the time dimension and preprocessed in the space dimension to obtain a multi-frame behavior feature set corresponding to the dynamic behavior sequence, wherein the multi-frame behavior feature set includes the skeleton key point features, environment-related object contour features and motion trajectory features of the target object in each video frame.

[0070] Temporal segmentation involves, for example, dividing a continuous video stream into discrete sets of video frames at fixed time intervals, such as extracting single frames every 0.04 seconds (corresponding to 25 frames per second), or adaptively segmenting between the start and end points of key dynamic actions to ensure that each video segment fully records a single behavioral cycle of the target object. Spatial preprocessing involves pixel-level processing and feature enhancement of single-frame images. This includes filtering algorithms to eliminate motion blur, histogram equalization to adjust for uneven illumination, and the application of denoising convolution kernels to reduce noise, thereby improving the robustness of subsequent feature extraction. The multi-frame behavioral feature set is the result of an aggregation of spatiotemporal features extracted from a preprocessed set of video frames. Skeletal keypoint features are extracted using deep learning-based human pose estimation algorithms (such as OpenPose or AlphaPose). These features are specifically represented as a set of 2D or 3D coordinates of human joints, such as the positions of a worker's head, shoulders, elbows, and knees in the image coordinate system. Environmental object contour features are identified using edge detection algorithms (such as the Canny operator or deep convolutional networks) to identify the contours of objects interacting with the target object in the video frame, such as the outline of a worker's handheld tool or the contact area between a forklift's forks and a pallet. Motion trajectory features are obtained using multi-target tracking algorithms (such as SORT or DeepSORT) to obtain a sequence of target object displacement vectors between consecutive frames, such as the coordinate sequence of a person's motion path from point A to point B. In implementation, skeleton detection, contour extraction, and trajectory tracking algorithms are executed in parallel on each frame of video data to generate a time-aligned multimodal feature set, providing standardized input for subsequent model processing.

[0071] Step S130: Input the multi-frame behavior feature set into a pre-trained abnormal behavior recognition model, and perform cross-modal correlation fusion on the skeleton key point features, environment-related object contour features, and motion trajectory features through the multimodal feature encoding layer in the abnormal behavior recognition model to generate a target behavior feature vector.

[0072] Pre-trained abnormal behavior recognition models, for example, are deep neural networks trained on large-scale annotated datasets (e.g., video libraries containing factory workers smoking, making phone calls, crossing boundaries, and performing illegal operations). Their architecture includes feature encoding, cross-modal fusion, and classification output modules. The multimodal feature encoding layer is the core component of the model responsible for processing heterogeneous features. It uses a multi-head attention mechanism or graph convolutional network to achieve semantic associations between features of different modalities. For example, skeletal keypoint features are mapped to node attributes in the graph structure, environmental object contour features are used as edge connection weights, and motion trajectory features are used to dynamically adjust the spatiotemporal dependencies of the graph. Cross-modal correlation fusion, for example, is the process of performing nonlinear transformations and weightings on features of different modalities within a unified feature space. For example, gating mechanisms are used to dynamically adjust the contribution of skeletal keypoint features and motion trajectory features, or cross-attention is used to calculate the potential impact of environmental object contour features on skeletal motion patterns. The target behavior feature vector is a fused, high-dimensional, dense vector whose dimensionality depends on the model design (e.g., 512 or 1024). Each element in the vector encodes the abstract semantics of the target object's behavior in the joint multimodal feature space. For example, the activation value of a specific dimension in the vector reflects whether the amplitude of a person's movement exceeds a safety threshold, or whether the spatial relationship between the object's outline and skeletal position meets standard operating requirements. During implementation, the system inputs a multi-frame behavioral feature set into the model in chronological order. The multimodal feature encoding layer performs feature fusion and aggregates temporal context information frame by frame, ultimately outputting a target behavior feature vector representing the overall behavioral pattern.

[0073] Step S140: performing similarity matching between the target behavior feature vector and a preset abnormal behavior feature library to determine whether the behavior type of the target object belongs to a predefined abnormal behavior category.

[0074] The pre-set abnormal behavior feature library is a database storing prototype vectors of known abnormal behavior categories. Each category corresponds to a predefined factory safety hazard (such as entering a high-risk area without a helmet, illegally crossing a conveyor belt, or smoking in a no-smoking area). Its prototype vectors are derived through clustering algorithms or deep feature statistics from classification models. Similarity matching, for example, involves calculating a distance metric (such as cosine similarity, Euclidean distance, or Mahalanobis distance) between the target behavior feature vector and each prototype vector in the library, and selecting the highest similarity value as the classification basis. The process of determining the behavior type involves two stages: threshold comparison and category determination. If the highest similarity exceeds a preset threshold (e.g., 0.85), the target behavior is considered to belong to the corresponding abnormal category; otherwise, it is classified as normal behavior. For example, if the cosine similarity between the target behavior feature vector and the prototype vector for the "climbing high-risk equipment" category reaches 0.92, the behavior is considered to pose a safety risk. In implementation, the abnormal behavior feature library needs to be regularly updated to incorporate new abnormal patterns. Updates can be made through incremental learning of new category prototypes or retraining the model to generate an updated feature space.

[0075] Step S150: If it is determined to belong to the abnormal behavior category, a corresponding warning signal is triggered according to the behavior type, and the warning signal and the associated video frame are sent to the target terminal device for real-time alarm display.

[0076] Warning signals are standardized alert instructions tied to abnormal behavior categories. They include the alarm level (e.g., a Level 1 alarm requires immediate intervention, while a Level 2 alarm requires manual review), the alarm location (monitoring device number and plant coordinates), and recommended action (e.g., notifying security personnel or disconnecting the device's power supply). Associated video frames, for example, include video clips covering key moments before and after the abnormal behavior (e.g., continuous frames from 5 seconds before the onset to 3 seconds after the end of the abnormal behavior), to assist with manual review and event backtracking. Target devices include, for example, the monitoring screen in the plant's central control room, mobile inspection terminals used by security personnel, or smartphone apps used by responsible individuals. Real-time alarm display requires low-latency transmission protocols (e.g., WebRTC or RTMP) to ensure that alarm information reaches the terminal within milliseconds. In specific implementation, triggering a warning signal automatically involves alarm logging, storage of associated video frames, and multi-terminal broadcasting. Simultaneously, pre-set control mechanisms are initiated (e.g., activating audible and visual alarms or locking access control systems in relevant areas). For example, when it is identified that "personnel have entered a dangerous machinery operation area", the system immediately sends an alarm message containing a location map and real-time video stream to the nearest inspection terminal in the area, and activates the on-site warning light to remind personnel to evacuate.

[0077] Through the above steps S110~S150, the embodiment of the present invention obtains real-time video stream data of the target factory area and divides it into a multi-frame behavior feature set, combines the multimodal feature fusion of skeletal key points, environmental object contours and motion trajectories, and generates a target behavior feature vector that can comprehensively characterize the behavior pattern. Then, through the matching and early warning mechanism of the abnormal behavior feature library, accurate and efficient abnormal behavior recognition and response are achieved. The cross-modal fusion of these three features, capturing human posture details through skeletal keypoint features, locating dangerous object interaction scenarios through environmental object contour features, and analyzing displacement patterns, overcomes the limitations of single visual features and constructs a complete feature profile of abnormal behavior from three dimensions: spatial interaction, object association, and motion pattern. This significantly improves the accuracy of identifying complex abnormal behaviors such as illegal smoking and dangerous operations. Furthermore, temporal segmentation preserves behavioral continuity, while spatial preprocessing extracts key features with high information density. This avoids redundant data interference while ensuring that the dynamic evolution of behavioral sequences is fully captured. This enables the abnormal behavior recognition model to quickly identify risky time periods and spatial regions, improving real-time response to sudden abnormal behaviors. Furthermore, differentiated warning signals are dynamically triggered based on feature matching results, and the feature library and model parameters are continuously optimized based on feedback from terminal devices, forming a closed-loop control mechanism of "identification-warning-verification-iteration". This not only reduces the impact of false alarms on the production process, but also enables early intervention at the early stages of violations through a predictive mechanism, significantly reducing the incidence of safety accidents in the factory. Skeletal keypoint features accurately identify illegal postures (such as holding dangerous objects), environmental object contour features correlate specific equipment violations (such as identifying unauthorized power connections), and motion trajectory features detect abnormal movement paths (such as entering dangerous areas). These three features work together to ensure stable identification of various risky behaviors in complex scenarios such as changing lighting and occlusion. The dynamic update strategy for the baseline feature vectors in the abnormal behavior feature library can adapt to changes in production line layouts and equipment configurations, avoiding the lag inherent in traditional static rule libraries. This ensures that the recognition model continues to adapt to evolving factory environments, forming a long-term, reliable intelligent security system.

[0078] As an embodiment, the step S120 of performing time dimension segmentation and spatial dimension preprocessing on the continuous video stream data to obtain a multi-frame behavior feature set corresponding to the dynamic behavior sequence includes:

[0079] Step S121: Divide the continuous video stream data into a plurality of video segments according to a preset time interval, each video segment containing a fixed number of continuous video frames.

[0080] For example, a preset time interval is the length of video segmentation that is pre-set based on the monitoring needs of the target factory area. Its value is determined by the temporal resolution requirements for target object behavior recognition. For example, it is set to 0.1 seconds in scenarios where rapid changes in movement need to be captured (such as detecting an arm-raising cigarette), and is extended to 0.5 seconds (corresponding to 2 frames per second) when monitoring low-frequency, long-period behaviors (such as people staying in restricted areas for a long time). A fixed number of continuous video frames is, for example, the upper limit of the number of frames contained in each video clip. For example, with a time interval of 0.04 seconds, a single video clip contains 30 frames of data, corresponding to 1.2 seconds of continuous monitoring footage, to ensure that the target object can complete a complete action cycle (such as the operational process from picking up a part to completing installation by an operator). In specific implementation, the system buffers and queues the input data through the video stream processing module, triggers the frame cutting operation at the preset time interval, and generates a temporally continuous and non-overlapping sequence of video clips.

[0081] Step S122: For each video frame, call the pre-trained skeleton key point detection network to extract the skeleton key point features of the target object, where the skeleton key point features include the three-dimensional coordinates of each joint point and the motion speed parameters between adjacent joint points.

[0082] The pre-trained skeleton key point detection network is, for example, a deep learning model trained based on a large-scale human body structure posture dataset (such as the COCO key point dataset). Its architecture can adopt the OpenPose framework based on convolutional neural networks or the ViTPose model based on Transformer, and has the ability to recover the three-dimensional joint coordinates of the target object from monocular or binocular video streams. The three-dimensional coordinates of each joint point are represented by the normalized spatial position data output by the model. For example, in human body posture estimation, the coordinates of the head joint point are (x head , y head , z head ), where z head The motion velocity parameters between adjacent joints are obtained by calculating the disparity of binocular cameras or deducing the depth estimation network of a monocular camera. For example, the motion velocity parameters between adjacent joints are instantaneous velocity vectors calculated based on the displacement of joints between consecutive video frames. For example, the three-dimensional velocity component (v) is obtained by dividing the difference between the elbow joint coordinates (x1, y1, z1) and (x2, y2, z2) in two adjacent frames (time difference Δt) by Δt. x , v y , v z In a personnel monitoring scenario at a liquor production plant, the system tracks the shoulder and wrist joints of workers in video frames. If a sudden increase in shoulder joint velocity and a sharp drop in wrist joint height are detected, it can preliminarily determine an abnormal tendency to throw dangerous objects. At this time, the skeletal key point features will include quantitative indicators of abnormal accelerated motion.

[0083] Step S123: Synchronously call the object contour segmentation network to perform contour recognition on objects that have spatial interaction with the target object in the video frame, and extract the contour features of the environment-related objects, wherein the contour features of the environment-related objects include object shape descriptors and relative distance parameters from the target object.

[0084] Object contour segmentation networks, for example, are neural network models optimized for semantic segmentation tasks (such as Mask R-CNN or U-Net). They use pixel-level classification to identify regions of objects in video frames that are in physical contact or spatial proximity with the target object, such as objects held by a person or cargo loaded on a transport vehicle. Object shape descriptors use mathematical methods such as Fourier descriptors or Hu invariant moments to parameterize the contour polygon. For example, a tool contour is discretized into 128 boundary points and its Fourier transform coefficients are calculated to form a 256-dimensional feature vector, which is used to distinguish between different categories of objects (such as a wrench and a hammer). Relative distance parameters are obtained by calculating the Euclidean distance between the target object's joint point (such as the center of a human hand) and the object's contour center of mass. If a binocular camera or depth sensor is used, the three-dimensional spatial distance can be directly obtained; in monocular video scenes, it must be estimated using a perspective projection model and known object dimensions. For example, when monitoring forklifts transporting pallets in a warehouse logistics area, the system identifies the pallet's outline and calculates the distance between it and the tip of the forklift's forks. When the distance exceeds a safety threshold (such as 2 meters) and the forks are not in the raised state, the contour features of the environment-related objects will contain geometric relationship data of the abnormal separation state, providing a basis for subsequent judgment of illegal operations.

[0085] In one embodiment, step S123, calling an object contour segmentation network to perform contour recognition on an object that spatially interacts with the target object in the video frame and extract the contour features of the environment-related object, may include the following steps:

[0086] Step S1231: calling the object contour segmentation network to locate the object area of ​​the video frame and generate the initial bounding box coordinates and confidence parameters of the target object.

[0087] The object contour segmentation network, for example, is a deep learning model trained on the Faster R-CNN or YOLOv5 architecture. It uses a region proposal network (RPN) or a single-shot detection mechanism to coarsely locate the target object in the input video frame and outputs the coordinates of a rectangular bounding box and a corresponding class confidence score. The initial bounding box coordinates represent the normalized coordinates of the target object's top-left and bottom-right corners in the image. For example, in a 1920×1080 video frame, the initial bounding box coordinates for a forklift are (x_min=0.35, y_min=0.42, x_max=0.68, y_max=0.79), corresponding to the rectangular region between pixel coordinates (672, 453) and (1305, 853). The confidence parameter is the model's estimated probability that the target object exists within the bounding box. For example, when the forklift detection confidence reaches 0.92, the bounding box is considered valid. In the personnel monitoring scenario of a liquor production plant, the system calls the object contour segmentation network on the video frames of the operating personnel to generate multiple initial bounding boxes. The bounding box with the highest confidence (such as 0.95) corresponds to the torso area of ​​the personnel, providing a benchmark for the subsequent expansion of the detection area.

[0088] Step S1232: Expanding the detection area by a preset ratio outward based on the initial bounding box coordinates to generate a candidate object search area, where the candidate object search area covers a spatial range within a preset radius around the target object.

[0089] The preset ratio is, for example, an expansion factor that is dynamically adjusted based on the target object category. For example, for a human target, the initial bounding box is expanded by 30% in width and 20% in height to cover the surrounding area that the person's arms could reach. The candidate object search area is calculated as follows: the width of the expanded bounding box = original width × (1 + 2 × horizontal expansion ratio), and the height = original height × (1 + 2 × vertical expansion ratio). The center point remains unchanged.

[0090] Step S1233: Call the candidate object detection network, perform a sliding window traversal on the pixels within the candidate object search area, and identify a set of candidate bounding boxes for all potential interactive objects. The candidate bounding box set includes the position coordinates and object category prediction results of each candidate object.

[0091] The candidate object detection network, for example, can employ a lightweight detection model based on MobileNetv3. It uses a sliding window mechanism to densely scan the candidate object search area with a stride of 8 pixels, outputting the class probability distribution and fine-tuned bounding box coordinates for each object within the window. The candidate bounding box set is then filtered using the non-maximum suppression (NMS) algorithm to remove redundant detection boxes with an overlap ratio exceeding a threshold (e.g., 0.5). The top K (e.g., K = 10) confidence-based candidate objects are retained. For example, in a video frame from a warehouse logistics area, candidate bounding boxes for a pallet (confidence 0.88), a shelf post (confidence 0.75), and a safety sign (confidence 0.63) are detected within the candidate object search area. Their position coordinates are stored in a normalized form in the set. The object category prediction result is a multi-class probability vector output by the softmax function. For example, the probability of the pallet class is 0.88, the shelf post is 0.75, and the safety sign is 0.63. The probabilities of all other classes are all below 0.1.

[0092] Step S1234: matching and screening is performed based on the object category prediction result and a preset interactive object category list, and candidate objects that do not belong to the list in the candidate bounding box set are eliminated to obtain a target interactive object set.

[0093] The preset list of interactive object categories is, for example, a list of object category names that have operational associations with the target object. For example, for operator targets, the list includes categories such as "safety helmets," "hand tools," and "chemical containers." For forklift targets, the list includes categories such as "pallets," "shelves," and "loading and unloading ramps." The matching screening process traverses the set of candidate bounding boxes and retains candidate objects whose category prediction results belong to the list and whose confidence exceeds a threshold (such as 0.6). In the refinery inspection scenario, the candidate bounding boxes output by the candidate object detection network include "valves" (0.82), "pipelines" (0.78), and "fire hydrants" (0.55). Fire hydrants are eliminated because they are not included in the interactive object category list and their confidence is lower than the threshold. The target interactive object set only retains the detection results of valves and pipes, ensuring that subsequent processing focuses on related objects.

[0094] Step S1235: inputting the image area corresponding to each candidate object in the target interactive object set into a pre-trained contour segmentation network to generate a precise contour mask of each candidate object, wherein the precise contour mask is a closed boundary describing the object shape by a binary pixel matrix.

[0095] The pre-trained contour segmentation network uses a U-Net architecture. Its encoder extracts image features through residual convolutional layers, while its decoder recovers pixel-level contour details through upsampling and skip connections. The precise contour mask is a binary matrix with the same resolution as the input image area, where foreground pixels (value 1) represent areas occupied by objects, and background pixels (value 0) represent areas without objects.

[0096] Step S1236: extracting the object shape descriptor based on the precise contour mask, wherein the object shape descriptor is encoded by a contour polygon vertex coordinate sequence and a curvature change feature, wherein the curvature change feature is generated by calculating an angle change rate of a line connecting adjacent vertices.

[0097] The coordinate sequence of the contour polygon vertices is obtained by, for example, simplifying the mask boundary by the Douglas-Peucker algorithm. For example, the original contour containing 500 boundary points is compressed into 50 key vertices, each of which is represented by (x i , y i ) coordinate pairs. The calculation method of curvature change characteristics is: traverse the vertex sequence and calculate the three adjacent vertices P i-1 , P i , P i+1 The angle θ formed i , and then θ i The first-order difference of the sequence is used to obtain the curvature change rate Δθ i In wrench contour analysis, the curvature change rate of the handle approaches zero, while the wrench mouth exhibits periodic high curvature changes. This feature can effectively distinguish different tool types. The object shape descriptor is ultimately encoded as a concatenation of normalized vertex coordinates and curvature change rates, for example, a 200-dimensional vector (50 vertices × 2 coordinates + 50 curvature values), which is used for subsequent similarity matching.

[0098] Step S1237: Obtain the coordinates of the trunk center point in the skeletal key point features of the target object, calculate the center of mass coordinates of the precise contour mask, and generate the relative distance parameter based on the three-dimensional space difference between the trunk center point coordinates and the center of mass coordinates.

[0099] The coordinates of the torso center point are, for example, the mean of the three-dimensional coordinates of the key points of the chest or pelvis region in the target object's skeleton key points. For example, the chest center point of a human target is the spatial average position of the midpoints of the left and right shoulder joints and the hip joints. The center of mass coordinates are obtained by calculating the mean of the three-dimensional coordinates (including depth information) of all foreground pixels in the precise contour mask. If a monocular camera is used, it is estimated by the scaling assumption. The relative distance parameters include the horizontal distance d x y=√((x t −x c ) 2 +(yt −y c ) 2 ) and the vertical distance d z =|z t −z c |, where (t) represents the center of the forklift's trunk and (c) represents the center of mass. In warehousing scenarios, if the horizontal distance between the forklift's trunk center and the pallet's center of mass exceeds 3 meters, the relative distance parameter will trigger an abnormal separation alarm, indicating a possible risk of cargo falling off.

[0100] Step S1238: combining the object shape descriptor and relative distance parameter of the same candidate object to form the environment-associated object contour feature.

[0101] The combination operation can be achieved by feature vector concatenation or attention weighted fusion, such as combining a 200-dimensional object shape descriptor with a 2-dimensional relative distance parameter (d xy , d z ) are concatenated into a 202-dimensional feature vector, which is then mapped to a unified 256-dimensional feature space through a fully connected layer. This feature combination mechanism ensures that the system simultaneously perceives both the geometric properties and spatial relationships of objects, significantly improving the accuracy of identifying abnormal interactive behaviors.

[0102] Step S124: Based on the temporal relationship of the video clips, track the position change of the target object in adjacent video frames to generate the motion trajectory feature, which includes a displacement direction sequence and an acceleration change curve.

[0103] An example of a temporal relationship is a strict time sequence between frames in a video clip. The system establishes an index through frame numbers or timestamps to ensure the causal consistency of the trajectory tracking algorithm in the time dimension. Position change tracking uses a multi-target tracking algorithm (such as DeepSORT) to predict the position of the target object in the next frame through a Kalman filter, and performs data association with the actual detection results to eliminate tracking loss caused by occlusion or illumination changes. The displacement direction sequence records the movement direction of the target object between consecutive frames in the form of a vector sequence. For example, in the XY plane coordinate system, the displacement vectors of 25 frames per second are arranged in time sequence to form a sequence of 25 groups of (dx, dy) data, which characterizes the time evolution pattern of the motion direction. The acceleration change curve is obtained by calculating the second-order difference of the displacement vector. For example, in the time interval Δt, according to the speed change Δ vx and Δ vyTangential and normal accelerations are calculated, and a graph of acceleration amplitude over time is plotted. In factory vehicle monitoring scenarios, if a transport vehicle's acceleration curve in a straight path exhibits high-frequency oscillations (e.g., acceleration fluctuating dramatically between ±2 m / s² within 0.5 seconds), these motion trajectory characteristics could indicate a risk of sudden braking or loss of control, triggering subsequent analysis.

[0104] In one embodiment, the step S124, based on the temporal relationship of the video clips, tracks the position change of the target object in adjacent video frames to generate the motion trajectory feature, includes:

[0105] Step S1241: extracting the spatial coordinates of the torso center point of the target object from each video frame of the video clip, and arranging them in the order of video frame acquisition time to generate an initial position coordinate sequence.

[0106] The spatial coordinates of the torso center can be obtained from the three-dimensional coordinate data output by the skeletal keypoint detection network. For example, in a human target, the torso center is determined by the spatial average coordinates of the midpoints between the left and right shoulder joints and the hip joints. For a forklift target, the geometric center of the chassis is used. The initial position coordinate sequence is a coordinate array sorted in ascending timestamp order. For example, a 30-frame video clip contains 30 sets of (x, y, z) data, with each frame separated by 0.033 seconds. In a factory vehicle crossing boundary detection scenario, this sequence accurately reflects the trajectory trend of a vehicle moving from a permitted driving area to a prohibited area, providing basic data for trajectory analysis.

[0107] Step S1242: performing difference calculation on the spatial coordinates of the torso center point of two adjacent video frames in the initial position coordinate sequence to obtain the position change difference of the target object between the adjacent video frames, wherein the position change difference includes a horizontal offset and a vertical offset.

[0108] The difference is calculated using vector subtraction, namely ΔP i = P i+1 − P i , where ΔP i This includes a horizontal offset (Δx, Δy) and a vertical offset Δz. For example, if the torso center point in frame i is (10.2m, 5.5m, 1.0m) and in frame i+1 is (10.5m, 5.3m, 1.0m), the position difference is (0.3m, -0.2m, 0.0m). In personnel monitoring in three-dimensional warehouses, the horizontal offset reflects the direction of movement, while the vertical offset can be used to detect abnormal three-dimensional behavior, such as climbing shelves. When Δz exceeds 0.2m for multiple consecutive frames, it indicates a potential risk of illegal climbing.

[0109] Step S1243: Determine the moving direction angle of the target object between each pair of adjacent video frames based on the proportional relationship between the horizontal offset and the vertical offset of the position change difference, and concatenate all moving direction angles in chronological order to generate the displacement direction sequence.

[0110] The moving direction angle is calculated by the inverse tangent function: θ i = arctan2(Δy i , Δx i ), the result is expressed in radians and normalized to the interval [0, 2π). For example, a horizontal displacement of Δx = 0.3m and Δy = -0.2m corresponds to an angle of 5.7596 radians (330 degrees), indicating a northwestward movement. The displacement direction sequence is an array of N-1 angle values ​​(N is the number of video frames).

[0111] Step S1244: Based on the ratio of the absolute value of the position change difference to the time interval between adjacent video frames, calculate the instantaneous movement rate of the target object between each pair of adjacent video frames, and arrange all instantaneous movement rates in chronological order to generate a rate change sequence.

[0112] Instantaneous movement rate v i = √(Δx i ² + Δy i ² + Δz i ²) / Δt, where Δt is the frame interval (e.g., 0.033 seconds). For example, when Δx = 0.3 m, Δy = -0.2 m, and Δz = 0.1 m, the instantaneous rate is √(0.09 + 0.04 + 0.01) / 0.033 ≈ 3.85 m / s. The rate change sequence is smoothed using a sliding window to eliminate noise. For example, a moving average filter with a window size of 5 is used. In a liquor factory inspection scenario, this can effectively distinguish between normal walking (1.2 m / s) and running (3 m / s) rates, providing a quantitative basis for abnormal behavior identification.

[0113] Step S1245: Perform a time window sliding calculation on the difference between two adjacent instantaneous moving rates in the rate change sequence to obtain the rate change difference of the target object in the continuous time window, and fit all the rate change differences in time sequence to generate the acceleration change curve.

[0114] For example, the rate change difference Δv i = v i+1 -v i , after the sliding average processing with a window size of 3, the smooth acceleration sequence a is obtained i = (Δv i-1 +Δv i +Δv i+1) / 3Δt. The acceleration change curve is fitted to discrete acceleration points using cubic spline interpolation. In logistics vehicle monitoring, this curve clearly shows the dynamic process of the vehicle from constant speed driving (a≈0) to emergency braking (a<-3m / s²). When the abnormal acceleration peak exceeds the threshold, an early warning is triggered.

[0115] Step S1246: Feature-binding each moving direction angle in the displacement direction sequence with the rate change difference at the corresponding time node in the acceleration change curve to form time-axis aligned fused trajectory data, and encoding the fused trajectory data into the action trajectory feature.

[0116] Feature binding is achieved by creating a timestamp index, for example, the i-th moving direction angle θ i and the i-th acceleration value a i The combination is (θ i , a i ) data pairs, forming N-2-dimensional fused trajectory data. The encoding process uses a recurrent autoencoder to compress the time series data into a 128-dimensional latent space vector. In arm motion monitoring, this vector can characterize the smoothness, directional stability, and acceleration compliance of the motion trajectory. When the degree of match between the latent space vector and the normal operating pattern library falls below a threshold, it is determined to be an abnormal trajectory feature, triggering a device shutdown and maintenance instruction. This encoding mechanism effectively integrates spatiotemporal motion features, significantly improving the ability to recognize complex trajectory patterns.

[0117] Step S125: combining the skeleton key point features, environment-related object contour features, and motion trajectory features corresponding to the same video frame to form the multi-frame behavior feature set.

[0118] Combination operations, for example, involve converting heterogeneous feature data into a unified format and concatenating them in spatial or channel dimensions. For example, skeletal keypoint features (256-dimensional vectors), environmental object outline features (128-dimensional vectors), and motion trajectory features (64-dimensional vectors) are concatenated along the feature dimension to form a 448-dimensional combined feature vector. Alternatively, these features can be mapped to higher-dimensional spaces and then weightedly fused using an attention mechanism. The multi-frame behavioral feature set is ultimately represented as a three-dimensional tensor structure, with the dimensions being the number of frames, the number of feature types, and the number of feature dimensions. For example, for a 30-frame video clip, if three feature types are extracted from each frame, each with 512 dimensions, the set shape is 30 × 3 × 512.

[0119] As an implementation manner, in step S122, before calling a pre-trained skeleton key point detection network to extract the skeleton key point features of the target object, the method provided by the embodiment of the present invention further includes the step of training the skeleton key point detection network, which may specifically include:

[0120] Step S101: Acquire a historical surveillance video dataset in a factory scene, and manually annotate each video frame in the historical surveillance video dataset, where the annotation content includes the joint position and joint motion label of the target object.

[0121] For example, historical surveillance video datasets are long-term video data sets collected from surveillance equipment deployed in the target factory. These datasets cover a variety of lighting conditions, equipment operating states, and human work scenarios. For example, 3,600 hours of video were collected over three consecutive months from the fermentation workshop of a liquor production plant, encompassing various scenarios such as human operation, vehicle transportation, and personnel inspections. Manual labeling uses professional labeling tools (such as LabelMe or CVAT) to label the joint points of target objects in each video frame. Joint point locations are recorded in pixel coordinates or 3D spatial coordinates, depending on the sensor type (monocular / binocular camera or RGB-D device with depth information). Joint point motion labels are the joint motion state classification results based on time series analysis. For example, arm rotation joints are labeled as "constant rotation" or "sudden stop and acceleration," and human knee joints are labeled as "flexion" or "extension." In the worker monitoring scenario at a liquor production plant, the labelers marked the coordinates of the workers' 17 human joints (including shoulders, elbows, wrists, hips, knees, etc.) in the video frames, and annotated joint motion labels based on the displacement between consecutive frames, such as "upper limb horizontal swing" or "lower limb climbing movement", to form structured training data.

[0122] Step S102: constructing an initial key point detection network, wherein the initial key point detection network includes a feature pyramid module and a three-dimensional coordinate regression module.

[0123] The feature pyramid module uses a multi-scale feature fusion architecture (such as HRNet or FPN) to extract local detail features and global semantic features of video frames through parallel convolution paths. For example, it generates feature maps at three scales: 1 / 4, 1 / 8, and 1 / 16 at a 512×512 pixel input resolution, respectively capturing the micro-texture of fingers, the outline of the person's torso, and the background information of the factory environment. The three-dimensional coordinate regression module consists of a fully connected layer and a coordinate transformation unit. Its input is the multi-scale feature vector output by the feature pyramid, and its output is the three-dimensional spatial coordinates (x, y, z) and motion speed parameters (v) of each joint point of the target object. x , v y ,v z In a 3D warehouse personnel detection scenario, the module calculates depth information using binocular camera parallax, maps 2D image coordinates to 3D coordinates in the warehouse coordinate system, and achieves 3D reconstruction of the forklift driver's hand position. During network initialization, the convolutional layer weights are assigned using the He normal distribution, and ImageNet pre-trained parameters are loaded to accelerate convergence.

[0124] Step S103: Input the video frames in the historical surveillance video dataset into the initial key point detection network, extract multi-scale spatial features through the feature pyramid module, and predict the three-dimensional coordinates and motion speed parameters of each joint point through the three-dimensional coordinate regression module.

[0125] Video frames undergo standardization preprocessing before input, including resolution scaling to the network input size (e.g., 512×512), pixel normalization to the range [-1, 1], and data augmentation (random rotation of ±15 degrees and brightness adjustment of ±20%). The feature pyramid module performs multi-level convolution and pooling operations on the preprocessed images. For example, in the HRNet architecture, a branched network exchanges high-dimensional information from feature maps at different scales to generate a feature vector that combines microscopic finger texture and macroscopic arm posture. The 3D coordinate regression module receives the feature vector and maps it to joint coordinates and velocity parameters through a fully connected layer. For example, the predicted 3D coordinates of the arm joints are (2.3m, 4.7m, 1.5m) and the motion velocity is (0.2m / s, -0.1m / s, 0.05m / s).

[0126] Step S104: Calculate the Euclidean distance loss value between the predicted coordinates and the labeled coordinates, and combine it with the difference loss value between the motion speed parameter and the joint motion label to generate a total training loss value.

[0127] Euclidean distance loss value L coord The calculation formula is: , where N is the total number of joint points, is the predicted coordinate, is the label coordinate. Difference loss value The smooth L1 loss function is used to calculate the deviation between the predicted speed and the annotated motion label. For example, when the joint motion label is "acceleration", if the predicted speed direction is opposite to the annotated direction, a high loss value will be generated. The total training loss value L total =αL coord +βL velocity , where α=0.7 and β=0.3 are hyperparameters determined by grid search.

[0128] Step S105: performing back propagation optimization on the initial key point detection network based on the total training loss value until the total training loss value converges to a preset threshold, thereby obtaining the pre-trained skeleton key point detection network.

[0129] Backpropagation optimization uses the Adam optimizer, with an initial learning rate set to 3e−4. A cosine annealing strategy adjusts the learning rate based on the validation set loss. During training, 32 video frames are input per batch. After forward propagation and loss calculation, the network weight parameters are updated using the chain rule. Convergence conditions are set as total training loss fluctuations less than 1e−4 for 10 consecutive training epochs and validation set accuracy exceeding 98%. In training on a smoking behavior dataset, after 120 epochs, the total training loss dropped from an initial 3.2 to 0.08 (with a preset threshold of 0.1). At this point, the network's mean error in arm joint coordinate predictions stabilized at 2.3 cm, and the velocity and direction matching accuracy reached 96.5%, meeting the accuracy requirements for real-time factory monitoring. Once the trained network is deployed on an edge computing device, it can perform high-precision joint detection of objects in live video streams, providing reliable feature input for abnormal behavior recognition.

[0130] As an embodiment, step S130, inputting the multi-frame behavior feature set into a pre-trained abnormal behavior recognition model, performing cross-modal correlation fusion on the skeleton key point features, environment-related object contour features, and motion trajectory features through the multimodal feature encoding layer in the abnormal behavior recognition model to generate a target behavior feature vector, may specifically include:

[0131] Step S131: inputting the skeleton key point features into the first feature encoding sub-network in the multimodal feature encoding layer to generate a first encoding feature vector, wherein the first feature encoding sub-network adopts a temporal convolution structure to capture the joint point motion pattern.

[0132] For example, a temporal convolutional architecture consists of a stack of neural network layers that extract features by sliding one-dimensional convolution kernels along the time axis. Different dilation coefficients (e.g., 1, 2, 4) are used to expand the receptive field to capture long-term joint motion patterns. The first feature encoding subnetwork takes as input a three-dimensional tensor of multi-frame skeletal keypoint features, with dimensions of time step × number of joints × coordinate dimensions (e.g., 30 × 17 × 3). The output is the first encoded feature vector (e.g., 256 dimensions) after multiple layers of dilated convolution and nonlinear activation.

[0133] Step S132: Inputting the contour features of the environment-associated object into the second feature encoding subnetwork in the multimodal feature encoding layer to generate a second encoding feature vector, wherein the second feature encoding subnetwork uses an attention mechanism to focus on the contour of the object that interacts most frequently with the target object.

[0134] For example, the attention mechanism is a computational module that dynamically assigns feature importance via a learnable weight matrix. Its query vector (Query) is derived from the skeletal keypoint features of the target object, and its key-value vector (Key-Value) is derived from the contour features of objects in the environment. Attention weights are generated by calculating the similarity between the query and the key. The second feature encoding subnetwork positionally encodes the object contour features in a single frame and then feeds this into a multi-head self-attention layer. This layer selects the contours of objects with the closest spatial distance to the target object and the longest contact time (e.g., the contour of a pallet with continuous contact by a forklift fork), while suppressing interference from irrelevant background objects. In personnel monitoring in a three-dimensional warehouse, if a worker remains within 0.5 meters of a shelf column for five consecutive frames, the weight of the corresponding shelf column in the attention weight matrix increases to 0.92, while the weight of a distant tool cart decreases to 0.05. This allows the second encoding feature vector to focus on encoding the shape and distance parameters of high-risk interactive objects. The network aggregates temporal attention weights using a gated recurrent unit (GRU), further enhancing the feature representation of frequently interacting objects.

[0135] Step S133: Inputting the motion trajectory feature into the third feature encoding sub-network in the multimodal feature encoding layer to generate a third encoding feature vector, wherein the third feature encoding sub-network adopts a recurrent neural network structure to analyze the continuity of the displacement direction sequence.

[0136] The recurrent neural network architecture uses a bidirectional long short-term memory (Bi-LSTM) network with 128 hidden units. The forward and backward LSTM units capture the forward evolution and reverse causal relationships of the displacement direction sequence, respectively. The third feature encoding subnetwork takes as input the displacement direction angle sequence (e.g., 29 angle values ​​for 30 frames) from the motion trajectory features and outputs a third-encoded feature vector (e.g., 128 dimensions) that incorporates temporal context. In a factory vehicle crossing boundary detection scenario, the Bi-LSTM network models the trajectory sequence of a forklift moving from a permitted area to a restricted area. If the displacement direction angle is detected to consistently deviate from the restricted area coordinates for eight consecutive frames (e.g., gradually shifting from 90 degrees to 135 degrees), the activation value of the dimension representing directional consistency in the third-encoded feature vector exceeds a threshold, indicating a potential violation path. A temporal attention mechanism is introduced in the network's output layer, assigning greater weight to features at trajectory inflection points, thereby increasing sensitivity to sudden steering behaviors.

[0137] Step S134: concatenate the first encoding feature vector, the second encoding feature vector, and the third encoding feature vector, and perform dimensionality reduction fusion through a fully connected layer to obtain the target behavior feature vector.

[0138] The concatenation operation concatenates the first encoded feature vector (256 dimensions), the second encoded feature vector (256 dimensions), and the third encoded feature vector (128 dimensions) along the feature dimension into a 640-dimensional composite vector. This is then fed into a two-layer fully connected network (with dimensions of 512 and 256, respectively). Overfitting is prevented by using a ReLU activation function and a Dropout layer (ratio 0.3). The final output is a 256-dimensional target behavior feature vector. During the dimensionality reduction process, the fully connected layer applies a nonlinear transformation to the cross-modal features using a weight matrix. For example, this enhances the implicit association between joint motion and object interaction, suppresses redundant information, and maximizes the class separability of the target behavior feature vector within the latent space.

[0139] As an implementation method, the training process of the abnormal behavior recognition model includes the following steps:

[0140] Step S201: Collect historical abnormal behavior video samples and normal behavior video samples in the factory area, extract the multi-frame behavior feature set for each video sample, and mark the corresponding abnormal behavior category label.

[0141] Historical abnormal behavior video samples cover safety incidents that occurred within the target factory over the past three years, including 16 types of abnormal behaviors, such as climbing shelves, smoking, and illegal use of electronic devices. Each type contains at least 200 samples, each lasting 5-30 seconds. Normal behavior video samples are randomly sampled from daily monitoring and include standard operating procedures, compliant inspection routes, and routine equipment operations. The multi-frame behavior feature set extraction process follows the preprocessing process of steps S120-S124, generating a structured dataset containing skeletal keypoint features, environmental object contour features, and motion trajectory features. Abnormal behavior category labels are encoded using one-hot encoding. For example, "entering a high-risk area without a helmet" corresponds to the label vector [1, 0, ..., 0], while "smoking" corresponds to [0, 1, ..., 0].

[0142] Step S202: constructing an initial abnormal behavior recognition model, wherein the initial abnormal behavior recognition model includes the multimodal feature encoding layer and the classifier layer.

[0143] The multimodal feature encoding layer consists of the first, second, and third feature encoding subnetworks defined in steps S131-S134, as well as a feature fusion module. The classifier layer uses a two-layer fully connected network (256→128→16) with a softmax function to output probability distributions for 16 categories of abnormal behavior. During model initialization, convolution kernel weights are initialized using a Xavier normal distribution, and the forget gate bias of the LSTM unit is set to 1.0 to mitigate gradient vanishing. The model input dimension is adapted to, for example, 30 frames × 3 modal features. The output layer is designed to classify 14 categories of abnormal behavior and one normal category. The dimensions of the fully connected layer are adjusted to meet actual requirements.

[0144] Step S203: Input the multi-frame behavior feature set into the initial abnormal behavior recognition model, output the target behavior feature vector through the multimodal feature encoding layer, and calculate the matching probability between the target behavior feature vector and the feature template corresponding to each abnormal behavior category through the classifier layer.

[0145] Matching probability is calculated using the cosine similarity metric. Each category in the classifier layer corresponds to a learnable feature template vector (e.g., 16 categories correspond to 16 256-dimensional vectors). The dot product between the target behavior feature vector and each template vector is calculated and then normalized to a probability value using Softmax. In the case of 3D warehouse personnel behavior recognition, when the input features represent the "throwing goods" behavior, the cosine similarity between the target behavior feature vector and the "throwing goods" template reached 0.91, and the Softmax output probability was 0.87, significantly higher than for other categories. During the model training phase, backpropagation is used to optimize the parameters of the template vector and feature encoding layer, ensuring that the feature vectors of similar samples are tightly clustered in the latent space.

[0146] Step S204: updating the parameters of the initial abnormal behavior recognition model based on the cross entropy loss value between the matching probability and the abnormal behavior category label.

[0147] The cross entropy loss calculation formula is: , where y c is the one-hot encoding of the true label, p c is the class probability output by the Softmax. The Adam optimizer was used for parameter update, with an initial learning rate of 1e−4, which decayed by a factor of 0.5 every 10 training epochs. An early stopping mechanism was introduced during training, terminating training if the validation set loss did not decrease for five consecutive epochs to prevent overfitting.

[0148] Step S205: introducing an adversarial training strategy, adding noise perturbation data to the multi-frame behavior feature set, and calculating the classification consistency loss value of the perturbed feature set and the original feature set, and performing joint optimization in combination with the cross entropy loss value until the model converges.

[0149] The adversarial training strategy can generate adversarial samples through the Fast Gradient Sign Method (FGSM), with the noise perturbation intensity ε set to 0.1, and adding perturbations along the loss gradient direction in the feature space. The classification consistency loss value L c onsist measures the KL divergence of the output probability distribution of the original feature set and the perturbation feature set, and the joint optimization objective function is L t otal=L c e+λL cOnsist (λ = 0.5). In the warehouse fire emergency drill data training, the adversarial sample simulates the coordinate offset of the skeleton key points caused by camera shaking. The model minimizes L c Onsist improves robustness to noise, ensuring that classification probability fluctuations remain within ±0.05 when encountering frame loss or sudden changes in illumination in real-world scenarios. Training is terminated when the total training loss fluctuates by less than 1e−5 for 20 consecutive cycles and the validation set accuracy exceeds 98.5%. The final model is deployed to the factory edge server for real-time abnormal behavior identification.

[0150] As an embodiment, the step S140, based on similarity matching between the target behavior feature vector and a preset abnormal behavior feature library, determines whether the behavior type of the target object belongs to a predefined abnormal behavior category, including:

[0151] Step S141: reading the baseline feature vectors of all predefined abnormal behavior categories from the abnormal behavior feature library, wherein the baseline feature vectors are obtained by centralizing the features extracted from the historical abnormal behavior data using a clustering algorithm.

[0152] The abnormal behavior feature library is a distributed database that stores benchmark feature vectors for all known abnormal behavior categories within the target factory. These benchmark feature vectors are generated by centrally processing multi-frame behavioral feature sets extracted from historical abnormal behavior case videos using a hierarchical clustering algorithm. For example, in a liquor production plant scenario, there are 12 abnormal behavior categories, including workers entering high-risk areas without helmets, smoking, and using mobile phones. Each category contains 3-5 clusters to address different manifestations of the same abnormal behavior. Centralized processing involves calculating the mean of the feature dimensions for the multi-frame behavioral feature sets within the same cluster. For example, the benchmark feature vector for a specific group of hand-raising behavior is a 256-dimensional vector, with each dimension taking the arithmetic mean of 500 historical feature samples within that group. Reading operations are implemented through the parallel query interface of distributed storage nodes. When an anomaly detection is triggered at the liquor production plant monitoring center, all benchmark feature vectors are loaded from the storage nodes into the memory cache, forming a search queue containing category labels, priority weights, and cluster group numbers to ensure real-time matching efficiency.

[0153] Step S142: Calculate the cosine similarity between the target behavior feature vector and each reference feature vector, and select the abnormal behavior categories corresponding to the first K reference feature vectors with the highest similarity as candidate categories.

[0154] Cosine similarity is calculated using the formula of the ratio of the vector dot product and the module length product: sim = (A·B) / (||A||×||B||), where A is the target behavior feature vector and B is the baseline feature vector. In the liquor production plant monitoring scenario, the target behavior feature vector has a similarity of 0.93 with the baseline feature vector for the "smoking" category and 0.62 with the "illegal power connection" category. The system selects the top three candidate categories when K=3. Dynamically adjust the threshold based on historical false alarm rates. For example, when the highest similarity between candidate categories falls below 0.85, the threshold is lowered by 5% to expand the search scope. After the candidate category list is generated, the system appends the associated cluster grouping number and severity level to the matching results for subsequent verification processes.

[0155] Step S143: performing a time dimension alignment check on the benchmark feature vector of the candidate category to determine whether the change trend of the target behavior feature vector in the continuous time window is consistent with the behavior pattern of the candidate category.

[0156] Temporal alignment verification is achieved using the Dynamic Time Warping (DTW) algorithm. This algorithm nonlinearly aligns the temporal variation curve of the target behavior feature vector with the typical pattern curve of the candidate category benchmark feature vector. A path cost is then calculated to assess trend consistency. For example, in detecting smoking behavior in a liquor production plant, the hand movement parameter variation curve of the target feature vector over 10 consecutive frames must maintain a DTW path cost less than 0.15 relative to the standard smoking pattern curve of the benchmark feature vector. If the height variation trend of the skeletal key points differs by more than 20% from the rising slope of the "climbing shelf" benchmark feature vector when a worker is detected entering a high-risk area, the temporal verification is considered a failure. A sliding window mechanism is introduced during the verification process, performing local alignment every five frames to ensure that sudden abnormal behaviors (such as rapid running for 0.5 seconds) are not misclassified as normal fluctuations.

[0157] Step S144: If the verification passes, the abnormal behavior category with the highest similarity among the candidate categories is determined as the behavior type of the target object; otherwise, the behavior type is marked as an unknown abnormal category and a manual review process is triggered.

[0158] The verification pass condition requires that both the similarity threshold and the DTW path cost threshold be met, for example, similarity ≥ 0.85 and DTW ≤ 0.2. In the personnel monitoring scenario of a liquor production plant, when the target behavior feature vector has a similarity of 0.91 with the "illegal smoking" baseline feature vector and DTW = 0.18, the system automatically marks the behavior as "illegal smoking" and triggers an alarm. If the similarity of an arm movement feature vector with the "smoking" baseline feature vector reaches 0.88 but DTW = 0.25 (exceeding the threshold of 0.2), the behavior is classified as an unknown anomaly category, and an alarm work order containing the video clip, feature vector, and matching details is simultaneously sent to the security officer's handheld terminal, requiring manual confirmation whether it is a new type of anomaly. After manual review, the multi-frame behavior feature set of the new anomaly type will be imported into the training set, triggering the incremental update process of the feature library.

[0159] As an implementation method, the abnormal behavior feature library can be constructed by the following steps:

[0160] Step S301: Collect videos of abnormal behavior cases that have occurred in the target factory area, where the abnormal behavior cases include illegal smoking, illegal use of electronic devices, and trespassing into dangerous areas.

[0161] Abnormal behavior videos are sourced from the target plant's security system's event record archive, covering all manually confirmed safety incidents from the past three years. Smoking violations require a 10-30 second video sequence of lighting, holding, and smoking a cigarette. Hazardous area intrusions require a video recording of the entire process, from the triggering of the boundary alarm to evacuation. The video capture equipment utilizes explosion-proof infrared cameras, ensuring 1080p / 25fps video clarity in high-risk environments. Depth sensor data is also included for 3D behavior reconstruction.

[0162] Step S302: Segment each abnormal behavior case video, extract a multi-frame behavior feature set within each time period, and mark the corresponding abnormal behavior category and severity level.

[0163] Segmentation processing divides continuous video into independent segments based on the start and end timestamps of the behavior. For example, a 2-minute video of illegal smoking can be divided into three sub-segments: the lighting phase (0-5 seconds), the smoking phase (6-45 seconds), and the cigarette butt discarding phase (46-50 seconds). The multi-frame behavioral feature set extraction process invokes the preprocessing flow of steps S120-S124 to generate feature data including skeletal key points, object outlines, and motion trajectories. Severity levels are divided into three levels based on the severity of the consequences of the behavior: Level 1 (immediate danger, such as entering a high-voltage area), Level 2 (potential danger, such as not wearing a helmet), and Level 3 (violation but low risk, such as using a mobile phone). In the case of a three-dimensional warehouse, smoking is marked as Level 2, and goods stacked too high and tilted are marked as Level 3. The system adjusts the alarm response speed based on the level.

[0164] Step S303: using a hierarchical clustering algorithm to group the multi-frame behavior feature sets of the same abnormal behavior category, and calculating the feature mean of each group as the reference feature vector.

[0165] The hierarchical clustering algorithm used the Ward variance minimization method, with the clustering termination criteria set at within-cluster distance less than 0.1 and between-cluster distance greater than 0.3. The baseline eigenvector for each group was obtained by calculating the mean of each dimension of the samples within the group. The clustering results were manually reviewed to confirm the rationality of the groupings, and outlier groups with high feature dispersion (e.g., groups with within-cluster sample standard deviations greater than 0.15) were eliminated.

[0166] Step S304: allocating a priority weight to each reference feature vector according to the severity level, and storing the reference feature vector and its priority weight in the abnormal behavior feature library.

[0167] Priority weight calculations are, for example, W = 1 + 0.5 × (3 - L), where L is the severity level (levels 1-3). The baseline feature vector weight for a level 1 anomaly is 2.0, for a level 2 anomaly is 1.5, and for a level 3 anomaly is 1.0. Data is stored in a columnar database in descending order of weight, prioritizing high-weight categories in the real-time matching process. For example, in the liquor manufacturer's feature database, the "smoking" category weight is set to 2.0, and the baseline feature vectors for its three clusters are stored in a hot memory area, ensuring matching latency below 10ms. Distributed storage nodes use a three-copy mechanism to prevent data loss, and a consistent hashing algorithm enables rapid location.

[0168] As an embodiment, step S141, reading the baseline feature vectors of all predefined abnormal behavior categories from the abnormal behavior feature library, wherein the baseline feature vectors are obtained by centralizing the features extracted from the historical abnormal behavior data using a clustering algorithm, may include the following steps:

[0169] Step S1411: Acquire historical abnormal behavior case video data within the target factory area, wherein the historical abnormal behavior case video data includes video stream segments marked with abnormal behavior categories and severity levels.

[0170] Video data on historical abnormal behavior cases is extracted from the Oracle database of the factory security system. This data includes video files, metadata tables (recording device number, timestamp, and geographic coordinates), and annotation information tables. A data cleansing process eliminates invalid videos with resolutions below 720p or with more than 30% keyframe loss to ensure that input quality meets feature extraction requirements. Video stream segments are stored in the H.265 encoding format, and decoding utilizes the GPU-accelerated FFmpeg library to improve processing efficiency.

[0171] Step S1412: performing frame-by-frame decoding and noise filtering on the historical abnormal behavior case video data to generate a pre-processed video frame set.

[0172] Frame-by-frame decoding can use the hardware-accelerated Video4Linux framework to convert the video stream into a sequence of independent frames in YUV420p format. Noise filtering can include Gaussian filtering (kernel size 5×5, σ=1.5) to remove high-frequency noise, median filtering (kernel size 3×3) to suppress impulse noise, and the CLAHE algorithm to enhance low-contrast areas.

[0173] Step S1413: Call the skeleton key point detection network and the object contour segmentation network for each video frame in the preprocessed video frame set to extract the skeleton key point features, environment-related object contour features and motion trajectory features to form a historical multi-frame behavior feature set.

[0174] The skeletal keypoint detection network uses the HRNet-W48 model, with an input size of 512×512, and outputs the three-dimensional coordinates and confidence scores of 17 human joints. The object contour segmentation network is implemented based on Mask R-CNN, generating pixel-level object masks and category labels. Motion trajectory features are extracted using the FairMOT tracking algorithm to extract displacement vectors between consecutive frames. Single-frame processing takes 35ms. The extracted skeletal features include the coordinates of the driver's hand and waist keypoints, environmental features include the Fourier descriptor of the pallet outline, and trajectory features record the forklift's displacement direction and angle per second. The historical multi-frame behavioral feature set is stored in chronological order in Apache Parquet columnar files, improving compression by 40% and accelerating data reading for clustering analysis.

[0175] Step S1414: Group the historical multi-frame behavior feature set according to the abnormal behavior category, apply a hierarchical clustering algorithm to the historical multi-frame behavior feature set of the same abnormal behavior category, and generate multiple cluster groups, each cluster group containing a multi-frame behavior feature subset with feature similarity higher than a preset threshold.

[0176] The grouping process divides the feature set into different directories based on the anomaly category ID. For example, the "smoking violation" category contains 1200 feature samples. Hierarchical clustering is implemented using AgglomerativeClustering in the scikit-learn library. The Manhattan distance is used as the distance metric to accommodate high-dimensional and sparse features, and the dendrogram cut height is set to 0.25.

[0177] Step S1415: performing mean calculation on the multi-frame behavior feature subset within each cluster group, and taking the arithmetic mean of each feature dimension as the reference feature vector of the cluster group.

[0178] The nanmean function in the Numpy library can be used to calculate the mean and handle missing values. For example, in the 256-dimensional feature vector for smoking behavior group A, the mean of dimension 128 (torque-related) is 0.76 and the standard deviation is 0.05. After generating the baseline feature vector, L2 normalization is performed to ensure the validity of the cosine similarity calculation. In the case of illegal climbing and connecting to power lines, the mean of the baseline feature vector in the altitude change dimension is +0.3 m / s, accurately reflecting the continuous upward movement trend. The baseline vector of each cluster group is supplemented with statistical information such as the number of samples and the maximum outlier distance for subsequent priority calculation.

[0179] Step S1416: The reference feature vector is associated with the corresponding abnormal behavior category, cluster group number and severity level and stored in a distributed storage node of the abnormal behavior feature library.

[0180] For example, distributed storage uses the Cassandra database, with a wide table structure containing fields such as feature vector (blob type), category ID (int), group number (varchar), and weight (float). The primary key is designed to be (category ID, group number), supporting fast searches by category.

[0181] Step S1417: When the target behavior feature vector is received, the baseline feature vectors of all abnormal behavior categories are loaded from the distributed storage node, and arranged in descending order according to the severity level to form a priority retrieval queue.

[0182] The loading process can utilize multi-threaded parallel reading, with each thread responsible for scanning a range of category IDs. For example, in the monitoring center, loading 120 baseline feature vectors takes 150ms, with the first-level anomaly category placed first when forming the priority queue. The queue memory structure is implemented using a max heap, with weight values ​​serving as heap keys, ensuring that high-priority features are prioritized for matching. The retrieval queue is accompanied by a last access timestamp, and a least-recently-used (LRU) strategy is used to eliminate low-priority features that have not been used for more than 30 days, maintaining memory usage within 8GB.

[0183] Step S1418: Traverse the benchmark feature vectors in the priority search queue, and perform similarity matching on the target behavior feature vector and each benchmark feature vector in the queue in sequence until a preset maximum number of matches is reached or the matching similarity exceeds a dynamic adjustment threshold.

[0184] The traversal algorithm uses an early termination strategy. When a similarity score exceeds 0.95, the result is immediately returned. Otherwise, matching continues until the queue is traversed or the limit of 100 matches is reached. The threshold is dynamically adjusted based on real-time system load. When CPU utilization exceeds 80%, the threshold is increased by 5% to reduce computational complexity. For example, if the target feature vector matches the 15th baseline feature vector (smoking category) with a similarity score of 0.96, the system terminates the matching process and returns the result. The entire matching round takes 22ms. A matching log records detailed process data for subsequent algorithm optimization and misdiagnosis analysis.

[0185] As an embodiment, in step S150, if it is determined that the abnormal behavior category is involved, triggering a corresponding warning signal according to the behavior type may specifically include:

[0186] Step S151: When the behavior type is the first type of abnormal behavior, a first-level warning signal is generated and the sound and light alarms in the target factory area are controlled to start, and an emergency warning message containing a screenshot of the behavior and positioning information is sent to the terminal device of the security manager.

[0187] The first category of abnormal behavior, for example, involves predefined abnormal behavior categories that pose an immediate danger, such as workers smoking, illegally using electronic devices, or illegally climbing to connect to power lines. The logic for generating a first-level warning signal is linked to the matching results of the benchmark feature vectors marked with the highest priority weight in the abnormal behavior feature library. This signal can be transmitted via the factory IoT control bus to the audible and visual alarms deployed in the target area, triggering them to activate a 120-decibel beeping frequency and a red strobe light pattern, covering an audible and visual warning within a 50-meter radius. An emergency warning message is constructed, containing the current timestamp, monitoring device ID, geographic coordinates, and a 5-second sequence of keyframes captured from the video stream (JPEG format with an 80% compression rate). This message is pushed via the MQTT protocol to the explosion-proof smart terminal devices equipped by security management personnel, triggering them to vibrate and highlight the screen. For example, in a liquor factory monitoring scenario, when a person is identified as smoking in a flammable and explosive area, the explosion-proof sound and light alarm at the top of the area is immediately activated, and an alarm message containing the person's facial features (after blurring) and the plan location map of the installation area is sent to the nearest multiple safety officers' handheld terminals. The location of the incident is simultaneously marked with a flashing red circle on the large-screen map in the central control room.

[0188] Step S152: When the behavior type is the second type of abnormal behavior, a second-level warning signal is generated and a behavior video clip and suggested handling measures are sent to the terminal device.

[0189] The second category of abnormal behavior, for example, involves predefined abnormal behaviors that pose a potential risk but do not require immediate intervention, such as a worker not wearing a safety helmet correctly or being detected holding an unlit cigarette in a no-smoking area. The generation of a secondary warning signal triggers an audible and visual alarm system consisting of a yellow warning light and an 80dB intermittent beep. The warning message is transmitted via the factory's dedicated 5G network as a 10-second H.265-encoded video clip of the behavior (resolution 1920×1080, frame rate 25fps) to the mobile device of the relevant responsible person. The video clip is accompanied by structured metadata including the start time and duration of the abnormal behavior and associated equipment status parameters.

[0190] Step S153: When the behavior type is an unknown abnormal category, a third-level warning signal is generated and the associated video stream is uploaded to the cloud analysis platform, and a multi-expert collaborative labeling mechanism is started to update the abnormal behavior feature library.

[0191] Unknown anomaly categories include behavior types whose similarity with all baseline feature vectors in the abnormal behavior feature library falls below a dynamic threshold (e.g., 0.75) and whose temporal dimension verification fails. This includes the first occurrence of a behavioral pattern or complex, illegal operations by untrained personnel. For example, a Level 3 warning signal activates a blue rotating light and a 60-decibel low-frequency beep to prevent excessive disruption to normal operations. Simultaneously, a complete video stream (HEVC encoded with depth information) containing the 30 seconds before and after the abnormal behavior and a multi-frame behavioral feature set are uploaded to the distributed object storage bucket of the cloud-based analysis platform. A multi-expert collaborative annotation mechanism can be employed to create annotation tasks through a RESTful API interface. The video stream is segmented into 5-second segments and randomly distributed to the terminals of at least several certified experts. Dedicated annotation tools are then used to independently annotate the abnormal behavior type, severity level, and key feature dimensions.

[0192] As an implementation manner, in step S153, starting a multi-expert collaborative annotation mechanism to update the abnormal behavior feature library may include the following sub-steps:

[0193] Step S1531: dividing the video stream data corresponding to the unknown anomaly category into multiple tasks to be labeled, and assigning them to multiple expert terminals for independent labeling.

[0194] Video stream segmentation utilizes an adaptive segmentation algorithm based on behavioral integrity, prioritizing segmentation at the start frame (e.g., the point of arm velocity mutation) and the end frame (e.g., the timestamp of the stop of the action) of an action, ensuring that each task to be labeled encompasses the complete cycle of abnormal behavior. The task allocation strategy utilizes a load balancing algorithm, taking into account the current workload of experts, their professional field compatibility, and historical labeling accuracy (e.g., equipment-related anomalies are prioritized for maintenance experts). Each task is assigned at least three experts for back-to-back labeling.

[0195] Step S1532: receiving the annotation results returned by each expert terminal, wherein the annotation results include behavior type suggestions and feature correction parameters.

[0196] Annotation results can be packaged in JSON format, including the abnormal behavior type (custom input is supported), confidence score (1-5), correction suggestions for key feature dimensions, and severity assessment. The system verifies the identity of the expert through digital signatures and hashes the submission timestamp to prevent tampering.

[0197] Step S1533: A majority voting algorithm is used to perform consistency check on the annotation results. If more than a preset proportion of experts suggest the same behavior type, the behavior type is added to the newly added category of the abnormal behavior feature library.

[0198] For example, the majority voting algorithm can set a threshold that requires at least 66.7% of experts to reach a consensus. When a tie occurs, the feature correction parameter similarity comparison is triggered. The process of adding new categories to the database includes generating a new baseline feature vector (taking the mean of the expert recommended features), setting the initial priority weight (based on the severity level), and updating the fully connected dimension of the classifier layer.

[0199] Step S1534: If the annotation results do not reach consistency, a secondary annotation process is initiated until an annotation result that meets the consistency condition is generated.

[0200] For example, the secondary labeling process expands the expert pool to more people, the system performs significance analysis on controversial feature dimensions, and automatically generates comparative views (such as superimposing the tremor frequency ranges labeled by different experts) to assist decision-making.

[0201] In an independently implementable implementation, the method provided in an embodiment of the present invention further includes a step of performing incremental learning on the abnormal behavior recognition model, which may specifically include:

[0202] Step S160: regularly collecting the warning processing records and manual review results fed back by the target terminal device, and screening out target case data with model recognition errors or confidence levels below a threshold.

[0203] Examples of target terminal devices include mobile inspection terminals, central control room consoles, and cloud-based alarm management platforms equipped by factory safety management personnel. The warning processing records they provide are stored in the form of structured logs, including fields such as the alarm trigger time, the processing personnel ID, the disposal measures, and the review conclusions. The manual review results are entered through the annotation interface of the terminal device. The engineer selects the correct category on the review interface and submits the revised label. The screening criteria for model recognition error cases include: cases that are manually reviewed as false positives or missed negatives after the warning signal is triggered, cases where the classifier output probability is lower than the dynamic confidence threshold (such as 0.65), and cases where the same abnormal category is matched three times in a row but the actual behavior does not match.

[0204] Step S170: re-extract features from the video stream in the target case data to generate an incremental training data set, and add a revised behavior type label to each data sample.

[0205] The feature re-extraction process uses the same preprocessing pipeline as the initial training phase, including a skeletal keypoint detection network to extract a 30-frame-per-second sequence of joint 3D coordinates, an object contour segmentation network to generate pixel-level masks, and a motion trajectory tracking algorithm to calculate displacement vectors. The construction of the incremental training dataset requires temporal expansion of the original video stream. For example, the video stream from 10 seconds before the target behavior to 5 seconds after the fault is resolved is re-segmented into 15-second segments containing the complete behavioral context, and a 256-dimensional multimodal feature vector is extracted for each segment. The corrected behavior type labels are updated based on manual review results. In the pipeline leak detection scenario at a liquor production plant, for 10 target cases where the original model misclassified "flange seal failure" as "pressure gauge failure," the system re-extracted features from 2 minutes of infrared thermal imaging video before and after the leak occurred. This generated an incremental dataset containing temperature gradient changes, vapor diffusion profiles, and the emergency actions of maintenance personnel. Each sample was annotated with the newly added "seal failure" category and assigned a unique hash identifier.

[0206] Step S180: Freeze the parameters of the multimodal feature encoding layer in the abnormal behavior recognition model, and only fine-tune the weights of the classifier layer.

[0207] Parameter freezing is achieved by setting the requires_grad attribute of the feature encoding layer to False, ensuring that the weights of the temporal convolutional network, attention mechanism module, and recurrent neural network components remain unchanged during incremental training. Fine-tuning of the classifier layer uses the SGD optimizer with a small learning rate (e.g., 1e-5), updating only the weight matrices and biases of the fully connected layers to prevent disruption of learned cross-modal feature association patterns.

[0208] Step S190: During the fine-tuning training process, an elastic weight merging algorithm is used to retain important parameters of the original classifier layer to avoid catastrophic forgetting of the learned abnormal behavior categories.

[0209] The execution logic of the elastic weight merging algorithm is to calculate the Fisher information matrix of each parameter of the original classifier layer on the historical training data, identify the weight channels that are critical to the classification of the mastered categories, and impose regularization constraints to protect the numerical changes of these channels when updating the parameters. In the specific implementation, the algorithm is for each weight parameter w i Calculate the importance score Ω i =1 / (F i +ε), where F i is the Fisher information, ε=1e-8 prevents division by zero error, and adds λ∑Ω to the loss function i (w i −w i old ) 2 The regularization term (λ=0.5) is used.

[0210] Based on the foregoing embodiments, an embodiment of the present invention provides an abnormal behavior identification and monitoring device. The various units included in the device, and the various modules included in each unit, can be implemented by a processor in a computer device; of course, they can also be implemented by a specific logic circuit; in the implementation process, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP) or a field programmable gate array (FPGA), etc.

[0211] Figure 2 A schematic diagram of the structure of an abnormal behavior recognition and monitoring device provided by an embodiment of the present invention is shown as follows: Figure 2 As shown, the abnormal behavior identification and monitoring device 200 includes:

[0212] The data acquisition module 210 is used to acquire continuous video stream data collected in real time by multiple monitoring devices in the target factory area, wherein the continuous video stream data includes a dynamic behavior sequence of at least one target object;

[0213] A feature extraction module 220 is configured to segment the continuous video stream data in the time dimension and preprocess the continuous video stream data in the space dimension to obtain a multi-frame behavior feature set corresponding to the dynamic behavior sequence, wherein the multi-frame behavior feature set includes the skeleton key point features, the contour features of the environment-related objects, and the motion trajectory features of the target object in each video frame;

[0214] A feature fusion module 230 is configured to input the multi-frame behavior feature set into a pre-trained abnormal behavior recognition model, and perform cross-modal correlation fusion of the skeleton key point features, environment-related object contour features, and motion trajectory features through a multimodal feature encoding layer in the abnormal behavior recognition model to generate a target behavior feature vector;

[0215] The behavior recognition module 240 is used to perform similarity matching between the target behavior feature vector and a preset abnormal behavior feature library to determine whether the behavior type of the target object belongs to a predefined abnormal behavior category;

[0216] The abnormal warning module 250 is used to trigger a corresponding warning signal according to the behavior type if it is determined to belong to the abnormal behavior category, and send the warning signal and the associated video frame to the target terminal device for real-time alarm display.

[0217] The description of the above device embodiment is similar to the description of the above method embodiment and has similar beneficial effects as the method embodiment. In some embodiments, the functions or modules included in the device provided by the embodiment of the present invention can be used to perform the method described in the above method embodiment. For technical details not disclosed in the device embodiment of the present invention, please refer to the description of the method embodiment of the present invention for understanding.

[0218] It should be noted that, in the embodiment of the present invention, if the above-mentioned AI-based factory abnormal behavior identification and monitoring method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention, or the part that contributes to the relevant technology, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a magnetic disk or an optical disk. In this way, the embodiment of the present invention is not limited to any specific hardware, software or firmware, or any combination of hardware, software and firmware.

[0219] An embodiment of the present invention provides a computer system including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, some or all of the steps in the above method are implemented.

[0220] An embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements some or all of the steps in the above method. The computer-readable storage medium may be transient or non-transient.

[0221] An embodiment of the present invention provides a computer program including computer-readable code. When the computer-readable code is run in a computer device, a processor in the computer device executes some or all of the steps for implementing the above method.

[0222] An embodiment of the present invention provides a computer program product, comprising a non-transitory computer-readable storage medium storing a computer program, wherein when the computer program is read and executed by a computer, some or all of the steps in the above method are implemented. The computer program product can be implemented in hardware, software, or a combination thereof. In some embodiments, the computer program product is embodied as a computer storage medium. In other embodiments, the computer program product is embodied as a software product, such as a software development kit (SDK).

[0223] It should be noted that the above descriptions of the various embodiments tend to emphasize the differences between the embodiments, and reference can be made to the similarities and similarities between the embodiments. The descriptions of the above embodiments of the device, storage medium, computer program, and computer program product are similar to the descriptions of the above-mentioned method embodiments and have similar beneficial effects as the method embodiments. For any technical details not disclosed in the embodiments of the device, storage medium, computer program, and computer program product of the present invention, please refer to the description of the method embodiments of the present invention for understanding.

[0224] Figure 3 A hardware entity diagram of a computer system provided by an embodiment of the present invention is as follows Figure 3 As shown, the hardware entity of the computer system 1000 includes: a processor 1001 and a memory 1002, wherein the memory 1002 stores a computer program that can be run on the processor 1001, and when the processor 1001 executes the program, the steps in the method of any of the above embodiments are implemented.

[0225] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0226] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed across multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the scheme of this embodiment.

[0227] In addition, all functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0228] Those skilled in the art will understand that all or part of the steps of implementing the above-mentioned method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, and other media that can store program codes.

[0229] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the relevant technology, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0230] The above description is only an embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with this technical field can easily think of changes or replacements within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. An AI-based method for identifying and monitoring abnormal behavior in a factory, characterized in that: The method comprises: Acquire continuous video stream data collected in real time by multiple monitoring devices within a target factory area, wherein the continuous video stream data includes a dynamic behavior sequence of at least one target object; Dividing the continuous video stream data into a plurality of video segments according to a preset time interval, each video segment comprising a fixed number of continuous video frames; For each video frame, a pre-trained skeleton key point detection network is called to extract the skeleton key point features of the target object. The skeleton key point features include the three-dimensional coordinates of each joint point and the motion speed parameters between adjacent joint points. Invoking an object contour segmentation network to locate the object region in the video frame and generate initial bounding box coordinates and confidence parameters of the target object; Expanding the detection area by a preset ratio outward based on the initial bounding box coordinates to generate a candidate object search area, where the candidate object search area covers a spatial range within a preset radius around the target object; Calling the candidate object detection network to perform a sliding window traversal of the pixels within the candidate object search area to identify a set of candidate bounding boxes for all potential interactive objects. The candidate bounding box set includes the position coordinates and object category prediction results of each candidate object; Matching and screening the object category prediction result with a preset interactive object category list, eliminating candidate objects that do not belong to the list in the candidate bounding box set, and obtaining a target interactive object set; Inputting the image region corresponding to each candidate object in the target interactive object set into a pre-trained contour segmentation network to generate an accurate contour mask of each candidate object, wherein the accurate contour mask is a closed boundary describing the object shape by a binary pixel matrix; Extracting an object shape descriptor based on the precise contour mask, wherein the object shape descriptor is encoded by a contour polygon vertex coordinate sequence and a curvature change feature, wherein the curvature change feature is generated by calculating an angle change rate of a line connecting adjacent vertices; Obtaining the coordinates of the center point of the torso in the skeleton key point features of the target object, calculating the coordinates of the center of mass of the precise contour mask, and generating a relative distance parameter based on the three-dimensional spatial difference between the coordinates of the center point of the torso and the coordinates of the center of mass; Combining the object shape descriptors and relative distance parameters of the same candidate object to form the contour features of the environment-related object; Extracting the spatial coordinates of the torso center point of the target object from each video frame of the video clip, and arranging them in the order of video frame acquisition time to generate an initial position coordinate sequence; performing difference calculation on the spatial coordinates of the torso center point of two adjacent video frames in the initial position coordinate sequence to obtain a position change difference of the target object between the adjacent video frames, wherein the position change difference includes a horizontal offset and a vertical offset; Determining the moving direction angle of the target object between each pair of adjacent video frames based on the proportional relationship between the horizontal offset and the vertical offset of the position change difference, and serially connecting all the moving direction angles in chronological order to generate a displacement direction sequence; Calculating the instantaneous movement rate of the target object between each pair of adjacent video frames based on the ratio of the absolute value of the position change difference to the time interval between adjacent video frames, and arranging all instantaneous movement rates in chronological order to generate a rate change sequence; Performing a time window sliding calculation on the difference between two adjacent instantaneous moving velocities in the velocity change sequence to obtain the velocity change difference of the target object in a continuous time window, and fitting all the velocity change differences in time sequence to generate an acceleration change curve; Binding each moving direction angle in the displacement direction sequence with the rate change difference at the corresponding time node in the acceleration change curve to form time-axis aligned fused trajectory data, and encoding the fused trajectory data into action trajectory features; Combine the skeleton key point features, environment-related object contour features, and motion trajectory features corresponding to the same video frame to form a multi-frame behavior feature set; Inputting the multi-frame behavior feature set into a pre-trained abnormal behavior recognition model, performing cross-modal correlation fusion on the skeleton key point features, environment-related object contour features, and motion trajectory features through a multimodal feature encoding layer in the abnormal behavior recognition model to generate a target behavior feature vector; Based on the similarity matching between the target behavior feature vector and a preset abnormal behavior feature library, determining whether the behavior type of the target object belongs to a predefined abnormal behavior category; If it is determined to belong to the abnormal behavior category, a corresponding warning signal is triggered according to the behavior type, and the warning signal and the associated video frame are sent to the target terminal device for real-time alarm display.

2. The method according to claim 1, characterized in that Before calling the pre-trained skeleton key point detection network to extract the skeleton key point features of the target object, the method further includes the step of training the skeleton key point detection network: Obtain a historical surveillance video dataset in a factory scenario, and manually annotate each video frame in the historical surveillance video dataset, where the annotation content includes the joint position and joint motion label of the target object; Constructing an initial key point detection network, wherein the initial key point detection network includes a feature pyramid module and a three-dimensional coordinate regression module; Input the video frames in the historical surveillance video dataset into the initial key point detection network, extract multi-scale spatial features through the feature pyramid module, and predict the three-dimensional coordinates and motion speed parameters of each joint point through the three-dimensional coordinate regression module; Calculating the Euclidean distance loss between the predicted coordinates and the labeled coordinates, and combining the difference loss between the motion speed parameter and the joint motion label to generate a total training loss; The initial key point detection network is back-propagated and optimized based on the total training loss value until the total training loss value converges to a preset threshold, thereby obtaining the pre-trained skeleton key point detection network.

3. The method according to claim 1, characterized in that The multi-frame behavior feature set is input into a pre-trained abnormal behavior recognition model, and cross-modal correlation and fusion of the skeleton key point features, environment-related object contour features, and motion trajectory features are performed through a multimodal feature encoding layer in the abnormal behavior recognition model to generate a target behavior feature vector, including: Inputting the skeleton key point features into the first feature encoding sub-network in the multimodal feature encoding layer to generate a first encoding feature vector, wherein the first feature encoding sub-network uses a temporal convolution structure to capture the joint point motion pattern; Inputting the contour features of the environment-associated object into a second feature encoding subnetwork in the multimodal feature encoding layer to generate a second encoded feature vector, wherein the second feature encoding subnetwork uses an attention mechanism to focus on the contours of objects that interact most frequently with the target object; Inputting the motion trajectory feature into a third feature encoding subnetwork in the multimodal feature encoding layer to generate a third encoding feature vector, wherein the third feature encoding subnetwork uses a recurrent neural network structure to analyze the continuity of the displacement direction sequence; The first encoding feature vector, the second encoding feature vector and the third encoding feature vector are concatenated and fused through a fully connected layer for dimensionality reduction to obtain the target behavior feature vector.

4. The method according to claim 3, characterized in that The training process of the abnormal behavior recognition model includes: Collect historical abnormal behavior video samples and normal behavior video samples in the factory area, extract the multi-frame behavior feature set for each video sample, and mark the corresponding abnormal behavior category label; Constructing an initial abnormal behavior recognition model, wherein the initial abnormal behavior recognition model includes the multimodal feature encoding layer and the classifier layer; Inputting the multi-frame behavior feature set into the initial abnormal behavior recognition model, outputting the target behavior feature vector through the multimodal feature encoding layer, and calculating the matching probability between the target behavior feature vector and the feature template corresponding to each abnormal behavior category through the classifier layer; Based on the cross entropy loss value between the matching probability and the abnormal behavior category label, updating the parameters of the initial abnormal behavior recognition model; An adversarial training strategy is introduced to add noise perturbation data to the multi-frame behavior feature set, and the classification consistency loss value of the perturbed feature set and the original feature set is calculated, and the cross entropy loss value is combined for joint optimization until the model converges.

5. The method according to claim 1, characterized in that The determining whether the target object's behavior type belongs to a predefined abnormal behavior category based on similarity matching between the target behavior feature vector and a preset abnormal behavior feature library includes: Reading baseline feature vectors of all predefined abnormal behavior categories from the abnormal behavior feature library, wherein the baseline feature vectors are obtained by centralizing features extracted from historical abnormal behavior data using a clustering algorithm; Calculate the cosine similarity between the target behavior feature vector and each benchmark feature vector, and select the abnormal behavior categories corresponding to the top K benchmark feature vectors with the highest similarity as candidate categories; Performing a time dimension alignment check on the benchmark feature vector of the candidate category to determine whether the change trend of the target behavior feature vector within a continuous time window is consistent with the behavior pattern of the candidate category; If the verification passes, the abnormal behavior category with the highest similarity among the candidate categories is determined as the behavior type of the target object; otherwise, the behavior type is marked as an unknown abnormal category and a manual review process is triggered.

6. The method according to claim 5, characterized in that The method for constructing the abnormal behavior feature library includes: Collect videos of abnormal behavior cases that have occurred in the target factory area, including illegal smoking, illegal use of electronic devices, and trespassing in dangerous areas; Each abnormal behavior case video is segmented, and a multi-frame behavioral feature set within each time period is extracted, and the corresponding abnormal behavior category and severity level are marked; A hierarchical clustering algorithm is used to group the multi-frame behavior feature sets of the same abnormal behavior category, and the feature mean of each group is calculated as the benchmark feature vector; A priority weight is assigned to each reference feature vector according to the severity level, and the reference feature vector and its priority weight are stored in the abnormal behavior feature library.

7. The method according to claim 1, characterized in that If it is determined that the behavior belongs to the abnormal behavior category, triggering a corresponding warning signal according to the behavior type includes: When the behavior type is the first type of abnormal behavior, a first-level warning signal is generated and the sound and light alarms in the target factory area are activated. At the same time, an emergency warning message containing a screenshot of the behavior and location information is sent to the terminal device of the security manager; When the behavior type is the second type of abnormal behavior, a second-level warning signal is generated and a video clip of the behavior and recommended handling measures are sent to the terminal device; When the behavior type is an unknown abnormal category, a level 3 warning signal is generated and the associated video stream is uploaded to the cloud analysis platform; The video stream data corresponding to the unknown anomaly category is divided into multiple tasks to be labeled, and assigned to multiple expert terminals for independent labeling; Receive the annotation results returned by each expert terminal, wherein the annotation results include behavior type suggestions and feature correction parameters; A majority voting algorithm is used to perform consistency check on the annotation results. If more than a preset proportion of experts suggest the same behavior type, the behavior type is added to the new category of the abnormal behavior feature library; If the annotation results do not reach consistency, a secondary annotation process is initiated until an annotation result that meets the consistency condition is generated.

8. The method according to claim 1, characterized in that The method further comprises the step of performing incremental learning on the abnormal behavior recognition model: Regularly collect warning processing records and manual review results fed back by the target terminal devices, and filter out target case data with model recognition errors or confidence levels below a threshold; Re-extracting features from the video stream in the target case data to generate an incremental training data set, and adding a revised behavior type label to each data sample; Freezing the parameters of the multimodal feature encoding layer in the abnormal behavior recognition model and only fine-tuning the weights of the classifier layer; During the fine-tuning training process, the elastic weight merging algorithm is used to retain the important parameters of the original classifier layer.

9. A computer system comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, wherein: When the processor executes the program, the steps of the method according to any one of claims 1 to 8 are implemented.

Images