Quick-response multi-worker dangerous action prediction and real-time early warning method and system

Abstract

The invention discloses a quick-response multi-worker dangerous action prediction and real-time early warning method and system, and belongs to the technical field of computer vision and deep learning, and the method comprises the steps: obtaining video frames of workers and an environment, carrying out the preprocessing, carrying out the real-time detection and tracking of a plurality of workers in the video frames through employing YOLOv5, generating and screening a high-confidence bounding box, and carrying out the cutting; extracting two-dimensional coordinates of the joints of the worker based on attitude estimation and generating a skeleton graph; carrying out feature modeling on a nonlinear relation between skeleton joints by adopting a KAN (Karan Area Network) so as to capture subtle attitude changes, and obtaining a high-dimensional feature vector of an action; time sequence modeling is carried out, future potential dangerous actions are predicted, danger scores are output, when the scores exceed a threshold value, early warning is triggered, and advanced intervention and real-time alarm of high-risk actions are achieved. According to the method, mutual interference in a multi-worker scene can be effectively reduced, the recognition sensitivity and real-time response capability of fine-grained action abnormity are improved, and the method is suitable for industrial production safety monitoring and accident prevention.

Description

Technical Field

[0001] This invention belongs to the fields of computer vision and deep learning, and specifically relates to a fast-response method and system for predicting and providing real-time early warning of dangerous actions by multiple workers. Background Technology

[0002] Workers in high-intensity work environments are prone to dangerous actions due to fatigue, improper operation, or lack of concentration, leading to safety accidents. Especially in complex multi-worker environments, while traditional safety monitoring systems rely on video surveillance and motion recognition technology, these systems face challenges in recognizing the simultaneous actions of multiple workers and complex backgrounds, making it difficult to maintain the effectiveness and accuracy of predictions. A more intelligent and efficient solution is urgently needed to improve worker safety. Therefore, accurately predicting workers' dangerous actions and providing timely warnings has become crucial for ensuring worker safety and improving productivity.

[0003] Traditional factory safety monitoring methods often suffer from insufficient recognition accuracy in complex multi-worker environments, difficulty in effectively distinguishing the independent actions of multiple workers, and insensitivity to subtle changes in movement, resulting in the inability to identify danger signals in advance. Furthermore, their poor real-time performance makes it difficult to process large-scale video data in a timely manner, affecting the effectiveness of early warning systems.

[0004] In summary, the main problems with current factory worker motion prediction are as follows:

[0005] 1. Complexity of multi-worker environments: In factory environments, multiple workers may appear in the same video frame simultaneously. Traditional image analysis methods struggle to accurately distinguish and track each worker in complex backgrounds, easily leading to interference between multiple workers and affecting the accuracy of motion recognition.

[0006] 2. Fine-grained analysis of motion changes is challenging: Fatigue-induced motion deformities often manifest as subtle postural changes, such as reduced range of motion or unstable limb posture. Traditional image feature extraction methods are not sensitive enough to these subtle changes and struggle to accurately capture early signals of dangerous movements.

[0007] 3. High Real-Time Requirements: In factory environments, dangerous actions can lead to serious safety accidents, therefore, motion recognition systems must have real-time detection and early warning capabilities. However, existing methods are easily limited by computing resources when processing large-scale video data, making it difficult to meet the real-time processing requirements. Summary of the Invention

[0008] The technical problem to be solved by this invention is to improve the accuracy of fall prediction and also to improve response efficiency.

[0009] This invention provides a fast-response method and system for predicting and providing real-time early warning of dangerous actions by multiple workers. By combining YOLOv5 target detection, skeleton graph generation, Kernel Activation Network (KAN) action recognition and Long Short-Term Memory Network (LSTM) time series modeling, a solution to the problem is proposed.

[0010] The present invention provides a method for predicting dangerous actions of multiple workers, comprising the following steps:

[0011] The continuous video frame sequence of the factory environment is preprocessed to identify and track workers in the video frames, and a bounding box for each worker is generated. The bounding box includes a confidence level, and low-confidence bounding boxes are filtered out by a preset confidence threshold.

[0012] Based on the filtered bounding boxes, the image region where each worker is located is cropped from the video frame;

[0013] Extract the coordinates of key human body points from the image region of each worker and generate a skeleton map;

[0014] High-dimensional motion feature vectors are extracted from the skeletal graph data;

[0015] Time series modeling is performed based on the high-dimensional action feature vector sequence to predict dangerous action scores.

[0016] Furthermore, the preprocessing includes resolution adjustment and pixel normalization of the video frames.

[0017] Furthermore, workers in the video frames are identified and tracked using the YOLOv5 model, generating bounding boxes for each worker.

[0018] Furthermore, based on the image region of each worker, the OpenPose algorithm is used for pose estimation, extracting 25 key points as human body key point coordinates to generate a skeleton map.

[0019] Furthermore, a high-dimensional motion feature vector is extracted from the skeletal graph data, specifically as follows:

[0020] The skeletal map data is input into the KAN module, which models the nonlinear relationships between joints through a learnable activation function and extracts high-dimensional motion feature vectors.

[0021] Furthermore, a spline function is used as the learnable activation function in the KAN module.

[0022] Furthermore, the high-dimensional action feature vector sequence is input into the Long Short-Term Memory (LSTM) network module to perform time series modeling and predict dangerous action scores.

[0023] Furthermore, the confidence threshold is 0.5.

[0024] This invention also proposes a method for early warning of dangerous actions by multiple workers, which involves: firstly obtaining a predicted dangerous action score based on the method for predicting dangerous actions by multiple workers according to this invention, setting a preset score threshold θ, and triggering an early warning if the dangerous action score exceeds the preset score threshold.

[0025] This invention also proposes a multi-worker hazardous action prediction system, comprising:

[0026] The YOLOv5 module is used to perform multi-worker target detection, tracking, and generate worker bounding boxes from the input video frames. The image within the bounding box is the worker region image.

[0027] The skeleton map generation module extracts key human body points from the worker area image and generates a skeleton map.

[0028] The KAN network module is used to extract high-dimensional motion feature vectors representing nonlinear relationships between joints from skeletal graph data.

[0029] The LSTM module is used to model high-dimensional action feature vector sequences and output dangerous action scores.

[0030] Beneficial Effects: This invention proposes a method and system for predicting and dynamically warning of dangerous actions based on computer vision and deep learning technologies, specifically designed for complex multi-worker factory environments. Through an innovative dynamic warning mechanism, this invention adjusts the alarm intensity and frequency based on worker dangerous action scores, ensuring timely intervention before high-risk actions occur. Compared to traditional methods, this system significantly improves detection accuracy, real-time performance, and warning flexibility, effectively reducing safety accidents caused by fatigue or distraction, and is widely applicable to safety management in industrial production environments.

[0031] The method and system of this invention can effectively solve three main problems in providing timely warnings before workers engage in dangerous actions:

[0032] Complex multi-worker environments: The YOLOv5 module, through real-time target detection technology, can accurately identify and separate multiple workers in a scene, avoiding the interference problems that occur in complex backgrounds with traditional methods. This module ensures that the motion data of each worker is correctly captured and used for subsequent analysis through precise worker positioning and tracking.

[0033] The challenge of fine-grained analysis of motion changes: The combined use of the skeletal graph module and the KAN network module provides powerful feature extraction capabilities for subtle motion changes caused by fatigue. The skeletal graph module first extracts the precise location of each worker's joint, and then uses the non-linear feature mapping of the KAN module to capture the complex relationships and subtle changes between joints. KAN can effectively identify minute changes in motion and is more sensitive to capturing potential danger signals than traditional methods.

[0034] High real-time requirements: The advantages of the LSTM module in time series modeling enable the system to predict worker movements in real time. LSTM can process high-dimensional feature sequences from large-scale video data, quickly determine whether a worker is in a dangerous situation, and trigger early warnings in a timely manner based on the prediction results. By combining efficient computing modules and intelligent alarm strategies, the system meets the requirements of real-time processing and effectively avoids latency issues caused by computing resource limitations.

[0035] Through the coordinated operation of various modules in the system of this invention, the dangers of workers' actions can be accurately identified, the safety of the factory environment can be improved, and potential safety accidents can be warned and prevented in a timely manner. Attached Figure Description

[0036] Figure 1 This is the overall flowchart of the multi-worker dangerous action prediction and real-time early warning method of the present invention;

[0037] Figure 2 This is a flowchart of worker target detection and area cutting;

[0038] Figure 3 This is a flowchart of the skeleton map generation and feature extraction process;

[0039] Figure 4 This is a flowchart of the action feature extraction process for the KAN module;

[0040] Figure 5 This is a flowchart of LSTM temporal modeling and dangerous action prediction; Detailed Implementation

[0041] The invention will be further explained below with reference to the accompanying drawings and specific embodiments.

[0042] This invention, based on existing artificial intelligence and computer vision technologies, proposes a solution to three problems in existing factory worker dangerous action prediction systems: the complexity of multi-worker environments, the difficulty of fine-grained analysis of action changes, and the high real-time requirements. This solution effectively addresses these problems and significantly improves the accuracy of predicting dangerous actions of factory workers in videos.

[0043] The innovation of this invention lies in combining YOLOv5 object detection, skeleton map generation, KAN network, and LSTM model to accurately identify dangerous actions in multi-worker environments and issue real-time warnings through an intelligent early warning mechanism. By efficiently processing multiple worker actions in complex backgrounds, this system improves prediction accuracy and real-time response capabilities, significantly enhancing the safety of factory workers in their work environments.

[0044] Specifically, this invention proposes a method and system for predicting and providing real-time early warning of dangerous actions by factory workers based on computer vision and deep learning technologies, specifically designed for multi-worker environments. The system combines YOLOv5 object detection, skeletal graph generation, Kernel Activation Network (KAN), and LSTM models to achieve accurate identification and hazard prediction of worker actions. The YOLOv5 module detects and tracks multiple workers, the skeletal graph module extracts key point data, the KAN network sensitively captures subtle changes in actions through nonlinear feature extraction, and the LSTM module predicts potential future dangerous actions through time-series modeling. The system processes video data in real time, generates hazard scores, and dynamically adjusts alarm intensity and frequency based on these scores, issuing timely alerts to remind workers to avoid dangerous operations. This technology effectively addresses the limitations of traditional safety monitoring systems, improves the accuracy and real-time response capabilities of factory safety management, reduces safety accidents caused by fatigue or inattention, and is widely applicable to industrial production environments.

[0045] The following is a further explanation using a specific example. Imagine a factory environment where workers are moving heavy objects. This invention's system monitors the workers' movements in real time using video surveillance and analyzes the dynamic changes in their movements to predict potential hazards.

[0046] In a factory environment, surveillance cameras installed in the work area capture real-time video of workers carrying heavy objects. Each frame contains the worker's position, posture, and surrounding environment. Continuous video frames are extracted from the factory surveillance video and input into the multi-worker environment hazardous action prediction and real-time early warning system of this invention. The system of this invention includes a YOLOv5 module, a skeleton graph generation module, a KAN network module, an LSTM module, and an early warning module.

[0047] YOLOv5 Module: The video frames input to the YOLOv5 module are preprocessed, including resolution adjustment and normalization. These preprocessed video frames become standardized input data adapted to the YOLOv5 module. The YOLOv5 module performs target detection on the input image, accurately identifying and tracking each worker, generating a set of bounding boxes containing worker position and size information. Low-confidence boxes are filtered using a confidence threshold to ensure detection accuracy and address the challenges of multiple targets and complex backgrounds.

[0048] Skeletal map generation module: Extracts the two-dimensional coordinates of the worker's joints using posture estimation techniques (such as OpenPose), converts them into a skeletal map, and identifies potential danger signals by analyzing subtle changes in movement (such as posture changes caused by fatigue).

[0049] KAN network module: By modeling the nonlinear relationship between joints through learnable activation functions, it can keenly capture subtle changes in worker movements. In particular, when workers experience subtle changes in movement such as fatigue, the KAN network can improve the system's ability to identify dangerous movements.

[0050] LSTM Module: This module analyzes the changing trends of worker movements through time series modeling to predict potentially dangerous actions in the future. When the LSTM model predicts a potential hazard, the system will issue a timely warning.

[0051] Early warning module: Once the system predicts the possibility of a dangerous action, the alarm mechanism will issue a warning, such as by vibrating a smart bracelet or helmet worn by the worker, so that the worker can adjust his or her actions in time.

[0052] like Figure 1 As shown, the present invention also provides a rapid response method for predicting and providing real-time early warning of dangerous actions by multiple workers, the specific steps of which are as follows:

[0053] Step 1: As Figure 2 As shown, surveillance cameras installed in the factory capture real-time video footage of workers at a fixed frame rate (e.g., 30 frames per second), generating a series of continuous time-series video frames, also called a video stream, represented as:

[0054]

[0055] in:

[0056] : Represents the video frame captured at time step t. Here, These are video frames captured by the factory's surveillance cameras at time step t.

[0057] T: Represents the total number of frames in the time series, i.e., the number of video frames obtained from the monitoring equipment.

[0058] These video frames contain raw footage of the workers and their work environment; the data is unprocessed and directly obtained from the monitoring equipment. Since the posture, position, and number of workers in different frames may change over time, data preprocessing is crucial for subsequent analysis.

[0059] To ensure the input data is compatible with the object detection model (such as YOLOv5) and to guarantee the efficiency and accuracy of the analysis, the original video frames need to undergo the following preprocessing steps: To accelerate the model's convergence and avoid problems in numerical computation, the pixel values ​​of the input image need to be normalized to the range [0, 1]. The normalization formula is as follows:

[0060]

[0061] in, The original video frames, with each pixel value ranging from 0 to 255. It is a normalized video frame, where all pixel values ​​are scaled to the range [0, 1].

[0062] For example, a pixel in the original frame has an RGB value of [200, 100, 50], which becomes [0.784, 0.392, 0.196] after normalization. By normalizing pixel values ​​to the range of [0, 1], we ensure that the input data is on a uniform scale, which helps improve the training efficiency and accuracy of the model.

[0063] After resolution adjustment and normalization, the original video frames are converted into standardized video frames:

[0064]

[0065] These standardized video frames provide compatible input data for the YOLO v5 module, while significantly improving subsequent processing efficiency.

[0066] Step 2: This invention selects YOLOv5 as the object detection algorithm primarily based on its high computational performance, excellent detection accuracy, good engineering practicality, and strong scalability. Compared to YOLOv3 and YOLOv4, YOLOv5 has a faster inference speed (1.5~2 times), a more lightweight model, and supports PyTorch training, facilitating optimization and deployment. Compared to YOLOv7 and YOLOv8, YOLOv5 maintains high detection accuracy while having lower computational overhead, enabling efficient operation on general-purpose GPUs and edge devices. Furthermore, YOLOv5 has comprehensive community support and pre-trained weights, making object detection more stable and significantly reducing application costs. Therefore, YOLOv5 balances speed, accuracy, and ease of use, making it the best choice for worker detection and tracking in this project.

[0067] 2.1 Detection Principle

[0068] The YOLOv5 model divides the input image into an S×S grid (e.g., a 13×13 grid), with each grid responsible for detecting targets within its coverage area. For each grid, the model predicts multiple bounding boxes and their respective categories.

[0069] YOLOv5 employs a global awareness approach for object detection, dividing the entire input image into a fixed-size grid and allowing each grid to independently predict targets within its coverage area. Specifically:

[0070] The input image is divided into an S×S grid (e.g., if S=13, a 13×13 grid is formed). Each grid is responsible for detecting targets within its covered area; that is, if the center point of a target falls within a grid, that grid will predict the target's bounding box and its category information. Each grid predicts multiple bounding boxes and assigns a confidence score to each bounding box to measure whether the bounding box contains a target.

[0071] This design allows YOLOv5 to improve its ability to detect small targets while maintaining detection speed and reducing the problem of duplicate detection.

[0072] 2.2 Detection Framework

[0073] In the YOLOv5 detection framework, each bounding box is described by five key parameters:

[0074] b=(x,y,w,h,p)

[0075] Where (x, y) are the center coordinates of the bounding box, which are normalized coordinates relative to the input image. These coordinates are normalized values ​​relative to the input image, i.e., within the range [0, 1]. Assuming the input image size is 416×416, the x and y coordinate values ​​within this range represent percentages relative to the image size. For example, if the target center point is located at 50% width and 60% height of the image, then x = 0.5, y = 0.6.

[0076] w, h: The width and height of the bounding box, which are also normalized values ​​relative to the size of the input image.

[0077] p: Confidence Score, which represents the probability that the box contains the target. It is usually between [0,1]. The higher the value, the greater the probability that the box contains the target.

[0078] For example, the bounding box of a target detected in a certain frame might be: b1=(150,200,50,100,0.85)

[0079] This means that the center point of the target is located at (150, 200). The target's width is 50, its height is 100, and the confidence level p = 0.85, indicating that there is an 85% probability that the bounding box actually contains the target.

[0080] Standardized video frames The data is input into the YOLOv5 model for each frame. The YOLO v5 module outputs several bounding boxes and confidence scores; object detection is performed using the YOLO v5 module for each frame at time step t. The YOLO v5 module outputs multiple bounding boxes and their confidence scores, denoted as:

[0081]

[0082] Where n represents the number of workers detected, which is also the number of bounding boxes output. It is the first The center coordinates of the worker's bounding box. , They are the first The width and height of the worker's bounding box. It is the first The confidence level corresponding to the bounding box of each worker.

[0083] Assuming that based on a certain video frame, the YOLO v5 module detects two worker targets, then:

[0084] ={(100,120,40,80,0.9),(250,300,60,90,0.8)}

[0085] First worker objective: Bounding box center coordinates: (100, 120), bounding box size: 40×80, confidence level: 0.9 (i.e., 90% probability of being a worker). Second worker objective: Bounding box center coordinates: (250, 300), bounding box size: 60×90, confidence level: 0.8 (i.e., 80% probability of being a worker).

[0086] This approach ensures that YOLOv5 can efficiently identify multiple targets in each frame of an image and provide accurate location, size, and confidence information for each target.

[0087] 2.3 Confidence Filtering

[0088] In YOLOv5 object detection, to further improve detection accuracy, the system uses a confidence filtering method to ensure that the output object bounding boxes are more accurate and to reduce false positives and false negatives. Specifically, a confidence threshold is set for each bounding box. Filtering is performed. Only when the confidence level is... The bounding box will be retained only if it is above the threshold, otherwise it will be discarded.

[0089] In this way, from the original detection set The system filters out bounding boxes that meet the criteria, removing those with low confidence levels or those unlikely to contain the target (such as workers), thus generating a new, more accurate set of bounding boxes. The specific steps are as follows:

[0090] (1) The original set of detection results is as follows:

[0091]

[0092] Each bounding box contains its location ( (center coordinates), size () and

[0093] (for width and height) and confidence level .

[0094] By comparing the confidence level of each bounding box with a preset threshold This determines whether to retain the bounding box. Only when... When the confidence level is higher than 0.5, the bounding box will be retained to ensure the accuracy of the detection. In this invention, the threshold is set to 0.5, that is, the bounding box will only be retained when the confidence level is higher than 0.5.

[0095] (2) The filtered bounding box set is:

[0096]

[0097] For example, suppose the set of original bounding boxes detected by the YOLOv5 model in a certain video frame is as follows: ={(100,120,40,80,0.9),(250,300,60,90,0.4)}

[0098] For this set, the system applied a confidence filtering step:

[0099] The confidence level of the first bounding box is 0.9, which exceeds the threshold of 0.5, so it is retained.

[0100] The confidence score of the second bounding box is 0.4, which is below the threshold of 0.5, so it is discarded.

[0101] The filtered bounding box set is as follows:

[0102] ={(100,120,40,80,0.9)}

[0103] This filtering step ensures that only bounding boxes with high confidence are retained, thereby improving the accuracy and reliability of subsequent analysis.

[0104] 2.4 Extraction of worker image regions

[0105] For each frame image The system according to Bounding boxes selected from the middle Extract the image region where the worker is located. The specific steps are as follows:

[0106] (1) Extracting bounding box information: from Obtain the bounding box coordinates of each worker. .

[0107] (2) Extracting the image region based on the bounding box: Based on the coordinates and dimensions of the bounding box, the image region where the worker is located is cropped. This region is the area in the original image corresponding to the worker target. The specific operation is as follows:

[0108] Based on, bounding box Calculate the coordinates of the top left corner of the bounding box. and the coordinates of the bottom right corner It can be calculated using the following formula:

[0109] - , - ,

[0110] + , + ;

[0111] (3) Cropping regions from the original image:

[0112] Use the calculated top left corner coordinates and the coordinates of the bottom right corner (from the original image) Cut out the area where the workers are located :

[0113]

[0114] This area It is an image region containing all the key information of the target worker, including the worker's shape, posture and other information, which provides accurate input for subsequent posture estimation.

[0115] Example: Suppose that at time step t=5, YOLOv5 detected... One worker, generating bounding boxes for:

[0116] ={(150,200,50,100),(300,400,60,120),(500,600,70,140)}

[0117] The system will use these bounding boxes to crop out the corresponding worker areas from the original image. .

[0118] final, It is a sub-image containing workers extracted from the original image, which will then be fed into a pose estimation algorithm (such as OpenPose) for further processing and analysis.

[0119] Step 3, Skeleton Map Generation

[0120] Skeletal map generation is one of the core steps of the entire system. Its function is to extract the joint positions of each worker, simplifying complex pose information into a structured skeletal representation. The overall flowchart is as follows: Figure 3 As shown.

[0121] 3.1 Two-dimensional skeleton coordinates

[0122] Based on the detection results of the YOLO v5 module, the image region where the worker is located is cropped from each bounding box. , represented as:

[0123]

[0124] in, and These are the height and width of the cropped image, and 3 represents the RGB color channels.

[0125] OpenPose is used to process the cropped image region. (in frame t) Using cropped images of individual workers, OpenPose performs pose estimation, enabling precise localization of key joints in the human body. The two-dimensional coordinates of each joint are represented by the following formula:

[0126] =

[0127] in:

[0128] It is the t-th frame. The worker's first One key point;

[0129] It indicates specific key body points (such as head, shoulders, elbows, wrists, knees, ankles, etc.).

[0130] and The coordinates of the key point are two-dimensional.

[0131] Collect the skeletal keypoints of each worker in each frame:

[0132]

[0133] Where J represents the total number of joints. The skeletal diagram consists of key points. The lines connecting them form a connected graph G=(V,E), where:

[0134] V is the set of vertices in the graph, containing all the skeletal keypoints, i.e. =

[0135] E is the set of edges of the graph, representing the connections between joints (e.g., head to shoulder, shoulder to elbow, etc.).

[0136] For example, assuming OpenPose detects a human skeleton model with 25 joints, then the set of joint positions of a worker in frame t=5 is... It can be represented as:

[0137]

[0138] This represents the two-dimensional coordinates of the worker's 25 key joints.

[0139] 3.2 Connection of Skeleton Diagram

[0140] After extracting all key points, the system connects these joints into a complete skeletal diagram based on human anatomy. The skeletal diagram can be represented as a connected graph G=(V,E): where:

[0141] The vertex set V contains all joint positions. ;

[0142] The edge set E represents the connection relationship between joints, such as head to shoulder, shoulder to elbow, elbow to wrist, etc.

[0143] For example, for a human body model with 25 joints, the vertex set V contains all the joint nodes, while the edge set E contains 24 predefined skeletal connections.

[0144] 3.3 Construction of the skeleton matrix:

[0145] For subsequent motion analysis and hazard prediction, the system organizes the skeletal data of each worker in each frame of the image into a two-dimensional matrix. The shape of this matrix is...

[0146] =

[0147] Where J represents the total number of joints, and They represent the first The worker The x and y coordinates of each joint in the t-th frame of the image.

[0148] This skeletal map data contains the positions of all joints of the worker in each frame, serving as the basis for subsequent motion analysis. If multiple workers are detected in time step t, the skeletal map data of each worker is aggregated, represented as...

[0149]

[0150] in, It represents the number of workers detected in frame t.

[0151] Skeletal map generation involves taking a cropped image of each worker as input and using a pose estimation model (OpenPose) to generate the coordinates and connections of skeletal keypoints, ultimately outputting a structured skeletal map. By using convolutional neural networks and heatmap techniques, the location of each keypoint can be accurately identified, and these keypoints can be connected to form a complete skeletal map.

[0152] Step 4: In predicting dangerous movements of factory workers, accurately identifying subtle changes in movement and potential hazardous behaviors is crucial. Traditional neural networks (such as multilayer perceptrons, MLPs) face limitations in handling such tasks due to their fixed activation functions, failing to flexibly adapt to complex nonlinear relationships between joints. To address this issue, this invention introduces Kernel Activation Network (KAN), a deep learning model that models complex dynamic relationships between joints using learnable activation functions (such as spline functions). KAN effectively improves the accuracy of feature extraction and movement recognition capabilities, particularly excelling in predicting subtle movements and hazardous behaviors. The flowchart of the module is as follows... Figure 4 As shown.

[0153] KAN (Kernel Activation Network) is a deep learning model that extracts motion features using learnable activation functions. Unlike traditional multilayer perceptrons (MLPs), KAN captures the nonlinear relationships between joints in a worker's skeletal diagram by applying learnable activation functions such as spline functions to each connection edge, thereby improving the expressive power of the feature space. Through KAN, the system can more accurately capture subtle changes in worker movements, particularly excelling in fine-grained motion analysis such as fatigue detection. The output of KAN is a high-dimensional feature vector representing the worker's current motion state, which is then used as input to temporal models (such as LSTM) for motion prediction and hazard identification.

[0154] The KAN model takes skeletal graph data of a worker's movements as input. This skeletal graph data consists of the coordinates of key points on the worker's body extracted from video frames using pose estimation algorithms (such as OpenPose). For example, if a worker is lifting a heavy object, the skeletal graph data will record the positions of key joints such as the worker's arms, legs, and torso. The skeletal diagram data of a worker at time step t can be represented as follows:

[0155] =

[0156] This skeletal data It includes the positions of all the worker's joints in each frame, serving as the basis for subsequent motion analysis.

[0157] KAN's core innovation lies in using learnable activation functions (such as spline functions) to capture the complex nonlinear relationships between joints. For each pair of joints in the target worker, spline functions are used to capture their dynamic interactions at time step t. This process helps KAN sensitively identify subtle changes in the worker's posture, especially minute dynamics caused by fatigue or abnormal movements. In traditional neural networks, activation functions are usually fixed (such as ReLU or Sigmoid), while KAN uses learnable activation functions (such as spline functions), which can flexibly adapt to different motion data.

[0158] 4.1 Spline Functions

[0159] Suppose that during the handling process, the worker's arms and legs may exhibit different dynamic changes (e.g., arm swinging, leg bending). KAN will model these dynamic relationships using spline functions, thereby capturing subtle changes in motion, especially when the worker's posture is impaired due to fatigue.

[0160] For any two key points, the... and Nonlinear relationship at time step t Represented by spline functions:

[0161]

[0162] in: and They are the first The joint and the first The coordinates of a joint at time step t.

[0163] The spline function, a nonlinear function, is used to model the dynamic interactions between joints. As an activation function, the spline function can be flexibly adjusted according to different motion data, thereby capturing nonlinear changes between joints, especially subtle pose changes caused by fatigue or distraction. This allows KAN networks to more accurately capture the interactions between joints and reflect local features in motion.

[0164] 4.2 Construction of High-Level Eigenvectors

[0165] In KAN, generating high-dimensional feature vectors is the core task of the entire model. By modeling the nonlinear relationship between each pair of joints, KAN generates a high-dimensional feature vector that contains the worker's current motion state. This feature vector not only describes the local action but also encompasses the overall motion pattern.

[0166] When there are J joints, in this specific implementation J=25. KAN sums the nonlinear relationships between each pair of joints, and finally generates a global high-dimensional feature vector. This high-dimensional feature vector comprehensively describes the worker's movement state. It integrates the dynamic relationships between each pair of joints and comprehensively describes the worker's motion state, reflecting the worker's overall motion pattern and helping the system identify all dynamics from subtle posture changes to violent movements.

[0167] The formula for generating high-dimensional feature vectors is:

[0168]

[0169] in:

[0170] The i-th worker at time step The high-dimensional feature vector represents the overall action state of the worker at that time step.

[0171] J: Indicates the total number of sections;

[0172] Joint j and joint The nonlinear relationship is modeled using learnable activation functions such as spline functions.

[0173] It is the first Skeletal map data of a worker at time step t, including the coordinates of all joints.

[0174] By summing the relationships between all joints, a global feature vector is generated, which integrates the interactions of each pair of joints and can comprehensively describe the worker's current motion dynamics.

[0175] KAN generates a high-dimensional feature vector that comprehensively describes a worker's motion state by combining local feature modeling and global feature fusion. During the handling process, a worker's arms and legs may exhibit different motion characteristics (such as arm swings and leg strides). KAN captures these motion details through local feature modeling and generates a comprehensive motion feature vector through global feature fusion.

[0176] Step 5: The LSTM module performs time-series modeling on the high-dimensional action feature vector output by the KAN module, achieving dynamic capture of worker action change trends and real-time prediction of hazardous actions. Through gating mechanisms such as forget gates, input gates, and output gates, the LSTM updates the hidden state and cell state at each time step, comprehensively considering past state information and current features to generate a sequence of hidden states containing complete time-series dynamics. These hidden states reflect the changing trends of worker actions, providing support for subsequent hazardous action scoring. The workflow of this module is as follows: Figure 5 As shown.

[0177] In this system, the LSTM (Long Short-Term Memory) module is primarily used to process time-series data of worker actions and predict potentially dangerous actions by capturing trends in these actions. By recursively processing the worker action features at each time step, LSTM can remember important historical information and suppress irrelevant interference, thus providing the system with timely action predictions and early warnings. The core advantage of LSTM lies in its gating mechanism, which effectively handles long-term dependencies in time-series data and predicts potential dangerous worker behaviors based on real-time data, thereby triggering the early warning system and ensuring worker safety.

[0178] The core of an LSTM network consists of three main gates: the forget gate, the input gate, and the output gate. Each gate determines how to adjust the input based on specific weights and biases. and the previous hidden state Update cell state and latent state.

[0179] 5.1 The Gate of Oblivion

[0180] The function of the forget gate is to determine which information should be retrieved from the cell state of the previous time step. "Forgetting" in the context of the input features at the current time step. and the hidden state of the previous moment To make a decision. The output of the forget gate. The calculation is as follows:

[0181]

[0182] This is the output of the forget gate at time step t. It is a vector ranging from [0,1], representing the proportion of each piece of information that is forgotten. Values ​​close to 1 indicate retention, and values ​​close to 0 indicate forgetting.

[0183] It is the weight matrix of the forget gate, which determines the degree of influence of the input.

[0184] It is the bias term of the forget gate.

[0185] It is the sigmoid activation function, used to restrict the output to the range [0, 1].

[0186] For example, when a worker is carrying a heavy object, their posture is relatively stable at a certain moment, and they do not show any signs of fatigue. At time step t−1, the worker's leg-supported posture is very stable, and the cellular state is... The system stores the stable information at that moment. When the worker changes their action at time step t, assuming the worker's posture does not change significantly, the output of the forget gate... It might be close to 1, for example, This means that 90% of the information is retained and 10% is forgotten.

[0187] 5.2 Input Gate

[0188] The input gate determines the input features at the current time step. What information needs to be updated to the cell state? The input gate calculation process consists of two parts: updating the input ratio. and the generation of candidate cell states. .

[0189]

[0190] It is the output of the input gate at time step t, representing the proportion of information to be updated to the cell state.

[0191] It is the weight matrix of the input gate.

[0192] It is the bias term of the input gate.

[0193] It is the sigmoid activation function.

[0194] Formula for calculating candidate cell states:

[0195]

[0196] The state is the candidate cell state. The tanh activation function is used to restrict the output to the range [-1, 1] to provide new candidate information.

[0197] It is the weight matrix of the candidate cell states.

[0198] It is a bias term for candidate cell states.

[0199] For example, at the next time step t, the worker begins to bend slightly due to fatigue, and the movement changes. If the change in the worker's movement is significant (e.g., needing to adjust posture due to back bending), the output of the input gate... It will approach 1, indicating that all the current new information needs to be added to the cell state.

[0200] Candidate cell state It is a candidate value of the current action feature, which may be an evaluation of the current action (such as bending over). After being processed by the tanh function, its value is kept in the range of [-1, 1].

[0201] 5.3 Update cell state

[0202] The cell state is the most important memory unit in LSTM, storing long-term information about the time series. Cell state updates are achieved by incorporating a forgetting gate. and input gate The output is complete.

[0203]

[0204] : The cell state preserved from the previous moment;

[0205] : Newly added status information at the current time step.

[0206] It's the updated cell state, combining previous state information with current input information. (Forget gate) The input gate determines how much of the previous state is forgotten. It determines how much of the current state to include.

[0207] For example, at time step t, the worker's bending motion puts more strain on the body, and the input gate will allow more bending information to enter the cell state. The forget gate, on the other hand, determines the proportion of steady-state information retained from the previous time step.

[0208] if =0.9 and =1, then the updated cell state It combines 90% of the previous state information with 100% of the current new input information.

[0209] 5.4 Output Gate

[0210] Finally, output gate The hidden state at the current moment is determined. How should it be updated, and based on the current cell state? Output generated:

[0211]

[0212] in: It is the output of the output gate at time step t, representing the proportion of the current hidden state update. It is the weight matrix of the output gate. It is the bias term of the output gate.

[0213] Then, based on the current cell state and the result of the output gate Calculate the hidden state :

[0214]

[0215] It is the hidden state at the current moment, which is output as time step t and used to predict the worker's action state.

[0216] For example, suppose that at the current time step, the worker's bending angle is relatively large, the output gate will determine how to generate the hidden state based on the cell state. The output gate controls the update ratio of the hidden state through the sigmoid function, ultimately generating the current hidden state. .

[0217] Step 6: Time Series Modeling and Forecasting

[0218] Through the gating mechanism described above, LSTM can model the time series of worker actions. The hidden state at each time step... It contains the changing trends of worker movements. These latent states are updated through recursive calculations, ultimately capturing potential danger signals in the time series.

[0219] LSTM computes the hidden state at each time step recursively. Capture dynamic trends over time:

[0220]

[0221] in:

[0222] It is the hidden state at the current moment, reflecting the changing trend of the worker's actions.

[0223] It represents the input features at the current moment.

[0224] Hidden state sequence It contains the changing trends of worker actions throughout the entire time series. By analyzing these latent states, LSTM can effectively predict future worker actions and identify potentially dangerous actions.

[0225] The LSTM model models the time series of worker actions, generating the hidden state at each time step t. The hidden state represents the current state of the worker's action. (LSTM hidden state) It will be passed to a fully connected layer, and a dangerous action score at that moment will be calculated using a sigmoid activation function. This score reflects the probability that a worker will perform a dangerous action.

[0226] Hidden states of LSTM It will be passed to the fully connected layer and used to calculate the dangerous action score at that moment through a sigmoid activation function:

[0227]

[0228] , These are the weight matrix and bias terms of the fully connected layer;

[0229] It is the dangerous action score at time step t, with a value range of [0, 1], representing the probability that the worker will perform a dangerous action.

[0230] Assuming at time step t=5, the LSTM model calculates the hazard score as follows: This means that there is an 80% probability that workers will engage in dangerous actions (such as bending over excessively, which may lead to falls or injuries).

[0231] By comparing the predicted dangerous action scores The alarm is triggered based on a preset threshold θ. For example, if the threshold θ = 0.7, an alarm will be triggered when the hazard score is... When the threshold is exceeded, the system triggers an alert:

[0232]

[0233] In the example above, assuming the system sets the threshold θ = 0.7, the calculated hazard score is... If the score exceeds the threshold, the system will trigger an alarm.

[0234] Through this real-time response mechanism, the system can greatly improve worker safety and avoid accidents caused by improper posture or fatigue.

[0235] In modern factory environments, several technical challenges exist, such as motion recognition in multi-worker environments, fine-grained motion change analysis, and high real-time requirements. To address these issues, this invention proposes an efficient worker motion recognition and early warning system by integrating a YOLO v5 module, a skeletal graph module, a KAN network module, an LSTM module, and an early warning module, as detailed below:

[0236] The complexity of multi-worker environments: In complex factory environments, multiple workers may appear in the same frame simultaneously, making it difficult for traditional image analysis methods to accurately distinguish and track each worker. The YOLO v5 module, with its powerful object detection and localization capabilities, can accurately identify and effectively track each worker in complex backgrounds, thus avoiding interference between multiple workers and ensuring the accuracy of motion recognition.

[0237] The challenge of analyzing fine-grained motion changes: Workers' motion changes under fatigue often manifest as subtle posture shifts, which traditional image feature extraction methods are insufficient for capturing. By extracting the two-dimensional coordinates of joints using a skeletal graph module and combining this with a KAN network module for non-linear feature mapping, we can accurately capture minute changes between joints. Especially under fatigue or unstable postures, KAN can sensitively identify potentially dangerous movements.

[0238] High real-time performance requirements: In factory environments, dangerous actions can quickly lead to serious safety accidents, therefore, motion recognition systems need to be both real-time and efficient. The LSTM module uses efficient time-series modeling to predict the dynamic changes in worker movements in real time, and combined with an early warning module, quickly generates a hazard score. When the hazard score exceeds a threshold, the system can promptly trigger an alarm to ensure the safety of workers and the surrounding environment.

[0239] In summary, by integrating multiple advanced technologies, this invention fully addresses the limitations of traditional methods in complex environments, achieving accurate identification in multi-worker environments, effective capture of fine-grained motion changes, and efficient processing of real-time early warnings, thus providing a precise and efficient solution for factory safety management.

Claims

1. A multi-worker dangerous action prediction method, characterized by, Includes the following steps: The continuous video frame sequence of the factory environment is preprocessed to identify and track workers in the video frames, and a bounding box for each worker is generated. The bounding box includes a confidence level, and low-confidence bounding boxes are filtered out by a preset confidence threshold. Based on the filtered bounding boxes, the image region where each worker is located is cropped from the video frame; Extract the coordinates of key human body points from the image region of each worker and generate a skeleton map; High-dimensional motion feature vectors are extracted from the skeletal graph data; Time series modeling is performed based on the high-dimensional action feature vector sequence to predict dangerous action scores.

2. The method of claim 1, wherein, The preprocessing includes resolution adjustment and pixel normalization of video frames.

3. The method of claim 1, wherein, Workers in video frames are identified and tracked using the YOLOv5 model, generating bounding boxes for each worker.

4. The method of claim 1, wherein, Based on the image region of each worker, the OpenPose algorithm is used for pose estimation, extracting 25 key points as human body key point coordinates to generate a skeleton map.

5. The method of claim 1, wherein, The high-dimensional motion feature vector is extracted from the skeletal map data, specifically as follows: The skeletal map data is input into the KAN module, which models the nonlinear relationships between joints through a learnable activation function and extracts high-dimensional motion feature vectors.

6. The method of claim 1, wherein, Spline functions are used as the learnable activation functions in the KAN module.

7. The method of claim 1, wherein, The high-dimensional action feature vector sequence is input into the Long Short-Term Memory (LSTM) network module to perform time series modeling and predict dangerous action scores.

8. The method of claim 1, wherein, The confidence threshold is 0.

5.

9. A method for early warning of dangerous actions by multiple workers, characterized in that, Based on the prediction method described in claim 1, a predicted dangerous action score is obtained, and a preset score threshold θ is set. If the dangerous action score exceeds the preset score threshold, an early warning is triggered.

10. A multi-worker hazardous action prediction system, characterized in that, include: The YOLOv5 module is used to perform multi-worker target detection, tracking, and generate worker bounding boxes from the input video frames. The image within the bounding box is the worker region image. The skeleton map generation module extracts key human body points from the worker area image and generates a skeleton map. The KAN network module is used to extract high-dimensional motion feature vectors representing nonlinear relationships between joints from skeletal graph data. The LSTM module is used to model high-dimensional action feature vector sequences and output dangerous action scores.

Images