Personnel operation evaluation method and system based on video timing joint analysis, and electronic device

Abstract

This invention discloses a method, system, and electronic device for personnel operation assessment based on video temporal joint analysis, relating to the field of power safety technology. This application involves acquiring video from the work site and extracting frames; extracting the sequence of key human points from each frame; dynamically defining risk areas for the head, torso, and hands based on these key points; calculating hand movement speed, distance from the risk area, and movement consistency characteristics; triggering refined behavior recognition when hand movement characteristics indicate a tendency to enter the risk area; extracting multi-scale temporal features before and after the trigger; and judging specific risk behaviors using a pre-trained action recognition model. This invention, through a two-level mechanism of "trend triggering + refined recognition," achieves early warning and accurate identification of risk behaviors such as removing helmets and gloves, effectively improving the real-time performance and preventative capabilities of work safety monitoring. This application realizes a shift from post-event identification to pre-event warning, resulting in better safety monitoring performance.

Description

Technical Field

[0001] This invention relates to the field of power safety technology, specifically to a method, system, and electronic equipment for personnel operation assessment based on video time-series joint analysis, used for on-site safety monitoring of live-line work. Background Technology

[0002] Distribution network areas are located at the end of the power grid, with numerous points and a wide distribution area. They develop and change rapidly, and their equipment is used in a wide range of scenarios. The communication technologies involved are diverse, and the standardization and intelligence levels of facilities and equipment are not high. The data models and interface protocols of terminal equipment are not unified, and there are barriers to cross-professional collaborative data sharing. The contradiction between efficiently, quickly, and accurately responding to customer needs and dealing with complex and ever-changing fault conditions on site is becoming increasingly prominent, resulting in problems such as difficulty in real-time control of distribution network operation, high difficulty in operation and maintenance management, and low efficiency in business operation.

[0003] The digital, fully-perceptive intelligent distribution area follows the "cloud-pipe-edge-terminal" architecture design of the distribution Internet of Things (IoT). It utilizes IoT technology, cloud platforms, intelligent integrated terminals for distribution areas, and smart IoT switches to collect and access various operational status and external environmental data of the low-voltage distribution network. This helps maintenance personnel to fully grasp the real-time operating status of equipment, promptly issue fault warnings and handle them, reduce the workload of on-site maintenance personnel, improve maintenance efficiency, and ensure power supply reliability.

[0004] Currently, most data monitoring in existing smart distribution areas is concentrated on the low-voltage side of the area. Data monitoring for the 10kV side of the distribution area can only be achieved by adding a 10kV sectionalizing switch. This solution, which applies medium-voltage power distribution equipment to low-voltage distribution areas, is often not widely adopted in digital distribution areas due to its high cost, resulting in the monitoring points on the 10kV side of the smart distribution area becoming blind spots in monitoring.

[0005] This invention discloses a data detection method for the 10kV side of a distribution transformer area, which can solve the problem of blind spots in the monitoring of 10kV side data in current digital distribution transformer areas. Compared with the traditional solution of adding a 10kV boundary switch, this detection method can save one-third of the cost and is conducive to practical application and promotion.

[0006] The prior art, a Chinese patent with publication number CN116046055A, discloses a system and method for environmental safety monitoring of distribution networks, specifically relating to live-line working in distribution networks. The system includes: a live-line working module configured to collect analog status information of transmission lines at the work site and convert the collected status information into digital status information via A / D conversion; an information integration module configured to centrally integrate the received digital status information from the live-line working module to obtain work site status data; a safety monitoring module configured to receive and monitor the work site status data from the information integration module, and send a sudden change warning signal when the detected sudden change in the work site status data exceeds a preset range; and a warning signal module configured to receive the sudden change warning signal from the safety monitoring module and issue an alarm signal. The live-line working module includes a voltage isolation unit. Summary of the Invention

[0007] The technical problem to be solved by the present invention is to address the shortcomings of the prior art by disclosing a method, system and electronic equipment for personnel operation assessment based on video time series joint analysis, which solves the problem that the prior art can only identify risky behaviors such as removing helmets and gloves after the fact and cannot achieve early warning.

[0008] To achieve the above objectives, the present invention provides a method, system, and electronic device for personnel performance evaluation based on video temporal joint analysis, comprising the following steps:

[0009] S1: Acquire video data from the live-line work site, and extract frames from the video data based on a dynamic frame extraction algorithm to obtain an image sequence;

[0010] S2: Preprocess each frame of the image sequence to obtain a preprocessed image sequence; extract human key points from each frame of the preprocessed image sequence based on VITPose and perform filtering and smoothing to obtain a human key point sequence containing hand key points.

[0011] S3: Define head risk region, torso risk region and hand risk region based on human body key points in each preprocessed image frame. At the same time, calculate hand motion features based on hand key points in the human body key point sequence. The motion features include hand motion speed, distance between hand and risk region, and motion consistency feature value.

[0012] S4: Based on the hand's movement characteristics and the risk area, determine whether the hand has a tendency to enter the risk area. If so, extract the target image sequence for the target time period from the video data based on the judgment time point.

[0013] S5: Extract multi-scale temporal features of human key points in the target image sequence, and input the multi-scale temporal features into a pre-built action recognition model to obtain risk recognition results;

[0014] The action recognition model extracts sample features through a feature extraction network containing spatiotemporal convolutional networks, spatial attention mechanisms, and temporal attention mechanisms, and is trained using a cross-entropy loss function.

[0015] Furthermore:

[0016] In step S2, each frame of the image sequence is preprocessed to obtain a preprocessed image sequence, including:

[0017] Each frame of the image sequence is converted to grayscale to obtain a grayscale image sequence;

[0018] Each frame of grayscale image in the grayscale image sequence is filtered to obtain a preprocessed image sequence;

[0019] In step S2, human key points are extracted from each frame of the preprocessed image sequence to obtain a human key point sequence, including:

[0020] Human body key points are collected in each frame of the preprocessed image sequence based on VITPose.

[0021] For each key point, filtering and smoothing are performed to obtain the human body key point sequence.

[0022] Furthermore:

[0023] In S3, the definition of the head risk area includes:

[0024] Extract the coordinates of key points on the head and calculate the average coordinates of these key points to obtain the head center point. Based on the head center point Constructing a high-risk area The head risk area The mathematical expression is:

[0025] Formula 1: ;

[0026] In Formula 1, The pixel coordinates of the head risk area are represented. The set radius value;

[0027] In S3, defining the torso risk area includes:

[0028] Extracting the coordinates of key points on the neck Coordinates of key points in the pelvis Based on the coordinates of the key neck points Coordinates of key points in the pelvis Constructing a risk zone for the torso The torso risk area The mathematical expression is:

[0029] Formula 2: ;

[0030] In formula 2, This represents the coordinates of a point within the risk area of ​​the torso. For linear interpolation parameters, For normal offset parameters, Coordinates of key points in the pelvis Constructing a risk zone for the torso The normal vector, To set the width value;

[0031] In S3, defining the hand risk area includes:

[0032] Extracting the coordinates of key points on the wrist Coordinates of key points on the thumb and the coordinates of the index finger key points Based on the coordinates of the wrist key points Coordinates of key points on the thumb and the coordinates of the index finger key points Calculate the first distance between the wrist key point and the index finger key point. The second distance between the wrist key point and the thumb key point And the angle of the line connecting the key point on the wrist to the key point on the index finger. and based on the first distance The second distance and the angle of the connecting line Constructing hand risk zones The hand risk area The mathematical expression is:

[0033] Formula 3:

[0034] ;

[0035] In formula 3, The coordinates of the key points on the wrist.

[0036] Furthermore:

[0037] The calculation of hand movement speed in S3 includes:

[0038] The calculation of hand motion features based on hand key points in the human body key point sequence includes:

[0039] Calculate the motion velocity from the coordinates of hand key points in any two adjacent frames. , ,in, Frame index, For the first The coordinates of the hand key points corresponding to the frame. For the first The coordinates of the hand key points corresponding to the frame. The time interval between adjacent frames;

[0040] Motion speed based on multiple adjacent frames Calculate hand movement speed The hand movement speed The mathematical expression is:

[0041] Formula 4: ,in, Number of frames;

[0042] Extract the hand region of the current hand from the last frame of data. And calculate the hand region of the current hand. With the target risk area distance ;

[0043] The target risk area is one of the head risk area, the torso risk area, and the target hand risk area.

[0044] The distance The mathematical expression is:

[0045] Formula 5: ;

[0046] In formula 5, As the first weight, Indicates the hand area With the target risk area Hausdorf distance, As the second weight, This indicates taking the minimum value. Indicates the hand area The point in the middle, Indicates the target risk area The point in the middle;

[0047] Extracting the motion speed of hand key points from multiple frames And extract the motion direction of the hand key points from multiple frames and the direction angle of the reference line. ;

[0048] The reference line is the line connecting the key hand points and the center point of the target risk area; the motion speed across multiple frames. and direction angle Normalization and standard deviation calculations are performed to obtain the speed consistency parameter. and direction consistency parameters and based on the speed consistency parameter and direction consistency parameters Calculate motion consistency eigenvalues ;

[0049] Among them, the motion consistency feature value The mathematical expression is:

[0050] Formula 6: ;

[0051] In formula 6, As the third weight, It is the fourth weight.

[0052] Furthermore:

[0053] In S4, the determination of whether the hand tends to enter the risk area based on the hand's movement characteristics and the risk area includes:

[0054] When the hand speed is greater than or equal to a preset speed threshold, the distance between the hand and the risk area is less than a preset distance threshold, and the motion consistency is greater than or equal to a preset consistency threshold, it is determined that the hand has a tendency to enter the risk area; otherwise, it is determined that the hand does not have a tendency to enter the risk area.

[0055] Furthermore:

[0056] In step S5, multi-scale temporal features of human key points in the target image sequence are extracted, including:

[0057] Constructing a human keypoint sequence based on target image sequences;

[0058] The sequence of human key points is input into a feature extraction network that includes a spatiotemporal convolutional network, a spatial attention mechanism, and a temporal attention mechanism. Multi-scale temporal features are extracted through multi-scale convolution.

[0059] Furthermore:

[0060] In step S5, the method for constructing the action recognition model includes:

[0061] S5.1, Obtain action sample data, wherein the action sample data includes image sequences of test subjects simulating various risky behaviors;

[0062] S5.2 Extract multi-scale temporal feature samples from the action sample data, and input the multi-scale temporal feature samples into the artificial neural network to obtain the probability of predicting risk behavior type;

[0063] S5.3 Calculate the probability of the predicted risk behavior type and the loss of the actual risk behavior type based on the pre-configured loss function, and adjust the parameters of the artificial neural network based on the backpropagation of the loss;

[0064] S5.4 Repeat steps S5.2-S5.3 until training is complete and the action recognition model is obtained.

[0065] Furthermore:

[0066] Formula 7: ;

[0067] In formula 7, Indicates loss, Index for dangerous behavior categories, The English for categories of dangerous behavior One-hot encoding representing the type of real risky behavior. To predict the probability of risky behavior types.

[0068] A personnel performance evaluation system based on video temporal joint analysis includes:

[0069] The acquisition module is used to acquire video data from the live-line working site; and to extract frames from the video data based on a dynamic frame extraction algorithm to obtain an image sequence.

[0070] The key point extraction module is used to preprocess each frame of the image sequence to obtain a preprocessed image sequence; and to extract human key points from each frame of the preprocessed image sequence to obtain a human key point sequence, wherein the human key points include hand key points.

[0071] The risk region extraction module is used to define risk regions based on human key points in each frame of preprocessed image, and to calculate hand motion features based on hand key points in the human key point sequence. The risk regions include a head risk region constructed based on head key points, a hand risk region constructed based on hand key points, and a torso risk region constructed based on torso key points.

[0072] The pre-judgment module is used to determine whether the hand has a tendency to enter the risk area based on the hand's movement characteristics and the risk area. If so, it extracts the target image sequence of the target time period from the video data based on the judgment time point.

[0073] The recognition module is used to extract multi-scale temporal features of human body key points in the target image sequence, and to recognize the multi-scale temporal features based on a pre-built action recognition model to obtain risk recognition results.

[0074] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the method described above.

[0075] The beneficial effects of this invention are as follows:

[0076] This invention discloses a personnel operation assessment method and system based on video temporal joint analysis. The method involves acquiring video of the work site and extracting frames; extracting the sequence of human key points from each frame; dynamically defining risk areas for the head, torso, and hands based on these key points; calculating hand movement speed, distance from the risk area, and movement consistency characteristics; triggering refined behavior recognition when hand movement characteristics indicate a tendency to enter the risk area; extracting multi-scale temporal features before and after the trigger; and determining specific risky behaviors using a pre-trained action recognition model. This invention, through a two-level mechanism of "trend triggering + refined recognition," achieves early warning and accurate identification of risky behaviors such as removing helmets and gloves, effectively improving the real-time performance and preventative capabilities of operational safety monitoring. This application represents a shift from post-event identification to pre-event warning, resulting in better safety monitoring performance. Attached Figure Description

[0077] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0078] Figure 1 This is a diagram illustrating an application scenario of a personnel performance evaluation method based on video temporal joint analysis, as shown in one embodiment of this application.

[0079] Figure 2 This is a flowchart illustrating a personnel performance evaluation method based on video temporal joint analysis in one embodiment of this application;

[0080] Figure 3 This is a flowchart illustrating the training and application of an action recognition model in one embodiment of this application.

[0081] Figure 4 This is a structural diagram of a personnel performance evaluation system based on video temporal joint analysis, as shown in one embodiment of this application.

[0082] Figure 5 A schematic diagram of the structure of a computer system suitable for implementing the electronic device of the present application is shown. Attached Figure Description

[0083] 110: Camera; 120: Cloud server; 130: On-site safety management personnel;

[0084] 501: Central Processing Unit; 502: Read-Only Memory (ROM); 503: Random Access Memory (RAM); 504: Bus; 505: Input / Output (I / O) Interface; 506: Input Section; 507: Output Section; 508: Storage Section; 509: Communication Section; 510: Driver; 511: Removable Media. Detailed Implementation

[0085] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0086] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0087] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0088] Example 1

[0089] like Figure 1 This is a diagram illustrating an application scenario of a personnel performance evaluation method based on video temporal joint analysis, as shown in one embodiment of this application. Figure 1 As shown, in this application, camera 110 captures original video at a distance of 1–3 meters from the work site. Frames extracted from the original video are sent to cloud server 120 for analysis and processing. The analysis and processing results are then sent to on-site safety management personnel 130 for real-time monitoring of hazardous points on-site. Furthermore, through the above scenario, a single safety officer can simultaneously monitor multiple live-line construction scenarios.

[0090] Example 2

[0091] Figure 2 This is a flowchart illustrating a personnel performance evaluation method based on video temporal joint analysis in one embodiment of this application, as shown below. Figure 2 The personnel job evaluation method based on video temporal joint analysis in this embodiment may include steps S1 to S2:

[0092] S1: Acquire video data from the live-line work site; and extract frames from the video data based on a dynamic frame extraction algorithm to obtain an image sequence;

[0093] The video stream is acquired by the on-site camera, and an adaptive frame-dropping algorithm (such as adjusting the frame rate according to the amplitude of motion in the image) is used to reduce the amount of data processing while ensuring that no information is lost, and output an image sequence. .

[0094] S2: Preprocess each frame of the image sequence to obtain a preprocessed image sequence; and extract human key points from each frame of the preprocessed image sequence to obtain a human key point sequence, wherein the human key points include hand key points.

[0095] Specifically, the preprocessing process includes:

[0096] S2.1, perform grayscale processing on each frame of the image sequence to obtain a grayscale image sequence;

[0097] S2.2, Filter each frame of grayscale image in the grayscale image sequence to obtain a preprocessed image sequence.

[0098] The above process converts each frame of the image to grayscale to reduce information dimensionality and improve the efficiency of subsequent operations, and uses preprocessing such as Gaussian filtering to reduce noise.

[0099] Methods for extracting human key point sequences include:

[0100] S2.3, Collect human body key points in each frame of the preprocessed image in the preprocessed image sequence based on VITPose;

[0101] S2.4, For each key point, filter and smooth it to obtain the human body key point sequence.

[0102] Specifically, the ViTPose high-precision model is used to detect 18 key points of the human body in the preprocessed image, and temporal filtering (such as Kalman filtering) is applied to the key point coordinate sequence to smooth out jitter, outputting a stable sequence of human body key points. .

[0103] S3, define risk regions based on human body key points in each preprocessed image frame, and calculate hand motion features based on hand key points in the human body key point sequence. The risk regions include a head risk region constructed based on head key points, a hand risk region constructed based on hand key points, and a torso risk region constructed based on torso key points.

[0104] To determine whether workers use their hands to perform dangerous actions such as removing hats, unbuttoning, and taking off gloves, this application first extracts the areas where hands must overlap with the protective clothing during these actions. Specifically, the head risk area is the area where the protective hat is located, the hand risk area is the area where the gloves are located, and the torso risk area is the area where the protective clothing closures (buttons, zippers, etc.) are located. The extraction process for these risk areas includes:

[0105] (1) Head risk area

[0106] Extract the coordinates of key points on the head and calculate the average coordinates of these key points to obtain the head center point. Based on the head center point Constructing a high-risk area The head risk area The mathematical expression is:

[0107] Formula 1: ;

[0108] In Formula 1, The pixel coordinates of the head risk area are represented. The set radius value;

[0109] When worn, a helmet or safety helmet is spatially fitted tightly to the wearer's head. In an image, the head region can be approximated as a circular or elliptical area. This method calculates the geometric center of the face or head by detecting multiple key points (such as the eyes, ears, and nose). Using this center as the center, and with a pre-defined radius sufficient to cover the size of a typical safety helmet... By drawing a circle, the dynamic risk area where the protective helmet is located can be defined on the image plane. The core principle is to use key points on the human body as anchor points to transform the static position of the protective equipment into a geometric area that dynamically changes with the human body.

[0110] Compared to the rectangular area of ​​the head detection frame, the circular area better fits the physical contour of the helmet, reduces the irrelevant areas introduced by the four corners of the rectangle, and makes risk assessment more focused.

[0111] (2) Risk area of ​​the trunk

[0112] Extracting the coordinates of key points on the neck Coordinates of key points in the pelvis Based on the coordinates of the key neck points Coordinates of key points in the pelvis Constructing a risk zone for the torso The torso risk area The mathematical expression is:

[0113] Formula 2: ;

[0114] In formula 2, This represents the coordinates of a point within the risk area of ​​the torso. For linear interpolation parameters, For normal offset parameters, Coordinates of key points in the pelvis Constructing a risk zone for the torso The normal vector, To set the width value;

[0115] The zippers, buttons, and other closures of protective suits are typically located along the front midline of the torso, extending from near the neck to the waist or pelvis. This method uses a zipper starting from a key point on the neck. To the key points of the pelvis The line segment is used to simulate the centerline of this closed element. The risk area is defined as a certain width extending to both sides of this line segment. The resulting narrow rectangular band. The orientation and length of the rectangle are determined by... and Real-time decision-making adapts to changes in posture such as bending over and turning, ensuring that the risk area is synchronized with the body's posture. Compared to using a large rectangle to frame the entire torso, this method defines an area that more closely matches the actual distribution of risk points, greatly reducing interference from non-risk areas and improving the signal-to-noise ratio for subsequent trend assessments.

[0116] (3) Hand risk areas

[0117] Extracting the coordinates of key points on the wrist Coordinates of key points on the thumb and the coordinates of the index finger key points Based on the coordinates of the wrist key points Coordinates of key points on the thumb and the coordinates of the index finger key points Calculate the first distance between the wrist key point and the index finger key point. The second distance between the wrist key point and the thumb key point And the angle of the line connecting the key point on the wrist to the key point on the index finger. and based on the first distance The second distance and the angle of the connecting line Constructing hand risk zones The hand risk area The mathematical expression is:

[0118] Formula 3:

[0119]

[0120] In formula 3, The coordinates of the key points on the wrist.

[0121] When the hand performs actions such as grasping or touching, the space it occupies is approximately an ellipse, and the direction of the ellipse is roughly the same as the direction of the fingers.

[0122] This method utilizes the wrist. Thumb tip and the tip of the index finger These three key points are used to fit the ellipse. The elliptical region fits the actual contour of the hand better than a point or circle. When the hand approaches a risk area, the overlap or distance calculation based on the ellipse can trigger an early and more accurate warning. In addition, the model naturally incorporates simple hand gestures (pointing), providing more dimensions of features for behavior understanding.

[0123] In addition, the hand movement features, including hand movement speed, distance between the hand and the risk area, and movement consistency feature values, are extracted through the following process:

[0124] S3.1 Extract the coordinates of the hand key points of the target hand from the human body key point sequence, and calculate the motion speed based on the hand key point coordinates of any two adjacent frames. ; and motion speed based on multiple adjacent frames Calculate hand movement speed ,

[0125] Among them, the hand movement speed The mathematical expression is: ;

[0126] Formula 4: ;

[0127] Formula 4, Frame index, For the number of frames, For the first The coordinates of the hand key points corresponding to the frame. For the first The coordinates of the hand key points corresponding to the frame. The time interval between adjacent frames;

[0128] Hand movement velocity is a fundamental physical quantity describing its motion state. This method estimates velocity by tracking the positional changes of specific hand keypoints (usually wrist keypoints due to their high stability) across consecutive video frames. For two adjacent frames, the velocity... Defined as a displacement vector Divide the modulus by the time interval To smooth out noise in single-frame calculations and reflect recent overall motion trends, the instantaneous velocity of consecutive T frames is calculated. Take the arithmetic mean to obtain the average hand movement speed. Its mathematical essence is discretized first-order difference and average filtering.

[0129] S3.2, Extract the hand region of the current hand from the last frame of data. And calculate the hand region of the current hand. With the target risk area distance The target risk area is one of the following: the head risk area, the torso risk area, and the risk area of ​​the object's hands. The distance... The mathematical expression is:

[0130] Formula 5: ;

[0131] In formula 5, As the first weight, Indicates the hand area With the target risk area Hausdorf distance, As the second weight, This indicates taking the minimum value. Indicates the hand area The point in the middle, Indicates the target risk area The point in the middle;

[0132] This step aims to quantify the spatial proximity between the hand and the target risk area (such as the head or torso). A fusion distance metric strategy is employed, combining Hausdorff distance and minimum point distance.

[0133] Hausdorff distance d_Hausdorff: Calculates the distance between two point sets (hand region). and risk areas A maximum and minimum distance between two shapes. It measures how difficult it is for two shapes to “overlap” or “match” each other, and is sensitive to the edges and shapes of the entire region.

[0134] Minimum point distance: Calculates the minimum Euclidean distance between all pairs of points in two regions. It directly reflects the absolute gap between the nearest points in the two regions.

[0135] Final distance It is the weighted sum of the two, with the weights being... and It can be adjusted according to the scenario; for example, when more attention is paid to the overall overlap trend, it can be increased. When focusing more on extreme proximity situations, it can increase .

[0136] A single Hausdorff distance can be sensitive to noise at the region's edges, while a single minimum point distance can lead to misjudgments due to fluctuations at individual points. Combining the two can more stably and comprehensively characterize complex spatial proximity relationships.

[0137] S3.3 Extract the motion speed of hand key points from multiple frames. And extract the motion direction of the hand key points from multiple frames and the direction angle of the reference line. The reference line is the line connecting the key hand points and the center point of the target risk area; the motion speed for multiple frames. and direction angle Normalization and standard deviation calculations are performed to obtain the speed consistency parameter. and direction consistency parameters and based on the speed consistency parameter and direction consistency parameters Calculate motion consistency eigenvalues The motion consistency feature value The mathematical expression is:

[0138] Formula 6:

[0139] In formula 6, As the third weight, It is the fourth weight.

[0140] Consistency characteristics are used to determine whether hand movements approaching risk areas have a clear purpose and stability. It is measured from two dimensions: consistency of speed magnitude and consistency of movement direction. Specifically, it includes:

[0141] Speed ​​consistency Calculate the instantaneous velocity of the most recent N frames. Normalized standard deviation The smaller the standard deviation, the smaller the velocity fluctuation and the smoother the motion; subtracting this value from 1 results in a larger value, which represents higher consistency.

[0142] Consistency of direction Calculate the angle between the motion direction of the most recent N frames and the target pointing (the line connecting the hand point to the center of the risk area, i.e., the "reference line"). Normalized standard deviation The smaller the fluctuation of the included angle, the more directly the hand movement trajectory points to the target; similarly, subtracting this value from 1 makes the larger value represent a more consistent direction.

[0143] Final consistency eigenvalues It is a weighted sum of the two. Weight and It can be used to adjust the contribution of speed and direction factors to overall consistency.

[0144] Unconscious limb movements or adjustments typically exhibit significant variations in speed and direction (low consistency), while conscious actions of picking up or manipulating an object usually follow a more stable and direct trajectory (high consistency). This characteristic is key to reducing false alarms. By using consistency as one of the triggering conditions, the alarm is ensured to be triggered only when the hand approaches the risk area "stablely" and "purposefully," greatly filtering out accidental movements.

[0145] S4. Based on the hand's movement characteristics and the risk area, determine whether the hand has a tendency to enter the risk area. If so, extract the target image sequence for the target time period from the video data based on the determination time point.

[0146] Specifically, when the hand speed is greater than or equal to a preset speed threshold, the distance between the hand and the risk area is less than a preset distance threshold, and the motion consistency is greater than or equal to a preset consistency threshold, it is determined that the hand has a tendency to enter the risk area; otherwise, it is determined that the hand does not have a tendency to enter the risk area.

[0147] This step is the core decision-making logic in the two-level early warning system described in this application. It utilizes the three quantitative features extracted above—hand movement speed (… ), distance between hands and risk areas ( ), motion consistency ( — and the preset threshold (speed threshold) Distance threshold Consistency threshold The comparison is performed using a logical AND condition. Its mathematical model is expressed as follows:

[0148] ;

[0149] The system determines that the hand is likely to enter the risk area only when all three of the above inequalities are true, and records that moment as the trigger point. This initiates the subsequent refined behavior recognition process. Otherwise, if the trend is deemed risk-free, the system remains in a low-computational-load monitoring state.

[0150] Specifically, based on the trigger time point Based on the baseline, the trigger time point is collected. The video then lasts approximately 1.5 seconds, and frame extraction is performed to obtain the target image sequence.

[0151] S5, extract multi-scale temporal features of human body key points in the target image sequence, and identify the multi-scale temporal features based on the pre-built action recognition model to obtain risk recognition results.

[0152] Example 3

[0153] Figure 3 This is a flowchart illustrating the training and application of an action recognition model in one embodiment of this application, as follows: Figure 3 As shown, this application pre-constructs an action recognition model based on spatiotemporal convolution, spatiotemporal attention, and multi-size features. This action recognition model is the core classifier used to precisely identify specific risk behavior categories after the front-end trend judgment is triggered. Its core principle lies in utilizing deep neural networks, especially the ability of spatiotemporal joint modeling, to automatically learn and extract deep dynamic patterns from video clips containing sequences of human key points that can identify different risk behaviors, such as "removing a helmet," "taking off gloves," and "unzipping a protective suit."

[0154] The model architecture principle is as follows:

[0155] Spatiotemporal Convolutional Network: This component forms the backbone of the model. Its spatial convolutional part is typically implemented using a Graph Convolutional Network (GCN). Within each frame, based on the human skeleton connectivity graph, it learns the spatial relationship patterns between joints, such as the coordination between the hand and head, and the elbow and shoulder when removing a helmet. The temporal convolutional part slides along the time dimension, learning the evolutionary patterns of the movement (such as the dynamic process of the hand moving from bottom to top and grasping). The combination of these two components enables joint modeling of the spatiotemporal texture of movements.

[0156] Spatial attention: Enables the model to dynamically focus on the body parts most relevant to the current recognition task. For example, when determining "take gloves", the attention mechanism will give higher weight to areas such as hands and wrists, while reducing attention to irrelevant parts such as legs.

[0157] Temporal attention: Enables the model to focus on the most discriminative keyframes or segments in an action sequence. For example, when recognizing "unbuttoning," the moment when a finger touches the button and applies force may be given higher importance.

[0158] Multi-size feature extraction: By using convolutional kernels of different sizes or extracting features at different network depths, the model can capture action information at different time scales. This is crucial for distinguishing between fast actions (quickly removing a hat) and slow actions (slowly unzipping a zipper).

[0159] Methods for constructing action recognition models include:

[0160] S1, acquire action sample data, wherein the action sample data includes image sequences of test subjects simulating various risky behaviors;

[0161] S2, extract multi-scale temporal feature samples from the action sample data, and input the multi-scale temporal feature samples into an artificial neural network to obtain the probability of predicting risky behavior types; the extraction process can be summarized as follows:

[0162] S21, Construct a human key point sequence based on human key points in the target image sequence;

[0163] S22, the sequence of human body key points is input into a feature extraction network that includes a spatiotemporal convolutional network, a spatial attention mechanism and a temporal attention mechanism, and multi-scale temporal feature samples are extracted through multi-scale convolution.

[0164] S3, calculate the probability of the predicted risk behavior type and the loss of the actual risk behavior type based on the pre-configured loss function, and adjust the parameters of the artificial neural network based on the backpropagation of the loss;

[0165] The mathematical expression for the loss function is:

[0166] Formula 7: ;

[0167] In formula 7, Indicates loss, Index for dangerous behavior categories, The English for categories of dangerous behavior One-hot encoding representing the type of real risky behavior. To predict the probability of risky behavior types.

[0168] S4. Repeat steps S2-S3 until training is complete and the action recognition model is obtained.

[0169] The cross-entropy loss function used in this application. It is the standard supervision signal for classification tasks. Mathematically, it means minimizing the difference between the probability distribution predicted by the model and the one-hot distribution of the true labels. When the predicted probability... With real labels The closer the match (i.e., for the true category) , The closer to 1), the greater the loss. The smaller the value, the better. Through the backpropagation algorithm, this loss value is used to adjust all parameters (weights and biases) in the network, driving the model to learn feature representations that can accurately distinguish different risky behaviors.

[0170] After obtaining the recognition model, the multi-scale temporal features of human body key points in the target image sequence are extracted in the manner described above and input into the action recognition model, so as to identify whether the worker's action is a risky action (such as "unbuttoning", "removing hat" etc.).

[0171] This invention discloses a personnel operation assessment method based on video temporal joint analysis. The method involves acquiring video of the work site and extracting frames; extracting the sequence of human key points from each frame; dynamically defining risk areas for the head, torso, and hands based on these key points; calculating hand movement speed, distance from the risk area, and movement consistency characteristics; triggering refined behavior recognition when hand movement characteristics indicate a tendency to enter the risk area; extracting multi-scale temporal features before and after the trigger; and determining specific risky behaviors using a pre-trained action recognition model. This invention, through a two-tiered mechanism of "trend triggering + refined recognition," achieves early warning and accurate identification of risky behaviors such as removing helmets and gloves, effectively improving the real-time performance and preventative capabilities of operational safety monitoring. This application represents a shift from post-event identification to pre-event warning, resulting in better safety monitoring performance.

[0172] Example 4

[0173] like Figure 4 As shown, this application also provides a personnel performance evaluation system based on video temporal joint analysis, including:

[0174] The acquisition module is used to acquire video data from the live-line working site; and to extract frames from the video data based on a dynamic frame extraction algorithm to obtain an image sequence.

[0175] The key point extraction module is used to preprocess each frame of the image sequence to obtain a preprocessed image sequence; and to extract human key points from each frame of the preprocessed image sequence to obtain a human key point sequence, wherein the human key points include hand key points.

[0176] The risk region extraction module is used to define risk regions based on human key points in each frame of preprocessed image, and to calculate hand motion features based on hand key points in the human key point sequence. The risk regions include a head risk region constructed based on head key points, a hand risk region constructed based on hand key points, and a torso risk region constructed based on torso key points.

[0177] The pre-judgment module is used to determine whether the hand has a tendency to enter the risk area based on the hand's movement characteristics and the risk area. If so, it extracts the target image sequence of the target time period from the video data based on the judgment time point.

[0178] The recognition module is used to extract multi-scale temporal features of human body key points in the target image sequence, and to recognize the multi-scale temporal features based on a pre-built action recognition model to obtain risk recognition results.

[0179] This invention discloses a personnel operation assessment system based on video temporal joint analysis. The system acquires video footage from the work site and extracts frames; extracts the sequence of human key points from each frame; dynamically defines risk areas for the head, torso, and hands based on these key points; calculates hand movement speed, distance from the risk area, and movement consistency characteristics; triggers refined behavior recognition when hand movement characteristics indicate a tendency to enter the risk area; extracts multi-scale temporal features before and after the trigger, and uses a pre-trained action recognition model to determine specific risky behaviors. This invention, through a two-tiered mechanism of "trend triggering + refined recognition," achieves early warning and accurate identification of risky behaviors such as removing helmets and gloves, effectively improving the real-time performance and preventative capabilities of operational safety monitoring. This application represents a shift from post-event identification to pre-event warning, resulting in better safety monitoring performance.

[0180] Example 5

[0181] like Figure 5 As shown, it illustrates a schematic diagram of the structure of a computer system suitable for implementing the electronic devices of the present application.

[0182] like Figure 5 As shown, the computer system specifically includes the following components:

[0183] The computer system includes a Central Processing Unit (CPU) 501, which can perform various appropriate actions and processes, such as executing the methods described in the above embodiments, based on programs stored in Read-Only Memory (ROM) 502 or programs loaded from storage portion 508 into Random Access Memory (RAM) 503. RAM 503 also stores various programs and data required for system operation. The CPU 501, ROM 502, and RAM 503 are interconnected via a bus 504. An Input / Output (I / O) interface 505 is also connected to the bus 504.

[0184] The following components are connected to I / O interface 505: an input section 506 including a keyboard, mouse, etc.; an output section 507 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 408 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to I / O interface 505 as needed. Removable media 511, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 510 as needed so that computer programs read from them can be installed into storage section 508 as needed.

[0185] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program including a computer program for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 509, and / or installed from removable medium 511. When the computer program is executed by central processing unit (CPU) 501, it performs various functions defined in the system of this application.

[0186] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying a computer-readable computer program. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0187] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0188] The units described in the embodiments of this application can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0189] Another aspect of this application provides a computer-readable storage medium storing a computer program thereon, which, when executed by a computer's processor, causes the computer to perform the method as described above. This computer-readable storage medium may be included in the electronic device described in the above embodiments, or it may exist independently and not assembled into the electronic device.

[0190] Another aspect of this application provides a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the various embodiments described above.

[0191] Furthermore, although the operations of the method of the present invention are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

Claims

1. A method for evaluating personnel performance based on video temporal joint analysis. Its features are, Including the following steps: S1: Acquire video data from the live-line work site, and extract frames from the video data based on a dynamic frame extraction algorithm to obtain an image sequence; S2: Preprocess each frame of the image sequence to obtain a preprocessed image sequence; Based on VITPose, human key points are extracted from each frame of the preprocessed image sequence and filtered and smoothed to obtain a human key point sequence containing hand key points. S3: Define head risk region, torso risk region and hand risk region based on human body key points in each preprocessed image frame. At the same time, calculate hand motion features based on hand key points in the human body key point sequence. The motion features include hand motion speed, distance between hand and risk region, and motion consistency feature value. S4: Based on the hand's movement characteristics and the risk area, determine whether the hand has a tendency to enter the risk area. If so, extract the target image sequence for the target time period from the video data based on the judgment time point. S5: Extract multi-scale temporal features of human body key points in the target image sequence, and input the multi-scale temporal features into a pre-built action recognition model to obtain risk recognition results; The action recognition model extracts sample features through a feature extraction network containing spatiotemporal convolutional networks, spatial attention mechanisms, and temporal attention mechanisms, and is trained using a cross-entropy loss function.

2. The personnel performance evaluation method based on video temporal joint analysis according to claim 1, Its features are, In step S2, each frame of the image sequence is preprocessed to obtain a preprocessed image sequence, including: Each frame of the image sequence is converted to grayscale to obtain a grayscale image sequence; Each frame of grayscale image in the grayscale image sequence is filtered to obtain a preprocessed image sequence; In step S2, human key points are extracted from each frame of the preprocessed image sequence to obtain a human key point sequence, including: Human body key points are collected in each frame of the preprocessed image sequence based on VITPose. For each key point, filtering and smoothing are performed to obtain the human body key point sequence.

3. The personnel performance evaluation method based on video temporal joint analysis according to claim 1, Its features are, In S3, the definition of the head risk area includes: Extract the coordinates of key points on the head and calculate the average coordinates of these key points to obtain the head center point. Based on the head center point Constructing a high-risk area The head risk area The mathematical expression is: Official 1: ; In Formula 1, The pixel coordinates of the head risk area are represented. The set radius value; In S3, defining the torso risk area includes: Extracting the coordinates of key points on the neck Coordinates of key points in the pelvis Based on the coordinates of the key neck points Coordinates of key points in the pelvis Constructing a risk zone for the torso The torso risk area The mathematical expression is: Official 2: ; In formula 2, This represents the coordinates of a point within the risk area of ​​the torso. For linear interpolation parameters, For normal offset parameters, Coordinates of key points in the pelvis Constructing a risk zone for the torso The normal vector, To set the width value; In S3, defining the hand risk area includes: Extracting the coordinates of key points on the wrist Coordinates of key points on the thumb and the coordinates of the index finger key points And based on the coordinates of the wrist key points Coordinates of key points on the thumb and the coordinates of the index finger key points Calculate the first distance between the wrist key point and the index finger key point. The second distance between the wrist key point and the thumb key point And the angle of the line connecting the key point of the wrist to the key point of the index finger. and based on the first distance The second distance and the angle of the connecting line Constructing hand risk zones The hand risk area The mathematical expression is: Formula 3: ； In formula 3, The coordinates of the key points on the wrist.

4. The personnel performance evaluation method based on video temporal joint analysis according to claim 1, Its features are, The hand movement speed is calculated in S3. include: The calculation of hand motion features based on hand key points in the human body key point sequence includes: Calculate the motion velocity from the coordinates of hand key points in any two adjacent frames. , ,in, Frame index, For the first The coordinates of the hand key points corresponding to the frame. For the first The coordinates of the hand key points corresponding to the frame. The time interval between adjacent frames; Motion speed based on multiple adjacent frames Calculate hand movement speed The hand movement speed The mathematical expression is: Formula 4: ,in, Number of frames; Extract the hand region of the current hand from the last frame of data. And calculate the hand region of the current hand. With the target risk area distance ; The target risk area is one of the head risk area, the torso risk area, and the target hand risk area. The distance The mathematical expression is: Official 5: ; In formula 5, As the first weight, Indicates the hand area With the target risk area Hausdorf distance, As the second weight, This indicates taking the minimum value. Indicates the hand area The point in the middle, Indicates the target risk area The point in the middle; Extracting the motion speed of hand key points from multiple frames And extract the motion direction of the hand key points from multiple frames and the direction angle of the reference line. ; The reference line is the line connecting the key hand points and the center point of the target risk area; the motion speed across multiple frames. and direction angle Normalization and standard deviation calculations are performed to obtain the speed consistency parameter. and direction consistency parameters and based on the speed consistency parameter and direction consistency parameters Calculate motion consistency eigenvalues ; Among them, the motion consistency feature value The mathematical expression is: Official 6: ; In formula 6, As the third weight, It is the fourth weight.

5. The personnel performance evaluation method based on video temporal joint analysis according to claim 1, Its features are, In S4, the determination of whether the hand tends to enter the risk area based on the hand's movement characteristics and the risk area includes: When the hand speed is greater than or equal to a preset speed threshold, the distance between the hand and the risk area is less than a preset distance threshold, and the motion consistency is greater than or equal to a preset consistency threshold, it is determined that the hand has a tendency to enter the risk area; otherwise, it is determined that the hand does not have a tendency to enter the risk area.

6. The personnel performance evaluation method based on video temporal joint analysis according to claim 1, Its features are, In step S5, multi-scale temporal features of human key points in the target image sequence are extracted, including: Constructing a human keypoint sequence based on target image sequences; The sequence of human key points is input into a feature extraction network that includes a spatiotemporal convolutional network, a spatial attention mechanism, and a temporal attention mechanism. Multi-scale temporal features are extracted through multi-scale convolution.

7. The personnel performance evaluation method based on video temporal joint analysis according to claim 1, Its features are, In step S5, the method for constructing the action recognition model includes: S5.1, Obtain action sample data, wherein the action sample data includes image sequences of test subjects simulating various risky behaviors; S5.2 Extract multi-scale temporal feature samples from the action sample data, and input the multi-scale temporal feature samples into the artificial neural network to obtain the probability of predicting risk behavior type; S5.3 Calculate the probability of the predicted risk behavior type and the loss of the actual risk behavior type based on the pre-configured loss function, and adjust the parameters of the artificial neural network based on the backpropagation of the loss; S5.4 Repeat steps S5.2-S5.3 until training is complete and the action recognition model is obtained.

8. The personnel performance evaluation method based on video temporal joint analysis according to claim 7, Its features are, The mathematical expression for the loss function is: Official 7: ; In formula 7, Indicates loss, Index for dangerous behavior categories, The English for categories of dangerous behaviors One-hot encoding representing the type of real risky behavior. To predict the probability of risky behavior types.

9. A personnel performance evaluation system based on video temporal joint analysis. The personnel performance evaluation method based on video temporal joint analysis as described in any one of claims 1 to 8 is adopted. Its features are, include: The acquisition module is used to acquire video data from the live-line work site. The video data is then processed by a dynamic frame extraction algorithm to obtain an image sequence. The key point extraction module is used to preprocess each frame of the image sequence to obtain a preprocessed image sequence; Human key points are extracted from each frame of the preprocessed image sequence to obtain a human key point sequence, wherein the human key points include hand key points. The risk region extraction module is used to define risk regions based on human key points in each frame of preprocessed image, and to calculate hand motion features based on hand key points in the human key point sequence. The risk regions include a head risk region constructed based on head key points, a hand risk region constructed based on hand key points, and a torso risk region constructed based on torso key points. The pre-judgment module is used to determine whether the hand has a tendency to enter the risk area based on the hand's movement characteristics and the risk area. If so, it extracts the target image sequence of the target time period from the video data based on the judgment time point. The recognition module is used to extract multi-scale temporal features of human body key points in the target image sequence, and to recognize the multi-scale temporal features based on a pre-built action recognition model to obtain risk recognition results.

10. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. Its features are, When the processor executes the program, it implements the steps of the method as described in any one of claims 1 to 8.

Images