Excavator-human collision accident warning method and system based on computer vision

Abstract

The present invention relates to a computer vision-based early warning method and system for excavator-human collision accidents. The present invention first uses computer vision algorithms such as target detection, 2D key point detection, and 3D posture recognition to perform 3D skeleton recognition of workers and excavators on the construction site video stream; secondly, based on the 3D motion posture of the excavator's 3D skeleton in the working state, the 3D dynamic danger zone around the excavator in the working state is delineated; finally, whether the key points of the worker's 3D skeleton have invaded the 3D dynamic danger zone around the excavator in the working state is judged as a basis for whether there is a danger of collision between the excavator and the human, and if there is a danger, an early warning is issued. Compared with the existing technology, the present invention constructs a 3D dynamic danger zone around the excavator in the working state through computer vision technology, improves the accuracy and effectiveness of the automatic early warning of the excavator-human collision accident, avoids the occurrence of misjudgment to the greatest extent, and is conducive to the intelligent safety management of the construction site.

Description

Technical Field

[0001] The present invention relates to the field of computer vision and construction safety technology, and in particular to a computer vision-based excavator-human collision accident early warning method and system. Background Art

[0002] The construction industry is labor-intensive. Project deadlines and existing technical requirements often require workers and heavy machinery to operate simultaneously, inevitably leading to a certain degree of human-machine interaction. Due to the varying skills and qualifications of workers, the high noise levels of the construction site environment, the visual blind spots associated with machine operation, and the inattention caused by fatigue, collisions and injuries are highly likely to occur when workers and heavy machinery are working together. Among construction machinery, excavators, with their high degree of freedom of movement and frequent interaction with workers during operation, have a high incidence of collisions between excavators and humans. Therefore, establishing an early warning system for collisions between workers and excavators is of great practical significance.

[0003] In recent years, with the advancement of computer software and hardware technologies and the increase in computing power, the application of computer vision and deep learning technologies in fields such as facial recognition and autonomous driving has been booming. In certain areas, such as object detection, recognition accuracy has approached or even exceeded that of humans. In the field of civil engineering construction safety management, there are already relatively mature computer vision-based systems for detecting the wearing of hard hats and protective clothing. Currently, the use of computer vision technology to detect unsafe worker behavior and unsafe mechanical equipment conditions is also increasing, demonstrating the advantage of computer vision technology in construction safety supervision without interfering with on-site production operations.

[0004] In the existing technology, the traditional excavator-human collision accident warning method based on on-site inspections by management personnel cannot achieve full coverage at all times, is easily affected by human subjective factors, is inefficient and has poor results.

[0005] Furthermore, collision prevention for heavy machinery and personnel at construction sites encompasses two main approaches: sensor-based and visual-based. Excavators are heavy construction machinery with flexible arms and a high degree of freedom. They move relatively little during operation, and their articulated arms experience frequent changes in posture. Therefore, when preventing and detecting collisions between excavators and personnel, the focus is often on locating the excavator bucket and identifying the arm's posture, rather than on the overall positioning of the excavator. This results in poor detection results. Summary of the Invention

[0006] The present invention aims to overcome the shortcomings of the aforementioned prior art by providing a computer vision-based excavator-to-person collision accident warning method and system. This method utilizes video footage captured by construction site monitoring facilities and computer vision algorithms such as image processing and deep learning to provide early warning of excavator-to-person collision accidents without disrupting on-site production operations. This reduces potential construction site accidents, prevents them, and improves construction site safety management.

[0007] The purpose of the present invention can be achieved by the following technical solutions:

[0008] A computer vision-based early warning method for an excavator-human collision accident includes the following steps:

[0009] Obtain construction site video stream;

[0010] Inputting the construction site video stream into an accident warning model to determine the dynamic danger zone of the excavator;

[0011] Determine whether there is anyone in the dynamic danger zone of the excavator, and if so, issue an early warning;

[0012] The construction of the accident warning model includes the following steps:

[0013] S1. Construct an excavator target detection model and a human target detection model; obtain an excavator construction site image dataset, and train and verify the excavator target detection model and the human target detection model;

[0014] S2. Build an excavator recognition model based on deep learning to extract the excavator's 2D key points and restore their 3D coordinates.

[0015] S3. Build a human body recognition model based on deep learning to extract 2D key points of the human body and restore their 3D coordinates;

[0016] S4. Based on the 3D coordinate restoration results of the key points of the excavator, quantitatively extract the operating status indicators of the excavator and delineate the 3D dynamic danger zone under the working state of the excavator;

[0017] S5. Use the PnP-based coordinate system method to unify the human coordinate system and the excavator coordinate system, and identify and monitor the behavior of people invading the 3D dynamic dangerous area when the excavator is working.

[0018] Furthermore, in step S1, the ACID construction image dataset is used as the basic dataset, and the basic dataset is expanded using Mosaic and CutMix data enhancement technologies;

[0019] Based on the excavator construction site image dataset, the Yolox convolutional neural network is trained to obtain an excavator target detection model and a human target detection model.

[0020] Furthermore, in step S2, constructing an excavator identification model based on deep learning includes the following steps:

[0021] The Hourglass neural network was trained using open source datasets and data augmentation methods to obtain an excavator 2D key point detection Hourglass model that can identify six key points on the excavator's arm and body.

[0022] Use modeling software to model the excavator, bind the excavator's motion during construction, and write a program to output the 2D coordinates of the six key points of the excavator in the image coordinate system and the 3D coordinates in the world coordinate system during the excavator's motion. Generate a data set that converts the coordinates of the excavator's 2D key points into 3D coordinates.

[0023] A Transformer model is trained based on a data set of the excavator 2D key point coordinates converted into 3D coordinates to obtain a Transformer model of the excavator 2D key point coordinates converted into 3D coordinates.

[0024] Furthermore, in step S3, constructing a human recognition model based on deep learning includes the following steps:

[0025] The Stacked Hourglass neural network is trained using the MSCOCO dataset to obtain a human 2D key point detection model.

[0026] The VideoPose3D neural network is trained using the Human3.6M dataset to obtain a human 3D skeleton recognition model.

[0027] Furthermore, in step S4, delineating the 3D dynamic danger zone in the working state of the excavator includes the following steps:

[0028] Using the Yolox-Hourglass-Transformer model, a 3D dynamic skeleton of the excavator during operation is obtained from the construction site video stream and then post-processed.

[0029] Using the Savgol filter, polynomial fitting is performed on the continuous signal points within the sliding window to filter out abnormal points in the 3D skeleton recognition results when the excavator is in operation.

[0030] The motion of the excavator in working state is decomposed into overall rotation and mechanical arm extension, and the motion state of the excavator is expressed using local coordinate system and global coordinate system;

[0031] The position parameters of the skeleton key points describing the motion posture of the excavator in the working state are converted into four angle parameters θ1, θ2, θ3, and θ4 through the global coordinate system and the local coordinate system. Among them, θ4 is used to express the overall rotation of the excavator in the global coordinate system, and θ1, θ2, and θ3 are used to express the extension of the excavator's manipulator in the xy plane of the local coordinate system.

[0032] Calculate the current angular velocity of the excavator's four angle parameters:

[0033]

[0034] Define the average worker reaction time Δt, and use the 3D area that the excavator's robotic arm may sweep during the worker's average reaction time as the 3D dynamic danger zone when the excavator is working. That is, calculate the excavator's motion trend based on Δt:

[0035]

[0036] The 3D area formed by the changes in the values ​​of the four angle parameters is obtained as the 3D dynamic danger area under the working state of the excavator.

[0037] Furthermore, the local coordinate system of the excavator takes the vertical plane where the mechanical arm is located as the xy plane, and the Y axis of the global coordinate system of the excavator coincides with the y axis of the local coordinate system. T and (X, Y, Z) in the global coordinate system T The coordinate conversion between them is:

[0038]

[0039]

[0040] Where boom_arm represents the key point of the middle hinge of the excavator's mechanical arm, and cab_boom represents the key point of the connection between the excavator's cab and the mechanical arm.

[0041] Furthermore, the calculation formulas for θ1, θ2, and θ3 are:

[0042]

[0043]

[0044]

[0045] Where right_bucket_end represents the key point of the right end of the excavator bucket, and left_bucket_end represents the key point of the left end of the excavator bucket.

[0046] Furthermore, in step S5, a coordinate system method based on PnP is used to unify the human body coordinate system and the excavator coordinate system, including the following steps:

[0047] If the homogeneous form of the 3D coordinates of a skeleton key point in the world coordinate system is P w =(X w ,Y w ,Z w ,1) T , and the homogeneous form of the 2D coordinates of the skeleton key point in the image pixel coordinate system is p = (u, v, 1) T When , the conversion relationship between the 3D coordinates and 2D coordinates of the skeleton key point is:

[0048]

[0049] Among them, K is the camera intrinsic parameter matrix,

[0050]

[0051] T is the transformation matrix from the world coordinate system to the camera coordinate system, and is the optimization parameter to be solved.

[0052]

[0053] Among them, R is the rotation matrix and t is the translation matrix;

[0054] Call the solvePnP function in the OpenCV computer vision library to solve the problem. The final optimized transformation matrix is ​​T * , the human coordinate system and the excavator coordinate system are converted to the camera coordinate system by the following formula:

[0055]

[0056] Furthermore, in step S5, identifying and monitoring the behavior of a person intruding into the 3D dynamic danger zone in the working state of the excavator includes the following steps:

[0057] By inversely transforming the human coordinate system and the excavator coordinate system into the camera coordinate system, the coordinates of the human body and the excavator are transformed from the camera coordinate system to the excavator coordinate system, so that the human coordinate system is incorporated into the excavator coordinate system;

[0058] Obtain the projection of the 3D dynamic danger zone around the excavator in the working state on the XZ plane of the global coordinate system and the projection on the xy plane of the local coordinate system respectively;

[0059] Obtain the projection of the key points of the human body 3D skeleton on the XZ plane of the global coordinate system and the projection on the xy plane of the local coordinate system respectively;

[0060] If the key points of the human body's 3D skeleton coincide with the projections of the 3D dynamic danger zone around the excavator in the working state on the XZ plane of the global coordinate system and the xy plane of the local coordinate system, it is considered that there is a hidden danger of a collision between the excavator and the worker, and an early warning is issued.

[0061] A computer vision-based early warning system for collisions between an excavator and a person, comprising a data acquisition module, a model training module, and an early warning module;

[0062] The data acquisition module is used to acquire the construction site video stream;

[0063] The model training module is used to train the accident warning model;

[0064] The early warning module is used to input the construction site video stream into the accident early warning model to determine the dynamic danger zone of the excavator; determine whether there is anyone in the dynamic danger zone of the excavator, and if so, issue an early warning;

[0065] The model training module constructs the accident warning model, which includes the following steps:

[0066] S1. Construct an excavator target detection model and a human target detection model; obtain an excavator construction site image dataset, and train and verify the excavator target detection model and the human target detection model;

[0067] S2. Build an excavator recognition model based on deep learning to extract the excavator's 2D key points and restore their 3D coordinates.

[0068] S3. Build a human body recognition model based on deep learning to extract 2D key points of the human body and restore their 3D coordinates;

[0069] S4. Based on the 3D coordinate restoration results of the key points of the excavator, quantitatively extract the operating status indicators of the excavator and delineate the 3D dynamic danger zone under the working state of the excavator;

[0070] S5. Use the PnP-based coordinate system method to unify the human coordinate system and the excavator coordinate system, and identify and monitor the behavior of people invading the 3D dynamic dangerous area when the excavator is working.

[0071] Compared with the prior art, the present invention has the following beneficial effects:

[0072] 1. This paper migrates and improves the existing computer vision algorithm for human 3D skeleton recognition, constructs an algorithm model for 2D key point recognition and 3D skeleton recognition of the excavator manipulator arm, and improves the accuracy of the 3D posture estimation algorithm under the working state of the excavator based on the monocular camera video stream.

[0073] 2. This invention uses the 3D coordinates of key points on the excavator skeleton and their motion states to establish a model for identifying and demarcating the 3D dynamic danger zone around the excavator while it is operating, enabling refined management of this 3D dynamic danger zone. A detection model for worker intrusion into the 3D dynamic danger zone around the excavator while it is operating was created, establishing a warning framework for excavator-human collision accidents, thus supporting refined and intelligent safety management at construction sites. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] Figure 1 It is a schematic diagram of the process of the present invention.

[0075] Figure 2 Schematic diagram of the excavator skeleton model in an embodiment of the present invention.

[0076] Figure 3 Schematic diagram of a human skeleton model in an embodiment of the present invention.

[0077] Figure 4 Schematic diagram of the global coordinate system and local coordinate system of an excavator in an embodiment of the present invention, where (a) is a three-dimensional view and (b) is a top view.

[0078] Figure 5 Schematic diagram of the skeleton posture angle parameters of the excavator in the working state in an embodiment of the present invention, wherein (c) is a schematic diagram of the mechanical arm when it is extended, and (d) is a schematic diagram of the whole body when it is rotated.

[0079] Figure 6 Schematic diagram of judging whether the key points of the worker's skeleton are located within the surrounding 3D dynamic danger zone when the excavator is in working condition in an embodiment of the present invention, wherein (e) is a schematic diagram of the local coordinate system xy plane, and (f) is a schematic diagram of the global coordinate system XZ plane. DETAILED DESCRIPTION

[0080] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0081] In response to the shortcomings of the existing technology, the present invention proposes a set of computer vision-based early warning systems for collision accidents between excavators and humans. First, through computer vision algorithms such as target detection, 2D key point detection, and 3D posture recognition, the 3D skeletons of workers and excavators are recognized in the construction site video stream; secondly, based on the 3D motion posture of the excavator's 3D skeleton in the working state, the 3D dynamic danger zone around the excavator in the working state is delineated; finally, it is judged whether the key points of the worker's 3D skeleton have invaded the 3D dynamic danger zone around the excavator in the working state, which serves as the basis for whether there is a danger of collision between the excavator and the human. If there is a danger, an early warning is issued. The present invention constructs a 3D dynamic danger zone around the excavator in the working state through computer vision technology. Under the condition of no intrusive interference with the front-line production operations at the construction site, it improves the accuracy and effectiveness of the automatic early warning of collision accidents between excavators and humans, avoids the occurrence of misjudgment to the greatest extent, and is conducive to the intelligent safety management of the construction site.

[0082] like Figure 1 As shown, a computer vision-based early warning method for an excavator-human collision accident includes the following steps:

[0083] Obtain construction site video stream;

[0084] Input the construction site video stream into the accident warning model to determine the dynamic danger zone of the excavator;

[0085] Determine whether there is anyone in the excavator's dynamic danger zone, and if so, issue an early warning;

[0086] The construction of the accident warning model includes the following steps:

[0087] S1. Build an excavator target detection model and a human target detection model; obtain an excavator construction site image dataset, and train and verify the excavator target detection model and the human target detection model;

[0088] S2. Build an excavator recognition model based on deep learning to extract the excavator's 2D key points and restore their 3D coordinates.

[0089] S3. Build a human body recognition model based on deep learning to extract 2D key points of the human body and restore their 3D coordinates;

[0090] S4. Based on the 3D coordinate restoration results of the key points of the excavator, quantitatively extract the operating status indicators of the excavator and delineate the 3D dynamic danger zone under the working state of the excavator;

[0091] S5. Use the PnP-based coordinate system method to unify the human coordinate system and the excavator coordinate system, and identify and monitor the behavior of people invading the 3D dynamic dangerous area when the excavator is working.

[0092] Specifically, the implementation process of the present invention includes the following steps:

[0093] In a computer vision-based excavator-human collision warning system, potential collisions are detected based on the video stream captured by a monocular camera. Therefore, the excavator and worker targets in the video must first be detected. Like other supervised deep learning algorithms, object detection algorithms are trained on a dataset, with parameters updated through backpropagation. The dataset and neural network model are the two most important components of this process.

[0094] For excavator target detection, this embodiment uses the open source ACID construction image dataset as the basic dataset for excavator target detection, and expands the basic image dataset through data enhancement technologies such as Mosaic and CutMix to form training and verification datasets that meet the training requirements of deep learning models.

[0095] The expanded ACID construction image dataset is used to train the Yolox convolutional neural network to obtain an excavator recognition model for construction sites with high accuracy and generalization performance.

[0096] After obtaining the excavator target in the construction site video stream, it is necessary to perform 2D key point detection and 3D skeleton recognition on the excavator.

[0097] For the 2D key point detection of the excavator, we use open source datasets and data enhancement methods to train the Hourglass neural network, and obtain the Hourglass model for 2D key point detection of the excavator, which can identify the six key points of the excavator's mechanical arm and fuselage.

[0098] For the 3D skeleton recognition task of the excavator, collecting training data is quite difficult. Therefore, we used C4D software to model the excavator and then bound the excavator's motion movements during construction. Then, using the Python secondary development interface provided by C4D software, we wrote a program to directly output the 2D coordinates of the excavator's six key points in the image coordinate system and the 3D coordinates in the world coordinate system during the movement.

[0099] Using the above dataset generation method, a large number of datasets can be generated for the task of converting the 2D key point coordinates of the excavator into 3D coordinates. The dataset is used to train the Transformer model to obtain the Transformer model for converting the 2D key point coordinates of the excavator into 3D coordinates. Figure 2 Table 1 shows the definitions of the six key points of the excavator.

[0100] Table 1 Definition of key points of excavator

[0101] Serial number variable name Chinese name (1) Body_end Excavator body tail end point (2) Cab_boom Connection point between cockpit and robotic arm (3) Boom_arm Middle hinge point of the robotic arm (4) Arm_bucket Connection point between the robotic arm and bucket (5) Left_bucket_end Left end point of bucket (6) Right_bucket_end Bucket right end point

[0102] For human target detection, 2D key point detection, and 3D skeleton recognition, a method similar to the excavator task mentioned above is used. This embodiment uses a human target detection model based on Yolox, a human 2D key point detection model based on Stacked Hourglass, and a human 3D skeleton recognition model based on VideoPose3D, and is trained using the MSCOCO dataset and the Human3.6M dataset. Figure 3 and Table 2 shows the definitions of 17 key points of the human body.

[0103] Table 2 Definition of key points of human body

[0104] Serial number variable name Chinese name (1) root pelvis (2) right_hip right hip (3) right_knee right knee (4) right_foot right foot (5) left_hip left hip (6) left_knee left knee (7) left_foot left foot (8) spine spine (9) thorax chest cavity (10) neck_base cervical spine (11) head head (12) left_shoulder left shoulder (13) left_elbow left elbow (14) left_wrist left wrist (15) right_shoulder right shoulder (16) right_elbow right elbow (17) right_wrist right wrist

[0105] After obtaining the 3D skeleton of the excavator, the next step is to delineate the 3D dynamic danger zone around the excavator during operation.

[0106] First, a Yolox-Hourglass-Transformer model is used to obtain the 3D dynamic skeleton of the excavator during operation from the construction site video stream. This is the 3D coordinate sequence of the key points of the skeleton during the excavator's operation, and then post-processed. This embodiment uses a Savgol filter to perform polynomial fitting on the continuous signal points within the sliding window to achieve a smoothing effect. This filter removes some abnormal points in the 3D skeleton recognition results during the excavator's operation, and also facilitates the subsequent extraction of the excavator's motion state.

[0107] After smoothing the 3D coordinate sequence of the excavator's skeleton key points with a Savgol filter, the excavator's kinematic parameters can be calculated. The excavator's skeleton has six key points, which, if analyzed independently, would have 18 degrees of freedom. However, the excavator's inherent articulation creates numerous constraints. Based on the excavator's connection structure and kinematic characteristics, the excavator's kinematic state can be decomposed into overall rotation and arm extension. Local and global coordinate systems are then used to better represent the excavator's kinematic state.

[0108] like Figure 4 As shown, XYZ is the global coordinate system and xyx is the local coordinate system. The local coordinate system of the excavator takes the vertical plane where the mechanical arm is located as the xy plane. The Y axis of the global coordinate system of the excavator coincides with the y axis of the local coordinate system. (x, y, z) in the local coordinate system T and (X, Y, Z) in the global coordinate system T The coordinate transformation between them is realized by the rotation angle θ obtained by formula (1) and the matrix transformation shown in formula (2).

[0109]

[0110]

[0111] Through the two coordinate systems of global coordinate system and local coordinate system, the original 18 position parameters of the 6 skeleton key points that describe the motion posture of the excavator in working state can be reduced to Figure 5 The four angle parameters θ1, θ2, θ3 and θ4 are shown. Since the length between each two key points of the excavator arm remains unchanged, only the above four angle parameters are needed to describe the skeleton posture and movement of the excavator in the working state. Among them, θ4 can express the overall rotation of the excavator in the global coordinate system, which is equal to Figure 4 The angle θ represents the rotation between the local coordinate system and the global coordinate system; θ1, θ2, and θ3 can express the extension of the excavator's manipulator arm in the xy plane of the local coordinate system, and can be calculated using the formulas shown in (3), (4), and (5).

[0112]

[0113]

[0114]

[0115] By reducing the 18 position parameters of the skeleton posture of the excavator in the working state to 4 angle parameters through the above transformation, the obtained 4 angle parameters are independent of each other, which is conducive to the subsequent description of the excavator's motion state.

[0116] The current angular velocity of the excavator's four angular parameters can be calculated using Equation (6). The excavator's motion trend can then be calculated using the first-order Taylor formula shown in Equation (7). Referring to existing research, this embodiment defines the 3D area that the excavator's robotic arm may sweep within the worker's average reaction time of 2.5 seconds as the 3D dynamic danger zone surrounding the excavator during operation. This is the 3D area formed by the changes in the values ​​of the four angular parameters calculated by substituting Δt = 2.5 seconds into Equation (7).

[0117]

[0118]

[0119] After obtaining the 3D dynamic danger zone around the excavator in working state and the 3D skeleton of the worker, the coordinate systems of the worker and the excavator are unified using the PnP algorithm to determine whether the worker has invaded the 3D dynamic danger zone around the excavator in working state.

[0120] The PnP algorithm (Perspective-n-Point) estimates the camera's position and motion using 3D-2D point pairs. Specifically, given n known point pairs in 3D space that correspond one-to-one to the 2D image, the camera's pose in 3D space can be estimated. Then, through an inverse coordinate transformation, the pose coordinates of these point pairs in the camera coordinate system can be determined. Because the excavator and the worker were captured at the same time by the same camera with known intrinsic parameters, the PnP method can unify both the excavator and the worker into a coordinate system based on the camera's position.

[0121] It is known that the homogeneous form of the 3D coordinates of a skeleton key point in the world coordinate system is P w =(X w ,Y w ,Z w ,1) T , and the homogeneous form of the 2D coordinates of the skeleton key point in the image pixel coordinate system is p = (u, v, 1) T When , the conversion relationship between the 3D coordinates and 2D coordinates of the skeleton key point is shown in formula (8).

[0122]

[0123] In formula (8), K is the camera intrinsic parameter matrix, which is known in practical problems;

[0124]

[0125] T is the transformation matrix from the world coordinate system to the camera coordinate system, and is the optimization parameter that needs to be solved;

[0126]

[0127] Among them, R is the rotation matrix and t is the translation matrix.

[0128] Regarding the PnP optimization solution method, this embodiment calls the solvePnP function in the OpenCV computer vision library to solve it, and the final optimized transformation matrix is ​​recorded as T * Get T * Afterwards, the coordinates of the worker and the excavator can be unified into the camera coordinate system through equation (9).

[0129]

[0130] In order to judge whether the key points of the worker's 3D skeleton invade the 3D dynamic danger zone around the excavator in the working state, first, the coordinates of the worker and the excavator can be transformed from the camera coordinate system to the excavator coordinate system through the inverse transformation of the above transformation, so that the worker coordinate system is incorporated into the excavator coordinate system; secondly, the projection of the 3D dynamic danger zone around the excavator in the working state on the XZ plane of the global coordinate system and the projection on the xy plane of the local coordinate system can be obtained respectively, as shown in FIG. Figure 6 As shown in the shaded part; Again, the projection of the worker's 3D skeleton key points on the global coordinate system XZ plane and the projection on the local coordinate system xy plane can be obtained respectively, as shown in Figure 6 As shown by points P1 and P2 in the figure; finally, if the projections of the worker's 3D skeleton key points on the global coordinate system XZ plane and the local coordinate system xy plane are both within the projections of the 3D dynamic danger zone around the excavator in the working state on these two plane coordinate systems, as shown in the figure. Figure 6 As shown by point P1 in the figure, it is considered that there is a hidden danger of collision between the excavator and the worker, and an early warning is issued.

[0131] The above describes in detail the preferred embodiments of the present invention. It should be understood that those skilled in the art can make numerous modifications and variations based on the concepts of the present invention without inventive effort. Therefore, any technical solutions that can be derived by those skilled in the art through logical analysis, reasoning, or limited experimentation based on the concepts of the present invention and the prior art should be within the scope of protection defined by the claims.

Claims

1. A computer vision-based early warning method for excavator-human collision accidents, characterized in that: The following steps are involved: Obtain construction site video stream; Inputting the construction site video stream into an accident warning model to determine the dynamic danger zone of the excavator; Determine whether there is anyone in the dynamic danger zone of the excavator, and if so, issue an early warning; The construction of the accident warning model includes the following steps: S1. Construct an excavator target detection model and a human target detection model; obtain an excavator construction site image dataset, and train and verify the excavator target detection model and the human target detection model; S2. Build an excavator recognition model based on deep learning to extract the excavator's 2D key points and restore their 3D coordinates. S3. Build a human body recognition model based on deep learning to extract 2D key points of the human body and restore their 3D coordinates; S4. Based on the 3D coordinate restoration results of the key points of the excavator, quantitatively extract the operating status indicators of the excavator and delineate the 3D dynamic danger zone under the working state of the excavator; S5. Use the coordinate system based on PnP to unify the human coordinate system and the excavator coordinate system to identify and monitor the behavior of people intruding into the 3D dynamic dangerous area of ​​the excavator in working state; In step S4, delineating the 3D dynamic danger zone in the working state of the excavator includes the following steps: Using the Yolox-Hourglass-Transformer model, a 3D dynamic skeleton of the excavator during operation is obtained from the construction site video stream and then post-processed. Using the Savgol filter, polynomial fitting is performed on the continuous signal points within the sliding window to filter out abnormal points in the 3D skeleton recognition results when the excavator is in operation. The motion of the excavator in working state is decomposed into overall rotation and mechanical arm extension, and the motion state of the excavator is expressed using local coordinate system and global coordinate system; The position parameters of the skeleton key points describing the motion posture of the excavator in the working state are converted into four angle parameters θ1, θ2, θ3, and θ4 through the global coordinate system and the local coordinate system. Among them, θ4 is used to express the overall rotation of the excavator in the global coordinate system, and θ1, θ2, and θ3 are used to express the extension of the excavator's manipulator in the xy plane of the local coordinate system. Calculate the current angular velocity of the excavator's four angle parameters: Define the average worker reaction time Δt, and use the 3D area that the excavator's robotic arm may sweep during the worker's average reaction time as the 3D dynamic danger zone when the excavator is working. That is, calculate the excavator's motion trend based on Δt: The 3D area formed by the changes in the values ​​of the four angle parameters is obtained as the 3D dynamic danger area under the working state of the excavator.

2. The computer vision-based early warning method for an excavator-human collision accident according to claim 1, characterized in that: In step S1, the ACID construction image dataset is used as the basic dataset, and the basic dataset is expanded using Mosaic and CutMix data enhancement technologies; Based on the excavator construction site image dataset, the Yolox convolutional neural network is trained to obtain an excavator target detection model and a human target detection model.

3. The computer vision-based early warning method for an excavator-human collision accident according to claim 1, characterized in that: In step S2, building an excavator identification model based on deep learning includes the following steps: The Hourglass neural network is trained using open source datasets and data augmentation methods to obtain an excavator 2D key point detection Hourglass model that can identify six key points on the excavator's arm and body. Use modeling software to model the excavator, bind the excavator's motion during construction, and write a program to output the 2D coordinates of the six key points of the excavator in the image coordinate system and the 3D coordinates in the world coordinate system during the excavator's motion. Generate a data set that converts the coordinates of the excavator's 2D key points into 3D coordinates. A Transformer model is trained based on a data set of the excavator 2D key point coordinates converted into 3D coordinates to obtain a Transformer model of the excavator 2D key point coordinates converted into 3D coordinates.

4. The computer vision-based early warning method for an excavator-human collision accident according to claim 1, characterized in that: In step S3, building a human recognition model based on deep learning includes the following steps: The Stacked Hourglass neural network is trained using the MSCOCO dataset to obtain a human 2D key point detection model. The VideoPose3D neural network is trained using the Human3.6M dataset to obtain a human 3D skeleton recognition model.

5. The computer vision-based early warning method for an excavator-human collision accident according to claim 1, characterized in that: The local coordinate system of the excavator takes the vertical plane where the mechanical arm is located as the xy plane. The Y axis of the global coordinate system of the excavator coincides with the y axis of the local coordinate system. The (x, y, z) in the local coordinate system T and (X, Y, Z) in the global coordinate system T The coordinate conversion between them is: Where boom_arm represents the key point of the middle hinge of the excavator's mechanical arm, and cab_boom represents the key point of the connection between the excavator's cab and the mechanical arm.

6. The computer vision-based early warning method for an excavator-human collision accident according to claim 1, characterized in that: The calculation formulas for θ1, θ2, and θ3 are: Where right_bucket_end represents the key point of the right end of the excavator bucket, and left_bucket_end represents the key point of the left end of the excavator bucket.

7. The computer vision-based early warning method for an excavator-human collision accident according to claim 1, characterized in that: In step S5, a coordinate system method based on PnP is used to unify the human body coordinate system and the excavator coordinate system, including the following steps: If the homogeneous form of the 3D coordinates of a skeleton key point in the world coordinate system is P w =(X w ,Y w ,Z w ,1) T , and the homogeneous form of the 2D coordinates of the skeleton key point in the image pixel coordinate system is p = (u, v, 1) T When , the conversion relationship between the 3D coordinates and 2D coordinates of the skeleton key point is: Among them, K is the camera intrinsic parameter matrix, T is the transformation matrix from the world coordinate system to the camera coordinate system, and is the optimization parameter to be solved. Among them, R is the rotation matrix and t is the translation matrix; Call the solvePnP function in the OpenCV computer vision library to solve the problem. The final optimized transformation matrix is ​​T * , the human coordinate system and the excavator coordinate system are converted to the camera coordinate system by the following formula:

8. The computer vision-based early warning method for an excavator-human collision accident according to claim 1, characterized in that: In step S5, identifying and monitoring the behavior of a person intruding into the 3D dynamic danger zone of the excavator in the working state includes the following steps: By inversely transforming the human coordinate system and the excavator coordinate system into the camera coordinate system, the coordinates of the human body and the excavator are transformed from the camera coordinate system to the excavator coordinate system, so that the human coordinate system is incorporated into the excavator coordinate system; Obtain the projection of the 3D dynamic danger zone around the excavator in the working state on the XZ plane of the global coordinate system and the projection on the xy plane of the local coordinate system respectively; Obtain the projection of the key points of the human body 3D skeleton on the XZ plane of the global coordinate system and the projection on the xy plane of the local coordinate system respectively; If the key points of the human body's 3D skeleton coincide with the projections of the 3D dynamic danger zone around the excavator in the working state on the XZ plane of the global coordinate system and the xy plane of the local coordinate system, it is considered that there is a hidden danger of a collision between the excavator and the worker, and an early warning is issued.

9. A computer vision-based early warning system for collisions between an excavator and a person, characterized in that: Including data acquisition module, model training module and early warning module; The data acquisition module is used to acquire the construction site video stream; The model training module is used to train the accident warning model; The early warning module is used to input the construction site video stream into the accident early warning model to determine the dynamic danger zone of the excavator; determine whether there is anyone in the dynamic danger zone of the excavator, and if so, issue an early warning; The model training module constructs the accident warning model, which includes the following steps: S1. Construct an excavator target detection model and a human target detection model; obtain an excavator construction site image dataset, and train and verify the excavator target detection model and the human target detection model; S2. Build an excavator recognition model based on deep learning to extract the excavator's 2D key points and restore their 3D coordinates. S3. Build a human body recognition model based on deep learning to extract 2D key points of the human body and restore their 3D coordinates; S4. Based on the 3D coordinate restoration results of the key points of the excavator, quantitatively extract the operating status indicators of the excavator and delineate the 3D dynamic danger zone under the working state of the excavator; S5. Use the coordinate system based on PnP to unify the human coordinate system and the excavator coordinate system to identify and monitor the behavior of people intruding into the 3D dynamic dangerous area of ​​the excavator in working state; In step S4, delineating the 3D dynamic danger zone in the working state of the excavator includes the following steps: Using the Yolox-Hourglass-Transformer model, a 3D dynamic skeleton of the excavator during operation is obtained from the construction site video stream and then post-processed. Using the Savgol filter, polynomial fitting is performed on the continuous signal points within the sliding window to filter out abnormal points in the 3D skeleton recognition results when the excavator is in operation. The motion of the excavator in working state is decomposed into overall rotation and mechanical arm extension, and the motion state of the excavator is expressed using local coordinate system and global coordinate system; The position parameters of the skeleton key points describing the motion posture of the excavator in the working state are converted into four angle parameters θ1, θ2, θ3, and θ4 through the global coordinate system and the local coordinate system. Among them, θ4 is used to express the overall rotation of the excavator in the global coordinate system, and θ1, θ2, and θ3 are used to express the extension of the excavator's manipulator in the xy plane of the local coordinate system. Calculate the current angular velocity of the excavator's four angle parameters: Define the average worker reaction time Δt, and use the 3D area that the excavator's robotic arm may sweep during the worker's average reaction time as the 3D dynamic danger zone when the excavator is working. That is, calculate the excavator's motion trend based on Δt: The 3D area formed by the changes in the values ​​of the four angle parameters is obtained as the 3D dynamic danger area under the working state of the excavator.

Images