A Construction Site Excavator Hazardous Operation Control System and Method Based on Behavior Recognition

Abstract

The present invention discloses a dangerous operation control system and method for a construction site excavator based on behavior recognition, belonging to the technical field of image processing. It includes a data acquisition module, a communication module, a behavior recognition module, a warning processing module, a hydraulic control execution module, and a human-computer interaction module. It solves the technical problems of insufficient real-time performance, inaccurate risk assessment, and lack of active intervention ability in traditional construction machinery safety control. The present invention can dynamically capture and evaluate safety risks in complex operation scenarios, realize a comprehensive safety assessment of the environment and behavior, actively restrict dangerous actions, and directly prevent accidents from occurring.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and particularly relates to a dangerous operation control system and method for construction site excavators based on behavior recognition. Background Technique

[0002] In the operation scenarios of construction machinery, especially on construction sites, once the operation of heavy equipment such as excavators is improper, it often leads to serious safety accidents, such as collisions, overturns, and even casualties. Traditional safety control methods mainly rely on physical fences, fixed safety signs, and manual supervision, and operators mostly rely on experience for judgment. This method is not only inefficient but also has problems such as misjudgment and response lag.

[0003] In recent years, with the rapid development of computer vision, deep learning, and intelligent control technologies, safety monitoring methods based on behavior recognition have gradually been applied to the safety control of construction machinery, but there are still key shortcomings in existing systems:

[0004] On the one hand, static rules and preset areas cannot real-time perceive the dynamically changing operation scenarios and complex operation behaviors;

[0005] On the other hand, due to environmental interference and the complexity of operation behaviors, existing methods are difficult to meet the actual engineering requirements in terms of real-time performance, accuracy, and robustness, and are only limited to passive alarm after discovering risks and lack the ability of active intervention, and cannot effectively prevent the occurrence of dangerous events. Summary of the Invention

[0006] The purpose of the present invention is to provide a dangerous operation control system and method for construction site excavators based on behavior recognition, which solves the technical problems of insufficient real-time performance, inaccurate risk assessment, and lack of active intervention ability in traditional construction machinery safety control.

[0007] To achieve the above purpose, the present invention adopts the following technical solutions:

[0008] A dangerous operation control system for construction site excavators based on behavior recognition includes a data acquisition module, a communication module, a behavior recognition module, an early warning processing module, a hydraulic control execution module, and a human-computer interaction module;

[0009] The data acquisition module is deployed on a drone and is used to real-time collect image data of the excavator operation site and send it to the backend server through the communication module;

[0010] The communication module is deployed on a drone and is used to provide a wireless data interaction channel between the drone and the backend server and encrypt the interaction data;

[0011] The behavior recognition module is deployed in the backend service and is used to preprocess the received image data. It extracts key features using a pre-trained MobileNet-V2 model, performs key point detection and the construction of a pose skeleton, outputs the excavator pose and key data, and calculates safety risk indicators.

[0012] The early warning processing module is deployed in the backend server and is used to calculate the risk value for evaluating safety risks according to the excavator pose and key data through a preset constraint function, compare the risk value with the theoretical operation radius to obtain the risk level, and generate an early warning signal.

[0013] The hydraulic control execution module is deployed in the backend server and is used to establish a data link with the embedded control unit inside the excavator, send control instructions to the embedded control unit to complete the posture adjustment of the excavator.

[0014] The human-computer interaction module is a client server connected to the backend server and is used to display the real-time monitoring interface, the UAV monitoring video, the behavior recognition result, the risk early warning status, and the hydraulic control execution situation.

[0015] Preferably, the UAV is a DJI Mavic 3 Enterprise type UAV.

[0016] Preferably, the UAV communicates with the backend service through a 5G network.

[0017] Preferably, the embedded control unit is a PLC controller inside the excavator, and the embedded control unit communicates with the backend server through a wireless local area network.

[0018] A method for controlling dangerous operations of construction site excavators based on behavior recognition includes the following steps:

[0019] Step 1: The data acquisition module collects real-time images of the excavator operation scene and the worker operation area, obtains image data, and sends it to the backend server through the communication module.

[0020] Step 2: Establish a behavior recognition module, an early warning processing module, and a hydraulic control execution module in the backend server.

[0021] The behavior recognition module retrieves the image data, preprocesses the image data to obtain a preprocessed image, extracts features and performs key point detection on the preprocessed image, and extracts the operation behavior actions of the excavator through an optimization algorithm and a constraint function to generate a behavior recognition result.

[0022] Step 3: The early warning processing module retrieves the behavior recognition result and, according to the preset constraint function , determines whether there is a safety risk: if yes, generates an early warning signal and executes Step 4; if no, returns to Step 2.

[0023] Step 4: The hydraulic control execution module generates an adjustment instruction according to the warning signal and the preset communication protocol, and sends the adjustment instruction to the embedded control unit of the excavator. After the embedded control unit finishes executing according to the adjustment instruction, it feeds back the execution result to the backend server;

[0024] Step 5: Deploy a client server. The client server communicates with the backend server through a network cable. Establish a human-machine interaction module in the client service. The human-machine interaction module obtains the preprocessed image, behavior recognition result, warning signal, and execution result generated by the backend server, and displays the real-time monitoring interface, drone monitoring video, behavior recognition result, risk warning status, and hydraulic control execution situation through the display screen.

[0025] Preferably, when performing Step 1, the data acquisition module is a high-definition camera deployed on the drone, and the acquired image data is static images;

[0026] The method for obtaining static images is: the video data collected by the high-definition camera is processed by extracting key frames from the video data to obtain static images;

[0027] The communication module is a 5G communication module built in the drone, and the image data is sent to the backend server through the 5G mobile network.

[0028] Preferably, when performing Step 2, the specific steps are as follows:

[0029] Step 2-1: The behavior recognition module retrieves the image data and preprocesses the image data, specifically including:

[0030] Step 2-1-1: Scale the image size to a fixed size, that is, a pixel size of 256×256;

[0031] Step 2-1-2: Normalize the pixel values of the image to the interval [0,1];

[0032] Step 2-1-3: Perform histogram equalization processing and contrast enhancement processing on the image to obtain the enhanced preprocessed image;

[0033] Step 2-1-4: Divide the preprocessed image into two data sets: a training set and a test set. The training set accounts for 70% of the total number of preprocessed images, and the test set accounts for 30%;

[0034] Step 2-2: The behavior recognition module constructs a pre-trained MobileNet-V2 model and uses the MobileNet-V2 model to extract the edge features of the excavator;

[0035] Using multiple layers of convolution, pooling, and activation functions to capture the structural information in the preprocessed image;

[0036] Step 2-3: Input the feature map output by the convolutional layer into the key point detection network to achieve multi-scale feature fusion;

[0037] Step 2-4: The key point detection network designs an output layer for each key part of the excavator and directly regresses the two-dimensional pixel coordinates of the key points. The key parts include the boom, the arm, and the bucket;

[0038] Predict the confidence value of each key point;

[0039] Step 2-5: Set a confidence threshold and compare the confidence value with the confidence threshold: If the confidence value is lower than the confidence threshold, return to the previous feature layer to further extract features; otherwise, execute Step 2-6;

[0040] Step 2-6: Use the non-maximum suppression method to filter redundant and noisy key points; Combine spatial coherence to adjust the predicted positions of the key points to satisfy geometric constraints and obtain the filtered key points;

[0041] Use the part affinity fields to connect the filtered key points to form the overall skeleton of the excavator;

[0042] Introduce an adjustment factor for the rotation angle between key points to reflect the correlation between key points;

[0043] The key points meet the following geometric constraints:

[0044] ;

[0045] ;

[0046] Among them, represents the predicted two-dimensional key point coordinates; represents the two-dimensional key point coordinates of the manually marked key points, that is, the true two-dimensional key point coordinates; represents the L2 vector norm, which is used to measure the error between the predicted coordinates and the true coordinates; represents the accumulation of the prediction errors between all key points, that is, the error metric for the overall skeleton reconstruction. By summing the errors of all key points, the accuracy of the overall model in pose prediction is evaluated; i and j represent indices, respectively identifying the numbers of different key points that make up the overall skeleton of the excavator, and i and j represent the key point pairs with direct connection relationships; is the total number of key points;

[0047] represents the translation degree of freedom adjustment parameter of the excavator in the plane, which is used to describe the translation correction during prediction; represents the rotation degree of freedom adjustment parameter of the excavator in the plane, which is used for key point rotation correction; Indicates the inter-frame key point correlation degree; Indicates the reasonable proportion range of the skeleton key points, used to ensure that the distance ratio between key points is within an appropriate range; Is a conversion function used to correct the two-dimensional coordinates predicted by the original model;

[0048] Step 2-7: Construct an operator intrusion function to determine whether to trigger an alarm. The operator intrusion function is as follows:

[0049] ;

[0050] Among them, Is the intrusion value; Is the attitude distance between the person and the excavator; Is the length index of the excavator skeleton; Is the direction influence coefficient, Is the theoretical operating radius of the excavator; Is a weighted linear combination function;

[0051] Preferably, when executing Step 3, the warning processing module retrieves The value of, and judges: when It indicates that there is a safety risk and a high-risk warning is triggered;

[0052] When Within the preset safety boundary threshold, a low-level warning is generated;

[0053] When Is greater than the preset safety boundary threshold, the system is in normal operation and no warning is triggered;

[0054] According to the above judgment results, a warning signal is generated.

[0055] Preferably, when executing Step 4, the hydraulic control execution module obtains the warning signal, determines the adjustment direction and control parameters of the hydraulic system for the boom attitude according to the preset safety state, and generates a control strategy:

[0056] When a high-risk warning occurs, immediately apply a brake to the boom or adjust the movement trajectory to reduce the speed and amplitude;

[0057] When a low-level warning occurs, take mild intervention and at the same time generate a prompt message to keep a distance;

[0058] The hydraulic control execution module presets a communication protocol for communicating with the embedded control unit of the excavator. According to the instructions specified in the communication protocol and combined with the control strategy, a control instruction is generated and the control instruction is sent to the embedded control unit of the excavator.

[0059] A risk control system and method for dangerous operations of construction site excavators based on behavior recognition according to the present invention solves the technical problems of insufficient real-time performance, inaccurate risk assessment, and lack of active intervention ability in traditional engineering machinery safety control. The present invention uses a patrol drone and a high-resolution sensor to collect operation site data in real time, and accurately recognizes the posture and key behaviors of the excavator through a deep learning model, and can dynamically capture and evaluate safety risks in complex operation scenarios. By using a customized constraint function to calculate risk indicators in real time, a comprehensive safety assessment of the environment and behavior is achieved. When potential dangers are detected, not only can an early warning be quickly issued, but also the six-degree-of-freedom movement of the excavator's boom can be actively intervened through a hydraulic control module to actively limit dangerous actions and directly prevent accidents from occurring, providing unprecedented safety protection capabilities for the engineering site. In view of the complexity of the actual working condition environment, the present invention designs a multi-level redundancy mechanism including data preprocessing, model fine-tuning, and control strategy optimization to ensure the efficient operation of the system, and is compatible with existing engineering machinery platforms and hydraulic control systems at the same time, facilitating popularization and application. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 is a schematic diagram of the risk control system for dangerous operations of construction site excavators according to the present invention;

[0061] Figure 2 is the main flow chart of the present invention;

[0062] Figure 3 is the flow chart of posture recognition of the present invention;

[0063] Figure 4 is the flow chart of early warning judgment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0064] Embodiment 1

[0065] A Figure 1 risk control system for dangerous operations of construction site excavators based on behavior recognition as shown includes a data acquisition module, a communication module, a behavior recognition module, an early warning processing module, a hydraulic control execution module, and a human-computer interaction module;

[0066] The data acquisition module is deployed on the drone and is used to collect image data of the excavator operation site in real time and send it to the backend server through the communication module;

[0067] The drone is a DJI Mavic 3 Enterprise drone. In this embodiment, the flight path covering the entire operation area of the excavator is set by programming to ensure no blind area monitoring. The flight altitude is fixed at 10 meters to ensure safety and obtain a wider field of view at the same time. The camera takes pictures at a 45° downward angle to reduce occlusion and distortion and cover the whole body of the excavator and the surrounding operation environment.

[0068] The UAV communicates with the backend service via a 5G network.

[0069] The communication module is deployed on the UAV and is used to provide a wireless data interaction channel between the UAV and the backend server and encrypt the interaction data.

[0070] The behavior recognition module is deployed in the backend service and is used to preprocess the received image data, extract key features using a pre-trained MobileNet-V2 model, perform key point detection and construction of the pose skeleton, output the excavator pose and key data, and calculate the safety risk index;

[0071] The early warning processing module is deployed in the backend server and is used to calculate the risk value for evaluating the safety risk through a preset constraint function based on the excavator pose and key data, compare the risk value with the theoretical operation radius to obtain the risk level, and generate an early warning signal;

[0072] The hydraulic control execution module is deployed in the backend server and is used to establish a data link with the embedded control unit inside the excavator, send control instructions to the embedded control unit to complete the pose adjustment of the excavator;

[0073] The embedded control unit is a PLC controller inside the excavator, and the embedded control unit communicates with the backend server via a wireless local area network.

[0074] The human-computer interaction module is a client server, which is connected to the backend server and is used to display the real-time monitoring interface, UAV monitoring video, behavior recognition results, risk early warning status, and hydraulic control execution status.

[0075] Embodiment 2

[0076] As Figures 2 - 4 shown, a method for controlling dangerous operations of a construction site excavator based on behavior recognition described in Embodiment 2 is implemented on the basis of a system for controlling dangerous operations of a construction site excavator based on behavior recognition described in Embodiment 1, and includes the following steps:

[0077] Step 1: The data acquisition module collects real-time images of the excavator operation scene and the worker operation area to obtain image data, and sends it to the backend server through the communication module;

[0078] When performing Step 1, the data acquisition module is a high-definition camera deployed on the UAV, and the collected image data is static images;

[0079] The method for obtaining static images is: the video data collected by the high-definition camera is used to obtain static images by extracting key frames from the video data;

[0080] The communication module is the built-in 5G communication module of the drone, which sends the image data to the backend server through the 5G mobile network.

[0081] Step 2: Establish a behavior recognition module, a warning handling module, and a hydraulic control execution module in the backend server;

[0082] The behavior recognition module retrieves the image data, preprocesses the image data to obtain a preprocessed image, extracts features and detects key points from the preprocessed image, and extracts the operation behavior actions of the excavator through an optimization algorithm and a constraint function to generate a behavior recognition result;

[0083] When executing Step 2, the specific steps are as follows:

[0084] Step 2-1: The behavior recognition module retrieves the image data and preprocesses the image data, specifically including:

[0085] Step 2-1-1: Scale the image size to a fixed size, that is, a pixel size of 256×256;

[0086] Step 2-1-2: Normalize the pixel values of the image to the interval [0,1];

[0087] Step 2-1-3: Perform histogram equalization processing and contrast enhancement processing on the image to obtain an enhanced preprocessed image;

[0088] Step 2-1-4: Divide the preprocessed image into two data sets: a training set and a test set. The training set accounts for 70% of the total number of preprocessed images, and the test set accounts for 30%;

[0089] Step 2-2: The behavior recognition module constructs a pre-trained MobileNet-V2 model and uses the MobileNet-V2 model to extract the edge features of the excavator;

[0090] Use multiple layers of convolution, pooling, and activation functions to capture the structural information in the preprocessed image;

[0091] In this embodiment, the MobileNet-V2 model constructed by the behavior recognition module adopts a depthwise separable convolution structure. After each convolutional layer, batch normalization and ReLU activation functions are followed, and then local features are extracted through a max-pooling layer. Stacked to form a multi-layer convolutional network for capturing the subtle structural information in the preprocessed image.

[0092] In the model construction stage, use a depthwise separable convolutional layer with a 3×3 convolutional kernel to extract the low-level features of the image. After batch normalization (BatchNormalization) and ReLU activation functions, local invariant features in the image are extracted through a max-pooling layer. This process progresses layer by layer to form multi-scale feature information.

[0093] Step 2-3: Input the feature map output by the convolutional layer into the key point detection network to achieve multi-scale feature fusion;

[0094] Step 2-4: The key point detection network designs an output layer for each key part of the excavator and directly regresses the two-dimensional pixel coordinates of the key points. The key parts include the boom, the arm, and the bucket;

[0095] Predict the confidence value of each key point;

[0096] In this embodiment, the output layer of the key point detection network, in addition to regressing the two-dimensional coordinates of each key point, also outputs the corresponding confidence value through the sigmoid activation function. This value reflects the probability of the existence of the key point. In this embodiment, the preset threshold is set to 0.5 to filter out key points with low probability and ensure more accurate subsequent skeleton reconstruction.

[0097] Step 2-5: Set a confidence threshold and compare the confidence value with the confidence threshold: If the confidence value is lower than the confidence threshold, return to the previous feature layer to further extract features; otherwise, execute Step 2-6;

[0098] In this embodiment, when the confidence of the detected key point is lower than the threshold, it will backtrack to the convolutional feature map of the previous layer, and use a refinement module (such as a small convolutional network or an upsampling module) to further enhance the local features of the image, so as to rediscover or confirm the key point information. This process adopts an iterative feedback mechanism to seek effective extraction of key points within the maximum number of iterations before making a decision.

[0099] Step 2-6: Use the non-maximum suppression method to filter out redundant and noisy key points; Combine spatial coherence to adjust the predicted positions of the key points to satisfy geometric constraints and obtain the filtered key points;

[0100] Use the part affinity fields to connect the filtered key points to form the overall skeleton of the excavator;

[0101] Introduce an adjustment factor for the rotation angle between key points to reflect the correlation between key points;

[0102] The key points conform to the following geometric constraints:

[0103] ;

[0104] ;

[0105] Among them, represents the predicted two-dimensional key point coordinates; represents the two-dimensional key point coordinates of the manually marked key points, that is, the true two-dimensional key point coordinates; Denotes the L2 vector norm, which is used to measure the error magnitude between the predicted coordinates and the true coordinates; Denotes the accumulation of the prediction errors between all key points, that is, the error metric for the entire skeleton reconstruction. By summing the errors of all key points, the accuracy of the overall model in pose prediction is evaluated; i and j represent indices, respectively identifying the numbers of different key points that make up the overall skeleton of the excavator. i and j represent pairs of key points with direct connection relationships; Is the total number of key points;

[0106] Denotes the translation degree-of-freedom adjustment parameter of the excavator in the plane, which is used to describe the translation correction during prediction; Denotes the rotation degree-of-freedom adjustment parameter of the excavator in the plane, which is used for key point rotation correction; Denotes the inter-frame key point correlation degree; Denotes the reasonable proportion interval of the skeleton key points, which is used to ensure that the distance proportion between key points is within a suitable range; Is a conversion function, which is used to correct the two-dimensional coordinates predicted by the original model;

[0107] In this embodiment, the principle of non-maximum suppression (NMS) is to describe in detail how to select the local maximum value among the candidate key points. For example, calculate the overlap degree between adjacent candidate points, select the point with the highest confidence, and suppress other overlapping values.

[0108] In this embodiment, during the process of constructing the skeleton of the excavator, the part affinity fields (PAFs) are used to connect the key points. The method includes calculating the connectivity scores for adjacent key point pairs, determining the effective connections according to the score levels, and at the same time introducing the rotation angle adjustment between key points to strengthen the skeleton consistency.

[0109] The calculation formula of PAFs is prior art, so it will not be described in detail.

[0110] After the non-maximum suppression processing, the present invention also adopts the part affinity fields (PAFs) method to construct the skeleton for the effective key points screened by NMS. Specifically, for each pair of candidate key points, calculate their connectivity scores, and introduce a rotation angle adjustment factor to make the reconstructed skeleton satisfy the preset geometric constraints in space. The IOU threshold in NMS and the PAF calculation are both dynamically adjusted based on the on-site data to ensure the robustness of the overall skeleton.

[0111] For example: after detecting multiple excavator key points (such as the boom, arm, bucket, etc.), the PAFs help determine which key points should be connected together to form the overall skeleton.

[0112] By calculating the association scores between each pair of candidate key points, the most reasonable connection relationships are screened out among a large number of candidate points, thereby reconstructing the posture skeleton of the excavator.

[0113] Using PAFs can help reduce the overall skeleton reconstruction deviation caused by the detection error of a single key point, and improve the accuracy and robustness of skeleton construction.

[0114] Step 2-7: Construct a worker intrusion function to determine whether to trigger an alarm. The worker intrusion function is as follows:

[0115] ;

[0116] where, is the intrusion value; is the posture distance between the person and the excavator; is the length index of the excavator skeleton; is the direction influence coefficient, is the theoretical operating radius of the excavator; is a weighted linear combination function.

[0117] In this embodiment, The specific formula of is as follows:

[0118] ;

[0119] where, is a preset weight system, and these weight coefficients need to be formulated and adjusted according to the actual scenario where the excavator is located;

[0120] represents the posture distance, and the specific calculation method is: takes the Euclidean distance between the key point of the excavator (such as the end of the boom) and the key position detected by the worker (such as the position of the worker, etc.).

[0121] In this embodiment, the worker bounding box is detected by an object detection model (such as MobileNet-SSD), and the center point of the bounding box is used as the worker position coordinate.

[0122] The calculation method of the direction influence coefficient is as follows:

[0123] Calculate the included angle difference between the operating direction of the excavator and the position of the worker, that is The direction influence coefficient, for example: if the excavator is facing the worker directly, then takes a larger value; on the contrary, if it deviates, it is smaller. The specific formula is as follows:

[0124] ;

[0125] Among them, is the included angle between the direction vectors, and normalize represents the normalization operation.

[0126] The calculation method of the length index of the excavator skeleton is as follows:

[0127] It is represented by summing the distances between the key points of each skeleton of the excavator , reflects the current extended or folded state of the machine, and in this embodiment, it is used as a dynamic index affecting the operation radius.

[0128] is the theoretical operation radius of the excavator:

[0129] The value of is calculated based on the mechanical parameters of the excavator at the time of factory shipment, such as the maximum extended length of the boom, the installation angle, and the safety redundancy distance.

[0130] In this embodiment, by setting the redundancy distance, the safety margin is ensured;

[0131] ;

[0132] Among them, is the maximum working radius of the boom calculated according to the factory parameters, is the increased redundancy distance for ensuring safety.

[0133] Step 3: The warning processing module retrieves the behavior recognition result and, according to the preset constraint function , determines whether there is a safety risk: if yes, generates a warning signal and executes Step 4; if no, returns to Step 2;

[0134] When executing Step 3, the warning processing module retrieves the value of and judges: when , it indicates that there is a safety risk and triggers a high-risk warning;

[0135] When is within the preset safety boundary threshold, a low-level warning is generated;

[0136] When is greater than the preset safety boundary threshold, the system is in a normal operation state and no warning is triggered;

[0137] According to the above judgment results, a warning signal is generated.

[0138] Step 4: The hydraulic control execution module generates an adjustment instruction according to the warning signal and the preset communication protocol, and sends the adjustment instruction to the embedded control unit of the excavator. After the embedded control unit finishes executing according to the adjustment instruction, it feeds back the execution result to the backend server;

[0139] When executing Step 4, the hydraulic control execution module obtains the warning signal, determines the adjustment direction and control parameters of the boom attitude according to the preset safety state, and generates a control strategy:

[0140] When a high-risk warning occurs, immediately apply brakes to the boom or adjust the movement trajectory to reduce the speed and amplitude;

[0141] When a low-level warning occurs, take mild intervention and generate a prompt message to keep a distance at the same time;

[0142] The hydraulic control execution module presets a communication protocol for communicating with the embedded control unit of the excavator. According to the instructions specified in the communication protocol and combined with the control strategy, it generates a control instruction and sends the control instruction to the embedded control unit of the excavator. The embedded control unit controls the hydraulic system to execute the control instruction.

[0143] In this embodiment, the control instruction is encapsulated in accordance with the preset communication protocol format and transmitted to the embedded control unit of the excavator (such as a PLC controller) through a wireless local area network or a dedicated fieldbus.

[0144] A custom simple communication protocol based on the JSON format is adopted. In practical applications, industrial standard protocols such as Modbus, CANopen, and EtherCAT can be used as the underlying transmission method according to site requirements.

[0145] The communication protocol format is as follows:

[0146] Data frame structure:

[0147] Header (message header): Identifies the starting part of the message, including the protocol version and message type, as follows:

[0148] "version": Protocol version (such as "1.0");

[0149] "messageType": Message type (instruction type, such as "CONTROL_COMMAND");

[0150] Timestamp: Used to record the time when the instruction is generated for on-site synchronization and fault tracing;

[0151] Command: Instruction name, used to identify a specific action. For example, "DANGER_INTERVENTION" represents emergency intervention;

[0152] Parameters (Control Parameters): Include specific execution parameters such as the target part, action type, force, operation mode, reference angle, and braking strategy, etc., as follows:

[0153] "targetArm": The target part (such as "upper arm");

[0154] "action": The action type (such as "deceleration braking" or "adjusting trajectory");

[0155] "force": The braking force or intervention force (for example, percentage or specific value);

[0156] "duration": The execution time or transition time (unit: millisecond);

[0157] "angleAdjustment": If the rotation angle needs to be adjusted;

[0158] Checksum: Used for data integrity verification, which can be calculated based on the message content using CRC or other hash algorithms.

[0159] Footer (Message End Indicator): Indicates the end of the message, such as the fixed string "EOF".

[0160] Example of protocol format:

[0161] {

[0162] "header":{

[0163] "version":"1.0",

[0164] "messageType":"CONTROL_COMMAND"

[0165] },

[0166] "timestamp":"2025-04-16T14:23:45Z",

[0167] "command":"DANGER_INTERVENTION",

[0168] "parameters":{

[0169] "targetArm":"upper arm",

[0170] "action":"deceleration braking",

[0171] "force": "80%", / / Emergency braking force at 80%

[0172] "duration": 100, / / Braking duration of 100 ms

[0173] "angleAdjustment": "-15" / / Adjust the boom angle 15 degrees to the left (negative value indicates counterclockwise) to avoid approaching workers

[0174] },

[0175] "checksum": "3f5e2a7b", / / Example checksum (actual calculation method can use algorithms such as CRC32)

[0176] "footer": "EOF"

[0177] }

[0178] Step 5: Deploy the client server. The client server communicates with the backend server via an Ethernet cable. Establish a human - machine interaction module in the client service. The human - machine interaction module obtains the pre - processed images, behavior recognition results, warning signals, and execution results generated by the backend server, and displays the real - time monitoring interface, drone monitoring video, behavior recognition results, risk warning status, and hydraulic control execution status through the display screen.

[0179] A dangerous operation control system and method for construction site excavators based on behavior recognition according to the present invention solve the technical problems of insufficient real - time performance, inaccurate risk assessment, and lack of active intervention ability in traditional construction machinery safety control. The present invention uses inspection drones and high - resolution sensors to collect operation site data in real - time, accurately identify the posture and key behaviors of the excavator through a deep - learning model, can dynamically capture and evaluate safety risks in complex operation scenarios, calculate risk indicators in real - time through customized constraint functions, achieve a comprehensive safety assessment of the environment and behavior. When detecting potential dangers, it can not only quickly issue warnings, but also perform real - time intervention on the six - degree - of - freedom movement of the excavator boom through the hydraulic control module, actively restrict dangerous actions, and directly prevent accidents from occurring, providing unprecedented safety protection capabilities for the engineering site. In view of the complexity of the actual working condition environment, the present invention designs multi - level redundancy mechanisms including data pre - processing, model fine - tuning, and control strategy optimization to ensure the efficient operation of the system, and is also compatible with existing construction machinery platforms and hydraulic control systems, facilitating popularization and application.

Claims

1. A construction site excavator dangerous operation control system based on behavior recognition, characterized in that: It includes a data acquisition module, a communication module, a behavior recognition module, an early warning processing module, a hydraulic control execution module, and a human-computer interaction module; The data acquisition module is deployed on the drone and is used to collect image data of the excavator operation site in real time and send it to the backend server through the communication module; The communication module is deployed on the drone and is used to provide a wireless data interaction channel between the drone and the backend server and encrypt the interaction data; The behavior recognition module is deployed in the backend service and is used to preprocess the received image data, extract key features using a pre-trained MobileNet-V2 model, perform key point detection and construction of the pose skeleton, output the excavator pose and key data, and calculate the safety risk index; The early warning processing module is deployed in the backend server and is used to calculate and evaluate the risk value of the safety risk through the operator intrusion function based on the excavator pose and key data, compare the risk value with the theoretical operation radius, obtain the risk level, and generate an early warning signal; The hydraulic control execution module is deployed in the backend server and is used to establish a data link with the embedded control unit inside the excavator and send control instructions to the embedded control unit to complete the pose adjustment of the excavator; The human-computer interaction module is a client server connected to the backend server and is used to display the real-time monitoring interface, drone monitoring video, behavior recognition result, risk early warning status, and hydraulic control execution situation.

2. The construction site excavator dangerous operation control system based on behavior recognition according to claim 1, characterized in that: The drone is a DJI Mavic 3 Enterprise drone.

3. The construction site excavator dangerous operation control system based on behavior recognition according to claim 1, characterized in that: The drone communicates with the backend service through a 5G network.

4. The construction site excavator dangerous operation control system based on behavior recognition according to claim 1, characterized in that: The embedded control unit is a PLC controller inside the excavator, and the embedded control unit communicates with the backend server through a wireless local area network.

5. A method for controlling dangerous operations of construction site excavators based on behavior recognition, which is applied to the system for controlling dangerous operations of construction site excavators based on behavior recognition according to any one of claims 1-4, and is characterized in that: It includes the following steps: Step 1: The data acquisition module collects real-time images of the excavator operation scenario and the worker operation area to obtain image data and sends it to the backend server through the communication module; Step 2: Establish a behavior recognition module, an early warning processing module, and a hydraulic control execution module in the backend server; The behavior recognition module retrieves the image data, preprocesses the image data to obtain a preprocessed image, extracts features and performs key point detection on the preprocessed image, extracts the operation behavior actions of the excavator through an optimization algorithm and geometric constraints, and generates a behavior recognition result; Step 3: The early warning processing module retrieves the behavior recognition result and determines whether there is a safety risk according to the function of the operator's intrusion , if yes, generate an early warning signal and execute Step 4; If not, return to Step 2; Step 4: The hydraulic control execution module generates an adjustment instruction according to the early warning signal and a preset communication protocol, sends the adjustment instruction to the embedded control unit of the excavator, and after the embedded control unit finishes executing according to the adjustment instruction, feeds back the execution result to the backend server; Step 5: Deploy a client server. The client server communicates with the backend server through a network cable. Establish a human-computer interaction module in the client service. The human-computer interaction module obtains the preprocessed image, behavior recognition result, early warning signal, and execution result generated by the backend server and displays the real-time monitoring interface, drone monitoring video, behavior recognition result, risk early warning status, and hydraulic control execution situation through a display screen.

6. The method for controlling dangerous operations of a construction site excavator based on behavior recognition according to claim 5, wherein: When performing step 1, the data acquisition module is a high-definition camera deployed on the drone, and the collected image data is static images; The method for obtaining static images is as follows: for the video data collected by the high-definition camera, static images are obtained by extracting key frames from the video data; The communication module is the 5G communication module built into the drone, and the image data is sent to the backend server through the 5G mobile network.

7. The method for controlling dangerous operations of a construction site excavator based on behavior recognition according to claim 6, wherein: When performing step 2, the specific steps are as follows: Step 2-1: The behavior recognition module retrieves the image data and preprocesses the image data. Specifically, it includes: Step 2-1-1: Scale the image size to a fixed size, that is, a pixel size of 256×256; Step 2-1-2: Normalize the pixel values of the image to the interval [0, 1]; Step 2-1-3: Perform histogram equalization processing and contrast enhancement processing on the image to obtain the enhanced preprocessed image; Step 2-1-4: Divide the preprocessed image into two data sets: a training set and a test set. The training set accounts for 70% of the total number of preprocessed images, and the test set accounts for 30%; Step 2-2: The behavior recognition module constructs a pre-trained MobileNet-V2 model and uses the MobileNet-V2 model to extract the edge features of the excavator; Using multiple layers of convolution, pooling, and activation functions to capture the structural information in the preprocessed image; Step 2-3: Input the feature map output by the convolutional layer into the key point detection network to achieve multi-scale feature fusion; Step 2-4: The key point detection network designs an output layer for each key part of the excavator and directly regresses the two-dimensional pixel coordinates of the key points. The key parts include the boom, arm, and bucket; Predict the confidence value of each key point; Step 2-5: Set a confidence threshold and compare the confidence value with the confidence threshold: if the confidence value is lower than the confidence threshold, return to the previous feature layer to further extract features; otherwise, execute step 2-6; Step 2-6: Use the non-maximum suppression method to filter redundant and noisy key points; combine spatial coherence to adjust the predicted positions of the key points to satisfy geometric constraints and obtain the filtered key points; Use the part affinity fields to connect the filtered key points to form the overall skeleton of the excavator; Introduce an adjustment factor for the rotation angle between key points to reflect the correlation between key points; The key points conform to the following geometric constraints: ； ； Among them, represents the predicted two-dimensional key point coordinates; represents the two-dimensional coordinates of the key points manually marked, that is, the true two-dimensional coordinates of the key points; represents the L2 vector norm, which is used to measure the error between the predicted coordinates and the true coordinates; represents the accumulation of the predicted errors between all key points, that is, the error metric of the entire skeleton reconstruction. By summing the errors of all key points, the accuracy of the overall model in pose prediction is evaluated; i and j represent indices, respectively identifying the numbers of different key points that make up the overall skeleton of the excavator; is the total number of key points; Represents the translation degree-of-freedom adjustment parameter of the excavator in the plane, which is used to describe the translation correction during prediction; Represents the rotation degree-of-freedom adjustment parameter of the excavator in the plane, which is used for key-point rotation correction; Represents the inter-frame key-point correlation degree; Represents the reasonable proportion interval of the skeleton key points, which is used to ensure that the distance ratio between the key points is within a suitable range; Is a conversion function, which is used to correct the two-dimensional coordinates predicted by the original model; Step 2-7: Construct a function for detecting the intrusion of operators and determine whether to trigger an alarm. The function for detecting the intrusion of operators is as follows: ； Among them, is the intrusion value; is the attitude distance between the person and the excavator; is the length index of the excavator skeleton; is the direction influence coefficient, is the theoretical operating radius of the excavator; is the weighted linear combination function.

8. The method for controlling dangerous operations of a construction site excavator based on behavior recognition according to claim 7, wherein: When executing step 3, the early warning processing module retrieves the value of, and judges that: when exists, it indicates that there is a security risk and a high-risk warning is triggered; When within the preset safety boundary threshold, a low-level warning is generated; When is greater than the preset safety boundary threshold, the system is in a normal operating state and no warning is triggered; Generate an alarm signal according to the above judgment result.

9. The method for controlling dangerous operations of a construction site excavator based on behavior recognition according to claim 8, characterized in that: When performing step 4, the hydraulic control execution module obtains the alarm signal, determines the adjustment direction and control parameters of the hydraulic system for the boom attitude according to the preset safety state, and generates a control strategy: When a high-risk alarm occurs, immediately apply braking to the boom or adjust the movement trajectory to reduce the speed and amplitude; When a low-level alarm occurs, take mild intervention and at the same time generate a prompt message to keep a distance; The hydraulic control execution module presets a communication protocol for communicating with the embedded control unit of the excavator, generates control instructions according to the instructions specified in the communication protocol and in combination with the control strategy, and sends the control instructions to the embedded control unit of the excavator.

Images