Construction site excavator dangerous operation management and control system and method based on behavior recognition

Abstract

The invention discloses a construction site excavator dangerous operation management and control system and method based on behavior recognition, and belongs to the technical field of image processing. The system comprises a data acquisition module, a communication module, a behavior recognition module, an early warning processing module, a hydraulic control execution module and a man-machine interaction module; the technical problems of insufficient real-time performance, inaccurate risk assessment and lack of active intervention capability in traditional engineering machinery safety management and control are solved, the safety risk in a complex operation scene can be dynamically captured and assessed, the comprehensive safety assessment of the environment and behaviors is realized, dangerous actions can be actively limited, and accidents can be directly prevented.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and particularly relates to a dangerous operation control system and method for construction site excavators based on behavior recognition. Background Art

[0002] In the operation scenarios of construction machinery, especially on construction sites, once the operation of heavy equipment such as excavators is improper, it often leads to serious safety accidents, such as collisions, overturns, and even casualties. Traditional safety control methods mainly rely on physical fences, fixed safety signs, and manual supervision, and operators mostly rely on experience for judgment. This method is not only inefficient but also has problems such as misjudgment and response lag.

[0003] In recent years, with the rapid development of computer vision, deep learning, and intelligent control technologies, safety monitoring methods based on behavior recognition have been gradually applied to the safety control of construction machinery, but there are still key shortcomings in the existing systems: On the one hand, static rules and preset areas cannot real-time perceive the dynamically changing operation scenarios and complex operation behaviors; On the other hand, due to environmental interference and the complexity of operation behaviors, the existing methods are difficult to meet the actual engineering requirements in terms of real-time performance, accuracy, and robustness, and are only limited to passive alarms after discovering risks and lack the ability of active intervention, and cannot effectively prevent the occurrence of dangerous events. Summary of the Invention

[0004] The purpose of the present invention is to provide a dangerous operation control system and method for construction site excavators based on behavior recognition, which solves the technical problems of insufficient real-time performance, inaccurate risk assessment, and lack of active intervention ability in traditional construction machinery safety control.

[0005] To achieve the above purpose, the present invention adopts the following technical solutions: A dangerous operation control system for construction site excavators based on behavior recognition, including a data acquisition module, a communication module, a behavior recognition module, an early warning processing module, a hydraulic control execution module, and a human-computer interaction module; The data acquisition module is deployed on the unmanned aerial vehicle (UAV) and is used to collect image data of the excavator operation site in real time and send it to the backend server through the communication module; The communication module is deployed on the UAV and is used to provide a wireless data interaction channel between the UAV and the backend server and encrypt the interaction data; The behavior recognition module is deployed in the backend service and is used to preprocess the received image data, extract key features using a pre-trained MobileNet-V2 model, perform key point detection and construction of the pose skeleton, output the excavator pose and key data, and calculate the safety risk index; The early warning processing module is deployed in the backend server. It is used to calculate the risk value for evaluating safety risks according to the excavator's posture and key data through a preset constraint function, compare the risk value with the theoretical operating radius to obtain the risk level, and generate an early warning signal. The hydraulic control execution module is deployed in the backend server. It is used to establish a data link with the embedded control unit inside the excavator and send control instructions to the embedded control unit to complete the posture adjustment of the excavator. The human-computer interaction module is a client server connected to the backend server. It is used to display the real-time monitoring interface, drone monitoring video, behavior recognition results, risk early warning status, and hydraulic control execution situation.

[0006] Preferably, the drone is a DJI Mavic 3 Enterprise type drone.

[0007] Preferably, the drone communicates with the backend service through a 5G network.

[0008] Preferably, the embedded control unit is a PLC controller inside the excavator, and the embedded control unit communicates with the backend server through a wireless local area network.

[0009] A method for controlling dangerous operations of construction site excavators based on behavior recognition includes the following steps: Step 1: The data acquisition module collects real-time images of the excavator operation scene and the worker operation area to obtain image data, and sends it to the backend server through the communication module. Step 2: Establish a behavior recognition module, an early warning processing module, and a hydraulic control execution module in the backend server. The behavior recognition module retrieves the image data, preprocesses the image data to obtain a preprocessed image, extracts features and key points from the preprocessed image, and extracts the operation behavior actions of the excavator through an optimization algorithm and a constraint function to generate a behavior recognition result. Step 3: The early warning processing module retrieves the behavior recognition result and, according to the preset constraint function , determines whether there is a safety risk: if yes, generates an early warning signal and executes Step 4; if no, returns to Step 2. Step 4: The hydraulic control execution module generates an adjustment instruction according to the early warning signal and a preset communication protocol, sends the adjustment instruction to the embedded control unit of the excavator, and after the embedded control unit finishes executing according to the adjustment instruction, feeds back the execution result to the backend server. Step 5: Deploy the client server. The client server communicates with the backend server via an Ethernet cable. Establish a human-computer interaction module in the client service. The human-computer interaction module obtains the preprocessed images, behavior recognition results, warning signals, and execution results generated by the backend server, and displays the real-time monitoring interface, drone monitoring video, behavior recognition results, risk warning status, and hydraulic control execution status through a display screen.

[0010] Preferably, when performing Step 1, the data acquisition module is a high-definition camera deployed on the drone, and the acquired image data is static images. The method for obtaining static images is as follows: For the video data collected by the high-definition camera, static images are obtained by extracting key frames from the video data. The communication module is a 5G communication module built into the drone, and the image data is sent to the backend server through the 5G mobile network.

[0011] Preferably, when performing Step 2, the specific steps are as follows: Step 2-1: The behavior recognition module retrieves the image data and preprocesses the image data, specifically including: Step 2-1-1: Scale the image size to a fixed size, i.e., 256×256 pixel size. Step 2-1-2: Normalize the pixel values of the image to the range [0,1]. Step 2-1-3: Perform histogram equalization processing and contrast enhancement processing on the image to obtain the enhanced preprocessed image. Step 2-1-4: Divide the preprocessed image into two data sets: a training set and a test set. The training set accounts for 70% of the total number of preprocessed images, and the test set accounts for 30%. Step 2-2: The behavior recognition module constructs a pre-trained MobileNet-V2 model and uses the MobileNet-V2 model to extract the edge features of the excavator. Use multiple layers of convolution, pooling, and activation functions to capture the structural information in the preprocessed image. Step 2-3: Input the feature map output by the convolutional layer into the keypoint detection network to achieve multi-scale feature fusion. Step 2-4: The keypoint detection network designs an output layer for each key part of the excavator and directly regresses the two-dimensional pixel coordinates of the keypoints. The key parts include the boom, arm, and bucket. Predict the confidence value of each keypoint. Step 2-5: Set a confidence threshold and compare the confidence value with the confidence threshold: If the confidence value is lower than the confidence threshold, return to the previous feature layer to further extract features; otherwise, perform Step 2-6. Step 2-6: Filter redundant and noisy key points using the non-maximum suppression method; combine spatial coherence to adjust the predicted positions of key points to meet geometric constraints and obtain the filtered key points; Connect the filtered key points using the partial affinity fields to form the overall skeleton of the excavator; Introduce an adjustment factor for the rotation angle between key points to reflect the correlation between key points; The key points meet the following geometric constraints: ; ; Among them, represents the predicted two-dimensional key point coordinates; represents the two-dimensional key point coordinates of the manually marked key points, that is, the true coordinates of the two-dimensional key points; represents the L2 vector norm, which is used to measure the error between the predicted coordinates and the true coordinates; represents the accumulation of the predicted errors between all key points, that is, the error metric for the entire skeleton reconstruction. By summing the errors of all key points, the accuracy of the overall model in pose prediction is evaluated; i and j represent indices, respectively identifying the numbers of different key points that make up the overall skeleton of the excavator, and i and j represent key point pairs with direct connection relationships; is the total number of key points; represents the translation degree-of-freedom adjustment parameter of the excavator in the plane, which is used to describe the translation correction during prediction; represents the rotation degree-of-freedom adjustment parameter of the excavator in the plane, which is used for key point rotation correction; represents the inter-frame key point correlation degree; represents the reasonable proportion interval of the skeleton key points, which is used to ensure that the distance ratio between key points is within an appropriate range; is a conversion function, which is used to correct the two-dimensional coordinates predicted by the original model; Step 2-7: Construct an operator intrusion function to determine whether to trigger an alarm. The operator intrusion function is as follows: ; Among them, is the intrusion value; is the pose distance between the person and the excavator; is the length index of the excavator skeleton; is the direction influence coefficient, is the theoretical operating radius of the excavator; is a weighted linear combination function; Preferably, when executing Step 3, the alarm processing module retrieves the value of, and judges: when When it is the case, it indicates the existence of a safety risk and triggers a high-risk warning; When Within the preset safety boundary threshold, a low-level warning is generated; When Greater than the preset safety boundary threshold, the system is in a normal operating state and no warning is triggered; According to the above judgment results, a warning signal is generated.

[0012] Preferably, when performing step 4, the hydraulic control execution module obtains the warning signal, determines the adjustment direction and control parameters of the boom attitude of the hydraulic system according to the preset safety state, and generates a control strategy: When a high-risk warning occurs, immediately apply a brake to the boom or adjust the movement trajectory to reduce the speed and amplitude; When a low-level warning occurs, take a mild intervention and at the same time generate a prompt message to keep a distance; A communication protocol for communicating with the embedded control unit of the excavator is preset in the hydraulic control execution module. According to the instructions specified in the communication protocol and combined with the control strategy, a control instruction is generated and sent to the embedded control unit of the excavator.

[0013] A dangerous operation control system and method for a construction site excavator based on behavior recognition according to the present invention solves the technical problems of insufficient real-time performance, inaccurate risk assessment, and lack of active intervention ability in traditional construction machinery safety control. The present invention uses a patrol unmanned aerial vehicle and a high-resolution sensor to collect operation site data in real time, accurately identifies the posture and key behaviors of the excavator through a deep learning model, can dynamically capture and evaluate safety risks in complex operation scenarios, and realizes a full-range safety assessment of the environment and behavior by calculating risk indicators in real time through a customized constraint function. When detecting potential dangers, it can not only quickly issue a warning, but also perform real-time intervention on the six-degree-of-freedom movement of the excavator boom through the hydraulic control module, actively restrict dangerous actions, and directly prevent accidents from occurring, providing unprecedented safety protection capabilities for the engineering site. In view of the complexity of the actual working condition environment, the present invention designs a multi-level redundancy mechanism including data preprocessing, model fine-tuning, and control strategy optimization to ensure the efficient operation of the system, and is compatible with existing construction machinery platforms and hydraulic control systems at the same time, facilitating popularization and application. Brief Description of the Drawings

[0014] Figure 1 is a schematic diagram of the dangerous operation control system of the construction site excavator of the present invention; Figure 2 is the main flow chart of the present invention; Figure 3 is the flow chart of the posture recognition of the present invention; Figure 4It is a flow chart of early warning judgment of the present invention. DETAILED DESCRIPTION

[0015] Example 1

[0016] Depend on Figure 1 A construction site excavator dangerous operation control system based on behavior recognition is shown, including a data acquisition module, a communication module, a behavior recognition module, an early warning processing module, a hydraulic control execution module and a human-computer interaction module; The data acquisition module is deployed on the drone to collect image data of the excavator operation site in real time and send it to the back-end server through the communication module; The drone is a DJI Mavic 3 Enterprise drone. In this embodiment, the flight path covering the entire operating area of ​​the excavator is set by programming to ensure no blind spot monitoring, the flight altitude is fixed at 10 meters, ensuring safety while obtaining a wider field of view, and the camera shoots at a 45° depression angle to reduce occlusion and distortion, covering the entire excavator and the surrounding operating environment.

[0017] The drone communicates with the backend service via a 5G network.

[0018] The communication module is deployed on the drone to provide a wireless data interaction channel between the drone and the back-end server and to encrypt the interaction data.

[0019] The behavior recognition module is deployed in the backend service to pre-process the received image data, extract key features using the pre-trained MobileNet-V2 model, perform key point detection and posture skeleton construction, output the excavator posture and key data, and calculate the safety risk index; The early warning processing module is deployed in the back-end server and is used to calculate the risk value of safety risk based on the excavator posture and key data through the preset constraint function, compare the risk value with the theoretical operating radius, obtain the risk level, and generate an early warning signal; The hydraulic control execution module is deployed in the back-end server and is used to establish a data link with the embedded control unit inside the excavator and send control instructions to the embedded control unit to complete the posture adjustment of the excavator; The embedded control unit is a PLC controller of the excavator content, and the embedded control unit communicates with the back-end server through a wireless local area network.

[0020] The human-computer interaction module is a client server, which is connected to the back-end server and is used to display the real-time monitoring interface, drone monitoring video, behavior recognition results, risk warning status and hydraulic control execution status.

[0021] Example 2

[0022] As Figures 2 - 4 shown, a method for controlling dangerous operations of construction site excavators based on behavior recognition described in Embodiment 2 is implemented on the basis of a system for controlling dangerous operations of construction site excavators based on behavior recognition described in Embodiment 1, and includes the following steps: Step 1: The data acquisition module collects real-time images of the excavator operation scene and the worker operation area, obtains image data, and sends it to the backend server through the communication module; When executing Step 1, the data acquisition module is a high-definition camera deployed on a drone, and the collected image data is static images; The method for obtaining static images is: the video data collected by the high-definition camera is used to obtain static images by extracting key frames from the video data; The communication module is a 5G communication module carried by the drone, and the image data is sent to the backend server through the 5G mobile network.

[0023] Step 2: Establish a behavior recognition module, a warning processing module, and a hydraulic control execution module in the backend server; The behavior recognition module retrieves the image data, preprocesses the image data to obtain preprocessed images, extracts features and key points from the preprocessed images, and extracts the operation behavior actions of the excavator through an optimization algorithm and a constraint function to generate a behavior recognition result; When executing Step 2, the specific steps are as follows: Step 2-1: The behavior recognition module retrieves the image data and preprocesses the image data, specifically including: Step 2-1-1: Scale the image size to a fixed size, that is, a pixel size of 256×256; Step 2-1-2: Normalize the pixel values of the image to the interval [0,1]; Step 2-1-3: Perform histogram equalization processing and contrast enhancement processing on the image to obtain an enhanced preprocessed image; Step 2-1-4: Divide the preprocessed image into two data sets: a training set and a test set. The training set accounts for 70% of the total number of preprocessed images, and the test set accounts for 30%; Step 2-2: The behavior recognition module constructs a pre-trained MobileNet-V2 model and uses the MobileNet-V2 model to extract the edge features of the excavator; Use multiple layers of convolution, pooling, and activation functions to capture the structural information in the preprocessed image; In this embodiment, the MobileNet-V2 model constructed by the behavior recognition module adopts a depthwise separable convolution structure. After each convolutional layer, batch normalization and the ReLU activation function are followed. Then, the max pooling layer is used to extract local features, and stacked to form a multi-layer convolutional network for capturing the subtle structural information in the preprocessed image.

[0024] In the model construction stage, the depthwise separable convolutional layer with a 3×3 convolutional kernel is used to extract the low-level features of the image. After batch normalization (BatchNormalization) and the ReLU activation function, the max pooling layer is used to extract the local invariant features in the image. This process progresses layer by layer to form multi-scale feature information.

[0025] Step 2-3: Input the feature map output by the convolutional layer into the keypoint detection network to achieve multi-scale feature fusion; Step 2-4: The keypoint detection network designs an output layer for each key part of the excavator and directly regresses the two-dimensional pixel coordinates of the keypoints. The key parts include the boom, arm, and bucket; Predict the confidence value of each keypoint; In this embodiment, the output layer of the keypoint detection network, in addition to regressing the two-dimensional coordinates of each keypoint, also outputs the corresponding confidence value through the sigmoid activation function. This value reflects the probability of the existence of the keypoint. In this embodiment, the preset threshold is set to 0.5 to filter out low-probability keypoints and ensure more accurate subsequent skeleton reconstruction.

[0026] Step 2-5: Set the confidence threshold and compare the confidence value with the confidence threshold: If the confidence value is lower than the confidence threshold, return to the previous feature layer to further extract features; otherwise, execute Step 2-6; In this embodiment, when the confidence of the detected keypoint is lower than the threshold, it will backtrack to the convolutional feature map of the previous layer, and use a refinement module (such as a small convolutional network or an upsampling module) to further enhance the local features of the image, so as to rediscover or confirm the keypoint information. This process adopts an iterative feedback mechanism to seek effective extraction of keypoints within the maximum number of iterations before making a decision.

[0027] Step 2-6: Use the non-maximum suppression method to filter out redundant and noisy keypoints; Combine spatial coherence to adjust the predicted positions of the keypoints to meet the geometric constraints and obtain the filtered keypoints; Use the part affinity fields to connect the filtered keypoints to form the overall skeleton of the excavator; Introduce an adjustment factor for the rotation angle between keypoints to reflect the correlation between keypoints; The keypoints meet the following geometric constraints: ; ; wherein, represents the predicted two-dimensional key point coordinates; represents the two-dimensional coordinates of the manually labeled key points, that is, the true two-dimensional coordinates of the key points; represents the L2 vector norm, which is used to measure the error between the predicted coordinates and the true coordinates; represents the accumulation of the prediction errors between all key points, that is, the error metric of the entire skeleton reconstruction. By summing the errors of all key points, the accuracy of the overall model in pose prediction is evaluated; i and j represent indices, respectively identifying the numbers of different key points that make up the overall skeleton of the excavator, and i and j represent key point pairs with a direct connection relationship; is the total number of key points; represents the translation degree-of-freedom adjustment parameter of the excavator in the plane, which is used to describe the translation correction during prediction; represents the rotation degree-of-freedom adjustment parameter of the excavator in the plane, which is used for key point rotation correction; represents the inter-frame key point correlation degree; represents the reasonable proportion interval of the skeleton key points, which is used to ensure that the distance ratio between key points is within a suitable range; is a conversion function, which is used to correct the two-dimensional coordinates predicted by the original model; In this embodiment, the principle of non-maximum suppression (NMS) is to describe in detail how to select the local maximum value among the candidate key points. For example, calculate the overlap degree between adjacent candidate points, select the point with the highest confidence, and suppress other overlapping values.

[0028] In this embodiment, during the process of constructing the skeleton of the excavator, the partial affinity fields (PAFs) are used to connect the key points. The method includes calculating the connectivity scores for adjacent key point pairs, determining the effective connections according to the score levels, and at the same time introducing the rotation angle adjustment between key points to strengthen the skeleton consistency.

[0029] The calculation formula of PAFs is prior art, so it will not be described in detail.

[0030] After the non-maximum suppression process, the present invention also adopts the partial affinity fields (PAFs) method to construct the skeleton for the effective key points screened by NMS. Specifically, for each pair of candidate key points, calculate their connectivity scores, and introduce a rotation angle adjustment factor to make the reconstructed skeleton satisfy the preset geometric constraints in space. The IOU threshold in NMS and the PAF calculation are both dynamically adjusted based on the on-site data to ensure the robustness of the overall skeleton.

[0031] For example: After detecting multiple key points of the excavator (such as the boom, arm, bucket, etc.), the PAFs help determine which key points should be connected together to form an overall skeleton.

[0032] By calculating the association scores between each pair of candidate key points, the most reasonable connection relationships are screened out among a large number of candidate points, thereby reconstructing the pose skeleton of the excavator.

[0033] Using PAFs can help reduce the deviation of the overall skeleton reconstruction caused by the detection error of a single key point, and improve the accuracy and robustness of skeleton construction.

[0034] Step 2 - 7: Construct the operator intrusion function to determine whether to trigger an alarm. The operator intrusion function is as follows: ; Among them, is the intrusion value; is the pose distance between the person and the excavator; is the length index of the excavator skeleton; is the direction influence coefficient, is the theoretical operating radius of the excavator; is the weighted linear combination function.

[0035] In this embodiment, The specific formula of is as follows: ; Among them, is the preset weight system, and these weight coefficients need to be formulated and adjusted according to the actual scenario where the excavator is located; represents the pose distance, and the specific calculation method is: takes the Euclidean distance between the key point of the excavator (such as the end of the boom) and the key position detected by the worker (such as the position of the worker, etc.).

[0036] In this embodiment, the worker's bounding box is detected through an object detection model (such as MobileNet - SSD), and the center point of the bounding box is used as the worker's position coordinates.

[0037] The calculation method of the direction influence coefficient is as follows: Calculate the included angle difference between the operating direction of the excavator and the position where the worker is located, that is is the direction influence coefficient. For example: If the excavator is facing the worker directly, then takes a larger value; conversely, if it deviates, it is smaller. The specific formula is as follows: ; Among them, is the included angle between the direction vectors, and normalize represents the normalization operation.

[0038] The calculation method of the length index of the excavator skeleton is as follows: It is represented by summing the distances between the key points of each skeleton of the excavator. , It reflects the current extended or folded state of the machine and is used as a dynamic index affecting the operation radius in this embodiment.

[0039] is the theoretical operation radius of the excavator: The value of is calculated based on the mechanical parameters of the excavator at the factory, such as the maximum extended length of the boom, the installation angle, and the safety redundancy distance.

[0040] In this embodiment, by setting the redundancy distance, the safety margin is ensured; ; Among them, is the maximum working radius of the boom calculated according to the factory parameters, is the increased redundancy distance for ensuring safety.

[0041] Step 3: The early warning processing module retrieves the behavior recognition result and judges whether there is a safety risk according to the preset constraint function , if yes, generates an early warning signal and executes Step 4; if no, returns to Step 2; When executing Step 3, the early warning processing module retrieves the value of, and judges: when , it indicates that there is a safety risk and triggers a high-risk early warning; When is within the preset safety boundary threshold, a low-level early warning is generated; When is greater than the preset safety boundary threshold, the system is in normal operation and no early warning is triggered; According to the above judgment results, an early warning signal is generated.

[0042] Step 4: The hydraulic control execution module generates an adjustment instruction according to the early warning signal and the preset communication protocol, and sends the adjustment instruction to the embedded control unit of the excavator. After the embedded control unit finishes executing according to the adjustment instruction, it feeds back the execution result to the backend server; When executing Step 4, the hydraulic control execution module obtains the early warning signal, determines the adjustment direction and control parameters of the hydraulic system for the boom attitude according to the preset safety state, and generates a control strategy: When a high-risk early warning occurs, immediately apply a brake to the boom or adjust the movement trajectory to reduce the speed and amplitude; When a low-level warning occurs, take mitigation intervention and generate a keep-distance prompt message at the same time; A communication protocol for communicating with the embedded control unit of the excavator is preset in the hydraulic control execution module. According to the instructions specified in the communication protocol and combined with the control strategy, a control instruction is generated and sent to the embedded control unit of the excavator. The embedded control unit controls the hydraulic system to execute the control instruction.

[0043] In this embodiment, the control instruction is encapsulated in accordance with a preset communication protocol format and transmitted to the embedded control unit of the excavator (such as a PLC controller) through a wireless local area network or a dedicated fieldbus.

[0044] A custom simple communication protocol based on the JSON format is adopted. In practical applications, industrial standard protocols such as Modbus, CANopen, and EtherCAT can be used as the underlying transmission method according to on-site requirements.

[0045] The communication protocol format is as follows: Data frame structure: Header (message header): Identifies the starting part of the message and contains the protocol version and message type, as follows: "version": Protocol version (such as "1.0"); "messageType": Message type (instruction type, such as "CONTROL_COMMAND"); Timestamp: Used to record the time when the instruction is generated for on-site synchronization and fault tracking; Command: Instruction name, used to identify a specific action. For example, "DANGER_INTERVENTION" represents emergency intervention; Parameters (control parameters): Include specific execution parameters, such as the target part, action type, force, operation mode, reference angle, and braking strategy, as follows: "targetArm": Target part (such as "boom"); "action": Action type (such as "deceleration braking" or "adjusting the trajectory"); "force": Braking force or intervention force (for example, percentage or specific value); "duration": Execution time or transition time (unit: millisecond); "angleAdjustment": If the rotation angle needs to be adjusted; Checksum: Used for data integrity verification and can be calculated using the CRC or other hash algorithms based on the message content.

[0046] Footer (Message End Indicator): Indicates the end of the message, e.g., the fixed string "EOF".

[0047] Example of protocol format: { "header":{ "version":"1.0", "messageType":"CONTROL_COMMAND" }, "timestamp":"2025-04-16T14:23:45Z", "command":"DANGER_INTERVENTION", "parameters":{ "targetArm":"Large Arm", "action":"Decelerate and Brake", "force":"80%", / / Emergency braking force at 80% strength "duration":100, / / Braking duration of 100 ms "angleAdjustment":"-15" / / Adjust the large arm angle 15 degrees to the left (negative value indicates counterclockwise) to avoid approaching workers }, "checksum":"3f5e2a7b", / / Example checksum (actual calculation method can use algorithms such as CRC32) "footer":"EOF" } Step 5: Deploy the client server. The client server communicates with the backend server via network cable. Establish a human-machine interaction module in the client service. The human-machine interaction module obtains the preprocessed images, behavior recognition results, warning signals, and execution results generated by the backend server, and displays the real-time monitoring interface, drone monitoring video, behavior recognition results, risk warning status, and hydraulic control execution status through the display screen.

[0048] A dangerous operation control system and method for construction site excavators based on behavior recognition according to the present invention solve the technical problems of insufficient real-time performance, inaccurate risk assessment, and lack of active intervention ability in traditional construction machinery safety control. The present invention uses inspection drones and high-resolution sensors to collect operation site data in real time, accurately identify the posture and key behaviors of excavators through a deep learning model, can dynamically capture and evaluate safety risks in complex operation scenarios, calculate risk indicators in real time through customized constraint functions, realize all-round safety assessment of the environment and behavior, when detecting potential dangers, can not only quickly issue warnings, but also perform real-time intervention on the six-degree-of-freedom movement of the excavator boom through a hydraulic control module, actively restrict dangerous actions, and directly prevent accidents from occurring, providing unprecedented safety protection capabilities for the engineering site. In view of the complexity of the actual working condition environment, the present invention designs multi-level redundancy mechanisms including data preprocessing, model fine-tuning, and control strategy optimization to ensure the efficient operation of the system, and is compatible with existing construction machinery platforms and hydraulic control systems at the same time, facilitating popularization and application.

Claims

1. A construction site excavator dangerous operation control system based on behavior recognition, characterized by: It includes data acquisition module, communication module, behavior recognition module, early warning processing module, hydraulic control execution module and human-computer interaction module; The data acquisition module is deployed on the drone to collect image data of the excavator operation site in real time and send it to the back-end server through the communication module; The communication module is deployed on the drone to provide a wireless data interaction channel between the drone and the backend server and implement data encryption for the interaction data; The behavior recognition module is deployed in the backend service to pre-process the received image data, extract key features using the pre-trained MobileNet-V2 model, perform key point detection and posture skeleton construction, output the excavator posture and key data, and calculate the safety risk index; The early warning processing module is deployed in the back-end server and is used to calculate the risk value of safety risk based on the excavator posture and key data through the preset constraint function, compare the risk value with the theoretical operating radius, obtain the risk level, and generate an early warning signal; The hydraulic control execution module is deployed in the back-end server and is used to establish a data link with the embedded control unit inside the excavator and send control instructions to the embedded control unit to complete the posture adjustment of the excavator; The human-computer interaction module is a client server, which is connected to the back-end server and is used to display the real-time monitoring interface, drone monitoring video, behavior recognition results, risk warning status and hydraulic control execution status.

2. A construction site excavator dangerous operation control system based on behavior recognition as claimed in claim 1, characterized in that: The drone is a DJI Mavic 3 Enterprise drone.

3. A construction site excavator dangerous operation control system based on behavior recognition as claimed in claim 1, characterized in that: The drone communicates with the backend service via a 5G network.

4. A construction site excavator dangerous operation control system based on behavior recognition as claimed in claim 1, characterized in that: The embedded control unit is a PLC controller of the excavator content, and the embedded control unit communicates with the back-end server through a wireless local area network.

5. A method for controlling dangerous operations of a construction site excavator based on behavior recognition, applied to the dangerous operations control system for a construction site excavator based on behavior recognition as described in any one of claims 1 to 4, characterized in that: The steps include: Step 1: The data acquisition module collects real-time images of the excavator operation scene and the worker's work area, obtains image data, and sends it to the back-end server through the communication module; Step 2: Establish a behavior recognition module, a warning processing module and a hydraulic control execution module in the back-end server; The behavior recognition module retrieves image data, preprocesses the image data to obtain a preprocessed image, performs feature extraction and key point detection on the preprocessed image, extracts the operating behavior of the excavator through optimization algorithms and constraint functions, and generates behavior recognition results; Step 3: The warning processing module retrieves the behavior recognition results and uses the preset constraint function , determine whether there is a security risk: if yes, generate a warning signal and execute step 4; If not, return to step 2; Step 4: The hydraulic control execution module generates an adjustment instruction according to the warning signal and the preset communication protocol, and sends the adjustment instruction to the embedded control unit of the excavator. After the embedded control unit completes the execution according to the adjustment instruction, it feeds back the execution result to the back-end server; Step 5: Deploy the client server. The client server communicates with the back-end server through the network cable. Establish a human-computer interaction module in the client service. The human-computer interaction module obtains the pre-processed images, behavior recognition results, warning signals and execution results generated by the back-end server, and displays the real-time monitoring interface, drone monitoring video, behavior recognition results, risk warning status and hydraulic control execution status through the display screen.

6. A method for controlling dangerous operations of a construction site excavator based on behavior recognition as claimed in claim 5, characterized in that: When executing step 1, the data acquisition module is a high-definition camera deployed on the drone, and the image data collected is a static image; The method for obtaining a static image is as follows: the video data collected by a high-definition camera is used to extract key frames from the video data to obtain a static image; The communication module is the drone’s built-in 5G communication module, which sends image data to the backend server through the 5G mobile network.

7. A method for controlling dangerous operations of a construction site excavator based on behavior recognition as claimed in claim 6, characterized in that: When executing step 2, the specific steps are as follows: Step 2-1: The behavior recognition module retrieves image data and pre-processes the image data, including: Step 2-1-1: Scale the image size to a fixed size, i.e. 256×256 pixels; Step 2-1-2: Normalize the pixel values ​​of the image to the interval [0,1]; Step 2-1-3: Perform histogram equalization and contrast enhancement on the image to obtain an enhanced pre-processed image; Step 2-1-4: Divide the preprocessed images into two data sets: a training set and a test set. The training set accounts for 70% of the total number of preprocessed images, and the test set accounts for 30%; Step 2-2: The behavior recognition module builds a pre-trained MobileNet-V2 model and uses the MobileNet-V2 model to extract the edge features of the excavator; Use multiple layers of convolution, pooling, and activation functions to capture structural information in preprocessed images; Step 2-3: Input the feature map output by the convolutional layer into the key point detection network to achieve multi-scale feature fusion; Step 2-4: The key point detection network designs an output layer for each key part of the excavator, and directly regresses the two-dimensional pixel coordinates of the key points. The key parts include the boom, forearm and bucket; Predict the confidence value of each key point; Step 2-5: Set a confidence threshold and compare the confidence value with the confidence threshold: if the confidence value is lower than the confidence threshold, return to the previous feature layer to further extract features; otherwise, execute step 2-6; Step 2-6: Use the non-maximum suppression method to filter redundant and noisy key points; combine the spatial coherence to adjust the predicted position of the key points so that they meet the geometric constraints and obtain the filtered key points; Use some affinity fields to connect the filtered key points to form the overall skeleton of the excavator; Introduce the adjustment factor of the rotation angle between key points to reflect the correlation between key points; Key points comply with the following geometric constraints: ； ； in, Represents the predicted two-dimensional key point coordinates; Represents the two-dimensional coordinates of the manually annotated key points, that is, the real coordinates of the two-dimensional key points; Represents the L2 vector norm, which is used to measure the error between the predicted coordinates and the true coordinates; It represents the accumulation of prediction errors between all key points, that is, the error measure of the entire skeleton reconstruction. By summing up the errors of all key points, the accuracy of the overall model in posture prediction is evaluated; i and j represent indexes, which respectively identify the numbers of different key points in the overall skeleton of the excavator. i and j represent key point pairs with direct connection relationships; is the total number of key points; It represents the adjustment parameter of the excavator's translational freedom in the plane, and is used to describe the translation correction during prediction; Represents the rotational freedom adjustment parameters of the excavator in the plane, which is used for key point rotation correction; Indicates the correlation between key points between frames; Indicates the reasonable ratio range of skeleton key points, which is used to ensure that the distance ratio between key points is within an appropriate range; is the conversion function used to correct the two-dimensional coordinates predicted by the original model; Step 2-7: Construct an operator intrusion function to determine whether to trigger an early warning. The operator intrusion function is as follows: ； in, is the intrusion value; is the posture distance between the personnel and the excavator; It is the length index of the excavator frame; is the directional influence coefficient, is the theoretical operating radius of the excavator; is a weighted linear combination function.

8. A method for controlling dangerous operations of a construction site excavator based on behavior recognition as claimed in claim 7, characterized in that: When executing step 3, the warning processing module calls The value of, judge: when When the alarm is raised, it indicates that there is a security risk and triggers a high-risk warning; when Within the preset safety boundary threshold, a low-level warning is generated; when If the value is greater than the preset safety boundary threshold, the system is in normal operation and no warning is triggered; Based on the above judgment results, a warning signal is generated.

9. A method for controlling dangerous operations of a construction site excavator based on behavior recognition as claimed in claim 8, characterized in that: When executing step 4, the hydraulic control execution module obtains the warning signal, determines the adjustment direction and control parameters of the hydraulic system for the boom posture according to the preset safety state, and generates a control strategy: When a high-risk warning is issued, brake the arm immediately or adjust the movement trajectory to reduce speed and amplitude; When a low-level warning is issued, mitigation intervention is taken and a reminder message to keep distance is generated; The hydraulic control execution module presets a communication protocol for communicating with the embedded control unit of the excavator. According to the instructions specified in the communication protocol and in combination with the control strategy, a control instruction is generated and sent to the embedded control unit of the excavator.

Images