A method and apparatus for monitoring personnel intrusion on a mobile crane

Abstract

The application provides a kind of automobile crane personnel intrusion monitoring method and device, belong to engineering machinery safety monitoring technical field.The method includes: synchronous acquisition contains the RGB image of pedestrian information, disparity map and three-dimensional point cloud data, and time-space registration is carried out and is preprocessed to establish training set;Personnel intrusion monitoring model of automobile crane is constructed, including feature extraction module, boom prediction module, pedestrian position prediction module and time-space registration module;Personnel intrusion monitoring model of automobile crane is trained using training set data;Collect construction site RGB image, disparity map and three-dimensional point cloud data, after time-space registration, input personnel intrusion monitoring model of automobile crane completed training according to the relationship between the position of pedestrian, motion direction and fence, determine intrusion risk.The application improves the pedestrian recognition accuracy in complex environments such as rain, fog and occlusion through the time-space attention feature fusion mechanism, and can provide early warning for potential intrusion behavior.

Description

Technical Field

[0001] This invention relates to a method and device for monitoring personnel intrusion into a truck crane, belonging to the field of engineering machinery safety monitoring technology. Background Technology

[0002] With the rapid advancement of urbanization and the intensive implementation of large-scale infrastructure projects in my country, the environments at various construction sites are becoming increasingly complex, and the number of scenarios involving cross-operation between humans and machinery is significantly increasing. Truck cranes, as indispensable heavy-duty lifting equipment in modern construction, are widely used in densely populated areas such as construction sites and transportation hubs due to their high mobility and wide operating range. However, during large-scale slewing, luffing, and lifting operations, the massive metal structure and dynamically moving parts of the crane create significant blind spots. Coupled with noise pollution and disorderly personnel movement at construction sites, ground workers are highly susceptible to accidentally entering the crane's dangerous operating radius (such as the boom rotation area, load swing area, and outrigger extension area) without noticing the equipment's movement. If personnel intrude into dangerous areas without timely warning, catastrophic consequences are highly likely: collisions between the crane's slewing mechanism or the load and personnel can cause serious mechanical injuries or even fatal accidents; load falls caused by emergency braking can lead to secondary injuries; accidents can also result in project delays, equipment damage, and huge compensation claims, seriously threatening the lives of construction workers and negatively impacting project management reputation and social stability.

[0003] Existing technologies primarily employ a single binocular camera combined with a YOLOv5s model for monitoring, identifying intrusions through a circular area of ​​fixed radius. This approach has the following key limitations:

[0004] 1. Binocular cameras rely on parallax ranging. In scenarios such as rain, fog, and backlight, the image quality deteriorates, leading to a significant increase in parallax calculation errors and a high rate of missed detections.

[0005] 2. Static fencing: A fixed circular plane area is used as the danger zone, without considering the impact of boom movement (such as rotation and luffing) and ground slope. When the boom rotates rapidly, the fencing update lags behind the actual danger zone, which can easily create a safety blind spot.

[0006] 3. Insufficient dynamic target recognition: Intrusion is judged only by the current location without combining the pedestrian's movement trajectory and intention (such as approaching the crane arm), resulting in delayed alarm response and difficulty in timely stopping dangerous behavior.

[0007] 4. Limited features: Relying solely on visual features to identify pedestrians, the recognition accuracy decreases in occluded scenarios (such as being blocked by barriers or goods) due to missing features.

[0008] Therefore, developing an intelligent monitoring and early warning system for personnel intrusion into truck cranes with real-time perception, accurate identification, and rapid response capabilities to proactively prevent human-machine conflict accidents has become an urgent technical requirement for improving the inherent safety level of construction sites and ensuring the safety of personnel. Summary of the Invention

[0009] The purpose of this invention is to provide a method and device for monitoring personnel intrusion into truck cranes, which solves the problems existing in the current monitoring of personnel intrusion into truck cranes and improves the accuracy of monitoring.

[0010] To achieve the above objectives, the present invention employs the following technical solution:

[0011] A method for detecting personnel intrusion into a truck crane includes the following steps:

[0012] Simultaneously acquire RGB images, disparity maps, and 3D point cloud data containing pedestrian information, perform spatiotemporal registration, and preprocess the data to build a training set;

[0013] A personnel intrusion detection model for a truck crane is constructed, comprising a feature extraction module, a boom prediction module, a pedestrian location prediction module, and a spatiotemporal registration module. The feature extraction module extracts visual features from RGB images and disparity maps, and extracts point cloud features from 3D point cloud data, incorporating a temporal and spatial attention mechanism to fuse the visual and point cloud features. The boom prediction module predicts the boom tip position based on an improved LSTM network and generates a 3D fence. The pedestrian location prediction module uses Kalman filtering combined with a Transformer to predict pedestrian positions using the fused visual and point cloud features. The spatiotemporal registration module performs spatiotemporal matching of the predicted pedestrian trajectory with the dynamic 3D fence to determine the intrusion risk.

[0014] Train the truck crane personnel intrusion detection model using the training set data, and obtain the trained truck crane personnel intrusion detection model;

[0015] RGB images, disparity maps, and 3D point cloud data of the construction site are collected, spatiotemporally registered, and then input into the trained truck crane personnel intrusion monitoring model to determine the intrusion risk based on the pedestrian's position, movement direction, and relationship with the fence.

[0016] Preferably, the spatiotemporal registration of the RGB image, disparity map, and 3D point cloud data is achieved through hardware triggering for time synchronization, sensor extrinsic parameters are obtained through hand-eye calibration, and the LiDAR point cloud is converted to the camera coordinate system for spatial registration.

[0017] Preferably, the feature extraction module uses an improved CSPDarknet network to extract visual features from RGB images and disparity maps, and uses Kalman filtering combined with Transformer to extract point cloud features from 3D point cloud data;

[0018] The improved CSPDarknet network introduces a coordinate attention module to enhance human region features, and processes the output features of the CSPDarknet network through the coordinate attention module to output visual features.

[0019] The Kalman filter, combined with the Transformer, divides the point cloud into voxels and extracts point cloud features containing voxel density and motion vectors.

[0020] The formula for fusing visual features and point cloud features is as follows:

[0021] ,

[0022] ,

[0023] ,

[0024] in, For cross-modal fusion features, This is the current timestamp. For spatial attention mask, As a visual feature, This is a 3×3 convolution operation. For max pooling, For activation function, For time attention weights, Point cloud features, among which The difference in features between adjacent frames It is a natural constant.

[0025] Preferably, the boom prediction module includes a boom tip position prediction module and a three-dimensional fence calculation module. The boom tip position prediction module predicts the boom tip position based on an improved LSTM network. The improved LSTM network incorporates an attention mechanism, and the prediction formula is as follows:

[0026] ,

[0027] in, for Predicted coordinates of the boom tip at all times. This is the current timestamp. To predict the timestamp, For improved LSTM networks, For LSTM hidden states, for Real-time coordinates of the boom tip for At all times, the boom elevation angle, for The angular velocity of the boom rotation at all times. for The constant boom luffing speed.

[0028] Preferably, the boom tip position prediction module includes: an input layer, an attention weight calculation module, an improved LSTM unit, and a prediction layer;

[0029] The input parameter set of the input layer: , , , And standardize the parameter set to ;

[0030] The attention weight calculation module performs weighted calculations on the parameter set to obtain the weighted boom parameter features; including:

[0031] The standardized parameter set The previous LSTM hidden state is mapped to query, key, and value; the query is the previous LSTM hidden state, and the key is... Through weight matrix Transformation, value Through weight matrix Transformation; It maps the standardized boom parameter set to a weight matrix of the keys. It is a weight matrix that maps the standardized boom parameter set to values;

[0032] Calculate attention score Attention weights are obtained by weight normalization using the Softmax function, and the parameter set is then weighted and fused using these attention weights. The calculation formula is as follows:

[0033] ,

[0034] in, For matrix multiplication, For transpose, Scaling factor For the hidden layer dimension, The timestamp of the current moment;

[0035] The improved LSTM unit utilizes the gating mechanism of LSTM to capture the long-term dependence of boom parameters, and replaces the original input of LSTM with weighted parameters;

[0036] Hidden states of the attention mechanism in the improved LSTM unit The updated formula is:

[0037] ,

[0038] in, For output gate, To renew memory cells, For scaling dot product attention functions, This represents the hidden state of the LSTM from the previous time step. for Real-time coordinates of the boom tip for At all times, the boom elevation angle, for The angular velocity of the boom rotation at all times. for The constant boom luffing speed.

[0039] The prediction layer hides the LSTM state. The coordinates are mapped to the predicted coordinates of the boom tip at time t+1.

[0040] Preferably, the three-dimensional fence calculation module includes:

[0041] Radial radius of three-dimensional fence The calculation formula is as follows:

[0042] ,

[0043] ,

[0044] in, for The basic danger radius for constant crane operation. for At any given time, the length of the boom is... This represents the actual lifting load of the truck crane during its current operation. The rated lifting load of the truck crane, The boom elevation angle of the truck crane. Predicting the linear velocity at the tip of the boom. This is the slope correction factor. The ground slope angle;

[0045] Three-dimensional fence height threshold The calculation formula is as follows:

[0046] ,

[0047] in, This is the reference height for the crane turntable. Predicting acceleration at the tip of the boom;

[0048] Using the predicted coordinates of the boom tip as the center point, a cylindrical three-dimensional fence is constructed based on the radial radius and height values.

[0049] Preferably, the prediction formula for predicting pedestrian positions is as follows:

[0050] ,

[0051] in, For pedestrians Predict coordinates at any time. , For pedestrians Actual coordinates at any given time For pedestrians Speed ​​at any moment For pedestrians Acceleration at all times This is the Kalman filter operator.

[0052] Preferably, the spatiotemporal registration module performs spatiotemporal matching between the predicted pedestrian trajectory and the dynamic 3D fence to determine the intrusion risk in the following specific way:

[0053] ,

[0054] in, Risk level, for Dynamic 3D fence at any time for A dynamic three-dimensional fence buffer zone is created at any given time, with the buffer zone extending 1 meter inside the fence. for The dynamic 3D fence core area is located at any given moment, and the core area is the region inside the buffer zone. The normal vector outside the fence boundary. This indicates that the pedestrian is moving towards the inside of the fence. This indicates that the pedestrian did not move inwards.

[0055] Preferably, the loss function during the training process of the truck crane personnel intrusion detection model is... as follows:

[0056] ,

[0057] ,

[0058] in, For 2D bounding box loss, For the 3D bounding box intersection-union ratio loss of LiDAR, For trajectory prediction error, The actual location of the pedestrian. , , These are the weighting coefficients.

[0059] A personnel intrusion detection device for a truck crane includes a processor and a memory storing program instructions, the processor being configured to execute the personnel intrusion detection method for the truck crane when the program instructions are executed.

[0060] The advantages of this invention are:

[0061] By using a spatiotemporal attention feature fusion mechanism, the accuracy of pedestrian recognition in complex environments such as rain, fog, and occlusion has been improved.

[0062] Based on an improved LSTM network-based crane trajectory prediction model, combined with parameters such as real-time load and ground slope, the generated dynamic 3D fence has a reduced boundary error compared to the actual hazardous area. This improvement solves the safety blind spot problem caused by crane movement in traditional static fences.

[0063] A pedestrian trajectory prediction model combining Kalman filtering and Transformer can provide early warning of potential intrusions, significantly reducing response time compared to traditional methods.

[0064] Hybrid loss function optimizes model accuracy: A hybrid loss function that integrates 2D / 3D intersection-union ratio loss (LCIoU / L3DIoU) with trajectory prediction error significantly improves the accuracy of dynamic target recognition. Even in scenarios with partial pedestrian occlusion or rapid movement, the model maintains stable performance, meeting the requirements of complex working conditions. Attached Figure Description

[0065] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof.

[0066] Figure 1 This is a schematic diagram of the method flow of the present invention.

[0067] Figure 2 This is a flowchart illustrating the operation of a personnel intrusion monitoring method for truck cranes based on multi-source sensing and dynamic fences, according to an embodiment of the present invention. Detailed Implementation

[0068] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0069] Example 1

[0070] like Figures 1-2 As shown, a method for detecting personnel intrusion into a truck crane includes the following steps:

[0071] S1: Simultaneously acquire RGB images, disparity maps, and 3D point cloud data containing pedestrian information, perform spatiotemporal registration, and preprocess the data to build a training set.

[0072] S2: Construct a personnel intrusion monitoring model for a truck crane, including a feature extraction module, a boom prediction module, a pedestrian position prediction module, and a spatiotemporal registration module. The feature extraction module extracts visual features from RGB images and disparity maps, and extracts point cloud features from 3D point cloud data, introducing a temporal and spatial attention mechanism to fuse visual features and point cloud features. The boom prediction module predicts the boom tip position based on an improved LSTM network and generates a 3D fence. The pedestrian position prediction module uses Kalman filtering combined with Transformer to predict pedestrian positions using fused visual features and point cloud features. The spatiotemporal registration module performs spatiotemporal matching between the predicted pedestrian trajectory and the dynamic 3D fence to determine the intrusion risk.

[0073] S3: Use the training set data to train the truck crane personnel intrusion detection model and obtain the trained truck crane personnel intrusion detection model.

[0074] S4: Collect RGB images, disparity maps, and 3D point cloud data from the construction site, perform spatiotemporal registration, and input them into the trained truck crane personnel intrusion monitoring model to determine the intrusion risk based on the pedestrian's position, movement direction, and relationship with the fence.

[0075] As a refinement of the above embodiments, step S1 includes:

[0076] Synchronous data acquisition: The binocular camera acquires RGB images and disparity maps, and the LiDAR acquires 3D point clouds. Time synchronization is achieved through hardware triggering.

[0077] Spatiotemporal registration: Sensor extrinsic parameters (rotation matrix, translation vector) are obtained through hand-eye calibration, and the LiDAR point cloud is transformed to the camera coordinate system. The formula is: ,in These are the three-dimensional coordinates in the camera coordinate system. The coordinates are three-dimensional coordinates in the lidar coordinate system. It is a 3×3 orthogonal rotation matrix. It is a 3×1 translation vector.

[0078] As a refinement of the above embodiments, the feature extraction module in step S2 includes:

[0079] (1) Visual features An improved CSPDarknet network is used, which enhances human region features through a coordinate attention module (CAM), resulting in an output dimension of [missing information]. (H and W are the height and width of the feature map,) (Number of channels). Point cloud features The point cloud is divided into 8cm×8cm×8cm voxels using Kalman filtering combined with a Transformer (VoxelTransformer). Spatiotemporal features (including voxel density and motion vectors) are extracted, and the output dimension is [missing information]. ( (Number of feature channels in the point cloud).

[0080] The core improvement of the CSPDarknet network is the addition of a coordinate attention module (CAM) to the feature extraction link of the original CSPDarknet network. The purpose is to enhance human body region features and suppress background interference (such as invalid information such as cranes, building materials, rain and fog at construction sites).

[0081] The original CSPDarknet network's feature extraction process was: "Input image → Convolutional layer → Cross-Stage Local Connection (CSP) → Downsampling → Output basic feature map". The improved process adds a CAM module after the "basic feature map output", forming: "Input image → Original CSPDarknet feature extraction → Generation of basic feature map → CAM module processing → Output of visual features enhanced with human characteristics". The mechanism of the CAM module is as follows:

[0082] ,

[0083] in, For the input feature map, This is a global average pooling operation; This is a 1×1 convolution operation.

[0084] Specifically, it includes:

[0085] Step 1: Global Average Pooling (GAP): Applying the base feature maps output by CSPDarknet Perform global average pooling to compress spatial dimensions, retain channel-level feature information, and focus on channel-level feature weight allocation.

[0086] Step 2: 1×1 convolution ( ): Perform a 1×1 convolution on the GAP output and adjust the number of channels (to reduce computational cost and fuse channel information).

[0087] Step 3: Sigmoid activation: The convolution result is mapped to the [0,1] interval through the Sigmoid function to generate coordinate attention weights (high weights are assigned to the channels corresponding to the human body region, and low weights are assigned to the background region).

[0088] Step 4: Feature Weighted Fusion: Combine the generated attention weights with the original base feature map. Perform element-wise multiplication to ultimately output visual features that enhance the characteristics of the human body region.

[0089] (2) Spatiotemporal attention fusion: Introducing time dimension weights and dynamically allocating cross-modal feature weights, the fusion formula is as follows:

[0090] ,

[0091] ,

[0092] ,

[0093] in, As a feature of fusion, This is the current timestamp. For spatial attention mask, As a visual feature, This is a 3×3 convolution operation. For max pooling, For activation function, For time attention weights, Point cloud features, among which The difference in features between adjacent frames It is a natural constant.

[0094] Fusion features It is used for "accurate detection of pedestrians in the work area". Through cross-modal fusion of vision and point cloud, it solves the problem of missed detection by single vision in rain, fog and occlusion scenarios, and ensures the recognition accuracy of pedestrian targets.

[0095] As a refinement of the above embodiments, the boom prediction module in step S2 includes:

[0096] (1) Crane trajectory prediction: An improved LSTM network (with attention mechanism) is used to predict the future trajectory. (This implementation takes 1 second) The position of the boom tip at that moment is predicted by the following formula:

[0097] ,

[0098] in, for Predicted coordinates of the boom tip at all times. This is the current timestamp. To predict the timestamp, For improved LSTM networks, For LSTM hidden states, for Real-time coordinates of the boom tip for At all times, the boom elevation angle, for The angular velocity of the boom rotation at all times. for The constant boom luffing speed.

[0099] The boom tip position prediction module includes: an input layer, an attention weight calculation module, an improved LSTM unit, and a prediction layer.

[0100] The input parameter set of the input layer: , , , And standardize the parameter set to .

[0101] The attention weight calculation module performs weighted calculations on the parameter set to obtain the weighted boom parameter features; including:

[0102] The standardized parameter set The previous LSTM hidden state is mapped to query, key, and value; the query is the previous LSTM hidden state, and the key is... Through weight matrix Transformation, value Through weight matrix Transformation; It maps the standardized boom parameter set to a weight matrix of the keys. It is a weight matrix that maps the standardized boom parameter set to values; both are used for the attention weight calculation of the boom tip position prediction module.

[0103] Calculate attention score Attention weights are obtained by weight normalization using the Softmax function, and the parameter set is then weighted and fused using these attention weights. The calculation formula is as follows:

[0104] ,

[0105] in, For matrix multiplication, For transpose, Scaling factor For the hidden layer dimension, This is the timestamp of the current moment.

[0106] The improved LSTM unit utilizes the gating mechanism of LSTM to capture the long-term dependencies of boom parameters, replacing the original input of the LSTM with weighted parameters; including:

[0107] Forgotten Gate :

[0108] ,

[0109] in, For the Sigmoid function, This is the weight matrix. For bias, For vector concatenation, For attention weights, This represents the hidden state of the LSTM at the previous time step.

[0110] Input gate :

[0111] ,

[0112] in, Here is the weight matrix of the input gate. This is the bias term for the input gate.

[0113] Candidate memory cells :

[0114] ,

[0115] in, Map values ​​to, The weight matrix for candidate memory cells, This refers to the bias term for candidate memory cells.

[0116] Memory cell renewal:

[0117] ,

[0118] in, For element-wise product, To renew memory cells, These are memory cells before they are renewed.

[0119] Output gate :

[0120] ,

[0121] in, Here is the weight matrix of the output gate. This is the bias term for the output gate.

[0122] Hidden states of the attention mechanism in the improved LSTM unit The updated formula is:

[0123] ,

[0124] in, For scaling dot product attention functions, This represents the hidden state of the LSTM from the previous time step. for Real-time coordinates of the boom tip for At all times, the boom elevation angle, for The angular velocity of the boom rotation at all times. for The constant boom luffing speed.

[0125] The prediction layer hides the LSTM state. The coordinates are mapped to the predicted coordinates of the boom tip at time t+1.

[0126] (2) Three-dimensional fence calculation: A three-dimensional fence with lead time is generated by combining the predicted trajectory, wherein the radial radius of the three-dimensional fence is... The calculation formula is as follows:

[0127] ,

[0128] ,

[0129] in, Let t be the basic danger radius for crane operation. for At any given time, the length of the boom is... This represents the actual lifting load of the truck crane during its current operation. The rated lifting load of the truck crane, The boom elevation angle of the truck crane. Predicting the linear velocity at the tip of the boom. This is the slope correction factor (with a value range of 0.1-0.3; the specific value can be adjusted according to the crane model (0.1-0.2 for small cranes and 0.2-0.3 for large cranes) or the on-site slope level; the default recommended value is 0.2 (balancing correction effect and precision control)). The slope angle is the ground slope angle.

[0130] The height threshold of the three-dimensional fence The calculation formula is as follows:

[0131] ,

[0132] in, This is the reference height for the crane turntable. Predict acceleration for the boom tip.

[0133] Using the predicted coordinates of the boom tip as the center point, a cylindrical three-dimensional fence is constructed based on the radial radius and height values.

[0134] It should be noted that the parameters for predicting boom motion come from the crane's own condition monitoring sensors and CAN bus.

[0135] As a refinement of the above embodiments, the pedestrian location prediction module in step S2 includes:

[0136] (1) Pedestrian trajectory prediction: Kalman filtering combined with Transformer is used to predict the future. (This embodiment uses 0.5s) Pedestrian position, the formula is:

[0137] ,

[0138] in, For pedestrians Predict coordinates at any time. For pedestrians Actual coordinates at any given time For pedestrians Speed ​​at any moment For pedestrians Acceleration at all times This is the Kalman filter operator.

[0139] Pedestrian trajectory prediction (Kalman filter + Transformer) requires information on pedestrian position, velocity, and acceleration detected by S2, and the accurate acquisition of this information depends on the fusion features of S2.

[0140] The specific combination structure of Kalman filtering and Transformer is as follows:

[0141] 1) Overall Collaborative Logic

[0142] Kalman filtering provides a denoised pedestrian state sequence, while Transformer captures the nonlinear long-range dependence of pedestrian motion. Together, they output a high-precision short-term trajectory prediction.

[0143] 2) Detailed structure and processing method of each module

[0144] 2-1) Module 1: Kalman Filtering—Observation Noise Removal and State Estimation:

[0145] It processes and collects RGB images, disparity maps, and 3D point cloud data containing pedestrian information, filters out noise, and outputs a clean pedestrian state sequence (including position, velocity, and acceleration) to provide high-quality input for the Transformer.

[0146] Input data: 2D coordinates of pedestrians output from binocular cameras; pedestrian motion parameters output from LiDAR.

[0147] Noise sources: sensor jitter caused by boom rotation (±0.1m position error), visual coordinate offset caused by rain and fog.

[0148] Processing flow:

[0149] In the prediction phase, the theoretical state at time t is predicted based on the pedestrian motion model (uniform velocity + acceleration correction):

[0150] ,

[0151] in, for Predict the state vector at each time step; Let be the state transition matrix, take This corresponds to a 50ms sampling interval; For the control matrix, take Adapting the effect of acceleration on position / velocity Let be the acceleration control value of the boom at time t.

[0152] Error covariance prediction:

[0153] ,

[0154] in, For process noise covariance, For error covariance, Transpose of the state transition matrix.

[0155] During the update phase, the predicted state is corrected using lidar observations to filter out noise:

[0156] Kalman gain : ,

[0157] in, For the observation matrix, take The dimensions of the observed values ​​are consistent with those of the state vector; To observe the noise covariance.

[0158] Status Update: ,

[0159] in, The output is the actual observation vector from the lidar, after correction. optimal state at any time .

[0160] Error covariance update: ,

[0161] in, To provide an error benchmark for the next iteration of the updated error covariance. It is an identity matrix.

[0162] Output: Generate a denoised sequence of pedestrian states.

[0163] 2-1) Module Two: Transformer—Nonlinear Motion Dependency Modeling

[0164] Based on the clean state sequence output by the Kalman filter, the nonlinear dependence of pedestrian motion (such as sudden changes in direction or deceleration) is learned, which makes up for the limitations of the linear motion assumption of the Kalman filter and optimizes the accuracy of trajectory prediction.

[0165] Input data: The state sequence output by the Kalman filter.

[0166] Network architecture design (adapting to short-term forecasting needs and simplifying the general Transformer architecture):

[0167] Position coding is used because the sequence length is short (only 5 frames). Fixed sinusoidal position coding is adopted.

[0168] A single-head self-attention layer focuses on the motion dependency between historical frames and the current frame, avoiding computational redundancy in multi-head attention: Query(Q): encoding vector of the current frame (time t) (1×64); Key(K): encoding vector of the past 5 frames (5×64); Value(V): encoding vector of the past 5 frames (5×64).

[0169] Perform attention weight calculation.

[0170] The feedforward network (FFN) consists of a 2-layer fully connected network (64→128→64), using ReLU as the activation function, processing the features of the attention output, and enhancing non-linear motion patterns (such as feature extraction of pedestrians changing from constant speed to deceleration).

[0171] Output layer: Linear mapping layer (64→2), which maps the feature vector output by the Transformer to the predicted values ​​of pedestrian motion parameters at t+0.5s.

[0172] 2-3) Module 3: Fusion Prediction—Final Trajectory Output

[0173] Combining the linear prediction of Kalman filtering with the nonlinear correction of Transformer, the final predicted coordinates of the pedestrian at t+0.5s are output. The processing logic is as follows:

[0174] Kalman filter linear prediction: based on the optimal state at time t Predict the base position at t+0.5s :

[0175] ,

[0176] in, for The position of the boom tip at all times. for The speed of the boom tip at all times, for The acceleration control amount of the boom at all times.

[0177] Transformer nonlinear correction: The predicted pedestrian motion parameters output by the Transformer are used to correct linear prediction errors (such as linear prediction deviations caused by sudden changes in pedestrian direction) to obtain the corrected pedestrian motion parameters. As output:

[0178] ;

[0179] in, These are the predicted values ​​for pedestrian motion parameters.

[0180] (2) Risk classification: Intrusion risk is determined by spatiotemporal matching, and the classification formula is as follows:

[0181] ,

[0182] in, Risk level, for Dynamic 3D fence at any time for A dynamic three-dimensional fence buffer zone is created at any given time, with the buffer zone extending 1 meter inside the fence. for The dynamic 3D fence core area is located at any given moment, and the core area is the region inside the buffer zone. The normal vector outside the fence boundary. This indicates that the pedestrian is moving towards the inside of the fence. This indicates that the pedestrian did not move inwards.

[0183] When the risk level is 0 (Risk=0), if the pedestrian's predicted position in the next 0.5 seconds is not within the dynamic 3D fence area at time t+1 (i.e., completely outside the fence boundary), and regardless of the pedestrian's direction of movement (towards or outwards from the fence), it is determined to be without intrusion risk. Continuous monitoring is performed without triggering any alarm devices.

[0184] When the risk level is 1 (Risk=1), the pedestrian's predicted position in the next 0.5 seconds is within the buffer zone (1m inside the fence, outside the core area), and the pedestrian's movement direction is not towards the inside of the fence (core area direction) (i.e. , The normal vector outside the fence boundary. For pedestrian speed (Directional component).

[0185] Triggering Level 1 Response (Low Risk Warning): Activate the yellow warning light (installed at the base of the boom) and simultaneously play a voice prompt (such as "Pay attention to people around you, pedestrians are near the safety boundary"), without any buzzer or power cut-off action.

[0186] When the risk level is 2 (Risk=2), it is considered a medium risk if any of the following conditions are met:

[0187] Condition 1: The pedestrian's predicted position in the next 0.5 seconds is within the buffer zone, and the pedestrian's movement direction is towards the inside of the fence (core area direction). ).

[0188] Condition 2: The pedestrian's predicted position in the next 0.5 seconds is within the core area (inside the buffer zone), and the pedestrian's movement direction does not move further inside the fence. ).

[0189] Trigger Level 2 Response (Medium Risk Warning): Activate the red flashing light (installed on the turntable) and simultaneously activate the 115dB buzzer (high-frequency warning sound) to remind pedestrians to evacuate immediately. There is no power cut-off action on the crane boom.

[0190] When the risk level is 3 (Risk=3), the pedestrian's predicted location in the next 0.5 seconds is within the core area (inside the buffer zone), and the pedestrian's movement direction is towards the inner part of the fence. At this point, the pedestrian had not only entered the high-risk core area, but was also approaching the even higher-risk crane operation center, which was judged to be extremely high risk.

[0191] Triggering Level 3 Response (Extremely High Risk Handling):

[0192] Hardware action: Cut off the boom luffing / lifting power (connected to the crane control system via a relay), forcibly stopping the boom movement.

[0193] Warning action: Simultaneously activate red flashing light + 115dB buzzer (continuing the strong warning of the level 2 warning).

[0194] Remote notification: Push alarm information (including real-time pedestrian location and crane status screenshots) to the administrator's mobile APP to facilitate remote monitoring and emergency response.

[0195] As a refinement of the above embodiment, step S3 uses a loss function that incorporates temporal errors to train the personnel intrusion detection model for truck cranes. The training process loss function... as follows:

[0196] ,

[0197] ,

[0198] in, For 2D bounding box loss, For the 3D bounding box intersection-union ratio loss of LiDAR, For trajectory prediction error, The actual location of the pedestrian. , , These are the weighting coefficients.

[0199] The weighting coefficients sum to 1, where: A value of 0.4-0.6 is used to balance the basic positioning accuracy of the two-dimensional bounding box; The value is 0.2-0.4, used to enhance the anti-environmental interference capability of the LiDAR 3D frame; A value of 0.1-0.3 is used to optimize the temporal accuracy of pedestrian trajectory prediction; the default value is recommended. =0.5、 =0.3、 =0.2, which can be finely adjusted according to the environmental complexity of crane operation scenarios (such as urban construction / port hoisting) to ensure that the hybrid loss function adapts to the core requirement of early warning of personnel intrusion under complex working conditions.

[0200] The core output of the boom prediction module is a dynamic 3D fence. By predicting the boom's trajectory 1 second from the crane's own parameters, a dynamic 3D fence (rather than a fixed fence) is generated that is synchronized with the boom's movement. Essentially, it defines the boundary of the hazardous area where the crane is operating, solving the problem of lag in existing static fence technology.

[0201] The core output of the feature extraction module is accurate pedestrian features and location information. By extracting enhanced pedestrian features through multimodal fusion, it ensures accurate pedestrian identification under complex conditions (rain, fog, occlusion) and obtains basic information such as the pedestrian's current position and speed, providing reliable input for the pedestrian location prediction module.

[0202] Example 2

[0203] This example system deployment scenario involves a 25-ton truck crane, implementing intrusion detection in a construction site. The system hardware deployment is as follows:

[0204] (1) Multimodal sensing layer:

[0205] The binocular camera, model ZED2 (120mm baseline, 2208×1242 resolution, 30fps frame rate), is mounted on the right front side of the crane turntable, 2.5m above the ground, with the lens facing the boom working area (horizontal field of view coverage of 120°).

[0206] The lidar is a 16-line lidar (model: RoboSenseRS-LiDAR-16, horizontal FOV 120°, point cloud density 200 points / m2, ranging range 0.5-100m), installed on the left side of the front of the turntable (0.8m away from the camera) to ensure that the field of view overlaps with the camera by ≥80%.

[0207] The auxiliary sensor, a tilt sensor (model: SCA100T, accuracy ±0.1°), is installed on the turntable horizontal platform to collect ground slope data; the CAN bus interface connects to the crane ECU to obtain boom parameters (length L, elevation angle) in real time. angular velocity ω, load wait).

[0208] (2) Data processing layer:

[0209] It uses NVIDIA Jetson AGX Orin (GPU: 32 TOPS computing power) + Xilinx K7 FPGA, deployed in the crane cab control cabinet, responsible for data fusion and model inference (processing latency ≤30ms).

[0210] (3) Decision-making and execution level:

[0211] The dynamic fence engine, integrated into the FPGA, receives CAN bus data every 50ms and updates the fence parameters.

[0212] The alarm system consists of three levels: Level 1 (yellow LED warning light + voice module "Beware of people around") installed at the base of the boom; Level 2 (red flashing light + 115dB buzzer) installed on the turntable; and Level 3 (relay output to cut off boom luffing / lifting power) connected to the crane control system. Remote alarms are pushed to the administrator's mobile app via a 4G module.

[0213] Method implementation steps:

[0214] Operation by boom rotation (angular velocity) Taking rainy / foggy conditions (visibility 50m) as an example, the specific steps are as follows:

[0215] Step 1: Multimodal Data Acquisition and Spatiotemporal Registration

[0216] The binocular camera acquires RGB images (including areas blurred by rain and fog) and parallax maps, while the lidar simultaneously acquires 3D point clouds (outputting clear point clouds through rain and fog).

[0217] Time synchronization: The timestamps of the camera and the LiDAR are aligned via GPIO trigger signals (error ≤ 1ms).

[0218] Spatial registration: Hand-eye calibration was performed using a checkerboard calibration board (12×10 grids, 50mm grid spacing) to obtain external parameters.

[0219] Rotation matrix Translation vector (Unit: m).

[0220] Transform the LiDAR point cloud to the camera coordinate system using the coordinate transformation formula: .

[0221] Step 2: Data Preprocessing and Feature Fusion

[0222] Point cloud denoising: 8cm×8cm×8cm voxel filtering is used to remove noise points caused by rain and fog (while preserving the outline of the human body point cloud).

[0223] Image enhancement: Adaptive histogram equalization is performed on the binocular images to improve contrast in rain and fog areas.

[0224] Feature extraction and fusion: visual features By improving the CSPDarknet+ coordinate attention module (CAM) extraction, the human contour features in the image are enhanced (to suppress background rain and fog interference); point cloud features are also improved. Voxel features were extracted using VoxelTransformer (human point cloud density ≈ 50 points / m², normal vector perpendicular to the ground); spatial attention mask was calculated. =0.3 (reduced visual weight in rain and fog areas), temporal attention weight =0.8 (significant pedestrian motion characteristics), fused features .

[0225] Step 3: Dynamic 3D Fence Generation

[0226] Crane boom parameters: Current , , , (Rated load) ), ground slope .

[0227] Crane trajectory prediction: Improved LSTM for predicting tip position in the next 1 second Calculate the predicted linear velocity acceleration .

[0228] Fence parameters: Radial radius:

[0229] ,

[0230] Height threshold:

[0231] ,

[0232] The fence is a cylindrical area with a radius of 6.03m and a height of 0.5-2.0m (the core area is...). The buffer zone is 5.03-6.03m long.

[0233] Step 4: Pedestrian Trajectory Prediction and Risk Assessment

[0234] Pedestrian Detection: Feature Fusion The coordinates of one pedestrian were identified. ,speed (Towards the boom).

[0235] Trajectory prediction: Kalman filter predicts position after 0.5s .

[0236] Risk assessment: Falling into the core area (5.1m≤5.03m) and =1.2×0.98>0 (moving inward), Risk level: 2.

[0237] Step 5: Hierarchical Response and Model Optimization

[0238] Triggering a Level 3 alarm: Power to the boom is cut off, a red flashing light and buzzer are activated, and an alarm message (including real-time location and screenshot) is pushed to the mobile app.

[0239] Model optimization: Record the detection data (pedestrians are correctly identified) in the rain and fog scene, and improve the robustness to similar scenes through iterative training using a hybrid loss function.

[0240] Example 3

[0241] This disclosure also provides a personnel intrusion monitoring device for a truck crane, including a processor and a memory. Optionally, the device may further include a communication interface and a bus. The processor, communication interface, and memory can communicate with each other via the bus. The communication interface can be used for information transmission. The processor can call logical instructions in the memory to execute the personnel intrusion monitoring method for a truck crane described in the above embodiments.

[0242] Furthermore, the logical instructions in the aforementioned memory can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium.

[0243] Memory, as a computer-readable storage medium, can be used to store software programs and computer-executable programs, such as the program instructions / modules corresponding to the methods in the embodiments of this disclosure. The processor executes the program instructions / modules stored in the memory to perform functional applications and data processing, thereby implementing the personnel intrusion monitoring method for truck cranes described above.

[0244] The memory may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the terminal device. Furthermore, the memory may include high-speed random access memory and may also include non-volatile memory.

[0245] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method of personnel intrusion monitoring for a mobile crane, characterized by, The method comprises the following steps: Synchronously collecting RGB images, disparity maps and three-dimensional point cloud data containing pedestrian information, and establishing a training set after time-space registration and preprocessing; The personnel intrusion monitoring model of the truck crane comprises a feature extraction module, a boom prediction module, a pedestrian position prediction module and a time-space registration module; the feature extraction module extracts visual features from the RGB images and the disparity maps, and extracts point cloud features from the three-dimensional point cloud data, and introduces a time and space attention mechanism to fuse the visual features and the point cloud features; the boom prediction module predicts the position of the tip of the boom based on an improved LSTM network, and generates a three-dimensional fence; the pedestrian position prediction module predicts the position of the pedestrian by fusing the visual features and the point cloud features through Kalman filtering combined with a Transformer; the time-space registration module matches the predicted trajectory of the pedestrian with the dynamic three-dimensional fence in time and space, and determines the intrusion risk; The personnel intrusion monitoring model of the truck crane is trained using the training set data, and a trained personnel intrusion monitoring model of the truck crane is obtained; The RGB images, the disparity maps and the three-dimensional point cloud data of the construction site are collected, and the intrusion risk is determined by inputting the data into the trained personnel intrusion monitoring model of the truck crane according to the relationship between the position, the moving direction of the pedestrian and the fence. The boom prediction module comprises a boom tip position prediction module and a three-dimensional fence calculation module; the boom tip position prediction module predicts the position of the tip of the boom based on an improved LSTM network; the improved LSTM network adds an attention mechanism, and the prediction formula is as follows: ， wherein, is the momentary boom tip predicted coordinate, is the current time stamp, is the predicted time stamp, is the improved LSTM network, is the LSTM hidden state, is the momentary boom tip actual coordinate, is the momentary boom elevation angle, is the momentary boom slew rate, is the momentary boom luffing speed; The three-dimensional fence calculation module comprises: Stereo fence radial radius The calculation formula is as follows: ， ， wherein, is the basic danger radius of the crane, is the current boom length of the crane, is the actual load of the crane, is the rated load of the crane, is the boom angle of the crane, is the predicted tip line speed of the boom, is the slope correction factor, is the ground slope angle. Stereographic fence height threshold The calculation formula is as follows: ， wherein, is the crane trolley reference height, is the jib tip predicted acceleration; A cylindrical three-dimensional fence is established with the predicted coordinates of the tip of the boom as the center point, according to the radial radius and the height threshold.

2. The method of personnel intrusion monitoring of a lorry loader according to claim 1, characterized in that, The time-space registration of the RGB images, the disparity maps and the three-dimensional point cloud data realizes time synchronization through hardware triggering, and realizes space registration by converting the laser radar point cloud to the camera coordinate system through hand-eye calibration to obtain the sensor external parameter.

3. The method of monitoring personnel intrusion into a vehicle crane according to claim 1, characterized in that, The feature extraction module extracts visual features from the RGB images and the disparity maps through an improved CSPDarknet network, and extracts point cloud features from the three-dimensional point cloud data through Kalman filtering combined with a Transformer; The improved CSPDarknet network introduces a coordinate attention module to strengthen the features of the human body region, and outputs visual features by processing the output features of the CSPDarknet network through the coordinate attention module; The Kalman filtering combined with the Transformer divides the point cloud into voxels, and extracts point cloud features containing voxel density and motion vector; The fusion formula of the visual features and the point cloud features is as follows: ， ， ， wherein, is a cross-modal fusion feature, is a current time stamp, is a spatial attention mask, is a visual feature, is a 3x3 convolution operation, is a max pooling, is an activation function, is a temporal attention weight, is a point cloud feature, wherein is a neighboring frame feature difference, is a natural constant.

4. The method of monitoring personnel intrusion into a vehicle crane according to claim 1, characterized in that, The boom tip position prediction module comprises an input layer, an attention weight calculation module, an improved LSTM unit and a prediction layer; The input layer input parameter set: , , , and normalizing the parameter set to ; The attention weight calculation module calculates the weighted parameters to obtain the weighted boom parameter features; and the improved LSTM unit replaces the original input of the LSTM with the weighted parameters by using the gating mechanism of the LSTM to capture the long-term dependence of the boom parameters. transforming the normalized parameter set to a query, a key, and a value; the query is the previous time step LSTM hidden state, the key is transformed by a weight matrix , and the value is transformed by a weight matrix ; is a weight matrix that maps the normalized crane parameter set to a key, is a weight matrix that maps the normalized crane parameter set to a value; The attention score is calculated The attention weight is obtained by weight normalization through a Softmax function, and the weighted jib parameter feature is obtained by weighting and fusing the parameter set using the attention weight. The calculation formula is as follows: ， wherein, is a matrix multiplication, is a transpose, is a scaling factor, is a hidden layer dimension, is a timestamp of the current time instant; The prediction formula for predicting the position of the pedestrian is as follows: The predicted layer maps the LSTM hidden state to the predicted coordinates of the jib tip at time t+1; the hidden state of the attention mechanism in the improved LSTM unit The update formula is: ， wherein, is an output gate, is an updated memory cell, and is a scaled dot-product attention function, is a previous time step LSTM hidden state, is is a crane tip actual coordinate at a time step, is is a crane boom elevation angle at a time step, is is a crane boom slew rate at a time step, is is a crane boom luffing speed at a time step.

5. The method of monitoring personnel intrusion into a vehicle crane according to claim 1, characterized in that, ​ ， wherein, is the pedestrian instantaneous coordinate, , is the pedestrian instantaneous coordinate, is the pedestrian instantaneous velocity, is the pedestrian instantaneous acceleration, is the Kalman filter operator.

6. The method of personnel intrusion monitoring for a vehicle crane according to claim 5, characterized in that, The spatio-temporal registration module spatio-temporally matches the pedestrian predicted trajectory with the dynamic three-dimensional fence, and determines the intrusion risk in the following specific manner: ， wherein, is the risk level, is is the dynamic three-dimensional fence at the moment, is is the dynamic three-dimensional fence buffer zone at the moment, the buffer zone is a 1m range inside the fence, is is the dynamic three-dimensional fence core area at the moment, the core area is the area inside the buffer zone, is the outward normal vector of the fence boundary, represents that the pedestrian moves to the inside of the fence, represents that the pedestrian does not move to the inside.

7. The method of personnel intrusion monitoring of a vehicle crane according to any of claims 1 - 6, characterized in that, The automobile crane personnel intrusion monitoring model training process loss function As follows: ， ， wherein, is a two-dimensional bounding box loss, is a laser radar 3D box intersection over union loss, is a trajectory prediction error, is a pedestrian actual position, , , is a weight coefficient.

8. A personnel intrusion monitoring device for a mobile crane, comprising a processor and a memory having stored program instructions, characterized in that, The processor is configured to, when running the program instructions, perform the personnel intrusion monitoring method of the truck crane as claimed in any one of claims 1-7.

Images