Crane operation safety protection method based on personnel trajectory prediction

Abstract

The invention relates to a crane operation safety protection method based on personnel trajectory prediction in the technical field of crane safety control. The crane operation safety protection method comprises the following steps: S1, acquiring personnel position and attitude data of a crane operation area in a multi-view manner; s2, predicting a future movement track of the person by adopting a space-time fusion neural network; s3, dynamically calculating a minimum safety distance and generating a safety boundary according to the crane motion parameters and the predicted trajectory; s4, adjusting the running state of the crane through a hierarchical control strategy, and triggering a manual intervention mechanism when a safety boundary is exceeded; the safety protection method can improve the protection effect and the operation efficiency and reduce the accident rate.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of crane safety control, in particular to a crane operation safety protection method based on personnel trajectory prediction. BACKGROUND

[0002] Traditional crane safety protection relies on manual visual observation or fixed area sensors such as infrared fences, which have the following defects: first, personnel cannot be predicted in real time when they mistakenly enter a dangerous area, and the response is lagging; second, static safety boundaries cannot adapt to dynamic operating environments such as personnel movement and cargo swinging; third, complete reliance on manual intervention leads to low operating efficiency and is prone to fatigue misjudgment. SUMMARY

[0003] In order to overcome the deficiencies in the background art and solve the existing technical problems, the present application discloses a crane operation safety protection method based on personnel trajectory prediction, which can improve the protection effect and operating efficiency and reduce the accident rate.

[0004] To achieve the above purpose, the present application adopts the following technical solutions:

[0005] A crane operation safety protection method based on personnel trajectory prediction, comprising the following steps: S1, multi-view collection of personnel position and posture data in the crane operation area; S2, prediction of personnel future motion trajectory using a space-time fusion neural network; S3, dynamic calculation of the minimum safety distance and generation of the safety boundary according to the crane motion parameters and the predicted trajectory; S4, adjustment of the crane operating state through a hierarchical control strategy, and triggering of the manual intervention mechanism when the safety boundary is exceeded.

[0006] Further, in S1, a plurality of binocular camera arrays are deployed on the crane body and the surrounding area to collect real-time RGB-D data in the operation area.

[0007] Further, in S1, a multi-source data space-time alignment engine in a three-dimensional polar coordinate system is constructed through a heterogeneous sensor matrix comprising a vision module, a radar, and a positioning base station.

[0008] Further, in S2, a space-time graph convolution network ST-GCN is used to analyze personnel skeletal key points, with a historical trajectory sequence as input, and an Attention-LSTM network is used to output a probability distribution trajectory, so as to be able to combine LSTM to predict personnel motion trajectory in the next 5-10 seconds.

[0009] Further, in S3, a four-dimensional space-time joint optimization model (X, Y, Z, Time) is established, the crane kinematic constraints: maximum angular velocity 2.5 rad / s, acceleration 0.3 m / s² are converted into the feasible region boundary of the convex optimization problem; the four-dimensional space-time joint optimization model is established, the variable u includes the control input of the crane, optimization is carried out in space-time, the target is to minimize the distance between the personnel position and the crane position, the change of the control input and the energy consumption term, the given objective function is:

[0010]

[0011] Where, ph(k): the probability distribution of the predicted position of the personnel at time k; pc(k): the position of the crane at time k, determined by the kinematic constraint; ||ph(k)-pc(k)||Q: the distance metric between the personnel and the crane, the weighting coefficient Q is used to control the relative importance of the target; ||Δu(k)||R: the cost of the change of the control input, the weighting coefficient R is used to adjust the input smoothness; Jenergy: the energy consumption optimization term; ρ: the weighting coefficient in multi-objective optimization.

[0012] Further, a hybrid planning strategy is developed, the improved artificial potential field method is used to generate a smooth path in the conventional working condition, and the model predictive control (MPC) is switched to for rolling horizon optimization in the conflict working condition, the planning period is ≤200 ms; the energy consumption-efficiency-safety multi-objective optimization algorithm is embedded, and the potential field parameter weight is adjusted online through the deep reinforcement learning PPO algorithm.

[0013] Further, in S4, the safety level and the control strategy are as follows: when the distance of the personnel from the safety boundary is greater than 3 m, the risk level is represented by green color, safety, and the crane is operated automatically; when the distance of the personnel from the safety boundary is 1-3 m, the risk level is represented by yellow color, early warning, and the crane enters the speed limiting mode; when the distance of the personnel from the safety boundary is less than 1 m, the risk level is represented by red color, danger, and the crane is braked urgently and notifies manual review.

[0014] By adopting the technical scheme as described above, the present application has the following beneficial effects:

[0015] The crane operation safety protection method based on personnel trajectory prediction disclosed in the present application realizes real-time risk quantification and active obstacle avoidance of the crane operation area through the multi-state perception, trajectory prediction and dynamic path planning technology, solves the hysteresis problem of the traditional manual monitoring, significantly improves the safety and automation level of the heavy machinery operation scene, and reduces the accident rate by 80% and improves the operation efficiency by 35% compared with the traditional manual monitoring mode. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 is a dynamic safety boundary plane schematic diagram of the present application;

[0017] Figure 2 is a three-dimensional schematic view of a dynamic security boundary of the present application; DETAILED DESCRIPTION

[0018] The technical solutions of the present application will be described below in conjunction with the accompanying drawings of the embodiments of the present application.

[0019] In conjunction with the accompanying drawings Figures 1-2 The crane operation safety protection method based on personnel trajectory prediction comprises the following steps:

[0020] Step one, multi-view acquisition of personnel position and posture data in the crane operation area, which can be specifically as shown in the accompanying drawings Figure 1 A plurality of binocular camera arrays are arranged on the crane body and the surrounding area, such as six groups of anti-shake zoom cameras are installed on both sides of the crane girder, the trolley and the track, with a resolution of 3840x2160 and a frame rate of 30fps, to collect real-time RGB-D data of the operation area. The RGB-D data refers to data containing both RGB (red, green, blue) images and depth images. The RGB image is usually a standard color image, while D represents depth information, which refers to the distance between each pixel point and the camera. It has been widely used in computer vision, robot perception, AR and VR fields.

[0021] According to the needs, in order to improve the data acquisition effect, a multi-source data space-time alignment engine in a three-dimensional polar coordinate system can be constructed through a heterogeneous sensing matrix containing a vision module, a radar and a positioning base station. Specifically, a heterogeneous sensing matrix can be deployed, including a binocular vision module (1920x1080@60Hz), a 77GHz millimeter wave radar (±0.1m ranging accuracy) and a UWB positioning base station (10cm level accuracy). The FPGA hardware synchronization mechanism is used to realize the timestamp alignment of multi-sensor data (±5ns synchronization accuracy), and the online calibration algorithm is used to compensate the sensor pose offset caused by mechanical vibration.

[0022] Step two, predicting the future trajectory of the personnel using a spatio-temporal fusion neural network; analyzing the personnel's skeletal key points using a spatio-temporal graph convolutional network (ST-GCN), inputting the historical trajectory sequence, and outputting the probability distribution trajectory through an Attention-LSTM network, so as to be able to combine the LSTM to predict the personnel's motion trajectory in the next 5-10 seconds; the ST-GCN is a graph convolutional network capable of processing spatio-temporal data, in this model, the spatial information (such as the personnel's skeletal key points) and the temporal information are represented as a graph structure, the network can process the spatio-temporal information in these graph structures, so as to extract more fine spatial and temporal dynamics, specifically, the ST-GCN will use a graph convolutional network to analyze the personnel's skeletal key points, such as the joint positions of the human body, the data of these key points are usually obtained through sensors or computer vision methods (such as pose estimation), in this way, the model can learn the personnel's motion patterns from the spatio-temporal graph; the LSTM is a neural network very suitable for time series prediction, it can effectively capture long-time span dependencies, and is suitable for time series modeling in trajectory prediction, however, a simple LSTM sometimes cannot pay enough attention to the key parts in the sequence, at this time, the Attention mechanism can play a complementary role, by combining Attention and LSTM, the model can more intelligently decide which moments in the sequence to pay attention to, such as an important turning point or target behavior, so as to improve the accuracy of trajectory prediction.

[0023] According to the needs, the spatio-temporal fusion neural network can also be set as a trajectory prediction model based on the graph attention mechanism (GAT), and the input includes the human body skeletal key point sequence, the environment point cloud data, and the crane kinematic state.

[0024] Or build a hierarchical prediction model, the bottom layer uses the OpenPose algorithm to extract 17 skeletal key points (processing speed ≥ 25 fps), the middle layer fuses the construction scene semantic segmentation information (Mask R-CNN implementation) through the spatio-temporal attention mechanism, and the top layer deploys the spatio-temporal causal convolutional network (TCN) for multi-modal trajectory prediction; then introduce an uncertainty quantification module, realize trajectory prediction confidence evaluation (output probability distribution instead of single trajectory) through Monte Carlo Dropout, and support risk-sensitive decision-making.

[0025] Step three, dynamically calculating the minimum safety distance and generating a safety boundary according to the crane motion parameters and the predicted trajectory;

[0026] According to the needs, a four-dimensional space-time joint optimization model (X, Y, Z, Time) is established, and the crane kinematic constraints: maximum angular velocity 2.5 rad / s, acceleration 0.3 m / s² are converted into the feasible region boundary of the convex optimization problem; the four-dimensional space-time joint optimization model is established, the variable u includes the control input of the crane, and optimization is performed in space-time, the goal is to minimize the distance between the personnel position and the crane position, the change of the control input and the energy consumption term, and the given objective function is:

[0027]

[0028] wherein, ph(k): the predicted position of the personnel at time k, expressed as a probability distribution, considering the uncertainty and random behavior of the personnel; pc(k): the position of the crane at time k, constrained by the kinematic constraints (maximum angular velocity 2.5 rad / s, maximum acceleration 0.3 m / s²) of the crane, and constrained by the motion model; ||ph(k)-pc(k)||Q: distance measurement between the personnel and the crane, the weighting coefficient Q is used to control the relative importance of the target; ||Δu(k)||R: cost of control input change, the weighting coefficient R is used to adjust the input smoothness; Jenergy: energy consumption optimization term; ρ: weighting coefficient in multi-objective optimization;

[0029] Since the motion of the crane is limited by the maximum angular velocity and acceleration, the motion of the crane can be constrained by the kinematic model, considering the nonlinear characteristics of the crane motion, these kinematic constraints are converted into the feasible region boundary in the convex optimization problem, through mathematical modeling, the kinematic constraints can be converted into constraint conditions, so that the path planning process conforms to the physical constraints in reality.

[0030] According to the needs, a hybrid planning strategy is developed, the improved artificial potential field method is used to generate a smooth path in normal working conditions, and the model predictive control (MPC) is switched to in conflict conditions for rolling horizon optimization, the planning period is ≤200 ms; the energy consumption-efficiency-safety multi-objective optimization algorithm is embedded, and the potential field parameter weight is adjusted online through deep reinforcement learning (PPO) algorithm;

[0031] The model predictive control (MPC) algorithm is used to optimize the safety boundary, and the objective function is:

[0032] wherein, Jtarget function, representing the total optimization cost; Ntotal number of time steps; phuman(k) predicted personnel position at time k; pc(k) crane expected position at time k; u(k) control amount, control input at time k; ||·||Euclidean norm, representing the distance or the size of the control amount; λweighting factor, used to adjust the importance of the position error term and the control input term.

[0033] Step four, adjust the crane operating state through the hierarchical control strategy, trigger the manual intervention mechanism when exceeding the safety boundary, send the hierarchical braking instruction to the crane PLC through the industrial bus (such as CANopen), and trigger the automatic emergency stop in the emergency state; Specifically, the safety level and control strategy are: when the distance between the personnel and the safety boundary is greater than 3m, the risk level is represented by green, which is safe, and the crane is operated automatically; When the distance between the personnel and the safety boundary is 1-3m, the risk level is represented by yellow, which is a warning, and the crane enters the speed limiting mode; When the distance between the personnel and the safety boundary is less than 1m, the risk level is represented by red, which is dangerous, the crane is braked urgently and the manual review is notified; A three-level fault-tolerant mechanism is constructed: first-level warning (audible and visual alarm + speed limitation), second-level pre-braking (reverse torque application), and third-level emergency braking (hydraulic brake + energy feedback).

[0034] The parts of the application not described in detail are prior art, and it is obvious to those skilled in the art that the application is not limited to the details of the above exemplary embodiments, and the application can be implemented in other specific forms without departing from the spirit or essential characteristics of the application; Therefore, from any point of view, the above examples should be regarded as exemplary and non-limiting, the scope of the application is defined by the appended claims rather than the above description, therefore all changes falling within the meaning and scope of the equivalent elements of the claims are intended to be included in the application, and any reference signs in the claims should not be regarded as limiting the content of the involved claims.

Claims

1. A crane operation safety protection method based on personnel trajectory prediction, characterized by: Includes the following steps: S1. Collect personnel position and posture data in the crane operation area from multiple perspectives; S2. Use a spatiotemporal fusion neural network to predict the future movement trajectory of people; S3. Based on the crane's motion parameters and predicted trajectory, dynamically calculate the minimum safe distance and generate a safety boundary; S4. Adjust the crane's operating status through a hierarchical control strategy, and trigger a manual intervention mechanism when the operating status exceeds the safety boundary.

2. The crane operation safety protection method based on personnel trajectory prediction according to claim 1, characterized in that: In S1, multiple binocular camera arrays are deployed on and around the crane body to collect RGB-D data of the work area in real time.

3. The crane operation safety protection method based on personnel trajectory prediction according to claim 1, characterized in that: in In S1, a multi-source data spatiotemporal alignment engine is constructed in a three-dimensional polar coordinate system through a heterogeneous sensing matrix that includes vision modules, radar, and positioning base stations.

4. The crane operation safety protection method based on personnel trajectory prediction according to claim 1, characterized in that: In S2, the spatiotemporal graph convolutional network ST-GCN is used to analyze the key points of the human skeleton. With the historical trajectory sequence as input, the probability distribution trajectory is output through the Attention-LSTM network, which can then be combined with LSTM to predict the human movement trajectory in the next 5 to 10 seconds.

5. The crane operation safety protection method based on personnel trajectory prediction according to claim 1, characterized in that: In S3, a four-dimensional spatiotemporal joint optimization model (X,Y,Z,Time) is established. The kinematic constraints of the crane—maximum angular velocity 2.5 rad / s and acceleration 0.3 m / s²—are transformed into the feasible region boundary of a convex optimization problem. The established four-dimensional spatiotemporal joint optimization model, with variable u including the crane's control input, optimizes in spatiotemporal space. The objective is to minimize the distance between the personnel position and the crane position, the change in control input, and the energy consumption term. The given objective function is: Where, ph(k): the probability distribution of the predicted position of the personnel at time k; pc(k): the position of the crane at time k, determined by kinematic constraints; ∥ph(k)-pc(k)∥Q: the distance metric between the personnel and the crane, with weighting coefficient Q used to control the relative importance of the objectives; ∥Δu(k)∥R: the cost of controlling input changes, with weighting coefficient R used to adjust input smoothness; Jenergy: the energy consumption optimization term; ρ: the weighting coefficient in multi-objective optimization.

6. The crane operation safety protection method based on personnel trajectory prediction according to claim 5, characterized in that: A hybrid planning strategy was developed. Under normal operating conditions, an improved artificial potential field method was used to generate a smooth path. Under conflict conditions, the model predictive control (MPC) was switched to perform rolling time-domain optimization with a planning cycle of ≤200ms. An energy consumption-efficiency-safety multi-objective optimization algorithm was embedded, and the potential field parameter weights were adjusted online through the deep reinforcement learning PPO algorithm.

7. The crane operation safety protection method based on personnel trajectory prediction according to claim 1, characterized in that: in In S4, the safety level and control strategy are as follows: when the distance between personnel and the safety boundary is greater than 3m, the risk level is represented by green, indicating safety, and the crane operates in full automatic mode; when the distance between personnel and the safety boundary is 1-3m, the risk level is represented by yellow, indicating a warning, and the crane enters speed-limited mode; when the distance between personnel and the safety boundary is less than 1m, the risk level is represented by red, indicating danger, and the crane brakes urgently and notifies a human to verify.

Images