A crane anti-collision detection method, system, device and medium

Abstract

The present invention discloses a method, system, device, and medium for anti-collision detection of a crane, relating to the technical field of computer platform load balancing. The method comprises collecting multimodal data of the crane and preprocessing the multimodal data; fusing the collected multimodal data through Kalman filtering to construct an LSTM-GRU prediction model; inputting the fused data into the LSTM-GRU prediction model to predict the crane's operating trajectory; and dynamically generating anti-collision control instructions based on the trajectory prediction results through an AI edge computing device, which the PLC receives and executes. The method of the present invention improves the accuracy of crane operating status perception based on multimodal data fusion, achieves accurate future trajectory calculation based on LSTM-GRU trajectory prediction, and implements dynamic anti-collision control based on the AI ​​edge computing device, effectively enhancing the intelligent level of anti-collision detection.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent control and safety detection of lifting machinery, and in particular to a method, system, equipment and medium for anti-collision detection of lifting machinery. Background Art

[0002] As key equipment in industrial production, logistics, transportation, and construction, cranes are widely used in ports, warehouses, construction sites, and large-scale manufacturing enterprises. However, due to their complex operating environments, large load variations, and nonlinear trajectory, cranes are prone to collisions, resulting in equipment damage, casualties, and economic losses. Therefore, implementing intelligent anti-collision detection for cranes to ensure safe operation has become a key technical issue that needs to be addressed in the industry.

[0003] Crane collision avoidance methods in the industry primarily include fixed sensor detection, rule-based collision warning, and manual monitoring. However, these methods suffer from limited sensor detection ranges and are susceptible to obstructions, resulting in blind spots. Sensors also have poor adaptability to ambient lighting, rain, fog, and dust, affecting detection accuracy. They can only passively sense the current state and cannot predict trajectories in advance, hindering collision avoidance during high-speed movement. Relying on fixed rules, they are unable to adapt to dynamic changes in varying operating conditions. They are also unable to predict future trajectories in advance, failing to effectively avoid the risk of sudden collisions during high-speed operation. False alarms can be a problem, and when cranes require precise operation in confined areas, they can trigger frequent false alarms, impacting operational efficiency. Relying on manual experience can lead to misjudgments and delayed responses. Operators experience high workloads and fatigue over extended periods, posing safety risks. Cranes cannot be precisely controlled in complex environments, especially when multiple cranes are working together, making collisions prone to misoperation. Summary of the Invention

[0004] In view of the above-mentioned problems, the present invention is proposed.

[0005] Therefore, the technical problem solved by the present invention is: the existing anti-collision detection method for lifting machinery has insufficient perception accuracy, weak trajectory prediction ability, unintelligent anti-collision strategy, and low real-time performance, as well as the problem of how to achieve high-precision trajectory prediction and dynamic anti-collision control based on multimodal data fusion and artificial intelligence technology.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions: a method for detecting anti-collision of a lifting machinery, comprising collecting multimodal data of the lifting machinery and preprocessing the multimodal data.

[0007] The collected multimodal data are fused through Kalman filtering to build an LSTM-GRU prediction model. The fused data is input into the LSTM-GRU prediction model to predict the operation trajectory of the lifting machinery.

[0008] Based on the trajectory prediction results, anti-collision control instructions are dynamically generated by the AI ​​edge computing device, and the PLC receives and executes the anti-collision control instructions.

[0009] Multimodal data fusion includes data fusion through Kalman filtering to eliminate measurement errors and improve state estimation accuracy.

[0010] Building an LSTM-GRU prediction model involves using LSTM to capture long-term trajectory trends and combining it with GRU to process short-term trajectory fluctuations, thereby achieving short-term and long-term trajectory prediction fusion.

[0011] Based on time series prediction, the crane's trajectory for the next N time steps is calculated in advance to determine potential collision risks.

[0012] The trajectory smoothing optimization method is used to optimize the trajectory calculation.

[0013] As a preferred solution of the anti-collision detection method for lifting machinery described in the present invention, the multimodal data of the lifting machinery is synchronously collected through AI cameras, lidars, absolute encoders, and inertial sensors, and the collected multimodal data of the lifting machinery is used as observation values ​​to construct a time series data set of the lifting machinery operation.

[0014] Multimodal data includes object category, location coordinates, reflection intensity, target distance, hook height, running speed, angular velocity, and acceleration.

[0015] As a preferred solution of the lifting machinery anti-collision detection method described in the present invention, the sensor data is aligned according to the highest frequency, and the low-frequency sensor data is filled by linear interpolation.

[0016] The aligned data is denoised based on particle filtering, and the denoised sensor data is used for outlier detection using the mean-standard deviation method. The abnormal parts are removed and filled again using linear interpolation. The multimodal data preprocessing process is repeated until no outlier data is detected in the outlier detection process.

[0017] Particle filter denoising involves simulating system states through multiple particles, dynamically estimating true signal values, and eliminating measurement noise.

[0018] Outlier detection involves calculating the mean of all data points, measuring the central tendency of the data, and using the standard deviation Measure the degree of dispersion of the data, determine the deviation of the data points from the mean based on the degree of dispersion, and identify data with excessive deviation as outliers.

[0019] As a preferred solution of the anti-collision detection method for lifting machinery described in the present invention, the pre-processed multimodal sensor data is fused based on Kalman filtering, the position, speed and acceleration information of the lifting machinery is set through the pre-processed multimodal data, the set position, speed and acceleration information of the lifting machinery is used as input data, and the operating state of the lifting machinery is predicted according to Newton's kinematic equations.

[0020] The state transition model is used to input the state information of the previous moment, combined with the preprocessed acceleration, to calculate the position, velocity and acceleration of the crane at the next moment and define the motion law.

[0021] The Kalman gain is calculated and the predicted state is corrected using the measurement data according to the Kalman gain to obtain the fused position information, velocity information and acceleration information.

[0022] Calculating the Kalman gain includes inputting a Kalman gain calculation formula by using a measurement matrix, a measurement noise covariance, and a state estimation covariance matrix to obtain the Kalman gain.

[0023] The covariance matrix of the state estimate measures the uncertainty of the predicted value and dynamically adjusts the confidence level of the prediction.

[0024] The measurement matrix maps the state variables to the measurement space and calculates the correction values.

[0025] The measurement noise covariance matrix is ​​set according to the sensor's own error characteristics to reflect the uncertainty of the measurement data and optimize the accuracy of state correction.

[0026] As a preferred solution of the lifting machinery anti-collision detection method described in the present invention, the fused position information, velocity information, and acceleration information data are used as input, the input position information, velocity information, and acceleration information data are normalized, and the normalized position information, velocity information, and acceleration information data form a time series.

[0027] The traditional LSTM prediction model is introduced into GRU calculation to construct GRU layer and fully connected layer.

[0028] The time series is used as input, the GRU layer adapts to the real-time motion changes of the crane, the LSTM layer outputs the overall trajectory trend of the crane, and the output results of the GRU layer and LSTM layer are combined to perform trajectory prediction calculation and output the predicted trajectory points.

[0029] The historical position, velocity, and acceleration data are input into the prediction model. Based on the mean square error calculation, a trajectory smoothing penalty term is introduced to construct a mean square error loss function. The historical trajectory prediction results are input into the mean square error loss function to calculate the error between the historical predicted trajectory and the actual trajectory.

[0030] The fusion result of multimodal data is input into the LSTM-GRU prediction model to output the operation trajectory of the lifting machinery.

[0031] As a preferred solution of the crane anti-collision detection method described in the present invention, the trajectory data output by the prediction model and the position information of the obstacle are used as input to calculate the minimum safe distance between the crane and the obstacle, and the safety distance threshold is set according to the minimum safe distance.

[0032] According to the set safety distance threshold, a control strategy is generated and converted into control instruction data.

[0033] Setting the safety distance threshold includes issuing an early warning signal when the distance between the crane and the obstacle is less than the warning distance.

[0034] When the distance between the crane and the obstacle is less than the braking distance, the braking operation is performed.

[0035] Generating a control strategy includes setting a control state according to a set safety distance threshold. When the distance between the crane and the obstacle is greater than the warning distance, the crane operates normally.

[0036] When the distance between the crane and the obstacle is between the braking distance and the warning distance, an audible and visual alarm will be issued.

[0037] When the distance between the crane and the obstacle is equal to the braking distance, a deceleration command is sent to the PLC to reduce the crane's operating speed.

[0038] When the distance between the crane and the obstacle is less than the braking distance, a braking command is sent to the PLC to stop the crane from working.

[0039] As a preferred solution of the crane anti-collision detection method described in the present invention, the control instruction data is input as input to the PLC control execution unit.

[0040] Under the normal operation instructions of the lifting machinery, the PLC controls the execution unit to maintain the original operating speed, does not trigger the alarm, does not adjust the inverter frequency, and does not trigger the braking system.

[0041] Under the sound and light alarm command, the PLC controls the execution unit to trigger the LED indicator to flash, the buzzer to sound, and the warning message to be displayed on the control panel or remote monitoring system. The crane speed will not be adjusted and the braking system will not be triggered.

[0042] Under the instruction to reduce the operating speed of the crane, the PLC controls the execution unit to adjust the speed, adjust the output frequency of the inverter, maintain the sound and light alarm, and do not trigger the brake system.

[0043] When the crane stops working, the PLC controls the execution unit to cut off the power supply of the crane drive motor, trigger the brake system, perform emergency braking, maintain the sound and light alarm, and send a shutdown report.

[0044] Another object of the present invention is to provide a crane anti-collision detection system, which can solve the problems of current crane anti-collision detection technology, such as insufficient perception accuracy, weak trajectory prediction capability, unintelligent anti-collision control strategy, and low real-time performance, through an AI edge computing solution based on multimodal data fusion and LSTM-GRU trajectory prediction.

[0045] As a preferred solution of the anti-collision detection system for lifting machinery described in the present invention, it includes: a data acquisition and preprocessing module, a data fusion and prediction module, and an instruction generation and execution module. The acquisition and preprocessing module is used to acquire multimodal data of the lifting machinery and preprocess the multimodal data. The data fusion and prediction module is used to fuse the collected multimodal data through Kalman filtering, construct an LSTM-GRU prediction model, and input the fused data into the LSTM-GRU prediction model to predict the operation trajectory of the lifting machinery. The instruction generation and execution module is used to dynamically generate anti-collision control instructions through the AI ​​edge computing device based on the trajectory prediction results, and the PLC receives and executes the anti-collision control instructions.

[0046] A computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement a step of a lifting machinery anti-collision detection method.

[0047] A computer-readable storage medium stores a computer program, which implements the steps of a lifting machinery anti-collision detection method when executed by a processor.

[0048] Beneficial effects of the present invention: The crane anti-collision detection method provided by the present invention is based on multimodal data fusion to improve the accuracy of crane operation status perception, based on LSTM-GRU trajectory prediction to achieve accurate future trajectory calculation, based on AI edge computing equipment to achieve dynamic anti-collision control, effectively improving the intelligence level of anti-collision detection, the present invention has achieved better results in crane operation safety, trajectory prediction accuracy and anti-collision response speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0050] Figure 1 This is an overall flow chart of a crane anti-collision detection method provided by the first embodiment of the present invention. DETAILED DESCRIPTION

[0051] To make the above-mentioned objects, features, and advantages of the present invention more clearly understood, the following detailed description of the specific embodiments of the present invention is given in conjunction with the accompanying drawings. It is obvious that the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in this field without creative work should fall within the scope of protection of the present invention.

[0052] Example 1, with reference to Figure 1 , as one embodiment of the present invention, provides a lifting machinery anti-collision detection method, comprising:

[0053] S1: Collect multimodal data of lifting machinery and preprocess the multimodal data.

[0054] Multimodal data of lifting machinery is collected synchronously through AI cameras, lidar, absolute encoders, and inertial sensors. The collected multimodal data of lifting machinery is used as observation values ​​to construct a time series dataset of lifting machinery operation.

[0055] Multimodal data includes object category, location coordinates, reflection intensity, target distance, hook height, running speed, angular velocity, and acceleration.

[0056] Furthermore, the sensor data is aligned according to the highest frequency, and the low-frequency sensor data is filled using linear interpolation. A preferred solution for linear interpolation filling is:

[0057] ;

[0058] in, represents the interpolated data points, Indicates time The original data at the moment, Indicates time The original data at the moment, Indicates a timestamp.

[0059] The aligned data is denoised based on particle filtering, and the denoised sensor data is used for outlier detection using the mean-standard deviation method. The abnormal parts are removed and filled again using linear interpolation. The multimodal data preprocessing process is repeated until no outlier data is detected in the outlier detection process.

[0060] Particle filtering denoising involves simulating system states through multiple particles, dynamically estimating true signal values, eliminating measurement noise, and improving data stability.

[0061] An optimal solution for particle filtering denoising is:

[0062] ;

[0063] in, represents the optimal state estimate after denoising, represents the number of particles, Indicates the The state of a particle at the previous moment, represents the control input, represents the process noise, represents the state transition equation, Indicates the The weight of a particle.

[0064] Outlier detection involves calculating the mean of all data points, measuring the central tendency of the data, and using the standard deviation Measures the degree of dispersion of data and determines the deviation of data points from the mean based on the degree of dispersion.

[0065] A preferred solution for outlier detection is:

[0066] ;

[0067] ;

[0068] in, represents the mean of the sample data, represents the number of samples, Indicates the data points, represents the standard deviation, Represents the deviation of each data point from the mean.

[0069] S2: The collected multimodal data is fused through Kalman filtering to build an LSTM-GRU prediction model. The fused data is input into the LSTM-GRU prediction model to predict the operation trajectory of the crane.

[0070] The pre-processed multimodal sensor data is fused based on Kalman filtering. The position, velocity and acceleration information of the lifting machinery are set through the pre-processed multimodal data. The set position, velocity and acceleration information of the lifting machinery are used as input data, and the operating status of the lifting machinery is predicted according to Newton's kinematic equations.

[0071] A preferred solution for predicting the operating status of a lifting machinery is:

[0072] ;

[0073] ;

[0074] ;

[0075] ;

[0076] in, Indicates the current time step The state estimation vector, represents the state transition matrix, represents the control input matrix, represents the process noise, Indicates the current time step The control input, represents the time step, 、 、 Indicates the acceleration values ​​on three axes.

[0077] The state transition model is used to input the state information of the previous moment, combined with the preprocessed acceleration, to calculate the position, velocity and acceleration of the crane at the next moment and define the motion law.

[0078] The Kalman gain is calculated and the predicted state is corrected using the measurement data according to the Kalman gain to obtain the fused position information, velocity information and acceleration information.

[0079] A preferred solution for state correction is:

[0080] ;

[0081] ;

[0082] in, represents the Kalman gain matrix, represents the state covariance matrix, represents the measurement matrix, represents the measurement noise covariance matrix, represents the final estimated state vector, represents the measurement value of the current time step, Represents the projection of the predicted state into the measurement space, Represents the Kalman gain Measurement update.

[0083] Calculating the Kalman gain includes inputting a Kalman gain calculation formula by using a measurement matrix, a measurement noise covariance, and a state estimation covariance matrix to obtain the Kalman gain.

[0084] Furthermore, a state transition model is adopted to input the state information of the previous moment and combine it with the preprocessed acceleration to calculate the position, velocity and acceleration of the crane at the next moment, define the motion law, and correct the predicted state according to the measurement data.

[0085] The Kalman gain is calculated by inputting the Kalman gain calculation formula through the measurement matrix, measurement noise covariance, and state estimation covariance matrix.

[0086] The covariance matrix of the state estimate is used to measure the uncertainty of the predicted value and dynamically adjust the confidence level of the prediction.

[0087] The measurement matrix is ​​used to map the state variables into the measurement space to ensure that the calculated correction values ​​are consistent with the data provided by the sensors.

[0088] The measurement noise covariance matrix is ​​set according to the sensor's own error characteristics to reflect the uncertainty of the measurement data and optimize the accuracy of state correction.

[0089] Furthermore, the fused position information, velocity information, and acceleration information data are used as input, and the input position information, velocity information, and acceleration information data are normalized, and the normalized position information, velocity information, and acceleration information data form a time series.

[0090] A preferred solution for composing a time series is:

[0091] ;

[0092] in, Represents the time series data matrix input to LSTM-GRU, Indicates the current time step location information, Indicates the current time step Speed ​​information, Current time step Acceleration information, Indicates the length of the time window.

[0093] The traditional LSTM prediction model is introduced into GRU calculation to construct GRU layer and fully connected layer.

[0094] A preferred solution for calculating the LSTM prediction model is:

[0095] ;

[0096] in, Indicates that LSTM is at time step The hidden state of represents the output gate of LSTM, Represents the memory cell state of LSTM, Indicates that long-term trajectory information is activated by the hyperbolic tangent function, Represents element-wise multiplication.

[0097] A preferred solution for introducing GRU calculation is:

[0098] ;

[0099] in, Indicates that GRU is at time step The hidden state of represents the update gate of GRU, represents the bias term of GRU, represents the input data of the current time step, represents the weight matrix of GRU, Represents the reset gate of GRU.

[0100] The time series is used as input, the GRU layer adapts to the real-time motion changes of the crane, the LSTM layer outputs the overall trajectory trend of the crane, and the output results of the GRU layer and LSTM layer are combined to perform trajectory prediction calculation and output the predicted trajectory points.

[0101] A preferred solution for combining the GRU layer and the LSTM layer is:

[0102] ;

[0103] in, Represents the predicted trajectory data for the next moment, represents the weight matrix of the output layer, represents the LSTM-GRU combination scaling factor, Represents the long-term trajectory features calculated by LSTM, Represents the short-term trajectory features calculated by GRU, Represents the bias term of the output layer.

[0104] An optimal solution for trajectory prediction is:

[0105] ;

[0106] in, Indicates the predicted future The trajectory data of time steps, Represents input-based Predict future trajectories.

[0107] The historical position, velocity, and acceleration data are input into the prediction model. Based on the mean square error calculation, a trajectory smoothing penalty term is introduced to construct a mean square error loss function. The historical trajectory prediction results are input into the mean square error loss function to calculate the error between the historical predicted trajectory and the actual trajectory.

[0108] An optimal solution for constructing the mean square error loss function is:

[0109] ;

[0110] in, represents the total loss function, represents the real future trajectory data, Represents: future trajectory data predicted by LSTM-GRU, represents the number of training data samples, represents the smoothing penalty coefficient, represents the second-order derivative of the predicted trajectory, Indicates the time window length for smoothing calculation.

[0111] S3: Based on the trajectory prediction results, the AI ​​edge computing device dynamically generates anti-collision control instructions, and the PLC receives and executes the anti-collision control instructions.

[0112] The trajectory data output by the prediction model and the location information of the obstacle are used as input to calculate the minimum safe distance between the crane and the obstacle, and the safety distance threshold is set based on the minimum safe distance.

[0113] A preferred method for calculating the minimum safe distance between a lifting machine and an obstacle is:

[0114] ;

[0115] in, Represents the crane and the obstacle at time step The Euclidean distance of 、 、 Denotes the crane at time step The three-dimensional coordinate position of , , Indicates that the obstacle is at time step The three-dimensional coordinate position of .

[0116] According to the set safety distance threshold, a control strategy is generated and converted into control instruction data.

[0117] Furthermore, setting a safety distance threshold includes sending an early warning signal when the distance between the crane and the obstacle is less than a warning distance.

[0118] When the distance between the crane and the obstacle is less than the braking distance, the braking operation is performed.

[0119] Generating a control strategy includes setting a control state according to a set safety distance threshold. When the distance between the crane and the obstacle is greater than the warning distance, the crane operates normally.

[0120] When the distance between the crane and the obstacle is between the braking distance and the warning distance, an audible and visual alarm will be issued.

[0121] When the distance between the crane and the obstacle is equal to the braking distance, a deceleration command is sent to the PLC to reduce the crane's operating speed.

[0122] When the distance between the crane and the obstacle is less than the braking distance, a braking command is sent to the PLC to stop the crane from working.

[0123] Furthermore, the control instruction data is taken as input and input into the PLC control execution unit.

[0124] Under the normal operation instructions of the lifting machinery, the PLC controls the execution unit to maintain the original operating speed, does not trigger the alarm, does not adjust the inverter frequency, and does not trigger the braking system.

[0125] Under the sound and light alarm command, the PLC controls the execution unit to trigger the LED indicator to flash, the buzzer to sound, and the warning message to be displayed on the control panel or remote monitoring system. The crane speed will not be adjusted and the braking system will not be triggered.

[0126] Under the instruction to reduce the operating speed of the crane, the PLC controls the execution unit to adjust the speed, adjust the output frequency of the inverter, maintain the sound and light alarm, and do not trigger the brake system.

[0127] A preferred solution for adjusting the inverter output frequency is:

[0128] ;

[0129] in, Indicates the new inverter output frequency of the crane. Indicates the current inverter output frequency of the crane. Indicates the speed adjustment factor.

[0130] When the crane stops working, the PLC controls the execution unit to cut off the power supply of the crane drive motor, trigger the brake system, perform emergency braking, maintain the sound and light alarm, and send a shutdown report.

[0131] Example 2 is an embodiment of the present invention, which provides a method for detecting anti-collision of a lifting machinery. In order to verify the beneficial effects of the present invention, scientific demonstration is carried out through economic benefit calculation and simulation experiments.

[0132] To verify the effectiveness of our proposed crane collision avoidance detection method based on LSTM-GRU trajectory prediction, this experiment was conducted in an indoor crane operating environment at a large manufacturing plant. The test site included multiple fixed obstacles, moving vehicles, human operators, and multiple cranes operating in parallel. The goal of the experiment was to evaluate the system's collision avoidance detection accuracy, trajectory prediction accuracy, and control response speed under different operating conditions.

[0133] First, the equipment and sensor configuration uses a bridge crane with a maximum lifting load of 10 tons and an operating speed of 1~3m / s.

[0134] The sensor uses an AI camera for target recognition, with a frame rate of 30 FPS. A LiDAR system detects obstacle distances with a scanning frequency of 10 Hz. An absolute encoder records hook position with a resolution of 0.01 m. An inertial unit (IMU) measures angular velocity with an acceleration accuracy of 0.01 m / s².

[0135] The experiment first collected and preprocessed crane operating data. Multimodal data, including crane position, velocity, acceleration, and obstacle distance, was synchronized. Particle filtering was used for denoising, reducing sensor errors and improving data quality. Linear interpolation was used to fill in low-frequency sensor data to ensure data time synchronization. Outlier detection was performed using the mean-standard deviation method, removing anomalous data and reinterpolating the data.

[0136] A Kalman filter is used to fuse the data and estimate the crane's real-time state. This fused data is then fed into the LSTM-GRU trajectory prediction model to predict the crane's trajectory for the next five seconds. Trajectory smoothing optimization is used to ensure a smooth prediction curve and reduce mutation errors.

[0137] The safe distance between the crane and the obstacle was calculated, with a warning threshold of 3m, a deceleration threshold of 2m, and a braking threshold of 1m. When the predicted trajectory approached an obstacle, the AI ​​computing device generated anti-collision control instructions. Key metrics such as trajectory error, false alarm rate, control response time, and collision rate were collected during the experiment. The experimental data is shown in Table 1.

[0138] Table 1 Experimental data table

[0139]

[0140] Example 3 is an embodiment of the present invention, which provides a lifting machinery anti-collision detection system, including a data acquisition and preprocessing module 100, a data fusion and prediction module 200, and an instruction generation and execution module 300.

[0141] S4: The acquisition and preprocessing module 100 is used to acquire multimodal data of the lifting machinery and preprocess the multimodal data.

[0142] It should also be noted that the data acquisition and preprocessing module 100 is responsible for sensing the state of the crane, collecting real-time data from sensors such as AI cameras, lidar, absolute encoders, and inertial sensors, including position, velocity, and acceleration information. To improve data quality, the module performs time alignment, noise removal, outlier detection, and data completion on the collected multimodal data to ensure the reliability and consistency of the input data. The module outputs high-quality, preprocessed sensor data, which is passed to the data fusion and prediction module 200.

[0143] S5: The data fusion and prediction module 200 is used to fuse the collected multimodal data through Kalman filtering, build an LSTM-GRU prediction model, and input the fused data into the LSTM-GRU prediction model to predict the operation trajectory of the lifting machinery.

[0144] It should also be noted that the data fusion and prediction module 200 is responsible for fusing preprocessed multimodal data and predicting the crane's operating trends in advance based on a trajectory prediction algorithm. A Kalman filter is used to fuse sensor data, eliminating measurement errors and constructing a unified state estimation model. An LSTM-GRU prediction model is used to input the fused data into the prediction model. Time series analysis is used to calculate future trajectory points and determine the crane's motion state at the next time step. The module outputs the predicted trajectory data, which is passed to the instruction generation and execution module 300.

[0145] S6: The instruction generation and execution module 300 is used to dynamically generate anti-collision control instructions through the AI ​​edge computing device based on the trajectory prediction results, and the PLC receives and executes the anti-collision control instructions.

[0146] It should also be noted that the command generation and execution module 300 is responsible for making intelligent collision avoidance decisions based on trajectory prediction data and controlling the crane's operating status. The AI ​​edge computing device analyzes the predicted trajectory and calculates the minimum safe distance between the crane and obstacles. Based on the safety threshold, the system dynamically generates collision avoidance control instructions, which are then executed by the PLC.

[0147] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the various embodiments of the method of the present invention. The aforementioned storage medium includes various media that can store program code, such as USB flash drives, mobile hard drives, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical disks.

[0148] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0149] More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk cartridge (magnetic devices), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting, or processing it in another suitable manner as necessary, and then storing it in a computer memory.

[0150] It should be understood that various aspects of the present invention may be implemented using hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods may be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one or a combination of the following technologies known in the art may be used: a discrete logic circuit having logic gates for implementing logic functions on data signals, an application-specific integrated circuit having suitable combinational logic gates, a programmable gate array (PGA), a field-programmable gate array (FPGA), etc. It should be noted that the above embodiments are intended only to illustrate the technical solutions of the present invention and are not intended to be limiting. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art will understand that modifications or equivalent substitutions may be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and such modifications are intended to be encompassed by the claims of the present invention.

Claims

1. A method for detecting anti-collision of a lifting machinery, characterized in that: include: Collect multimodal data of lifting machinery and pre-process the multimodal data; The collected multimodal data is fused through Kalman filtering to build an LSTM-GRU prediction model. The fused data is input into the LSTM-GRU prediction model to predict the operation trajectory of the crane. Based on the trajectory prediction results, the AI ​​edge computing device dynamically generates anti-collision control instructions, which the PLC receives and executes. Multimodal data fusion includes data fusion through Kalman filtering to eliminate measurement errors and improve state estimation accuracy; Building an LSTM-GRU prediction model involves using LSTM to capture long-term trajectory trends and combining it with GRU to process short-term trajectory fluctuations, achieving a fusion of short-term and long-term trajectory predictions. Based on time series prediction, the crane's trajectory is calculated N time steps in the future to determine potential collision risks; Adopt trajectory smoothing optimization method to optimize trajectory calculation; The multimodal data collection of lifting machinery includes: The multimodal data of the crane is collected synchronously through AI cameras, lidar, absolute encoders, and inertial sensors. The collected multimodal data of the crane is used as observation values ​​to construct a time series dataset of the crane operation. Multimodal data includes object category, location coordinates, reflection intensity, target distance, hook height, running speed, angular velocity, and acceleration; The preprocessing of the multimodal data includes: Align the sensor data according to the highest frequency, and use linear interpolation to fill in the low-frequency sensor data; The aligned data is denoised using a particle filter. The denoised sensor data is then used for outlier detection using the mean-standard deviation method. The outliers are removed and filled again using linear interpolation. The multimodal data preprocessing process is repeated until no outlier data is detected in the outlier detection process. Particle filter denoising includes simulating the system state through multiple particles, dynamically estimating the true signal value, and eliminating measurement noise; Outlier detection involves calculating the mean of all data points, measuring the central tendency of the data, and using the standard deviation Measure the degree of dispersion of the data, determine the deviation of the data points from the mean based on the degree of dispersion, and identify data with excessive deviation as outliers; The multimodal data fusion includes: The pre-processed multi-modal sensor data is fused based on Kalman filtering. The position, velocity and acceleration information of the crane are set through the pre-processed multi-modal data. The set position, velocity and acceleration information of the crane are used as input data to predict the operating status of the crane according to Newton's kinematic equations. The specific prediction of the operating status of the lifting machinery is as follows: ； ； ； ； in, Indicates the current time step The state estimation vector, represents the state transition matrix, represents the control input matrix, represents the process noise, Indicates the current time step The control input, represents the time step, 、 、 Indicates the acceleration values ​​on three axes; The state transition model is used to input the state information of the previous moment, combined with the pre-processed acceleration, to calculate the position, velocity and acceleration of the crane at the next moment and define the motion law; Calculate the Kalman gain and use the measurement data to correct the predicted state based on the Kalman gain to obtain the fused position information, velocity information, and acceleration information; Calculating the Kalman gain includes inputting the Kalman gain calculation formula through the measurement matrix, the measurement noise covariance, and the state estimation covariance matrix to obtain the Kalman gain; The covariance matrix of the state estimate measures the uncertainty of the predicted value and dynamically adjusts the confidence level of the prediction; The measurement matrix maps the state variables to the measurement space and calculates the correction values; The measurement noise covariance matrix is ​​set according to the sensor's own error characteristics to reflect the uncertainty of the measurement data and optimize the accuracy of state correction; The construction of the LSTM-GRU prediction model includes: The fused position information, velocity information, and acceleration information data are used as input, the input position information, velocity information, and acceleration information data are normalized, and the normalized position information, velocity information, and acceleration information data are used to form a time series; Introducing the traditional LSTM prediction model into GRU calculation, constructing the GRU layer and the fully connected layer; The time series is used as input, and the GRU layer adapts to the real-time motion changes of the crane. The LSTM layer outputs the overall trajectory trend of the crane. The output results of the GRU layer and LSTM layer are combined to perform trajectory prediction calculation and output the predicted trajectory points. The historical position, velocity, and acceleration data are input into the prediction model. Based on the mean square error calculation, a trajectory smoothing penalty term is introduced to construct a mean square error loss function. The historical trajectory prediction results are input into the mean square error loss function to calculate the error between the historical predicted trajectory and the actual trajectory. The fusion result of multimodal data is input into the LSTM-GRU prediction model to output the operation trajectory of the lifting machinery.

2. The anti-collision detection method for a lifting machinery according to claim 1, wherein: The dynamic generation of anti-collision control instructions by the AI ​​edge computing device includes: The trajectory data output by the prediction model and the location information of the obstacle are used as input to calculate the minimum safe distance between the crane and the obstacle, and the safety distance threshold is set based on the minimum safe distance. Generate a control strategy based on the set safety distance threshold and convert the control strategy into control instruction data; Setting the safety distance threshold includes issuing an early warning signal when the distance between the crane and the obstacle is less than the warning distance; When the distance between the crane and the obstacle is less than the braking distance, the braking operation is performed; Generating a control strategy includes setting the control state according to the set safety distance threshold. When the distance between the crane and the obstacle is greater than the warning distance, the crane operates normally. When the distance between the crane and the obstacle is between the braking distance and the warning distance, an audible and visual alarm will be issued; When the distance between the crane and the obstacle is equal to the braking distance, a deceleration command is sent to the PLC to reduce the crane's operating speed; When the distance between the crane and the obstacle is less than the braking distance, a braking command is sent to the PLC to stop the crane from working.

3. The anti-collision detection method for a lifting machinery according to claim 2, wherein: The PLC receives and executes anti-collision control instructions including: Take the control instruction data as input and input it into the PLC control execution unit; Under normal operation instructions of the lifting machinery, the PLC controls the execution unit to maintain the original operating speed, does not trigger the alarm, does not adjust the inverter frequency, and does not trigger the brake system; Under the sound and light alarm command, the PLC controls the execution unit to trigger the LED indicator to flash, the buzzer to sound, and the warning message to be displayed on the control panel or remote monitoring system, without adjusting the crane speed or triggering the brake system; Reduce the crane's operating speed. The PLC controls the execution unit to adjust the speed and the inverter output frequency, maintaining the sound and light alarms without triggering the brake system. When the crane stops working, the PLC controls the execution unit to cut off the power supply of the crane drive motor, trigger the brake system, perform emergency braking, maintain the sound and light alarm, and send a shutdown report.

4. A system using the lifting machinery anti-collision detection method according to any one of claims 1 to 3, characterized in that: It includes a data acquisition and preprocessing module (100), a data fusion and prediction module (200), and an instruction generation and execution module (300); The acquisition and preprocessing module (100) is used to acquire multimodal data of the lifting machinery and preprocess the multimodal data; The data fusion and prediction module (200) is used to fuse the collected multimodal data through Kalman filtering, build an LSTM-GRU prediction model, and input the fused data into the LSTM-GRU prediction model to predict the operation trajectory of the lifting machinery; The instruction generation and execution module (300) is used to dynamically generate anti-collision control instructions through the AI ​​edge computing device according to the trajectory prediction result, and the PLC receives and executes the anti-collision control instructions.

5. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the lifting machinery anti-collision detection method according to any one of claims 1 to 3 are implemented.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the lifting machinery anti-collision detection method according to any one of claims 1 to 3 are implemented.

Images