Traveling crane hoisting safety detection method and system based on multi-sensor calibration

Abstract

The invention relates to the technical field of sensors, in particular to a travelling crane hoisting safety detection method and system based on multi-sensor calibration, a camera and a radar serve as a multi-sensor unit, timestamp synchronization is carried out on data collected in real time, data association is carried out on detection information of continuous frames, and the detection result is obtained. The method comprises the following steps of: keeping correlation consistency of front and back frames of a detected object, clustering, filtering and optimizing steel coil point clouds, calculating the length, width and terrain clearance of a steel coil, calculating the outer contour of the point clouds, realizing fusion of a plurality of multi-sensor units, splicing and fusing the generated point clouds, and converting the point clouds into a unified coordinate system. According to images and point cloud information collected by the visual radar sensors at the two ends of the travelling crane, the positions of the pedestrian, the suspension arm and the steel coil can be accurately captured and calculated, and the relevant parameters of the pedestrian, the suspension arm and the steel coil are calculated. And when the pedestrian moves into a dangerous range of the hung steel coil, the alarm on the driving control board card automatically gives out a ringing alarm.

Description

Technical Field

[0001] The present invention relates to the field of sensor technology, and in particular to a method and system for detecting the safety of crane-mounted objects based on multi-sensor calibration. Background Art

[0002] In the scenario where a vehicle is moving with a suspended object, attention should be paid to the safety of people near the suspended object. At this time, it is necessary to calculate whether the person is within the alarm range.

[0003] By fusing the data of multiple sensors, we can combine the advantages of different sensors and play the role of their strengths and avoid their weaknesses. Multi-sensor calibration is the basis of data fusion, which is to unify the camera pixel coordinate system and the lidar coordinate system.

[0004] Existing positioning systems based on multi-sensor fusion may be disturbed in complex environments, resulting in reduced positioning accuracy and stability. Delays may occur when processing large amounts of sensor data and synchronizing data, leading to safety issues for people near the load when the vehicle is carrying the load. Summary of the invention

[0005] (I) Purpose of the invention

[0006] In order to solve the technical problems existing in the background technology, the present invention proposes a method and system for safety detection of overhead cranes based on multi-sensor calibration, which can effectively improve the safety of people near the overhead cranes in the scenario where the overhead cranes are moving.

[0007] (II) Technical solution

[0008] In order to solve the above problems, the first aspect of the present invention provides a method for detecting safety of crane-mounted objects based on multi-sensor calibration, comprising:

[0009] Step S01, calibrating the transformation matrix of the multi-sensor and the multi-sensor unit, completing the fusion and splicing of the point cloud after three-dimensional reconstruction to achieve coordinate unification;

[0010] Step S02, extracting the plane equation parameters of the maximum point cloud plane, calculating the transformation matrix from the point cloud to the specified plane, and saving it as a priori parameters;

[0011] Step S03, inputting the camera and lidar point cloud data soft-synchronized by the algorithm, back-projecting the radar point cloud to the pixel plane through the calibrated camera radar extrinsic matrix and up-sampling to obtain a dense depth map;

[0012] Step S04, extracting semantic information of the personnel, the boom and the steel coil using the acquired depth map, and tracking the semantic information;

[0013] Step S05, performing semantic 3D reconstruction on the acquired semantic information to generate an instance point cloud with semantic information of personnel, boom and steel coil;

[0014] Step S06, detecting the movement directions of the crane and the boom using a feature point method and multi-view geometric constraints;

[0015] Step S07, obtaining the obstacle height in the direction of the boom and the object being hoisted to trigger the obstacle avoidance mechanism;

[0016] Step S08, extracting the point cloud of the plane where the personnel are located, and calculating the personnel escape space;

[0017] Step S09, calculating the distance from the person to the outer contour of the hanging object;

[0018] Step S10, calculate the shortest distance from the centroid of each person's point cloud to the outer contour of the steel coil, compare it with the preset safety distance threshold, and output the alarm result.

[0019] Preferably, in step S01, the calibration includes manual calibration and automatic calibration, wherein manual calibration refers to projecting the point cloud of the laser radar onto a two-dimensional plane, and manually adjusting the Euler angles to match the point cloud features with the image features one by one to obtain a transformation matrix.

[0020] Preferably, in step S03, acquiring the depth map includes:

[0021] Each camera image is used as a calibration timestamp;

[0022] Sort the received radar point cloud data before and after the camera image according to the timestamp;

[0023] Define the radar point cloud data within a certain time range before and after the camera image timestamp as the corresponding synchronization data;

[0024] Multiple synchronized radar point cloud data are superimposed and projected onto the pixel plane to obtain a depth map.

[0025] Preferably, in the step S06, after extracting the boom semantics, the following steps are performed:

[0026] Extract and match feature points of specific areas of the two frames of images input by the camera installed on the vehicle;

[0027] Use multi-view geometric constraints to filter matching feature points;

[0028] The movement directions of the carriage and the boom are calculated separately through the feature point vectors.

[0029] Preferably, in step S07, the boom, hanging object point cloud and environment point cloud are input, and the point cloud at the same height is intercepted in the direction of movement of the boom. If there is an obstacle point cloud in the direction of movement, the obstacle avoidance mechanism is triggered, otherwise it runs normally.

[0030] Preferably, in step S08, the reconstructed three-dimensional point cloud is input, and the plane where each person is located is extracted as a passable area using the RANSAC-based plane extraction algorithm, and a search is conducted in the area to determine whether there is an area where the distance to the outer contour of the hanging object is greater than a set safety distance threshold.

[0031] Preferably, in step S09, the input hanging object point cloud data is subjected to Euclidean clustering, radius filtering and statistical filtering to obtain point cloud data with noise removed, the principal vector, center of mass and length and width of the hanging object point cloud are calculated by principal component analysis, the hanging object point cloud is projected onto the same plane, and the nearest neighbor search algorithm is used to find the nearest point from the point cloud to the center of mass of the person, and finally the distance is used as the horizontal distance from the person to the nearest point of the outer contour of the hanging object.

[0032] Preferably, it also includes dividing different safety ranges according to pre-settings, based on the height of the suspended objects and whether there is escape space. When the distance from a person to the outer contour of the suspended objects is less than a safety distance threshold, a continuous alarm warning is triggered until the detected person leaves the danger range and stops.

[0033] The second aspect of the present invention proposes a safety detection system for crane hoisting objects based on multi-sensor calibration, comprising:

[0034] The calibration module is used to calibrate the transformation matrix of multiple sensors and multi-sensor units, and complete the point cloud fusion and splicing after 3D reconstruction to achieve coordinate unification;

[0035] The ground extraction module is used to extract the plane equation parameters of the maximum point cloud plane, calculate the transformation matrix from the point cloud to the specified plane, and save it as a priori parameters;

[0036] The first generation module is used to input the camera and lidar point cloud data of the algorithm soft synchronization, back-project the radar point cloud to the pixel plane through the calibrated camera radar extrinsic matrix and up-sample to obtain a dense depth map;

[0037] A semantic module, used to extract semantic information of personnel, booms and steel coils using the acquired depth map, and track the semantic information;

[0038] The second generation module is used to perform semantic 3D reconstruction on the acquired semantic information to generate an instance point cloud with semantic information of personnel, boom and steel coil;

[0039] A detection module is used to detect the movement direction of the crane and the boom using a feature point method and multi-view geometric constraints;

[0040] Obstacle avoidance module, used to obtain the height of obstacles in the direction of the boom and the load, and trigger the obstacle avoidance mechanism;

[0041] The first calculation module is used to extract the point cloud of the plane where the personnel are located and calculate the escape space of the personnel;

[0042] The second calculation module is used to calculate the distance from the personnel to the outer contour of the hanging object;

[0043] The third calculation module is used to calculate the shortest distance from the centroid of each personnel point cloud to the outer contour of the steel coil, compare it with the preset safety distance threshold, and output the alarm result.

[0044] The above technical solution of the present invention has the following beneficial technical effects:

[0045] The present invention can accurately capture and calculate the positions of pedestrians, booms and steel coils based on the images and point cloud information collected by the visual radar sensors at both ends of the vehicle, and calculate the relevant parameters of pedestrians, booms and steel coils through target detection algorithms and multi-target tracking algorithms. When a pedestrian moves into the dangerous range of the suspended steel coil, the alarm on the vehicle control board automatically sounds a bell to warn, and the alarm sound is lifted when the pedestrian leaves the dangerous range. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 is a flow chart of the method of the present invention;

[0047] Figure 2 The figure is a flow chart of an embodiment of the method of the present invention. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings. It should be understood that these descriptions are only exemplary and are not intended to limit the scope of the present invention. In addition, in the following description, the description of well-known structures and technologies is omitted to avoid unnecessary confusion of the concept of the present invention.

[0049] refer to Figure 1-2 , the present invention is described in detail below.

[0050] In step S01, a camera and a radar in the present invention are used as a multi-sensor unit to synchronize the timestamps of the data collected in real time. In the present invention, multi-sensor calibration refers to the use of a camera and a laser radar as a multi-sensor unit, and the transformation matrix from the radar to the camera needs to be calibrated, and the multi-sensor unit refers to the transformation matrix from the camera in a multi-sensor unit to another multi-sensor unit. By pre-calibrating the transformation matrix, the fusion and splicing of the point cloud after three-dimensional reconstruction can be completed to realize the coordinate system. In multi-sensor calibration, manual calibration and automatic calibration methods are used at the same time to complete the calculation of the transformation matrix; wherein manual calibration is to project the point cloud of the laser radar on a two-dimensional plane, and manually adjust the Euler angle to match the point cloud features with the image features one by one to obtain an accurate transformation matrix. In the calibration of multi-sensor units, the common view area in the point clouds generated by the two sensors is found, and the precise registration algorithm is used to calculate the transformation matrix. For two point clouds with a common view area, this algorithm first uses a pass-through filter to cut out the common view area of ​​one of the point clouds, then optimizes the point cloud through voxel filtering and statistical filtering, and finally calculates the relative transformation matrix of the two point clouds through the point cloud nearest point iterative algorithm as the calibration parameters of the two multi-sensor units.

[0051] In step S02, after the calibration is completed and the spliced ​​point cloud of the unified coordinate system is obtained, the plane equation parameters of the largest point cloud plane are extracted, and the transformation matrix from the point cloud to the specified plane is calculated and saved as a priori parameters. Specifically, the plane equation parameters of the largest plane are extracted by the point cloud plane fitting algorithm based on the Ransac algorithm, and then the transformation matrix from the point cloud to the specified plane is calculated by the method of minimizing the distance from the point to the plane in the nearest iterative point algorithm.

[0052] In step S03, the camera and lidar point cloud data soft-synchronized by the algorithm are input, and the radar point cloud is back-projected to the pixel plane through the calibrated camera radar extrinsic matrix and up-sampled to obtain a dense depth map. Specifically, the acquisition of the depth map includes taking each camera image as a calibration timestamp; sorting the received radar point cloud data before and after the camera image according to the timestamp; defining the radar point cloud data within a certain time range before and after a camera image timestamp as the corresponding synchronization data; and superimposing and projecting multiple synchronized radar point cloud data to the pixel plane to obtain a depth map.

[0053] In step S04, the semantic information of the personnel, the boom and the steel coil is extracted using the acquired depth map, and the semantic information is tracked;

[0054] In step S05, semantic three-dimensional reconstruction is performed on the acquired semantic information to generate an instance point cloud with semantic information of personnel, booms and steel coils.

[0055] In step S06, the moving directions of the vehicle and the boom are detected using the feature point method and multi-view geometric constraints. Specifically, after the object detector extracts the semantics of the boom in step S04, feature points are first extracted and matched for specific areas of the input two frames of the camera mounted on the vehicle, and then the multi-view geometric constraints are used to filter the matched feature points, and finally the moving directions of the vehicle and the boom are calculated separately through the feature point vectors.

[0056] In step S07, the obstacle height in the direction of the boom and the suspended object is obtained to trigger the obstacle avoidance mechanism. Specifically, after the reconstruction of the semantic point cloud is completed, the boom, suspended object point cloud and environmental point cloud are first input, and then the point cloud at the same height in the direction of the boom movement is intercepted. If there is an obstacle point cloud in the direction of movement, the obstacle avoidance mechanism is triggered, otherwise it runs normally.

[0057] In step S08, the point cloud of the plane where the personnel are located is extracted to calculate the personnel's escape space. Specifically, the escape space refers to the relationship between the personnel's passable area on the plane where they are located and the set safe distance to the hanging object, that is, if there is an area in the personnel's passable range that is greater than the safe distance to the hanging object, it is defined as the existence of escape space, otherwise there is no escape space. Input the reconstructed three-dimensional point cloud, first use the RANSAC-based plane extraction algorithm to extract the plane where each person is located as a passable area. Then search in this area whether there is an area where the distance to the outer contour of the hanging object is greater than the set safe distance.

[0058] In step S09, the distance from the person to the outer contour of the hanging object is calculated. Specifically, the hanging object point cloud is clustered and filtered to determine and calculate the distance from the person to the outer contour of the hanging object. First, the input hanging object point cloud data is subjected to Euclidean clustering, radius filtering and statistical filtering to obtain point cloud data with noise removed. Secondly, the principal vector, center of mass and length and width of the hanging object point cloud are calculated by principal component analysis. Then, the hanging object point cloud is projected to the same plane, and the nearest neighbor search algorithm is used to find the point cloud to the nearest point of the person's center of mass. Finally, the distance is used as the horizontal distance from the person to the nearest point of the outer contour of the hanging object.

[0059] In step S10, the shortest distance from the center of mass of each person's point cloud to the outer contour of the steel coil is calculated, and compared with the preset safety distance threshold, and the alarm result is output. Specifically, according to the pre-set, different safety ranges are divided according to the height of the hanging object and whether there is escape space. When the distance from the person to the outer contour of the hanging object is less than the safety range, the algorithm sends an alarm signal to the board connected to the alarm through the serial port, triggering a continuous alarm ringing alarm until the detected person leaves the dangerous range and stops.

[0060] It should be noted that, in the present invention, for the parts with common view areas, the semantic labels and Euclidean distances are used for fusion to ensure the consistency of the global instance point cloud.

[0061] The second aspect of the present invention provides a safety detection system for crane-mounted objects based on multi-sensor calibration, comprising:

[0062] The calibration module is used to calibrate the transformation matrix of multiple sensors and multi-sensor units, complete the fusion and splicing of point clouds after 3D reconstruction, so as to achieve coordinate unification; the ground extraction module is used to extract the plane equation parameters of the maximum point cloud plane, calculate the transformation matrix of the point cloud to the specified plane, and save it as a priori parameters; the first generation module is used to input the camera and lidar point cloud data of the algorithm soft synchronization, back-project the radar point cloud to the pixel plane through the calibrated camera radar external parameter matrix and up-sample it to obtain a dense depth map; the semantic module is used to use the acquired depth map to extract the semantic information of personnel, crane arms and steel coils, and track the semantic information; the The second generation module is used to perform semantic 3D reconstruction on the acquired semantic information to generate an instance point cloud with semantic information of personnel, boom and steel coil; the detection module is used to detect the movement direction of the vehicle and boom using the feature point method and multi-view geometric constraints; the obstacle avoidance module is used to obtain the obstacle height in the forward direction of the boom and the suspended object to trigger the obstacle avoidance mechanism; the first calculation module is used to extract the point cloud of the plane where the personnel are located and calculate the escape space for the personnel; the second calculation module is used to calculate the distance from the personnel to the outer contour of the suspended object; the third calculation module is used to calculate the shortest distance from the centroid of each personnel point cloud to the outer contour of the steel coil, and compare it with the preset safety distance threshold to output the alarm result.

[0063] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments, and the above embodiments and descriptions are only preferred examples of the present invention, and are not intended to limit the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, and these changes and improvements all fall within the scope of the present invention to be protected.

Claims

1. A method for detecting the safety of crane-mounted objects based on multi-sensor calibration, characterized in that: include: Step S01, calibrating the transformation matrix of the multi-sensor and the multi-sensor unit, completing the fusion and splicing of the point cloud after three-dimensional reconstruction to achieve coordinate unification; Step S02, extracting the plane equation parameters of the maximum point cloud plane, calculating the transformation matrix from the point cloud to the specified plane, and saving it as a priori parameters; Step S03, inputting the camera and lidar point cloud data soft-synchronized by the algorithm, back-projecting the radar point cloud to the pixel plane through the calibrated camera radar extrinsic matrix and up-sampling to obtain a dense depth map; Step S04, extracting semantic information of the personnel, the boom and the steel coil using the acquired depth map, and tracking the semantic information; Step S05, performing semantic 3D reconstruction on the acquired semantic information to generate an instance point cloud with semantic information of personnel, boom and steel coil; Step S06, detecting the movement directions of the crane and the boom using a feature point method and multi-view geometric constraints; Step S07, obtaining the obstacle height in the direction of the boom and the object being hoisted to trigger the obstacle avoidance mechanism; Step S08, extracting the point cloud of the plane where the personnel are located, and calculating the personnel escape space; Step S09, calculating the distance from the person to the outer contour of the hanging object; Step S10, calculate the shortest distance from the centroid of each person's point cloud to the outer contour of the steel coil, compare it with the preset safety distance threshold, and output the alarm result.

2. The method for detecting safety of crane-mounted objects based on multi-sensor calibration according to claim 1 is characterized in that: In the step S01, the calibration includes manual calibration and automatic calibration, wherein the manual calibration refers to projecting the point cloud of the laser radar onto a two-dimensional plane, and manually adjusting the Euler angles to match the point cloud features with the image features one by one to obtain a transformation matrix.

3. The method for detecting safety of crane-mounted objects based on multi-sensor calibration according to claim 1 is characterized in that: In the step S03, the acquisition of the depth map includes: Each camera image is used as a calibration timestamp; Sort the received radar point cloud data before and after the camera image according to the timestamp; Define the radar point cloud data within a certain time range before and after the camera image timestamp as the corresponding synchronization data; Multiple synchronized radar point cloud data are superimposed and projected onto the pixel plane to obtain a depth map.

4. The method for safety detection of crane-mounted objects based on multi-sensor calibration according to claim 1 is characterized in that: In the step S06, after the boom semantics are extracted, the following steps are performed: Extract and match feature points of specific areas of the two frames of images input by the camera installed on the vehicle; Use multi-view geometric constraints to filter matching feature points; The movement directions of the carriage and the boom are calculated separately through the feature point vectors.

5. The method for safety detection of crane-mounted objects based on multi-sensor calibration according to claim 1 is characterized in that: In step S07, the boom, the hanging object point cloud and the environment point cloud are input, and the point cloud at the same height is intercepted in the direction of the boom movement. If there is an obstacle point cloud in the direction of movement, the obstacle avoidance mechanism is triggered, otherwise it runs normally.

6. The method for safety detection of crane-mounted objects based on multi-sensor calibration according to claim 1 is characterized in that: In step S08, the reconstructed three-dimensional point cloud is input, and the plane where each person is located is extracted as a passable area using the RANSAC-based plane extraction algorithm, and a search is conducted in the area to determine whether there is an area where the distance to the outer contour of the hanging object is greater than a set safety distance threshold.

7. The method for detecting safety of crane-mounted objects based on multi-sensor calibration according to claim 1 is characterized in that: In step S09, the input hanging object point cloud data is subjected to Euclidean clustering, radius filtering and statistical filtering to obtain point cloud data with noise removed, the principal vector, center of mass and length and width of the hanging object point cloud are calculated by principal component analysis, the hanging object point cloud is projected onto the same plane, the nearest neighbor search algorithm is used to find the point cloud closest to the center of mass of the person, and finally the distance is used as the horizontal distance from the person to the nearest point of the outer contour of the hanging object.

8. The method for safety detection of crane-mounted objects based on multi-sensor calibration according to claim 7 is characterized in that: It also includes dividing different safety ranges according to pre-settings, based on the height of the suspended objects and whether there is escape space. When the distance from a person to the outer contour of the suspended objects is less than the safety distance threshold, continuous alarm warnings are triggered until the detected person leaves the danger range and stops.

9. The crane crane safety detection system based on multi-sensor calibration is characterized by: include: The calibration module is used to calibrate the transformation matrix of multiple sensors and multi-sensor units, and complete the point cloud fusion and splicing after 3D reconstruction to achieve coordinate unification; The ground extraction module is used to extract the plane equation parameters of the maximum point cloud plane, calculate the transformation matrix from the point cloud to the specified plane, and save it as a priori parameters; The first generation module is used to input the camera and lidar point cloud data of the algorithm soft synchronization, back-project the radar point cloud to the pixel plane through the calibrated camera radar extrinsic matrix and up-sample to obtain a dense depth map; A semantic module, used to extract semantic information of personnel, booms and steel coils using the acquired depth map, and track the semantic information; The second generation module is used to perform semantic 3D reconstruction on the acquired semantic information to generate an instance point cloud with semantic information of personnel, boom and steel coil; A detection module is used to detect the movement direction of the crane and the boom using a feature point method and multi-view geometric constraints; Obstacle avoidance module, used to obtain the height of obstacles in the direction of the boom and the load, and trigger the obstacle avoidance mechanism; The first calculation module is used to extract the point cloud of the plane where the personnel are located and calculate the escape space of the personnel; The second calculation module is used to calculate the distance from the personnel to the outer contour of the hanging object; The third calculation module is used to calculate the shortest distance from the centroid of each personnel point cloud to the outer contour of the steel coil, compare it with the preset safety distance threshold, and output the alarm result.

Images