Vision and laser dual-redundancy anti-collision system and anti-collision method

Abstract

The invention discloses a vision and laser dual-redundancy anti-collision system and an anti-collision method, and relates to the technical field of crane anti-collision, the vision and laser dual-redundancy anti-collision system comprises a data acquisition layer and a data processing layer, the data acquisition layer comprises a plurality of laser radars and industrial cameras, the laser radars acquire three-dimensional point cloud data, and the industrial cameras acquire picture information of multiple visual angles; the data processing layer comprises a camera network, a laser radar network and a 3D detector, the camera network extracts a first feature from the picture information, the first feature comprises a multi-view two-dimensional feature and a multi-view inference-based three-dimensional feature, the laser radar network extracts a second feature from the point cloud data, and the second feature comprises a three-dimensional structure feature; the first feature and the second feature are fused and then input into a 3D detector, and the 3D detector identifies target information and performs system control and / or early warning based on the identified target information. The effects of accurately detecting the moving speed and the moving direction of the moving target and avoiding the obstacle are achieved.

Description

Technical Field

[0001] This invention relates to the field of crane collision avoidance technology, and in particular to a visual and laser dual-redundant collision avoidance system and method. Background Technology

[0002] Currently, rubber-tired gantry cranes are the main loading and unloading equipment at container terminals, and their efficient operation directly affects the overall operational efficiency of the terminal. However, during the process of moving the gantry crane to container positions according to operational instructions, safety protection faces multiple severe challenges.

[0003] For example, from an operational perspective, when operating the crane in the cab, the driver's view is obstructed by the crane's own mechanical structure, creating natural blind spots. This makes it impossible to achieve full-range, blind-spot-free monitoring of the crane's route and surrounding environment, making it difficult to predict potential safety hazards. When operating from a remote control room, the driver must simultaneously monitor multiple screens for operational information, resulting in highly fragmented attention. This not only makes it easy to miss crucial safety warnings but also delays reaction time to emergencies, significantly increasing operational risks. Meanwhile, the complexity of the crane's operating area further exacerbates safety pressures. The operating area includes frequently moving internal and external container trucks, as well as truck drivers and equipment maintenance personnel who may enter the area at any time. This creates numerous overlapping work conditions, and any negligence could lead to serious accidents such as equipment collisions or personal injuries. Therefore, accurately addressing the safety challenges of protecting the crane's direction of travel and constructing a comprehensive, blind-spot-free safety guarantee system has become paramount in strengthening the safety defenses of container terminal loading and unloading operations. Summary of the Invention

[0004] The purpose of this invention is to provide a vision and laser dual-redundant collision avoidance system and method. By fusing vision and 3D laser point cloud data, it accurately detects the speed and direction of movement of moving targets. Using deep learning technology, it judges the movement trend of the machine and warns of abnormal situations within a specific range. It can communicate with PLCs and other devices to control the trolley to perform obstacle avoidance operations.

[0005] The above-mentioned technical objective of the present invention is achieved through the following technical solution:

[0006] A dual-redundant visual and laser collision avoidance system includes a data acquisition layer and a data processing layer. The data acquisition layer includes several LiDARs and industrial cameras. The LiDARs acquire 3D point cloud data, and the industrial cameras acquire image information from multiple perspectives. The data processing layer includes a camera network, a LiDAR network, and a 3D detector. The camera network extracts a first feature from the image information, which includes multi-view 2D features and 3D features inferred from the multi-view perspective. The LiDAR network extracts a second feature from the point cloud data, which includes 3D structural features. The first and second features are fused and input into the 3D detector. The 3D detector identifies target information and performs system control and / or early warning based on the identified target information.

[0007] Furthermore, the lidar and / or industrial camera are provided in at least two pairs, located at opposite ends of the forward direction.

[0008] Furthermore, the target information identified by the 3D detector includes category, three-dimensional position, size, and orientation.

[0009] Furthermore, the camera network extracts features from the input image information through a convolutional neural network to obtain a multi-scale two-dimensional feature representation that includes target edges, texture, and category semantic information.

[0010] Furthermore, the camera network processes image information from different perspectives in parallel and forms multi-view two-dimensional features through perspective alignment and feature mapping mechanisms.

[0011] Furthermore, the camera network combines the calibration parameters and geometric constraints of industrial cameras to infer the depth and position information of the target in three-dimensional space, generating three-dimensional features based on multi-view inference.

[0012] Furthermore, the first and second features are first fused and sampled, and then a unified feature is generated through an early / mid-term / late-term fusion strategy and input into the 3D detector.

[0013] This application also discloses a collision avoidance method, including the following steps: After identifying the target information, the collision avoidance logic control stage is carried out. First, the target distance d1 measured by the industrial camera is read, and then the target distance d2 measured by the lidar is read. Using d1 and d2 as inputs, it is determined whether the target has entered different safety threshold ranges. Based on the safety threshold range entered, the risk level of the target is obtained, and the corresponding alarm prompt is triggered.

[0014] Furthermore, the process for determining different safety threshold ranges includes, Step S21: Determine whether d1 < r1 or d2 < r1 holds. If so, it is determined that the target has entered the most dangerous area, and a high-frequency alarm prompt is triggered. If not, proceed to Step S22; Step 22: Determine whether d1 < r2 or d2 < r2 holds. If so, trigger a medium-frequency alarm prompt. If not, proceed to Step 23; Step 23: Determine whether d1 < r3 or d2 < r3 holds. If so, trigger a low-frequency alarm prompt. If not, proceed to Step 24; Step 24: When neither d1 nor d2 reaches the r3 threshold, but the system still detects a long-distance obstacle, give a warning prompt through the screen; Where r1 is the emergency braking threshold, r2 is the deceleration stop threshold, r3 is the warning threshold, and r1 < r2 < r3.

[0015] Furthermore, when the industrial camera fails, the lidar independently completes obstacle positioning and distance measurement; When the lidar fails, the industrial camera completes obstacle positioning and distance measurement through the depth inference of multi-view images.

[0016] In summary, the present invention has the following beneficial effects: Focusing on multi-sensor fusion technologies such as lidar and industrial cameras, it achieves accurate detection of the moving speed and direction of moving targets. Using deep learning technology, it judges the movement trend of the machine and warns of abnormal situations within a specific range, and provides a PLC communication interface according to the safety strategies of different protection areas, enabling communication control with the PLC to perform obstacle avoidance operations for the trolley; It avoids the risk of single-sensor technology failures. Through the dual-redundancy design of integrating 3D lidar and industrial cameras, it obtains point cloud data and image data. Combining deep learning technology, it achieves an obstacle detection accuracy rate of ≥99.5% and a false alarm rate of anti-collision warning of ≤1%. The perception accuracy is 0.1m, the effective detection distance is 50m, and the alarm response time is ≤300ms. It can accurately identify various obstacles and perform real-time dynamic detection; In addition, it has an IP66 protection level, a working temperature range of -40°C to 65°C, and a humidity adaptability of 100% non-condensation, with strong environmental adaptability. In addition, it supports all brands such as Siemens, ABB, Fuji, and Yaskawa, with wide compatibility, and the laser and vision systems are mutually backup to cope with single-point failures;​​​​​​​This is a hardware layout diagram of a vision and laser dual-redundant collision avoidance system of the present invention applied to a tire-mounted bridge; Figure 2 This is a schematic diagram of a vision and laser dual-redundant collision avoidance system according to the present invention; Figure 3 This is a schematic diagram of the anti-collision logic in a vision and laser dual-redundancy anti-collision system of the present invention; In the diagram, 1 is the first lidar, 2 is the second lidar, 3 is the third lidar, 4 is the fourth lidar, 5 is the first industrial camera, 6 is the second industrial camera, 7 is the third industrial camera, 8 is the fourth industrial camera, 9 is the first audible and visual alarm, 10 is the second audible and visual alarm, 11 is the third audible and visual alarm, 12 is the fourth audible and visual alarm, and 13 is the sensing workstation. Detailed Implementation

[0018] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings. These embodiments do not constitute a limitation of the present invention. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of this application.

[0019] A dual-redundant collision avoidance system using both vision and laser, such as Figure 1 and Figure 2 As shown, it includes a data acquisition layer and a data processing layer. The data acquisition layer includes several lidar and industrial cameras. The lidar emits laser pulses and measures the time it takes for them to reflect back from the surface of the object to generate accurate three-dimensional information about the position and shape of the obstacle, thereby acquiring three-dimensional point cloud data. The industrial cameras acquire image information from multiple perspectives to acquire the three-dimensional position and shape information of the obstacle, providing the system with high-precision spatial data. In this embodiment, it is applied to a tire-type bridge, and at least two pairs of lidar and industrial cameras are provided, located at both ends of the forward direction. The lidar is mounted on the bottom of the two-way gate legs via brackets. There are four lidars: the first lidar 1 (right side of the shore), the second lidar 2 (left side of the shore), the third lidar 3 (right side of the sea), and the fourth lidar 4 (left side of the sea). They are used to generate accurate three-dimensional information about the position and shape of obstacles. After the protective cover is installed, the height and angle are adjusted according to the actual situation on site. The industrial cameras are mounted on the two-way gate legs by brackets, covering the area in the direction of the large vehicle's movement. There are four industrial cameras: the first industrial camera 5 (left side of the shore), the second industrial camera 6 (right side of the sea), the third industrial camera 7 (left side of the sea), and the fourth industrial camera 8 (right side of the shore). After the protective covers are installed, the height and angle of each camera are adjusted according to the actual site conditions.

[0020] like Figure 2As shown, the data processing layer includes a camera network, a LiDAR network, and a 3D detector to optimize the quality of raw data and extract core information from the sensors. An industrial camera acquires image information and inputs it into a camera network. The camera network extracts a first feature from the image information. The first feature includes multi-view two-dimensional features and three-dimensional features inferred from the multi-view perspective. The camera network is a deep learning-based visual perception network that uses a convolutional neural network (CNN) to extract features from the input image information, obtaining a multi-scale two-dimensional feature representation that includes target edges, texture, and category semantic information. Under multi-camera configuration, the camera network processes image information from different perspectives in parallel and forms a multi-view two-dimensional feature set through perspective alignment and feature mapping mechanisms. At the same time, the camera network combines the calibration parameters and geometric constraints of the industrial camera to further infer the depth and position information of the target in three-dimensional space, generating auxiliary three-dimensional features based on perspective inference.

[0021] like Figure 2 As shown, the lidar network extracts a second feature from point cloud data, which includes three-dimensional structural features. The lidar network is a deep learning-based point cloud processing network used to analyze and model point cloud data acquired by 3D lidar. The network extracts the spatial structural features of obstacles, including the target's three-dimensional position, size, shape, and spatial distribution information, by processing the point cloud data through voxelization, projection, or point-level feature encoding. A sensor workstation 13 is installed on the upright bracket inside the electrical room of the tire-type field bridge. It is used to deploy 3D detectors, etc. The algorithm identifies the target's category, three-dimensional position, size and attitude. The first feature and the second feature are fused and input into the 3D detector. The 3D detector identifies the target information (including category, three-dimensional position, size and attitude) and performs system control and / or early warning based on the identified target information. The device for early warning (to enable coordinated control between equipment and trigger active obstacle avoidance) includes several audible and visual alarms installed at the legs of the large vehicle doors. There are four audible and visual alarms: the first audible and visual alarm 9 (right side of the shore), the second audible and visual alarm 10 (left side of the shore), the third audible and visual alarm 11 (right side of the sea), and the fourth audible and visual alarm 12 (left side of the sea). These are used to alert the system after it detects an obstacle, so as to avoid accidents. In some embodiments, it also assists technicians in completing system deployment and debugging work by configuring and debugging visualization software, so that the system can better adapt to the field operation environment.

[0022] This application also discloses a collision avoidance method, including the following steps: The hardware involved (3D LiDAR, industrial camera, and sensing workstation 13, etc., for data acquisition, data processing, system control and early warning) is installed at the corresponding position of the tire crane. During the movement of the tire crane trolley according to the operation instructions, the environmental data is collected and processed by the two dual sensors, industrial camera and 3D LiDAR, and high-precision target detection is achieved by combining their respective advantages. The specific process is as follows: First, data acquisition is carried out in parallel: industrial cameras acquire two-dimensional images from multiple perspectives, and 3D LiDAR acquires three-dimensional point cloud data. The two together constitute the multi-source samples of the system. Then, the two types of data are respectively entered into the corresponding sensor networks for preprocessing and feature extraction. In image information preprocessing, preprocessing is performed first, including image normalization, color correction, contrast adjustment, etc., to improve image quality. Then, the preprocessed visual data is processed by image processing algorithms, including applying deep neural networks to perceive obstacle information from the image, and then extracting multi-view two-dimensional features and three-dimensional features based on multi-view inference. In point cloud data preprocessing, preprocessing is performed first, including noise removal, point cloud density unification, and filtering, to improve data quality and reduce computation. Then, the preprocessed point cloud data is processed by algorithms such as clustering and feature extraction to achieve obstacle perception, recognition and localization, and to extract the three-dimensional structural features of the point cloud data.

[0023] Subsequently, the system fuses and samples the multi-view 2D and 3D features output by the camera network with the 3D structural features output by the LiDAR network, integrating the advantages of visual semantics and spatial accuracy to generate more expressive fused features. Visual and point cloud features are then combined using an early / mid / late-stage fusion strategy to generate unified features, which are directly input into the 3D detector. The algorithm then identifies the target's category, 3D position, size, and pose.

[0024] like Figure 3 As shown, after identifying the target information, the system completes environmental perception and three-dimensional target localization. Then, the system enters the collision avoidance logic control stage and completes the logical decision-making according to the following process. Step 10: After the system starts up, it first reads the target distance d1 measured by the visual recognition module (industrial camera), and then reads the target distance d2 measured by the lidar. The system uses the two sensing data d1 and d2 as inputs to determine whether the target has entered different safety threshold ranges. Based on the safety threshold range, the risk level of the target is obtained and the corresponding alarm prompt is triggered, thereby realizing graded risk control and response command output. Step 20: Determine the collision avoidance level based on the distance between the obstacle and the vehicle; the determination process for different safety threshold ranges and collision avoidance levels includes, Step S21: Determine whether d1 < r1 or d2 < r1 (emergency braking threshold) holds. If so, it is determined that the target has entered the most dangerous area, triggering danger level 4 and a high-frequency alarm prompt. If not, proceed to step S22; Step 22: Determine whether d1 < r2 or d2 < r2 (deceleration stop threshold) holds. If so, it is determined to be danger level 3 and a medium-frequency alarm prompt is triggered. If not, proceed to step 23; Step 23: Determine whether d1 < r3 or d2 < r3 (warning threshold) holds. If so, it is determined to be danger level 2 and a low-frequency alarm prompt is triggered. If not, proceed to step 24; Step 24: When neither the target distances d1 nor d2 reach the r3 threshold, but the system still detects a long-distance obstacle, it is classified as danger level 1 and a warning prompt is given through the screen; Step 25: If all the above conditions are not met, it means that no obstacle has entered the警戒区域 currently, and the system sets the danger level to 0 and enters the normal operation mode.

[0025] Among them, r1 < r2 < r3; the anti-collision area based on the distance range can be accurately set by inputting specific distances in the configuration interface; in some embodiments, the deceleration stop area can also be custom-configured according to the image drawing range; the system alarm conditions can also be logged and video-recorded to support retrieving alarm data within a time period such as within 14 days.

[0026] When a single-point failure occurs, a detection distance of at least 15m needs to be maintained to ensure safety redundancy; in some embodiments, after subtracting a safety redundancy distance from d1 or d2, the above-mentioned step S20 is judged again; Specifically, if any device of the industrial camera or 3D lidar fails, the other sensor system needs to immediately start the independent working mode - when the industrial camera fails, the lidar relies on its stable three-dimensional space detection ability to independently complete obstacle positioning and distance measurement; when the lidar fails, the industrial camera completes obstacle positioning and distance measurement through the depth inference of multi-view images, ensuring that the obstacle perception function required for basic anti-collision is not affected.

[0027] When the system is operating normally, it not only improves the perception efficiency through the parallel mode of dual-sensor data processing, but also compensates for the limitations of a single sensor with the feature fusion technology; in the single-point failure scenario, the redundant design of the dual sensors enables the system to maintain the advantages of multi-sensor fusion while still being able to independently complete the basic environment perception and safety protection functions relying on a single sensor, thus achieving an overall improvement in efficiency, accuracy, and reliability.

[0028] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art can make various modifications or equivalent substitutions to the present invention within the scope of its essence and protection. Such modifications or equivalent substitutions should also be considered to fall within the protection scope of the present invention.

Claims

1. A vision and laser dual redundant collision avoidance system characterized by: It includes a data acquisition layer and a data processing layer. The data acquisition layer includes several lidars and industrial cameras. The lidars collect three-dimensional point cloud data, and the industrial cameras collect picture information from multiple perspectives. The data processing layer includes a camera network, a lidar network, and a 3D detector. The camera network extracts first features from the picture information. The first features include multi-perspective two-dimensional features and three-dimensional features inferred based on multi-perspectives. The lidar network extracts second features from the point cloud data. The second features include three-dimensional structural features. After the first features and the second features are fused, they are input into the 3D detector. The 3D detector identifies target information and performs system control and / or warning based on the identified target information.

2. A vision and laser dual redundant collision avoidance system according to claim 1, characterized in that: At least two pairs of the lidars and / or industrial cameras are provided and are respectively located at both ends of the forward direction.

3. The visual and laser dual redundant collision avoidance system of claim 1 or 2, wherein: The target information identified by the 3D detector includes category, three-dimensional position, size, and pose.

4. The vision and laser dual redundant collision avoidance system of claim 1, wherein: The camera network performs feature extraction on the input picture information through a convolutional neural network to obtain a multi-scale two-dimensional feature representation containing target edge, texture, and category semantic information.

5. The visual and laser dual redundant collision avoidance system of claim 1 or 4, wherein: The camera network processes the picture information from different perspectives in parallel and forms multi-perspective two-dimensional features through a perspective alignment and feature mapping mechanism.

6. The visual and laser dual-redundant collision avoidance system according to claim 5, characterized in that: The camera network combines the calibration parameters of the industrial camera and the geometric constraint relationship to infer the depth and position information of the target in the three-dimensional space and generates three-dimensional features inferred based on multi-perspectives.

7. The visual and laser dual-redundant collision avoidance system according to claim 1, characterized in that: The first features and the second features are first fusion sampled, and then a unified feature is generated through an early / mid / late fusion strategy and input into the 3D detector.

8. A collision avoidance method, based on the visual and laser dual-redundant collision avoidance system as described in claim 1, characterized in that: It includes the following steps. After identifying the target information, enter the anti-collision logic control stage. First, read the target distance d1 measured by the industrial camera, and then read the target distance d2 measured by the lidar. Taking d1 and d2 as inputs, judge whether the target enters different safety threshold ranges. According to the entered safety threshold ranges, obtain the risk level of the target and trigger the corresponding alarm prompt.

9. The anti-collision method according to claim 8, characterized in that: The judgment process of different safety threshold ranges includes: Step S21, judge whether d1 < r1 or d2 < r1 holds. If so, it is determined that the target has entered the most dangerous area and a high-frequency alarm prompt is triggered. If not, execute step S22; Step 22, judge whether d1 < r2 or d2 < r2 holds. If so, trigger a medium-frequency alarm prompt. If not, execute step 23; Step 23, judge whether d1 < r3 or d2 < r3 holds. If so, trigger a low-frequency alarm prompt. If not, execute step 24; Step 24, when both d1 and d2 do not reach the r3 threshold, but the system still detects a long-distance obstacle, give a warning prompt through the screen; Among them, r1 is the emergency braking threshold, r2 is the deceleration stop threshold, r3 is the warning threshold, and r1 < r2 < r3.

10. An anti-collision method according to claim 8, characterized in that: When the industrial camera fails, the lidar independently completes obstacle positioning and distance measurement; When the lidar fails, the industrial camera completes obstacle positioning and distance measurement through the depth inference of multi-perspective images.

Images