Yard anti-collision system based on 3D vision

Through a three-dimensional vision-based anti-collision system, the multi-view three-dimensional reconstruction algorithm and artificial intelligence controller are used to generate electronic virtual maps, solving the problems of blind spots and high costs of traditional lidar in complex environments, and achieving more efficient Container yard anti-collision protection.

CN113192199BActive Publication Date: 2024-11-12AEROSPACE INTELLIGENT MFG (SHANGHAI) TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202110610685.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-06-01
Publication Date
2024-11-12
Estimated Expiration
2041-06-01

AI Technical Summary

Technical Problem

In the complex outdoor lighting environment, traditional yard anti-collision system is difficult to accurately and quickly convert the plane image into a three-dimensional point cloud using lidar alone, resulting in blind spots in view angles and high costs, and it is impossible to effectively detect the misalignment of containers in adjacent yards in the direction of large vehicles, resulting in prevention. Collision protection vulnerability.

Method used

Adopt a yard collision prevention system based on three-dimensional vision, and use two digital camera arrays to take multi-view high-definition photos, generate electronic virtual maps through deep convolutional neural networks and multi-view three-dimensional reconstruction algorithms, and combine artificial intelligence controllers to perform spreader load paths. Planning to achieve high-precision anti-collision protection.

Benefits of technology

It improves the detection capability of nearby container misalignment, provides better anti-collision protection, reduces system costs, improves container operation efficiency, and avoids collision accidents between spreaders and containers.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113192199B_ABST
    Figure CN113192199B_ABST
Patent Text Reader

Abstract

The present invention belongs to the field of artificial intelligence algorithms, and in particular to a yard anti-collision system based on three-dimensional vision. For the existing yard anti-collision systems, laser radars are used alone. A single laser radar has a large visual blind spot, and multiple expensive laser radars have high costs, resulting in the problem that cost and efficiency cannot be taken into account at the same time. The following scheme is now proposed, which includes two groups of camera arrays, industrial control equipment, a yard crane and an artificial intelligence controller. The camera array is arranged on the top of the yard crane, an electrical room is arranged in the yard crane, the industrial control equipment is installed in the electrical room, and the artificial intelligence controller is connected to the two groups of camera arrays. The present invention uses a digital camera to take pictures of different angles of the yard below the yard crane or the quay crane, and then uses a multi-view three-dimensional reconstruction algorithm and an artificial intelligence algorithm to generate an electronic virtual map of the entire yard below, which can provide better protection for the dislocation of containers in adjacent bays and has better cost performance.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence algorithm technology, and in particular to a three-dimensional vision-based yard anti-collision system. Background Art

[0002] The yard anti-collision system, also known as the yard anti-bowling system (hereinafter referred to as LCPS), is mainly used for three functions in the operation of spreaders in automated or semi-automated container port terminals. The anti-collision protection function for the trolley and truck direction: in the forward direction of the trolley, if the position of the empty spreader or the spreader with load is lower than the obstacle, the LCPS will calculate the horizontal distance between the obstacle and the spreader load (which may be empty) according to the current trolley position read by the PLC, and limit the trolley's moving speed when it is close, or automatically stop the trolley when it is very close, and give the emergency stop reason. Because of the minimum distance recommended for rising, the trolley anti-collision lock is unlocked only when the operator or automatic control system raises the spreader load above the obstacle. The obstacles detected by LCPS, in addition to the trolley direction, must also realize the anti-collision protection function of the spreader and the obstacles in the adjacent yard in the direction of the large vehicle. For example, when the adjacent yard in the direction of the large vehicle is misaligned or the large vehicle is tilted, the containers in the adjacent yard will protrude into the yard, causing safety hazards for the operation of the trolley; Optimal path planning: On the basis of the above functions, LCPS should further perform intelligent optimization on the entire container path. Planning, depending on the degree of automation, is divided into basic functions, advanced functions and high-level functions. In the basic function, LCPS can provide the obstacle height information of the target position to the semi-automatic system. By comparing it with the lifting height value, the distance to the top of the task box is displayed in real time on the main screen of the driver's cab to facilitate the driver to determine the distance to drop the box. At the same time, LCPS can calculate the parabolic trolley spreader running route for the specified target container position, and give operation suggestions and operation restrictions, that is, when the trolley is at different positions and the spreader is at different heights, LCPS limits the running speed of the trolley and spreader to achieve intelligent soft landing. The control system in the advanced function has a higher degree of automation, so the parabolic running route can be directly automatically operated, and the operator only needs to monitor it, that is, the spreader automatically runs parabolically to a lower height above the target container position. In the high-level function, the LCPS system will intelligently analyze the container stacking situation in the yard or on the cargo ship, transmit it to the TOS system, and give selection suggestions for the target container position. In addition, the operator can easily reset the target container position to any new yard in the yard or on the cargo ship through the interactive interface;

[0003] Due to the lack of effective technical means in the past, the flat images taken by the camera could not be accurately and quickly converted into three-dimensional point clouds in complex outdoor lighting environments. Therefore, traditional yard anti-collision systems all use laser radars alone, using the principle of laser scanning and ranging to obtain real-time data on a scan line below, and regard the single-line data as the height distribution of the container below to plan the movement path of the spreader and achieve the purpose of anti-collision. However, in practice, there are hundreds or thousands of possible situations for the height distribution of containers under the spreader in the yard after permutation and combination, and a single laser radar has a large visual blind spot and may not be able to detect the misalignment of containers in adjacent yards in the direction of the truck, resulting in anti-collision protection loopholes. Although the loopholes can be compensated by increasing the number of laser radars, because the data of laser radars is very unintuitive, it takes developers a lot of time and management costs to manually debug and optimize the synthesis of multi-radar data. In addition, the scanning range of cheap laser radars is limited and they are easily disturbed by complex outdoor weather such as rain, snow, fog and haze. This leads to the traditional LCPS using laser radars. It is necessary to combine multiple expensive laser radars, and cost and efficiency cannot be taken into account. Summary of the invention

[0004] The purpose of the present invention is to solve the problem that the yard collision avoidance system existing in the prior art all uses laser radar alone. A single laser radar has a large visual blind spot, and multiple expensive laser radars are expensive, resulting in the disadvantage that cost and performance cannot be taken into account at the same time. A yard collision avoidance system based on three-dimensional vision is proposed.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] The yard anti-collision system based on three-dimensional vision includes two groups of camera arrays, industrial control equipment, yard crane and artificial intelligence controller. The camera array is arranged on the top of the yard crane. An electrical room is arranged in the yard crane. The industrial control equipment is installed in the electrical room. The artificial intelligence controller is connected to the two groups of camera arrays. A yard crane PLC is arranged in the electrical room. The industrial control equipment is connected to the yard crane PLC. The camera array uses a digital camera. The digital camera is used to continuously shoot downward the stacks in the container yard and send the taken multi-view high-definition photo groups to the artificial intelligence controller. The artificial intelligence controller is used to perform high-precision digital three-dimensional reconstruction of the photo group to generate an electronic virtual map of the entire yard below, and then plan the load path of the spreader on the electronic virtual map to generate control parameters of the spreader and the trolley.

[0007] Preferably, the high-precision digital 3D reconstruction process is as follows: first, a deep convolutional neural network instance segmentation algorithm is used to identify the corners of the container as feature points, and the instance label number of each feature point is obtained. Then, for each two pictures, the feature points with the same label number are the same corner features of the same container photographed, thereby forming paired points. Then, the paired points are subjected to an optimization calculation without baseline constraints, and the intrinsic matrix is ​​iteratively calculated. Then, the intrinsic matrix is ​​used to convert each two two-dimensional images into a textured three-dimensional point cloud. Since the above paired points are not only used for stereoscopic vision, but also for Figure 3 3D reconstruction is also used to find the common point cloud parts of every two 3D point clouds. Finally, all point clouds are spliced ​​together and the electronic virtual map generation of the entire container yard is completed.

[0008] Preferably, the spreader load path planning process is as follows: first, a pattern recognition algorithm is used to obtain the coordinates of each corner of the container below as a control point column, and then an optimization method is used to calculate an equation of a plane parabola that is higher than all control points and as short as possible. Finally, the part of the parabola that is higher than the spreader's maximum rising height is cut off and replaced with a horizontal straight line segment to obtain the final spreader load motion trajectory that will not collide with the stack and has a relatively low running time.

[0009] Compared with the prior art, the present invention has the following beneficial effects:

[0010] The load anti-collision system for spreaders used in container yard operations of the present invention uses a digital camera to take pictures of the yard below the yard crane or the quay crane from different angles, and then uses a multi-view camera to Figure 3 The three-dimensional reconstruction algorithm and the artificial intelligence algorithm generate an electronic virtual map of the entire lower yard, and then perform parabolic container path planning on the electronic virtual map, so as to prevent the occurrence of accidents in which the spreader collides and breaks up the container stacking, and improve the container operation efficiency. Compared with the traditional method of multiple laser radars, the present invention can provide better protection for the dislocation of containers in adjacent bays, and also has better cost performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 This is a schematic diagram of the component connection structure of the three-dimensional vision-based yard anti-collision system proposed by the present invention;

[0012] Figure 2 A schematic diagram of the working process of the three-dimensional vision-based yard anti-collision system proposed by the present invention;

[0013] Figure 3 This is a schematic diagram of the operation of the three-dimensional vision-based yard collision avoidance system proposed by the present invention;

[0014] Figure 4This is a schematic diagram of the operation of the three-dimensional vision-based yard collision avoidance system proposed by the present invention;

[0015] Figure 5 The electronic virtual floor plan of the yard is a three-dimensional vision-based yard anti-collision system proposed by the present invention. Figure 3 Dimensional reconstruction renderings.

[0016] In the picture: 1 camera array, 2 industrial control equipment, 3 yard crane, 4 camera group, 5 shore crane. DETAILED DESCRIPTION

[0017] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0018] Embodiment 1

[0019] Reference Figure 1-5 The yard anti-collision system based on three-dimensional vision includes two groups of camera arrays 1, industrial control equipment 2, yard crane 3 and artificial intelligence controller. The camera array 1 is arranged on the top of the yard crane 3, an electrical room is arranged in the yard crane 3, the industrial control equipment 2 is installed in the electrical room, the artificial intelligence controller is connected with the two groups of camera arrays 1, a yard crane PLC is arranged in the electrical room, and the industrial control equipment 2 is connected with the yard crane PLC. The camera used in the camera array 1 is a digital camera, which sends the multi-view high-definition photo group taken to the artificial intelligence controller, which performs high-precision digital three-dimensional reconstruction on the photo group to generate an electronic virtual map of the entire lower yard, and then performs spreader load path planning on the electronic virtual map, thereby generating control parameters of the spreader and the trolley.

[0020] In this embodiment, the high-precision digital 3D reconstruction process is as follows: first, a deep convolutional neural network instance segmentation algorithm is used to identify the corners of the container as feature points, and the instance label number of each feature point is obtained. Then, for each two images, the feature points with the same label number are the same corner features of the same container photographed, thereby forming paired points. Then, the paired points are subjected to baseline-free optimization calculations, and the intrinsic matrix is ​​iteratively calculated. Then, this intrinsic matrix is ​​used to convert each two two-dimensional images into a textured three-dimensional point cloud. Since the above paired points are not only used for stereoscopic vision, but also for Figure 3 3D reconstruction is also used to find the common point cloud parts of every two 3D point clouds. Finally, all point clouds are spliced ​​together and the electronic virtual map generation of the entire container yard is completed.

[0021] In this embodiment, the spreader load path planning process is as follows: first, a pattern recognition algorithm is used to obtain the coordinates of the corners of the container below as a control point column, and then an optimization method is used to calculate a plane parabola equation that is higher than all control points and as short as possible. Finally, the part of the parabola that is higher than the spreader's maximum rising height is cut off and replaced with a horizontal straight line segment to obtain the final spreader load movement trajectory that will not collide with the stack and has a relatively low running time.

[0022] Embodiment 2

[0023] Reference Figure 1-2 A yard anti-collision system based on three-dimensional vision includes an artificial intelligence controller and a digital camera. The digital camera is installed on the crossbeam above the yard crane or the quay crane in the form of a multi-camera array or a mobile track to take a multi-view high-definition photo group of the stacked containers in the yard directly below, and send it to the artificial intelligence controller. The artificial intelligence controller is installed in the electrical room of the yard crane or the quay crane. The digital cameras are connected to the artificial intelligence controller through a network cable to an optical fiber. The artificial intelligence controller is connected to the yard crane or the quay crane PLC. If the digital camera is installed in a camera array, it is ensured that the container at any position in the yard is photographed by at least a pair of adjacent digital cameras with the same focal length, that is, covered in their common field of view. If it is installed in a mobile track, the camera continuously shoots video to cover all containers within the moving range, so that the image of any container is finally taken by two cameras with different installation positions, or in different frames of the same video, thereby obtaining at least two image pairs with large parallax;

[0024] In this embodiment, after obtaining the parallax image pair, the deep neural network instance segmentation algorithm is used to find the four corners of all containers in each image as important labeled feature points. In the next step, a pair of feature points with the same label in the image pair is regarded as the pairing points of the same corner in different view images. Then, the pre-calibrated single-camera internal parameters and the optimized multi-view imaging equation are used to calculate the binocular intrinsic matrix of each image pair. With the intrinsic matrix, all pixels in the common area of ​​the paired images can be paired based on the grayscale features and baseline constraints, and then the paired images are converted into a three-dimensional point cloud with color texture. The order of each point cloud is determined by the camera array installation method or the order of camera movement. Moreover, because each point cloud has a common part with each other, the latter point cloud can be continuously spliced ​​with the previous point cloud, and finally the point cloud of the entire yard is completed, which is the required electronic virtual map of the yard.

[0025] In the present implementation, after having the electronic virtual map, in order to avoid omissions, the three-dimensional corner coordinates of all containers are calculated again, which are all potential collision points that must be avoided. These three-dimensional coordinates are projected onto the two-dimensional plane formed by the descent of the mobile spreader of the yard crane or the quay crane trolley, and then the constrained plane parabola fitting algorithm is applied to obtain the parabola equation, and finally the part of the parabola equation that is higher than the spreader's rising limit is flattened into a straight line segment, thus completing a path planning. Preferably, the total distance of the above-planned parabola and straight line path is less than when there is no system, the driver can visually observe the straight rise, straight translation and straight descent operation path, and avoid the spreader load "bowling" container stack accident caused by the misalignment of the shell position without this system or other imperfect systems, so that it not only ensures safety but also improves operation efficiency. Through this embodiment, it is explained that the spreader load anti-collision system is respectively installed on the yard crane and the quay crane, and is used to prevent the spreader with or without containers from colliding with the container in the yard due to misoperation or poor observation when operating on the ground container yard and the container wheel yard during the operation process.

[0026] Embodiment 3

[0027] Reference Figure 3 The yard anti-collision system based on three-dimensional vision includes an industrial control device 2 and a camera array 1. A camera array 1 for shooting downward is installed on the top crossbeam of the entire yard crane 3. The camera array 1 is composed of 22 digital cameras, 11 on the left and 11 on the right (relative to the direction of the vehicle). Since the left and right are completely symmetrical, the 11 digital cameras on the left are taken as an example. The 11 cameras are divided into two groups. One group is 5 telephoto cameras, which use a longer focal length to shoot the first and second-story containers and ground signs in the yard from front to back (relative to the direction of the vehicle). The other group is 6 close-up cameras, which use a shorter focal length to shoot the third, fourth, and fifth-story containers from front to back. The spacing between the cameras in the same group in the two groups is half of the long side of the field of view when shooting the nearest object of interest. Therefore, such installation ensures that in this embodiment, at the current position of the yard crane in the yard, containers of any height and position can be covered by the public field of view of at least two cameras in the same group, and thus are suitable for the multi-view system of the present invention. Figure 3 Dimensional reconstruction algorithm.

[0028] In this implementation, the industrial control equipment 2 is installed in the electrical room of the crane 3. The industrial control equipment 2 uses the optical fiber camera array 1 to receive the work site data. The industrial control equipment 2 is connected to the crane PLC in the electrical room with a network cable.

[0029] In the present implementation, the specific number of cameras in the camera array 1 should be determined according to the different types of yard cranes. For example, for a certain type of yard crane that only stacks three layers of containers but has a longer span, the telephoto camera group may not be used, thereby reducing the number of cameras in the camera array to only 12.

[0030] Embodiment 4

[0031] Reference Figure 4 The yard anti-collision system based on three-dimensional vision includes an industrial control device 2 and a camera group 4. The industrial control device 2 is installed in the electrical room of the quay crane 5. The industrial control device 2 uses an optical fiber camera group 4 to receive the work site data. The industrial control device 2 is connected to the quay crane PLC in the electrical room with a network cable.

[0032] In the present embodiment, since the beam on the sea side of the quay crane is generally very long, in order to reduce the number of cameras used in the camera array, five cameras with different focal lengths are integrated and installed on the moving track of one side of the beam. The five cameras are responsible for shooting from the bottom of the cabin without boxes to the highest 18 layers of boxes that this model of quay crane can lift. The camera groups are arranged completely symmetrically on the other side of the beam, so that a total of 10 cameras are used in this embodiment. Similarly, for different quay crane shapes, the number of cameras used can be increased or decreased according to actual conditions.

[0033] The above are only preferred specific implementation modes of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical solutions and inventive concepts of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A three-dimensional vision-based yard anti-collision system, comprising two sets of camera arrays (1), industrial control equipment (2), a yard crane (3) and an artificial intelligence controller, characterized in that: The camera array (1) is arranged on the top of the yard crane (3), an electrical room is arranged in the yard crane (3), the industrial control equipment (2) is installed in the electrical room, the artificial intelligence controller is connected to the two groups of camera arrays (1), the yard crane PLC is arranged in the electrical room, the industrial control equipment (2) is connected to the yard crane PLC, the camera array (1) uses a digital camera, the digital camera is used to continuously shoot downwards the stacks in the container yard, and send the shot multi-view high-definition photo group to the artificial intelligence controller, the artificial intelligence controller is used to perform high-precision digital three-dimensional reconstruction of the photo group, generate an electronic virtual map of the entire yard below, and then perform spreader load path planning on the electronic virtual map, thereby generating control parameters of the spreader and the trolley; The high-precision digital 3D reconstruction process is as follows: first, a deep convolutional neural network instance segmentation algorithm is used to identify the corners of the container as feature points, and the instance label number of each feature point is obtained. Then, for each two pictures, the feature points with the same label number are the same corner features of the same container photographed, thereby forming paired points. Then, the paired points are subjected to an optimization calculation without baseline constraints, and the intrinsic matrix is ​​iteratively calculated. Then, the intrinsic matrix is ​​used to convert each two two-dimensional images into a textured three-dimensional point cloud. Since the above paired points are not only used for three-dimensional reconstruction of stereoscopic views, but also complete the generation of an electronic virtual map of the entire container yard; The spreader load path planning process is as follows: first, a pattern recognition algorithm is used to obtain the coordinates of each corner of the container below as a control point column, and then an optimization method is used to calculate a plane parabola equation that is higher than all control points and as short as possible. Finally, the part of the parabola that is higher than the spreader's maximum rising height is cut off and replaced with a horizontal straight line segment to obtain the final spreader load movement trajectory that will not collide with the stack and has a relatively low running time.