Box placing safety system based on multi-sensor fusion

Abstract

The utility model discloses a box placing safety system based on multi-sensor fusion. The system comprises a first group of laser radar sensors used for obtaining a first scanning result; the second group of laser radar sensors are used for obtaining a second scanning result; the first, second and third groups of cameras are respectively used for acquiring first, second and third groups of images; the truck head anti-pressing safety detection unit judges whether an obstacle exists above the truck plate according to the first and second scanning results, judges whether the truck head enters the position below the container according to the third group of images, and judges whether the truck head is complete according to the second group of images; the truck board foreign matter safety detection unit is used for judging whether foreign matters exist on the truck boards according to the first and second scanning results and the third group of images; and the control unit sends out a warning or operation stop signal according to the judgment result. According to the box placing safety system based on multi-sensor fusion provided by the invention, through fusion of detection results of the laser radar sensor and the camera, the safety during box placing is greatly improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of hoisting system transportation, and in particular to a box placing safety system based on multi-sensor fusion. BACKGROUND

[0002] At present, in the process of port automation operation, the port crane performs operation on the external truck for placing a box. Many safety problems are prone to occur in the process of placing a box, such as the truck head of the external truck being pressed, foreign matter existing on the truck plate of the external truck, pedestrians existing beside the external truck, and the container being damaged after the placing of the box is completed.

[0003] Therefore, it is necessary to provide a box placing safety system based on multi-sensor fusion to solve the above problems. CONTENT OF THE INVENTION

[0004] The present application provides a box placing safety system based on multi-sensor fusion, which greatly improves the safety during placing a box by fusing the detection results of the laser radar sensor and the camera.

[0005] The present application provides a box placing safety system based on multi-sensor fusion, which comprises:

[0006] a plurality of laser radar sensors, wherein a first group of laser radar sensors is arranged on a girder above a lane line, and a second group of laser radar sensors is arranged on a truck plate of a truck;

[0007] a plurality of cameras, wherein a first group of cameras is arranged on the girder above the lane line, a second group of cameras is arranged on a spreader, and a third group of cameras is arranged on the side of a trolley;

[0008] the first group of laser radar sensors are used for scanning the truck, the crane and a yard to obtain a first scanning result;

[0009] the second group of laser radar sensors are used for scanning the truck plate to obtain a second scanning result;

[0010] the first group of cameras are used for acquiring a first group of images, and the first group of images are truck head images;

[0011] the second group of cameras are used for acquiring a second group of images, and the second group of images are truck head images, truck images, crane images and yard images;

[0012] the third group of cameras are used for acquiring a third group of images, and the third group of images are truck head images, truck plate images and container images;

[0013] a data acquisition unit is used for acquiring the first scanning result, the second scanning result, the first group of images, the second group of images and the third group of images;

[0014] a front anti-crushing safety detection unit connected with the data acquisition unit, which judges whether there is an obstacle above the truck plate according to the first scanning result and the second scanning result, judges whether the truck head enters under the container according to the third group of images, judges whether the truck head is complete according to the second group of images, and judges whether the position of the truck head is reasonable according to the first group of images;

[0015] a truck plate foreign matter safety detection unit connected with the data acquisition unit, which judges whether there is a foreign matter on the truck plate according to the first scanning result, the second scanning result and the third group of images;

[0016] a control unit connected with the front anti-crushing safety detection unit and the truck plate foreign matter safety detection unit, which sends a warning reminder or a work stop signal according to the judgment results of the front anti-crushing safety detection unit and the truck plate foreign matter safety detection unit.

[0017] Preferably, a personnel foreign matter safety detection unit is further included, which is connected with the data acquisition unit, judges whether there is a personnel foreign matter in the gap between the truck and the spreader or between the truck and the yard according to the first scanning result, judges whether there is a personnel foreign matter between the truck and the spreader or between the truck and the yard according to the second group of images, and judges whether there is a personnel foreign matter around the truck head according to the first group of images.

[0018] Preferably, a guide plate anti-crushing safety detection unit is further included, which is connected with the data acquisition unit, judges whether the relative position of the truck plate and the container is reasonable according to the first scanning result and the second scanning result, judges whether the relative position of the truck plate and the container is reasonable according to the third group of images, and judges whether the relative position of the truck plate and the container is reasonable according to the second group of images.

[0019] Preferably, the first group of laser radar sensors and the second group of laser radar sensors are planar array laser radars.

[0020] Preferably, the first group of cameras, the second group of cameras and the third group of cameras are monocular cameras or binocular cameras.

[0021] Preferably, a display unit is further included, which is connected with the control unit, and is used for displaying warning reminder information or work stop information.

[0022] Preferably, a voice broadcast unit is further included, which is connected with the control unit, and is used for broadcasting a warning reminder or a work stop.

[0023] Preferably, the first group of lidar sensors includes at least two lidar sensors, and the second group of lidar sensors includes at least five lidar sensors.

[0024] Preferably, the first group of cameras includes at least one camera, the second group of cameras includes at least four cameras, and the third group of cameras includes at least two cameras.

[0025] This application offers the following advantages over existing technologies: A container loading safety system based on multi-sensor fusion is provided, comprising: multiple lidar sensors, wherein a first set of lidar sensors is mounted on a beam above the lane line, and a second set of lidar sensors is mounted on the truck bed; multiple cameras, wherein a first set of cameras is mounted on a beam above the lane line, a second set of cameras is mounted on a spreader, and a third set of cameras is mounted on the side of the truck; the first set of lidar sensors is used to scan the truck, crane, and yard to obtain a first scan result; the second set of lidar sensors is used to scan the truck bed to obtain a second scan result; the first set of cameras is used to acquire a first set of images, which are truck front images; the second set of cameras is used to acquire a second set of images, which are truck front images, truck images, crane images, and yard images; the third set of cameras is used to acquire a third set of images, which are truck front images, truck bed images, and container images; and a data acquisition unit, which is used to acquire the first... The system includes a scanning result, a second scanning result, a first set of images, a second set of images, and a third set of images; a truck front anti-crush safety detection unit connected to the data acquisition unit, which determines whether there are obstacles above the truck bed based on the first and second scanning results, whether the truck front has entered under the container based on the third set of images, whether the truck front is intact based on the second set of images, and whether the position of the truck front is reasonable based on the first set of images; a truck bed foreign object safety detection unit connected to the data acquisition unit, which determines whether there are foreign objects on the truck bed based on the first, second, and third sets of images; and a control unit connected to both the truck front anti-crush safety detection unit and the truck bed foreign object safety detection unit, which issues warnings or work stop signals based on the judgment results of these units. By fusing the detection results of the lidar sensor and the camera, the safety during container placement is greatly improved.

[0026] Furthermore, it also includes a personnel and foreign object safety detection unit, which is connected to the data acquisition unit. The personnel and foreign object safety detection unit determines whether there are personnel or foreign objects in the gap between the container truck and the lifting device, and between the container truck and the storage yard, based on the first scanning result. It also determines whether there are personnel or foreign objects between the container truck and the lifting device, and between the container truck and the storage yard, based on the second set of images. Finally, it determines whether there are personnel or foreign objects around the container truck cab based on the first set of images, thereby ensuring the safety of personnel when loading containers.

[0027] Furthermore, it also includes a guide plate anti-pressure safety detection unit, which is connected to the data acquisition unit. The guide plate anti-pressure safety detection unit determines whether the relative position of the truck pallet and the container is reasonable based on the first scan result and the second scan result, determines whether the relative position of the truck pallet and the container is reasonable based on the third set of images, and determines whether the relative position of the truck pallet and the container is reasonable based on the second set of images, thereby ensuring the accuracy of the truck pallet position and preventing container breakage during container placement. Attached Figure Description

[0028] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0029] Figure 1 This is a schematic diagram of the structure of the box placement safety system based on multi-sensor fusion in the embodiments of this application;

[0030] Figure 2 This is a schematic diagram of a box placement safety system based on multi-sensor fusion in another embodiment of this application.

[0031] The accompanying drawings have illustrated specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to specific embodiments. Detailed Implementation

[0032] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0033] The spreader lowers the container onto the truck bed according to the task. Special attention must be paid to the safety of the truck cab during this process. In the standard operating procedure for container loading and unloading, the truck must follow a designated route and stop at the loading / unloading area. Once the truck is accurately positioned, the rail-mounted crane moves along a pre-set track to the work station and performs container lifting operations using the automated spreader system. During the container lowering operation, unexpected displacement of the truck will lead to two safety hazards: firstly, it will significantly prolong the loading / unloading cycle, causing the container alignment accuracy to deviate from acceptable limits; secondly, due to the disruption of the dynamic matching relationship between the lifting system and the transport vehicle, abnormal contact between the spreader components and the truck cab may occur. This sudden mechanical action may generate impact loads exceeding the equipment's capacity, posing a serious safety threat to the operator in the cab.

[0034] To address the aforementioned issues, this application provides a box placement safety system based on multi-sensor fusion, which significantly improves the safety of box placement by fusing detection results from lidar sensors and cameras.

[0035] Figure 1 This is a schematic diagram of the structure of the box placement safety system based on multi-sensor fusion in the embodiments of this application. See now. Figure 1 This application provides a box placement safety system based on multi-sensor fusion, comprising:

[0036] Multiple lidar sensors, wherein the first set of lidar sensors 11 is set on the beam above the lane line, and the second set of lidar sensors 12 is set on the truck bed;

[0037] Multiple cameras, of which the first group of cameras 13 is set on the main beam above the lane line, the second group of cameras 14 is set on the crane, and the third group of cameras 15 is set on the side of the trolley.

[0038] The first set of lidar sensors 11 is used to scan trucks, cranes, and storage yards to obtain the first scan result;

[0039] The second set of lidar sensors 12 is used to scan the truck pallet to obtain a second scan result;

[0040] The first set of cameras 13 is used to acquire a first set of images, which are images of the truck cab.

[0041] The second set of cameras 14 is used to acquire a second set of images, which are images of truck cabs, trucks, cranes, and storage yards.

[0042] The third set of cameras 15 is used to acquire a third set of images, which are images of the truck cab, truck pallet, and container.

[0043] Data acquisition unit 2 is used to acquire the first scan result, the second scan result, the first set of images, the second set of images, and the third set of images;

[0044] The front anti-pinch safety detection unit 31 is connected to the data acquisition unit. The front anti-pinch safety detection unit 31 determines whether there are obstacles above the truck bed based on the first scan result and the second scan result, determines whether the truck head has entered under the container based on the third set of images, determines whether the truck head is intact based on the second set of images, and determines whether the position of the truck head is reasonable based on the first set of images.

[0045] The vehicle board foreign object safety detection unit 32 is connected to the data acquisition unit. The vehicle board foreign object safety detection unit 32 determines whether there is a foreign object on the loading truck board based on the first scan result, the second scan result and the third set of images.

[0046] The control unit 4 is connected to the front anti-pressure safety detection unit 31 and the vehicle board foreign object safety detection unit 32 respectively, and issues warning reminders or operation stop signals based on the judgment results of the front anti-pressure safety detection unit 31 and the vehicle board foreign object safety detection unit 32.

[0047] Specifically, by fusing data from multiple sets of lidar sensors and multiple sets of cameras, the safety of truck-side operations can be perceived in real time from multiple directions and angles, achieving multiple protections for various complex scenarios in truck-side operations.

[0048] In specific implementation, the first group of lidar sensors 11 and the second group of lidar sensors 12 are area array lidars.

[0049] In specific implementation, the first group of cameras 13, the second group of cameras 14 and the third group of cameras 15 are monocular cameras or binocular cameras.

[0050] In a specific implementation, a display unit is also included, which is connected to the control unit 4. The display unit is used to display warning reminders or work stop information.

[0051] In specific implementation, a voice broadcasting unit is also included, which is connected to the control unit 4. The voice broadcasting unit is used to broadcast warnings or reminders or to stop the operation.

[0052] In a specific implementation, the first group of lidar sensors 11 includes at least two lidar sensors, and the second group of lidar sensors 12 includes at least five lidar sensors.

[0053] In a specific implementation, the first group of cameras 13 includes at least one camera, the second group of cameras 14 includes at least four cameras, and the third group of cameras 15 includes at least two cameras.

[0054] Figure 2 This is a schematic diagram of a box placement safety system based on multi-sensor fusion in another embodiment of this application. (See now) Figure 2 This application provides a box placement safety system based on multi-sensor fusion, comprising:

[0055] Multiple lidar sensors, wherein the first set of lidar sensors 11 is set on the beam above the lane line, and the second set of lidar sensors 12 is set on the truck bed;

[0056] Multiple cameras, of which the first group of cameras 13 is set on the main beam above the lane line, the second group of cameras 14 is set on the crane, and the third group of cameras 15 is set on the side of the trolley.

[0057] The first set of lidar sensors 11 is used to scan trucks, cranes, and storage yards to obtain the first scan result;

[0058] The second set of lidar sensors 12 is used to scan the truck pallet to obtain a second scan result;

[0059] The first set of cameras 13 is used to acquire a first set of images, which are images of the truck cab.

[0060] The second set of cameras 14 is used to acquire a second set of images, which are images of truck cabs, trucks, cranes, and storage yards.

[0061] The third set of cameras 15 is used to acquire a third set of images, which are images of the truck cab, truck pallet, and container.

[0062] Data acquisition unit 2 is used to acquire the first scan result, the second scan result, the first set of images, the second set of images, and the third set of images;

[0063] The front anti-pinch safety detection unit 31 is connected to the data acquisition unit 2. The front anti-pinch safety detection unit 31 determines whether there are obstacles above the truck bed based on the first scan result and the second scan result, determines whether the truck head has entered under the container based on the third set of images, determines whether the truck head is intact based on the second set of images, and determines whether the position of the truck head is reasonable based on the first set of images.

[0064] The vehicle board foreign object safety detection unit 32 is connected to the data acquisition unit 2. The vehicle board foreign object safety detection unit 32 determines whether there is a foreign object on the loading truck board based on the first scan result, the second scan result and the third set of images.

[0065] The personnel and foreign object safety detection unit 33 is connected to the data acquisition unit 2. The personnel and foreign object safety detection unit 33 determines whether there are personnel or foreign objects in the gap between the truck and the spreader and between the truck and the yard based on the first scanning result, determines whether there are personnel or foreign objects between the truck and the spreader and between the truck and the yard based on the second set of images, and determines whether there are personnel or foreign objects around the truck cab based on the first set of images.

[0066] The guide plate anti-pressure safety detection unit 34 is connected to the data acquisition unit 2. The guide plate anti-pressure safety detection unit 34 determines whether the relative position of the truck pallet and the container is reasonable based on the first scan result and the second scan result, determines whether the relative position of the truck pallet and the container is reasonable based on the third set of images, and determines whether the relative position of the truck pallet and the container is reasonable based on the second set of images.

[0067] The control unit 4 is connected to the front anti-pressure safety detection unit 31, the platform foreign object safety detection unit 32, the personnel foreign object safety detection unit 33, and the guide plate anti-pressure safety detection unit 34 respectively. Based on the judgment results of the front anti-pressure safety detection unit 31, the platform foreign object safety detection unit 32, the personnel foreign object safety detection unit 33, and the guide plate anti-pressure safety detection unit 34, it issues warning reminders or operation stop signals.

[0068] Specifically, in port operations, there is a type of container truck with guide plates on both sides of the truck bed. During the container placement process, it is necessary to check whether the container is placed incorrectly on the guide plates to avoid the bottom of the container being penetrated.

[0069] In summary, the container placement safety system based on multi-sensor fusion provided in this application includes: multiple lidar sensors, wherein a first set of lidar sensors is installed on the beam above the lane line, and a second set of lidar sensors is installed on the truck bed; multiple cameras, wherein a first set of cameras is installed on the beam above the lane line, a second set of cameras is installed on the spreader, and a third set of cameras is installed on the side of the truck; the first set of lidar sensors is used to scan the truck, crane, and yard to obtain a first scan result; the second set of lidar sensors is used to scan the truck bed to obtain a second scan result; the first set of cameras is used to acquire a first set of images, which are truck front images; the second set of cameras is used to acquire a second set of images, which are truck front images, truck images, crane images, and yard images; the third set of cameras is used to acquire a third set of images, which are truck front images, truck bed images, and container images; and a data acquisition unit, which is used to acquire the first scan result, the second set of lidar sensors, the third ... The system includes two sets of scan results, a first set of images, a second set of images, and a third set of images; a truck front anti-crush safety detection unit connected to the data acquisition unit, which determines whether there are obstacles above the truck bed based on the first and second scan results, whether the truck front has entered under the container based on the third set of images, whether the truck front is intact based on the second set of images, and whether the position of the truck front is reasonable based on the first set of images; a truck bed foreign object safety detection unit connected to the data acquisition unit, which determines whether there are foreign objects on the truck bed based on the first, second, and third set of images; and a control unit connected to both the truck front anti-crush safety detection unit and the truck bed foreign object safety detection unit, which issues warnings or work stop signals based on the judgment results of these units. By fusing the detection results of the lidar sensor and the camera, the safety during container placement is greatly improved.

[0070] Furthermore, it also includes a personnel and foreign object safety detection unit, which is connected to the data acquisition unit. The personnel and foreign object safety detection unit determines whether there are personnel or foreign objects in the gap between the container truck and the lifting device, and between the container truck and the storage yard, based on the first scanning result. It also determines whether there are personnel or foreign objects between the container truck and the lifting device, and between the container truck and the storage yard, based on the second set of images. Finally, it determines whether there are personnel or foreign objects around the container truck cab based on the first set of images, thereby ensuring the safety of personnel when loading containers.

[0071] Furthermore, it also includes a guide plate anti-pressure safety detection unit, which is connected to the data acquisition unit. The guide plate anti-pressure safety detection unit determines whether the relative position of the truck pallet and the container is reasonable based on the first scan result and the second scan result, determines whether the relative position of the truck pallet and the container is reasonable based on the third set of images, and determines whether the relative position of the truck pallet and the container is reasonable based on the second set of images, thereby ensuring the accuracy of the truck pallet position and preventing container breakage during container placement.

[0072] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the claims.

[0073] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A box placement safety system based on multi-sensor fusion, characterized in that, include: Multiple lidar sensors are used, with the first set of lidar sensors mounted on the beam above the lane lines and the second set of lidar sensors mounted on the truck bed. Multiple cameras are used, with the first set of cameras mounted on the main beam above the lane line, the second set of cameras mounted on the crane, and the third set of cameras mounted on the side of the trolley. The first set of lidar sensors is used to scan trucks, cranes, and storage yards to obtain the first scan results; The second set of lidar sensors is used to scan the truck pallet to obtain a second scan result; The first set of cameras is used to acquire a first set of images, which are images of the truck cab. The second set of cameras is used to acquire a second set of images, which are images of truck cabs, trucks, cranes, and storage yards. The third set of cameras is used to acquire a third set of images, which are images of the truck cab, truck pallet, and container. A data acquisition unit is used to acquire the first scan result, the second scan result, the first set of images, the second set of images, and the third set of images; The front anti-pinch safety detection unit is connected to the data acquisition unit. The front anti-pinch safety detection unit determines whether there are obstacles above the truck bed based on the first scan result and the second scan result, determines whether the truck head has entered under the container based on the third set of images, determines whether the truck head is intact based on the second set of images, and determines whether the position of the truck head is reasonable based on the first set of images. A foreign object safety detection unit is connected to the data acquisition unit. The foreign object safety detection unit determines whether there is a foreign object on the truck bed based on the first scan result, the second scan result and the third set of images. The control unit is connected to the front anti-pressure safety detection unit and the vehicle panel foreign object safety detection unit respectively, and issues warning reminders or operation stop signals based on the judgment results of the front anti-pressure safety detection unit and the vehicle panel foreign object safety detection unit.

2. The box placement safety system based on multi-sensor fusion according to claim 1, characterized in that, It also includes a personnel and foreign object safety detection unit, which is connected to the data acquisition unit. The personnel and foreign object safety detection unit determines whether there are personnel or foreign objects in the gap between the truck and the spreader, or between the truck and the yard, based on the first scanning result. It determines whether there are personnel or foreign objects between the truck and the spreader, or between the truck and the yard, based on the second set of images. It also determines whether there are personnel or foreign objects around the truck cab based on the first set of images.

3. The box placement safety system based on multi-sensor fusion according to claim 1, characterized in that, It also includes a guide plate anti-pressure safety detection unit, which is connected to the data acquisition unit. The guide plate anti-pressure safety detection unit determines whether the relative position of the truck pallet and the container is reasonable based on the first scan result and the second scan result, determines whether the relative position of the truck pallet and the container is reasonable based on the third set of images, and determines whether the relative position of the truck pallet and the container is reasonable based on the second set of images.

4. The box placement safety system based on multi-sensor fusion according to claim 1, characterized in that, The first group of lidar sensors and the second group of lidar sensors are area array lidar.

5. The box placement safety system based on multi-sensor fusion according to claim 1, characterized in that, The first group of cameras, the second group of cameras, and the third group of cameras are either monocular cameras or binocular cameras.

6. The box placement safety system based on multi-sensor fusion according to claim 1, characterized in that, It also includes a display unit, which is connected to the control unit, and the display unit is used to display warning reminders or work stop information.

7. The box placement safety system based on multi-sensor fusion according to claim 1, characterized in that, It also includes a voice broadcast unit, which is connected to the control unit, and the voice broadcast unit is used to broadcast warnings or reminders or to stop the operation.

8. The box placement safety system based on multi-sensor fusion according to claim 1, characterized in that, The first group of lidar sensors includes at least two lidar sensors, and the second group of lidar sensors includes at least five lidar sensors.

9. The box placement safety system based on multi-sensor fusion according to claim 1, characterized in that, The first group of cameras includes at least one camera, the second group of cameras includes at least four cameras, and the third group of cameras includes at least two cameras.

Images