Real-time 3D modeling crane operation automatic collision avoidance system and method based on multi-detection technology fusion

Abstract

The present invention discloses a real-time three-dimensional modeling crane operation automatic collision avoidance system based on the fusion of multiple detection technologies, which is used to enable the crane to transport the suspended object from a starting position to an end position along a preset transmission path. The preset transmission path is divided into a plurality of transmission sub-areas from the starting position to the end position in sequence. The automatic collision avoidance system comprises a first laser radar sensor, a second laser radar sensor, a third laser radar sensor and a video detection camera arranged at different fixed positions on the boom of the crane; the positions of the first laser radar sensor, the second laser radar sensor and the third laser radar sensor are arranged so that the scanning ranges of the three cover the next transmission sub-area during the movement of the boom and there is an overlapping area, and the visual range of the video detection camera covers the next transmission sub-area during the movement of the boom.

Description

Technical Field

[0001] The present invention relates to the technical field of hoisting, and in particular to a real-time three-dimensional modeling crane operation automatic collision avoidance system and method based on the fusion of multiple detection technologies. Background Art

[0002] At present, the cranes commonly used on offshore platforms are generally offshore pedestal cranes. However, due to their long booms, wide operating coverage radius, large amplitude range (15°~80°), and 360° rotation range, as well as the large number of well repairing equipment on the top deck of offshore platforms, operators are restricted by the operating environment and cannot accurately judge the relative position between the crane and obstacles at certain times and in certain areas. Therefore, the booms of offshore pedestal cranes are prone to collisions with other equipment, resulting in safety accidents, which in turn cause accidents and paralysis of the crane. Summary of the invention

[0003] The purpose of the present invention is to provide a real-time three-dimensional modeling crane operation automatic collision avoidance system and method based on the fusion of multiple detection technologies, which can effectively solve the above-mentioned technical problems existing in the prior art.

[0004] On the one hand, an embodiment of the present application discloses a real-time three-dimensional modeling crane operation automatic collision avoidance system based on the fusion of multiple detection technologies, which is used to enable the crane to transport the suspended object from the starting position to the end position according to a preset transmission path, and the preset transmission path is divided into multiple transmission sub-areas from the starting position to the end position in sequence. The automatic collision avoidance system includes a first laser radar sensor, a second laser radar sensor, a third laser radar sensor and a video detection camera arranged at different fixed positions on the boom of the crane; the positions of the first laser radar sensor, the second laser radar sensor and the third laser radar sensor are arranged so that the scanning ranges of the three cover the next transmission sub-area during the movement of the boom and there is an overlapping area, and the visual range of the video detection camera covers the next transmission sub-area during the movement of the boom;

[0005] When the hanging object is at the starting position or any transmission sub-area, radar data of the next transmission sub-area is collected by the first laser radar sensor, the second laser radar sensor, and the third laser radar sensor respectively, and video data of the next transmission sub-area is collected by the video detection camera;

[0006] Performing a first data fusion on each frame of radar data collected by the first laser radar sensor, the second laser radar sensor, and the third laser radar sensor at the same time to obtain fused target radar data;

[0007] When it is determined that there is an obstacle in the target radar data, the position and size information of the obstacle is obtained, and based on the position and size information of the obstacle, it is determined whether the obstacle falls within the preset transmission path of the next transmission sub-area;

[0008] If yes, the obstacle radar data in the target radar data is fused for the second time with the obstacle image data identified by the video data of the corresponding frame at the same time, to obtain the fused target fusion data;

[0009] The target fusion data is identified by using a pre-trained image recognition system to identify the type of the obstacle, and a collision prediction calculation is performed for the next transmission sub-area based on the type, position, size information of the obstacle and the outer dimensions of the hanging object. When the collision prediction calculation result shows that a collision will occur along the preset transmission path of the next transmission sub-area, a collision warning signal is output, and based on the type, position, size information of the obstacle and the outer dimensions of the hanging object, the preset transmission path of the next transmission sub-area is adjusted to avoid a collision; wherein the outer dimensions of the hanging object are obtained by pre-measuring and inputting the outer dimensions of the hanging object.

[0010] As an improvement of the above solution, the same moment means that the time difference between the acquired radar frames or between the acquired radar frames and the video frames is less than or equal to the time threshold.

[0011] As an improvement on the above scheme, when adjusting the preset transmission path of the next transmission sub-area, the end point position of the preset transmission path of the next transmission sub-area is kept unchanged while ensuring that no collision occurs; if the end point position of the preset transmission path of the next transmission sub-area is adjusted, then when the suspended object enters the next transmission sub-area, the starting position of the preset transmission path of the next transmission sub-area is changed to the end point position of the preset transmission path of the next transmission sub-area after the adjustment, and the preset transmission path of the next transmission sub-area is predicted and calculated and adjusted in combination with the data collected by the first lidar sensor, the second lidar sensor, the third lidar sensor and the video detection camera.

[0012] As an improvement of the above scheme, when the obstacle is judged to be a dynamic obstacle based on the position information of the obstacle in the target fusion data of each frame of the next transmission sub-area, its movement trend is estimated according to the motion trajectory of the dynamic obstacle, and the preset transmission path of the next transmission sub-area is adjusted in combination with the movement trend.

[0013] On the other hand, an embodiment of the present application discloses a real-time three-dimensional modeling crane operation automatic collision avoidance method based on the fusion of multiple detection technologies, which is applicable to the automatic collision avoidance system to enable the crane to transport the suspended object from the starting position to the end position according to a preset transmission path, and the preset transmission path is divided into multiple transmission sub-areas from the starting position to the end position in sequence, and the automatic collision avoidance system includes a first laser radar sensor, a second laser radar sensor, a third laser radar sensor and a video detection camera arranged at different fixed positions on the boom of the crane; the positions of the first laser radar sensor, the second laser radar sensor and the third laser radar sensor are arranged so that the scanning ranges of the three cover the next transmission sub-area during the movement of the boom and there is an overlapping area, and the visual range of the video detection camera covers the next transmission sub-area during the movement of the boom; wherein, the real-time three-dimensional modeling crane operation automatic collision avoidance method based on the fusion of multiple detection technologies includes the steps of:

[0014] S1. When the hanging object is at the starting position or any transmission sub-area, radar data of the next transmission sub-area is collected by the first laser radar sensor, the second laser radar sensor, and the third laser radar sensor respectively, and video data of the next transmission sub-area is collected by the video detection camera;

[0015] S2, performing a first data fusion on each frame of radar data collected by the first laser radar sensor, the second laser radar sensor, and the third laser radar sensor at the same time to obtain fused target radar data;

[0016] S3, when it is determined that there is an obstacle in the target radar data, obtaining the position and size information of the obstacle, and determining whether the obstacle falls within a preset transmission path of the next transmission sub-area based on the position and size information of the obstacle;

[0017] S4, if yes, then perform a second data fusion on the obstacle radar data in the target radar data and the obstacle image data identified in the video data of the corresponding frame at the same time to obtain fused target fusion data;

[0018] S5. Use a pre-trained image recognition system to identify the target fusion data, thereby identifying the type of the obstacle, and perform a collision prediction calculation for the next transmission sub-area based on the type, position, size information of the obstacle and the outer dimensions of the hanging object. When the collision prediction calculation result shows that a collision will occur along the preset transmission path of the next transmission sub-area, a collision warning signal is output, and based on the type, position, size information of the obstacle and the outer dimensions of the hanging object, the preset transmission path of the next transmission sub-area is adjusted to avoid a collision; wherein the outer dimensions of the hanging object are obtained by pre-measuring and inputting the outer dimensions of the hanging object.

[0019] As an improvement of the above solution, the same moment means that the time difference between the acquired radar frames or between the acquired radar frames and the video frames is less than or equal to the time threshold.

[0020] As an improvement of the above-mentioned scheme, when adjusting the preset transmission path of the next transmission sub-area, the end point position of the preset transmission path of the next transmission sub-area is kept unchanged while ensuring that no collision occurs; if the end point position of the preset transmission path of the next transmission sub-area is adjusted, then when the suspended object enters the next transmission sub-area, the starting position of the preset transmission path of the next transmission sub-area is changed to the end point position of the preset transmission path of the next transmission sub-area after the adjustment, and the preset transmission path of the next transmission sub-area is adjusted in combination with the data collected by the first lidar sensor, the second lidar sensor, the third lidar sensor and the video detection camera.

[0021] As an improvement of the above scheme, when the obstacle is judged to be a dynamic obstacle based on the position information of the obstacle in the target fusion data of each frame of the next transmission sub-area, its movement trend is estimated according to the motion trajectory of the dynamic obstacle, and the collision prediction calculation and adjustment are performed on the preset transmission path of the next transmission sub-area in combination with the motion trend.

[0022] On the other hand, an embodiment of the present application discloses an electronic device, which includes a processor and a memory, wherein the memory is used to store a computer program, the computer program includes program instructions, and the processor is configured to execute the above-mentioned real-time three-dimensional modeling crane operation automatic collision avoidance method based on the fusion of multiple detection technologies.

[0023] In another aspect, the present application discloses a computer program product or a computer program, wherein the computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the above-mentioned real-time three-dimensional modeling crane operation automatic collision avoidance method based on multi-detection technology fusion.

[0024] In summary, the embodiment of the present invention provides a real-time three-dimensional modeling crane operation automatic collision avoidance system and method based on the fusion of multiple detection technologies, which is designed in multiple directions by designing multiple laser radar sensors on the boom. In addition, it is also combined with a video detection camera to achieve 360-degree monitoring. In this way, not only can the distance or size of the obstacle be determined by the laser radar sensor, but it can also be combined with video detection to clearly know what the obstacle is, which is better for avoiding obstacles. Secondly, the present invention uses a crane to transport the hanging object from the starting position to the end position according to a preset transmission path, and divides the preset transmission path from the starting position to the end position into multiple transmission sub-areas in sequence, and then collects and obtains the data of the next transmission sub-area during the movement of the boom through the first laser radar sensor, the second laser radar sensor, the third laser radar sensor and the video detection camera, and obtains the data of each frame of radar data collected by the first laser radar sensor, the second laser radar sensor and the third laser radar sensor at the same time. The fused target radar data is obtained, and when it is determined that there is an obstacle in the target radar data, the position and size information of the obstacle is obtained, and based on the position and size information of the obstacle, it is determined whether the obstacle falls into the preset of the next transmission sub-area. If so, the obstacle radar data in the target radar data is fused for the second time with the obstacle image data identified in the video data of the corresponding frame at the same time to obtain the fused target fusion data; the target fusion data is identified by a pre-trained image recognition system to identify the type of the obstacle, and the collision prediction calculation of the next transmission sub-area is performed based on the type, position, size information of the obstacle and the outer dimensions of the hanging object. When the collision prediction calculation result shows that a collision will occur along the preset transmission path of the next transmission sub-area, a collision warning signal is output, and the preset transmission path of the next transmission sub-area is adjusted based on the type, position, size information of the obstacle and the outer dimensions of the hanging object to avoid collision. Therefore, the crane can effectively and safely transport the hanging object to the terminal position. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solution of the present invention, the drawings required for use in the implementation mode will be briefly introduced below. Obviously, the drawings described below are only some implementation modes of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0026] Figure 1 A structural block diagram of a real-time 3D modeling crane operation automatic collision avoidance system based on multi-detection technology fusion provided in an embodiment of the present invention.

[0027] Figure 2A flowchart of an automatic collision avoidance method for crane operation based on real-time three-dimensional modeling and fusion of multiple detection technologies provided in a preferred embodiment of the present invention.

[0028] Figure 3 Schematic diagram of the structure of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0030] The following will introduce in detail the real-time three-dimensional modeling crane operation automatic collision avoidance system based on the fusion of multiple detection technologies proposed in the embodiment of the present application in conjunction with the accompanying drawings.

[0031] refer to Figure 1 An embodiment of the present invention provides a real-time three-dimensional modeling crane operation automatic collision avoidance system based on the fusion of multiple detection technologies, which is used to enable the crane to transport the suspended objects from the starting position to the end position according to a preset transmission path, wherein the preset transmission path is divided into multiple transmission sub-areas from the starting position to the end position in sequence.

[0032] Specifically, Figure 1 As shown, the automatic collision avoidance system includes a first laser radar sensor 11, a second laser radar sensor 12, a third laser radar sensor 13 and a video detection camera 14 arranged at different fixed positions on the boom of the crane. The positions of the first laser radar sensor 11, the second laser radar sensor 12 and the third laser radar sensor 13 make their scanning ranges cover the next transmission sub-area during the movement of the boom and there is an overlapping area, and the visual range of the video detection camera 14 covers the next transmission sub-area during the movement of the boom.

[0033] It can be understood that the first laser radar sensor 11, the second laser radar sensor 12, the third laser radar sensor 13 and the video detection camera 14 are arranged at fixed positions in different directions on the boom to achieve 360-degree monitoring. In addition, the relative positions of the first laser radar sensor 11, the second laser radar sensor 12, the third laser radar sensor 13 and the video detection camera 14 remain unchanged. Moreover, the first laser radar sensor 11, the second laser radar sensor 12, and the third laser radar sensor 13 cover the next transmission sub-area during the movement of the boom and there is an overlapping area. After the first data fusion, the target radar data obtained is fused for the overlapping part and the non-overlapping part is spliced, thereby reducing the range of the laser detection blind area.

[0034] Understandable, such as Figure 1 As shown, the automatic collision avoidance system also includes a background control device 15, which is used to receive and process the data collected by the first laser radar sensor 11, the second laser radar sensor 12, the third laser radar sensor 13 and the video detection camera 14 (protection fusion, etc.), and control the corresponding functional components on the crane to perform corresponding operations according to the data processing results to drive the boom to move the suspended object. It can be understood that the corresponding functional components on the crane are familiar to existing technicians, and the description is omitted here.

[0035] The working principle and process of the real-time 3D modeling crane operation automatic collision avoidance system based on multi-detection technology fusion provided by the embodiment of the present invention are described in detail below:

[0036] When the suspended object is at the starting position or any transmission sub-area, the radar data of the next transmission sub-area is collected by the first laser radar sensor 11, the second laser radar sensor 12, and the third laser radar sensor 13, and the video data of the next transmission sub-area is collected by the video detection camera. That is, when the suspended object is at the starting position of the preset transmission path, the radar data of the first transmission sub-area is collected by the first laser radar sensor 11, the second laser radar sensor 12, and the third laser radar sensor 13, and the video data of the first transmission sub-area is collected by the video detection camera 14; when the suspended object is at the first transmission sub-area, the radar data of the second transmission sub-area is collected by the first laser radar sensor 11, the second laser radar sensor 12, and the third laser radar sensor 13, and the video data of the second transmission sub-area is collected by the video detection camera 14, and so on, until the suspended object is transported to the terminal position of the preset transmission path. It can be understood that the data collected by the first laser radar sensor 11, the second laser radar sensor 12, the third laser radar sensor 13 and the video detection camera 14 are sent to the control device 15 in real time for data processing. It is understandable that since the boom drives the hanging object to move from the starting position of the preset transmission path, the first laser radar sensor 11, the second laser radar sensor 12, the third laser radar sensor 13 and the video detection camera 14 arranged at fixed positions in different directions on the boom move with the boom and collect data all the time, and send the collected data to the control device 15 in real time, that is, the control device 15 processes the received data by area (sub-area). It is understandable that when the hanging object is at the starting position or any transmission sub-area, the data of two or even more transmission sub-areas can be collected by the first laser radar sensor 11, the second laser radar sensor 12, the third laser radar sensor 13 and the video detection camera 14, as long as the visual range of the first laser radar sensor 11, the second laser radar sensor 12, the third laser radar sensor 13 and the video detection camera 14 can be covered and the acquisition frame rate meets the conditions.

[0037] Specifically, when the control device 15 receives data from the first laser radar sensor 11, the second laser radar sensor 12, the third laser radar sensor 13 and the video detection camera 14, the processing process is as follows:

[0038] First, each frame of radar data collected by the first laser radar sensor 11, the second laser radar sensor 12, and the third laser radar sensor 13 at the same time is subjected to a first data fusion to obtain fused target radar data; then, the fused target radar data is identified, and when it is determined that there is an obstacle in the target radar data, the position and size information of the obstacle is obtained, and based on the position and size information of the obstacle, it is determined whether the obstacle falls into the preset transmission path of the next transmission sub-area; if so, the obstacle radar data in the target radar data is subjected to a second data fusion with the obstacle image data identified in the video data of the corresponding frame at the same time to obtain fused target fusion data; finally, the target fusion data is identified by using a pre-trained image recognition system to identify the type of the obstacle, and a collision prediction calculation is performed for the next transmission sub-area based on the type, position, size information of the obstacle and the outer dimensions of the hanging object; when the collision prediction calculation result shows that a collision will occur if the output is output along the preset transmission path of the next transmission sub-area, a collision warning signal is output, and based on the type, position, size information of the obstacle and the outer dimensions of the hanging object, the preset transmission path of the next transmission sub-area is adjusted to avoid a collision. The outer dimensions of the hanging object are obtained by measuring and inputting the outer dimensions of the hanging object in advance, and the collision prediction of obstacles is calculated for the different outer dimensions of the hanging object.

[0039] It can be understood that the same moment means that the time difference between the acquired radar frames or between the acquired radar frames and the video frames is less than or equal to the time threshold.

[0040] A preferred solution can be adopted, that when adjusting the preset transmission path of the next transmission sub-area, the end point position of the preset transmission path of the next transmission sub-area is kept unchanged while ensuring that no collision occurs; if the end point position of the preset transmission path of the next transmission sub-area is adjusted, then when the suspended object enters the next transmission sub-area, the starting position of the preset transmission path of the next transmission sub-area is changed to the end point position of the preset transmission path of the next transmission sub-area after the adjustment, and the preset transmission path of the next transmission sub-area is predicted and calculated and adjusted based on the data collected by the first lidar sensor, the second lidar sensor, the third lidar sensor and the video detection camera.

[0041] In addition, when the obstacle is judged to be a dynamic obstacle based on the position information of the obstacle in the target fusion data of each frame of the next transmission sub-area, its movement trend is estimated according to the movement trajectory of the dynamic obstacle, and the preset transmission path of the next transmission sub-area is adjusted in combination with the movement trend.

[0042] refer to Figure 2 , the embodiment of the present application discloses a real-time 3D modeling crane operation automatic collision avoidance method based on the fusion of multiple detection technologies, which is applicable to the automatic collision avoidance system to enable the crane to transport the suspended object from the starting position to the end position according to a preset transmission path, and the preset transmission path is divided into multiple transmission sub-areas from the starting position to the end position in sequence, and the automatic collision avoidance system includes a first laser radar sensor, a second laser radar sensor, a third laser radar sensor and a video detection camera arranged at different fixed positions on the boom of the crane; the positions of the first laser radar sensor, the second laser radar sensor and the third laser radar sensor are set so that the scanning ranges of the three cover the next transmission sub-area during the movement of the boom and there is an overlapping area, and the visual range of the video detection camera covers the next transmission sub-area during the movement of the boom; wherein, the real-time 3D modeling crane operation automatic collision avoidance method based on the fusion of multiple detection technologies includes the steps of:

[0043] S1. When the hanging object is at the starting position or any transmission sub-area, radar data of the next transmission sub-area is collected by the first laser radar sensor, the second laser radar sensor, and the third laser radar sensor respectively, and video data of the next transmission sub-area is collected by the video detection camera;

[0044] S2, performing a first data fusion on each frame of radar data collected by the first laser radar sensor, the second laser radar sensor, and the third laser radar sensor at the same time to obtain fused target radar data;

[0045] S3, when it is determined that there is an obstacle in the target radar data, obtaining the position and size information of the obstacle, and determining whether the obstacle falls within a preset transmission path of the next transmission sub-area based on the position and size information of the obstacle;

[0046] S4, if yes, then perform a second data fusion on the obstacle radar data in the target radar data and the obstacle image data identified in the video data of the corresponding frame at the same time to obtain fused target fusion data;

[0047] S5. Use a pre-trained image recognition system to identify the target fusion data, thereby identifying the type of the obstacle, and perform a collision prediction calculation for the next transmission sub-area based on the type, position, size information of the obstacle and the outer dimensions of the hanging object. When the collision prediction calculation result shows that a collision will occur along the preset transmission path of the next transmission sub-area, a collision warning signal is output, and based on the type, position, size information of the obstacle and the outer dimensions of the hanging object, the preset transmission path of the next transmission sub-area is adjusted to avoid a collision; wherein the outer dimensions of the hanging object are obtained by pre-measuring and inputting the outer dimensions of the hanging object.

[0048] As an improvement of the above solution, the same moment means that the time difference between the acquired radar frames or between the acquired radar frames and the video frames is less than or equal to the time threshold.

[0049] As an improvement of the above-mentioned scheme, when adjusting the preset transmission path of the next transmission sub-area, the end point position of the preset transmission path of the next transmission sub-area is kept unchanged while ensuring that no collision occurs; if the end point position of the preset transmission path of the next transmission sub-area is adjusted, then when the suspended object enters the next transmission sub-area, the starting position of the preset transmission path of the next transmission sub-area is changed to the end point position of the preset transmission path of the next transmission sub-area after the adjustment, and the preset transmission path of the next transmission sub-area is adjusted in combination with the data collected by the first lidar sensor, the second lidar sensor, the third lidar sensor and the video detection camera.

[0050] As an improvement of the above scheme, when the obstacle is judged to be a dynamic obstacle based on the position information of the obstacle in the target fusion data of each frame of the next transmission sub-area, its movement trend is estimated according to the motion trajectory of the dynamic obstacle, and the collision prediction calculation and adjustment are performed on the preset transmission path of the next transmission sub-area in combination with the motion trend.

[0051] The specific implementation method of the real-time three-dimensional modeling crane operation automatic collision avoidance method based on the fusion of multiple detection technologies in this embodiment can refer to the description of the real-time three-dimensional modeling crane operation automatic collision avoidance system based on the fusion of multiple detection technologies in the above embodiment, which will not be repeated here.

[0052] like Figure 3 As shown, an embodiment of the present invention provides an electronic device 300, including a memory 310 and a processor 320, wherein the memory 310 is used to store one or more computer instructions, and the processor 320 is used to call and execute the one or more computer instructions, so as to realize any of the above-mentioned real-time three-dimensional modeling crane operation automatic collision avoidance methods based on the fusion of multiple detection technologies.

[0053] That is, the electronic device 300 includes: a processor 320 and a memory 310, in which computer program instructions are stored, wherein when the computer program instructions are executed by the processor, the processor 320 executes any of the above-mentioned real-time three-dimensional modeling crane operation automatic collision avoidance methods based on the fusion of multiple detection technologies.

[0054] Furthermore, if Figure 3 As shown, the electronic device 300 further includes a network interface 330 , an input device 340 , a hard disk 350 , and a display device 360 ​​.

[0055] The above-mentioned interfaces and devices can be interconnected through a bus architecture. The bus architecture can be a bus and bridge that can include any number of interconnected buses. Specifically, one or more central processing units (CPUs) represented by the processor 320 and various circuits of one or more memories represented by the memory 310 are connected together. The bus architecture can also connect various other circuits such as peripheral devices, voltage regulators, and power management circuits together. It can be understood that the bus architecture is used to achieve connection and communication between these components. In addition to the data bus, the bus architecture also includes a power bus, a control bus, and a status signal bus, which are all well known in the art, so they will not be described in detail herein.

[0056] The network interface 330 can be connected to a network (such as the Internet, a local area network, etc.), obtain relevant data from the network, and save it in the hard disk 350.

[0057] The input device 340 can receive various instructions input by the operator and send them to the processor 320 for execution. The input device 340 can include a keyboard or a pointing device (for example, a mouse, a trackball, a touch pad or a touch screen, etc.).

[0058] The display device 360 ​​can display the result obtained by the processor 320 executing the instruction.

[0059] The memory 310 is used to store programs and data necessary for the operation of the operating system, as well as data such as intermediate results during the calculation process of the processor 320.

[0060] It is understood that the memory 310 in the embodiment of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. The memory 310 of the apparatus and method described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0061] In some implementations, the memory 310 stores the following elements, executable modules or data structures, or a subset thereof, or an extended set thereof: an operating system 311 and application programs 312 .

[0062] The operating system 311 includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., which are used to implement various basic services and process hardware-based tasks. The application 312 includes various application programs, such as a browser, etc., which are used to implement various application services. The program for implementing the method of the embodiment of the present invention can be included in the application 312.

[0063] The processor 320, when calling and executing the application and data stored in the memory 310, specifically, the program or instruction stored in the application 312, when the hanging object is in the starting position or any transmission sub-area, respectively collects radar data of the next transmission sub-area through the first laser radar sensor, the second laser radar sensor, and the third laser radar sensor, and collects video data of the next transmission sub-area through the video detection camera; performs a first data fusion on each frame of radar data collected by the first laser radar sensor, the second laser radar sensor, and the third laser radar sensor at the same time to obtain fused target radar data; obtains the position and size information of the obstacle when it is determined that there is an obstacle in the target radar data, and determines whether the obstacle falls into the preset transmission sub-area of ​​the next transmission sub-area based on the position and size information of the obstacle path; if so, the obstacle radar data in the target radar data is fused for the second time with the obstacle image data identified in the video data of the corresponding frame at the same time to obtain the fused target fusion data; the target fusion data is identified by using a pre-trained image recognition system to identify the type of the obstacle, and a collision prediction calculation is performed for the next transmission sub-area based on the type, position, size information of the obstacle and the outer dimensions of the hanging object; when the collision prediction calculation result shows that a collision will occur if the output is output along the preset transmission path of the next transmission sub-area, a collision warning signal is output, and based on the type, position, size information of the obstacle and the outer dimensions of the hanging object, the preset transmission path of the next transmission sub-area is adjusted to avoid a collision; wherein the outer dimensions of the hanging object are obtained by pre-measuring and inputting the outer dimensions of the hanging object.

[0064] The real-time three-dimensional modeling crane operation automatic collision avoidance method based on multi-detection technology fusion disclosed in the above embodiment of the present invention can be applied to the processor 320, or implemented by the processor 320. The processor 320 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 320. The above processor 320 can be a general processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a ready-made programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, and can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of the present invention. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiment of the present invention can be directly embodied as a hardware decoding processor to be executed, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 310, and the processor 320 reads the information in the memory 310 and completes the steps of the above method in combination with its hardware.

[0065] It is understood that the embodiments described herein can be implemented in hardware, software, firmware, middleware, microcode or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), general purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described in the present application or a combination thereof.

[0066] For software implementation, the techniques described herein can be implemented by modules (eg, procedures, functions, etc.) that perform the functions described herein. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or outside the processor.

[0067] Specifically, the processor 320 is also used to read the computer program and execute any of the above-mentioned methods for automatic collision avoidance of crane operations based on real-time three-dimensional modeling and fusion of multiple detection technologies.

[0068] The present application also provides a computer-readable storage medium, which stores a computer program. The computer program includes program instructions. When the program instructions are executed by a processor, the processor executes the above method, such as executing the method executed by the above electronic device, which is not repeated here.

[0069] Optionally, the storage medium involved in the present application, such as computer-readable storage medium, may be non-volatile or volatile.

[0070] Optionally, the computer-readable storage medium may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to the use of blockchain nodes, etc. Among them, the blockchain referred to in this application is a new application mode of computer technologies such as distributed data storage, point-to-point transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods, each of which contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. The blockchain may include the underlying blockchain platform, the platform product service layer, and the application service layer.

[0071] It should be noted that, for the above-mentioned various method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this application.

[0072] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0073] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may be physically included separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0074] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform some steps of the sending and receiving method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store program codes.

[0075] Compared with the prior art, the embodiment of the present invention provides a real-time three-dimensional modeling crane operation automatic collision avoidance system and method based on the fusion of multiple detection technologies, which is designed in multiple directions by designing multiple laser radar sensors on the boom. In addition, it is also combined with a video detection camera to achieve 360-degree monitoring. In this way, not only can the distance or size of the obstacle be determined by the laser radar sensor, but it can also be combined with video detection to clearly know what the obstacle is, which is better for avoiding obstacles. Secondly, the present invention uses a crane to transport the hanging object from the starting position to the end position according to a preset transmission path, and divides the preset transmission path from the starting position to the end position into multiple transmission sub-areas in sequence, and then collects and obtains the data of the next transmission sub-area during the movement of the boom through the first laser radar sensor, the second laser radar sensor, the third laser radar sensor and the video detection camera, and obtains the data of each frame of radar data collected by the first laser radar sensor, the second laser radar sensor and the third laser radar sensor at the same time. The fused target radar data is obtained, and when it is determined that there is an obstacle in the target radar data, the position and size information of the obstacle is obtained, and based on the position and size information of the obstacle, it is determined whether the obstacle falls into the preset of the next transmission sub-area. If so, the obstacle radar data in the target radar data is fused for the second time with the obstacle image data identified in the video data of the corresponding frame at the same time to obtain the fused target fusion data; the target fusion data is identified by a pre-trained image recognition system to identify the type of the obstacle, and the collision prediction calculation of the next transmission sub-area is performed based on the type, position, size information of the obstacle and the outer dimensions of the hanging object. When the collision prediction calculation result shows that a collision will occur along the preset transmission path of the next transmission sub-area, a collision warning signal is output, and the preset transmission path of the next transmission sub-area is adjusted based on the type, position, size information of the obstacle and the outer dimensions of the hanging object to avoid collision. Therefore, the crane can effectively and safely transport the hanging object to the terminal position.

[0076] The above disclosures are only some preferred embodiments of the present invention, which certainly cannot be used to limit the scope of rights of the present invention. Ordinary technicians in this field can understand that all or part of the processes of the above embodiments and equivalent changes made according to the claims of the present invention are still within the scope of the invention.

Claims

1. A real-time 3D modeling crane operation automatic collision avoidance system based on the fusion of multiple detection technologies, used to enable the crane to transport the suspended object from the starting position to the end position according to a preset transmission path, characterized in that: The preset transmission path is divided into a plurality of transmission sub-areas from the starting position to the end position in sequence, and the automatic collision avoidance system includes a first laser radar sensor, a second laser radar sensor, a third laser radar sensor and a video detection camera arranged at different fixed positions on the boom of the crane; the positions of the first laser radar sensor, the second laser radar sensor and the third laser radar sensor are arranged so that the scanning ranges of the three cover the next transmission sub-area during the movement of the boom and there is an overlapping area, and the visual range of the video detection camera covers the next transmission sub-area during the movement of the boom; When the hanging object is at the starting position or any transmission sub-area, radar data of the next transmission sub-area is collected by the first laser radar sensor, the second laser radar sensor, and the third laser radar sensor respectively, and video data of the next transmission sub-area is collected by the video detection camera; Performing a first data fusion on each frame of radar data collected by the first laser radar sensor, the second laser radar sensor, and the third laser radar sensor at the same time to obtain fused target radar data; When it is determined that there is an obstacle in the target radar data, the position and size information of the obstacle is obtained, and based on the position and size information of the obstacle, it is determined whether the obstacle falls within the preset transmission path of the next transmission sub-area; If yes, the obstacle radar data in the target radar data is fused for the second time with the obstacle image data identified by the video data of the corresponding frame at the same time, to obtain the fused target fusion data; The target fusion data is identified by using a pre-trained image recognition system to identify the type of the obstacle, and a collision prediction calculation is performed for the next transmission sub-area based on the type, position, size information of the obstacle and the outer dimensions of the hanging object. When the collision prediction calculation result shows that a collision will occur if the output is along the preset transmission path of the next transmission sub-area, a collision warning signal is output, and the preset transmission path of the next transmission sub-area is adjusted based on the type, position, size information of the obstacle and the outer dimensions of the hanging object to avoid a collision; wherein the outer dimensions of the hanging object are obtained by pre-measuring and inputting the outer dimensions of the hanging object; When the obstacle is judged to be a dynamic obstacle based on the position information of the obstacle in the target fusion data of each frame of the next transmission sub-area, its movement trend is estimated according to the movement trajectory of the dynamic obstacle, and the collision prediction calculation and adjustment are performed on the preset transmission path of the next transmission sub-area in combination with the movement trend.

2. The real-time 3D modeling crane operation automatic collision avoidance system based on multi-detection technology fusion according to claim 1 is characterized in that: The same moment means that the time difference between the acquired radar frames or between the acquired radar frames and the video frames is less than or equal to the time threshold.

3. The real-time 3D modeling crane operation automatic collision avoidance system based on multi-detection technology fusion according to claim 1 is characterized in that: When adjusting the preset transmission path of the next transmission sub-area, the end point position of the preset transmission path of the next transmission sub-area is kept unchanged while ensuring that no collision occurs; If the end position of the preset transmission path of the next transmission sub-area is adjusted, when the suspended object enters the next transmission sub-area, the starting position of the preset transmission path of the next transmission sub-area is changed to the end position of the preset transmission path of the next transmission sub-area after the adjustment, and the preset transmission path of the next transmission sub-area is adjusted in combination with the data collected by the first lidar sensor, the second lidar sensor, the third lidar sensor and the video detection camera.

4. A real-time three-dimensional modeling crane operation automatic collision avoidance method based on the fusion of multiple detection technologies, which is applicable to the automatic collision avoidance system to enable the crane to transport the suspended object from the starting position to the end position according to a preset transmission path, and the preset transmission path is divided into multiple transmission sub-areas from the starting position to the end position in sequence. The automatic collision avoidance system includes a first laser radar sensor, a second laser radar sensor, a third laser radar sensor and a video detection camera arranged at different fixed positions on the boom of the crane; the positions of the first laser radar sensor, the second laser radar sensor and the third laser radar sensor are set so that the scanning ranges of the three cover the next transmission sub-area during the movement of the boom and there is an overlapping area, and the visual range of the video detection camera covers the next transmission sub-area during the movement of the boom; it is characterized in that The method for automatic collision avoidance of crane operation based on real-time three-dimensional modeling and fusion of multiple detection technologies comprises the following steps: S1. When the hanging object is at the starting position or any transmission sub-area, radar data of the next transmission sub-area is collected by the first laser radar sensor, the second laser radar sensor, and the third laser radar sensor respectively, and video data of the next transmission sub-area is collected by the video detection camera; S2, performing a first data fusion on each frame of radar data collected by the first laser radar sensor, the second laser radar sensor, and the third laser radar sensor at the same time to obtain fused target radar data; S3, when it is determined that there is an obstacle in the target radar data, obtaining the position and size information of the obstacle, and determining whether the obstacle falls within a preset transmission path of the next transmission sub-area based on the position and size information of the obstacle; S4, if yes, then perform a second data fusion on the obstacle radar data in the target radar data and the obstacle image data identified in the video data of the corresponding frame at the same time to obtain fused target fusion data; S5. Use a pre-trained image recognition system to identify the target fusion data, thereby identifying the type of the obstacle, and perform collision prediction calculation for the next transmission sub-area based on the type, position, size information of the obstacle and the outer dimensions of the hanging object. When the collision prediction calculation result shows that a collision will occur if the output is along the preset transmission path of the next transmission sub-area, a collision warning signal is output, and the preset transmission path of the next transmission sub-area is adjusted based on the type, position, size information of the obstacle and the outer dimensions of the hanging object to avoid a collision; wherein the outer dimensions of the hanging object are obtained by pre-measuring and inputting the outer dimensions of the hanging object; Among them, when the obstacle is judged to be a dynamic obstacle based on the position information of the obstacle in the target fusion data of each frame of the next transmission sub-area, its movement trend is estimated according to the motion trajectory of the dynamic obstacle, and the collision prediction calculation and adjustment are performed on the preset transmission path of the next transmission sub-area in combination with the motion trend.

5. The method for automatic collision avoidance of crane operation based on real-time 3D modeling and fusion of multiple detection technologies according to claim 4 is characterized in that: The same moment means that the time difference between the acquired radar frames or between the acquired radar frames and the video frames is less than or equal to the time threshold.

6. The method for automatic collision avoidance of crane operation based on real-time 3D modeling and fusion of multiple detection technologies according to claim 4 is characterized in that: When adjusting the preset transmission path of the next transmission sub-area, the end point position of the preset transmission path of the next transmission sub-area is kept unchanged while ensuring that no collision occurs; If the end position of the preset transmission path of the next transmission sub-area is adjusted, when the suspended object enters the next transmission sub-area, the starting position of the preset transmission path of the next transmission sub-area is changed to the end position of the preset transmission path of the next transmission sub-area after the adjustment, and the preset transmission path of the next transmission sub-area is adjusted in combination with the data collected by the first lidar sensor, the second lidar sensor, the third lidar sensor and the video detection camera.

7. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory is used to store a computer program, the computer program comprises program instructions, and the processor is configured to call the program instructions to execute the method according to any one of claims 4 to 6.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor executes the method according to any one of claims 4 to 6.

Images