Ship unloader grab bucket anti-collision system based on three-dimensional space visual capture algorithm

Abstract

The invention provides a ship unloader grab bucket anti-collision system based on a three-dimensional space visual capture algorithm, which comprises a data acquisition module for monitoring and surveying a site, a data transmission module and a data modeling module, and is characterized in that three-dimensional map information of a cabin is complemented through the data acquisition module and the data modeling module; according to the system, the grab bucket is accurately controlled, meanwhile, full-link cooperation is achieved in the full-automatic operation process through the data transmission module, therefore, the system can be suitable for any ship type and cabin, mechanical carrying data and external position early warning are in high-degree coordination and uniformity, and the working efficiency is improved. Therefore, the operation signal and the fault information can be detected more quickly in the operation process of the system.

Description

Technical Field

[0001] The invention relates to the field of ship unloader grab buckets, and in particular to a ship unloader grab bucket anti-collision system based on a three-dimensional space visual capture algorithm. Background Art

[0002] The grab bucket of a ship unloader is a mechanical device used to load and unload ship cargo. It usually consists of a large crane and a grab bucket. The grab bucket is a clamp-like device that can be opened and closed to clamp, lift and dump cargo, such as coal, ore, grain, etc. The grab bucket of a ship unloader is usually used in ports, docks and other places to efficiently load and unload ship cargo, improve loading and unloading efficiency, and reduce manual labor;

[0003] The existing anti-collision measures for the grab bucket of ship unloaders mainly rely on the driver's observation and judgment of the grab bucket's position and movement trajectory during operation, as well as some semi-automatic anti-collision warning radars and simple mechanical anti-collision devices. However, the mechanical anti-collision device lacks intelligent monitoring and is unable to determine the grab bucket's position and detect its movement trajectory in real time. As a result, the mechanical anti-collision device can only detect the collision between the grab bucket and the fixed device, but has poor collision detection effect between the grab bucket and other objects. It is impossible to take timely measures to prevent further damage when a collision occurs. In addition, the information transmission method of the semi-automatic anti-collision warning radar is backward, the transmission efficiency is low, and the collision warning information transmission process is unstable, which makes it very easy for grab bucket collision accidents to occur due to untimely information reception. Summary of the invention

[0004] In view of this, the purpose of the present invention is to propose a ship unloader grab anti-collision system based on a three-dimensional space visual capture algorithm to solve the technical problems that the existing ship unloader grab anti-collision measures cannot determine the grab position and detect its movement trajectory in real time, and the information transmission is slow, resulting in the mechanical anti-collision device can only detect the collision between the grab and the fixing device, and the collision detection effect between the grab and other objects is poor.

[0005] Based on the above purpose, the present invention provides a ship unloader grab anti-collision system based on a three-dimensional space visual capture algorithm, comprising the following modules:

[0006] A data acquisition module for on-site monitoring and surveying, which uses a 3D radar detector and binocular monitoring equipment;

[0007] A data transmission module, which specifically includes an information transmission part between the survey equipment and the ship unloader and a data transmission part between the ship unloader and the central control server;

[0008] The data modeling module includes the digital twin 3D model part of the ship's cabin and the data sharing part of the central control server and the operating ship unloader.

[0009] Furthermore, the three-dimensional radar detector is used to scan the hatch data of the current operation, and to emit laser to detect the position information of the mechanical grab.

[0010] Furthermore, the specific application steps of the three-dimensional scanning radar are as follows:

[0011] S1: laser emission;

[0012] S2: laser scanning;

[0013] S3: Data processing: Calculate the three-dimensional coordinates of the target object;

[0014] S4: Data output: The processed data can be output to a computer or other device through a communication interface for further processing or application.

[0015] Furthermore, the binocular monitoring device is installed on the moving part of the ship unloader, and the binocular vision is used for image acquisition and image correction to eliminate distortion and error.

[0016] Furthermore, it is characterized in that the specific operation steps of the binocular monitoring device are as follows:

[0017] S1, dual-target positioning;

[0018] S2, image acquisition: obtain two images of the same scene through a binocular camera;

[0019] S3, image correction: preprocessing the acquired image;

[0020] S4. 3D reconstruction: obtain 3D information of the object.

[0021] Furthermore, the installation of the binocular survey camera and the 3D scanning radar is synchronously equipped with an outdoor client terminal device for realizing fixed-point communication.

[0022] Furthermore, the channel for data communication between the ship unloader and the central server arranged in the port central control command center adopts optical fiber to ensure the validity and accuracy of the data.

[0023] Furthermore, the establishment of the digital twin 3D model of the ship cabin includes the following steps:

[0024] S1, select the aerial image dataset;

[0025] S2, configure model parameters;

[0026] S3, basic training;

[0027] S4, sparse training;

[0028] S5. Channel pruning.

[0029] Beneficial effects of the present invention:

[0030] The present invention completes the three-dimensional map information of the cabin through the data scanning and acquisition module and the data modeling module, realizes precise control of the grab, and realizes full-link collaboration in the fully automatic operation process through the data transmission module. Therefore, the system can be applied to any ship type and cabin, and the mechanical carrying data and external position warning are in high coordination and unity. Therefore, the operation signal and fault information can be detected more quickly during the operation of the system, which solves the existing anti-collision measures for the grab of the ship unloader, and the technical problem that the grab position cannot be determined and its motion trajectory cannot be detected in real time, and the information transmission is slow, resulting in the mechanical anti-collision device can only detect the collision between the grab and the fixing device, and the collision detection effect between the grab and other objects is not good. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only for the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0032] Figure 1 It is a schematic diagram of the overall module of the system of the present invention;

[0033] Figure 2 It is a structural schematic diagram of the ship unloader of the present invention;

[0034] Figure 3 The working principle diagram of the three-dimensional radar detector of the present invention;

[0035] Figure 4 It is the monitoring principle diagram of the binocular monitoring device of the present invention;

[0036] Figure 5 It is a model network structure diagram of the present invention;

[0037] Figure 6 is the system learning rate function formula of the present invention;

[0038] Figure 7 The grab bucket anti-collision algorithm of the present invention;

[0039] The numbers in the figure are:

[0040] 1. Ship unloader; 11. Driver's cab; 12. Crane arm; 13. Three-dimensional radar detector; 14. Binocular monitoring equipment; 2. Grab bucket; 3. Ship; 31. Hatch opening. DETAILED DESCRIPTION

[0041] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific embodiments.

[0042] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the present invention should be understood by people with ordinary skills in the field to which the present invention belongs. The "first", "second" and similar words used in the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0043] To further understand the present invention, the present invention will be described in detail below with reference to the accompanying drawings.

[0044] Combination Figure 1 and 2 The present invention provides a grab anti-collision system for a ship unloader 1 based on a three-dimensional space visual capture algorithm, comprising the following modules:

[0045] A data scanning and acquisition module for monitoring and surveying the site, which is applied with a three-dimensional radar detector 13 and a binocular monitoring device 14, for accurately collecting the position data of the grab bucket 2 and the hull and taking real-time and accurate monitoring pictures of the server background, and collecting two-dimensional and three-dimensional information of the work site at the same time. The three-dimensional radar detector 13 is fixedly installed at the lower end of the driver's cab 11;

[0046] Through two solid-state three-dimensional radar detectors 13, one of which scans the data of the hatch opening 31 currently in operation, the shape data of the hatch opening 31 and the surface data of the ore layer of the open part of the hatch opening 31 can be measured. The data includes the X / Y / Z three-dimensional point cloud coordinates of the four corners of the hatch opening 31 and the three-dimensional point cloud coordinates of the surface of the ore layer of the open part of the hatch opening 31 (the cloud coordinates are used to determine the specific volume of the hatch opening 31 to be drawn later). The update of the data of the hatch opening 31 will be updated twice according to each operation cycle, namely the data after the grab 2 leaves the hatch opening 31, and the data before the grab 2 returns to the hatch opening 31. The update frequency is about once every 30 seconds. The updated data will be sent back to the onboard server PC in the form of an array through the Ethernet communication interface;

[0047] Another solid-state laser three-dimensional radar detector 13 can detect the position information of the mechanical grab 2, which is mainly achieved by the scanning and positioning function of the laser of the three-dimensional radar detector 13. The three-dimensional radar detector 13 can scan the surrounding environment in real time and generate three-dimensional point cloud data, which can be used to detect the position and posture of the grab 2. In this way, the movement and operation of the grab 2 can be accurately controlled to avoid collision with surrounding objects;

[0048] Combination Figure 3 The specific application steps of the three-dimensional radar detector 13 are as follows:

[0049] S1: Laser emission: Solid-state lasers generate laser light, which is processed by the optical system to form a laser beam with a certain shape and characteristics;

[0050] S2: Laser scanning: The laser beam passes through scanning elements such as MEMS micromirrors or OPA optical phased arrays to achieve vertical and horizontal scanning of the laser beam. Laser reception: The reflected laser beam is received by the receiver and converted into an electrical signal;

[0051] S3: Data processing: After the electrical signal is processed, the distance, speed and other information of the target object can be obtained, and the three-dimensional coordinates of the target object can be calculated by combining the vertical and horizontal position information of the target object obtained by the scanning element;

[0052] S4: Data output: The processed data can be output to a computer or other device through a communication interface for further processing or application.

[0053] The binocular monitoring device 14 is installed on the moving part of the ship unloader 1 (such as the boom 12 or the pusher rake, etc.). The binocular monitoring device 14 is specifically a dual 360° pan-tilt camera, which is used to perform binocular vision function to realize the whole cabin ground Figure 3 It not only collects dimensional information, but also synchronizes the AI ​​recognition model to conduct obstacle target survey, supplements the information of the inner wall of the cabin and the cargo pile on the inner wall, and makes up for the defect of incomplete cabin information of the old generation of ship unloader 1 fully automatic system, and completely solves the technical pain points of poor safety and low efficiency of the automatic link caused by incomplete cabin information of the old generation of ship unloader 1;

[0054] The binocular vision function first uses binocular positioning (see Figure x) to input two checkerboard images to obtain the intrinsic reference matrix K, relative rotation matrix R and translation matrix T of the two cameras. Then, the binocular camera is used for image acquisition to obtain two images of the same scene. Then, the image is corrected to eliminate distortion and error and ensure that the left and right image rows are aligned.

[0055] In the stereo matching stage, the system searches for the projection points of the same object in the left and right images, i.e., matching points. Finally, through 3D reconstruction, the 3D coordinates of the object are calculated using the pixel coordinates of the matching points and the internal and external parameters of the camera using the triangulation principle, thereby obtaining the 3D information of the object.

[0056] Combination Figure 4 , the specific operation steps of the binocular monitoring device 14 are as follows:

[0057] S1, dual-target positioning: input two chessboard images of image acquisition, output the internal parameter matrix K of the two cameras, and the relative R, T. The internal parameters include the focal length fx, fy, the horizontal and vertical pixel numbers cx, cy that differ between the center pixel coordinates of the image and the pixel coordinates of the image origin, etc. The external parameters include the rotation matrix R and the translation matrix T;

[0058] S2, image acquisition: acquiring two images of the same scene through the binocular monitoring device 14;

[0059] S3, Image correction: Preprocess the acquired image to eliminate distortion and error so that the lines of the left and right images are aligned. Stereo matching: Find the corresponding matching points in the left and right images, that is, find the projection points of the same object in the left and right images;

[0060] S4, 3D reconstruction: According to the pixel coordinates of the matching points and the internal and external parameters of the camera, the 3D coordinates of the object are calculated through the triangulation principle to obtain the 3D information of the object;

[0061] Since the three-dimensional map information of the cabin is completed, the new generation of grab 2 bucket control technology realizes full closed-loop control, which greatly improves the safety and reliability of grab 2 bucket reclaiming during fully automatic operation. It is an effective method to comprehensively improve the safety and reliability of fully automatic operation and ensure efficiency. We can accurately control the depth of the swing bucket by accurately controlling the bucket swing angle θ, thereby achieving the accuracy of bucket reclaiming. This is something that cannot be achieved by visual manual bucket reclaiming and open-loop automatic bucket reclaiming. Therefore, the safety of bucket reclaiming during fully automatic operation has made a qualitative leap;

[0062] The active collision avoidance algorithm of grab bucket 2 of ship unloader 1 is based on known information including cabin height, cabin opening 31 width, cabin opening 31 length, material height in cabin, and cabin inner wall depth. The height difference L between the cabin material surface and cabin opening 31 can be obtained by subtracting the material height from the cabin height, and the depth difference D can be obtained by subtracting the cabin buckle edge from the cabin inner wall depth. The hypotenuse T 2 =L 2 +Depth D 2 , we can get the maximum unloading angle θ of grab 2 (please refer to Figure 7 ).

[0063] A data transmission module, which specifically includes an information transmission part between the survey equipment and the ship unloader 1 and a data transmission part between the ship unloader 1 and the central control server;

[0064] Data transmission between survey equipment and ship unloader 1: The survey equipment is a binocular survey camera. The installation of the binocular survey camera and the three-dimensional radar detector 13 is synchronized with an outdoor 5G CPE (Customer Premise Equipment). The 5G CPE meets multiple wireless standards such as 5G NR / LTE / WCDMA and can access ISPs such as China Mobile / China Telecom / China Unicom. An outdoor 5GCPE is installed on the ship unloader 1 to form a private 5G LAN with the pusher and rake to achieve fixed-point communication. It can achieve stable and high-speed transmission of survey data, real-time playback of background monitoring images, and the analysis and deconstruction of workflow data by the background AI visual model in the first time;

[0065] Data transmission between ship unloader 1 and the background central server: There are two channels for data communication between ship unloader 1 and the central server set up in the port's central control and command center. The first is the airborne optical fiber, and the second is the 5G CPE. The two communication channels are hot standby for each other. When any communication link is disconnected, the other communication link will automatically switch the communication data. The self-check time of communication diagnosis is 1 second. When the communication diagnosis fails and the delay is 4 seconds, if it is not restored, the data connection will be immediately switched to the second link. Taking into account the synchronization factor, the central server set up in the central control room and each ship unloader 1 are equipped with two sets of software background communication services. The software database of the central server uses dual-channel data for mutual redundant backup to verify the received data to ensure the validity and accuracy of the data;

[0066] The data transmission part between the survey equipment and the ship unloader 1 and the data transmission part between the ship unloader 1 and the back-end central server realizes full-link collaboration in the fully automatic operation process. Therefore, this system can be applied to any ship type and cabin, has the advantages of not being picky about ship type and cabin, and does not need to establish a huge database to memorize ship type and cabin type;

[0067] In addition, since this system is based on the support of 5G wireless communication technology, the position detection of the airborne three-dimensional radar detector 13 is a dual linkage detection of real-time data and airborne PLC interlocking signals, so the mechanical carrying data and external position warning are in a high degree of coordination and unity. Therefore, during the operation of the system, the operation signal and fault information can be detected more quickly, so as to achieve the purpose of significantly reducing maintenance costs and improving work and maintenance efficiency.

[0068] Data modeling module, which includes a digital twin 3D model of the ship's cabin and a data sharing part between the central control server and the ship unloader 1;

[0069] By starting the ship unloader 1 trolley, the ship unloader 1 grab 2 enters the preparatory working state, the visual AI analysis model in the onboard server is ready to run and automatically analyze the data, the central server transmits the hull and cabin data to the AI ​​model for simulation matching, and the ship unloader 1 grab 2 synchronizes the motion data and preparatory variables with the arm detection system in the cabin;

[0070] The establishment of a digital twin 3D model of a ship cabin includes the following steps:

[0071] S1. Select a dataset of 2,000 aerial images, including cabin images in various scenarios, with a certain number of empty samples. Set training samples and validation samples in a ratio of 9:1, and organize the dataset into VOCdevkit format. Label the cabins in the dataset and convert the format to YOLO format;

[0072] S2. Configure model parameters. According to the training requirements and graphics card performance constraints, set the input image size to 461×416 and the initial learning rate to 0.00261. Use the learning rate cosine annealing strategy to reduce the learning rate as the number of iterations increases.

[0073] Change the YOLO layer category to 1, and change the depth of the convolution kernel of the previous YOLO layer to 18 accordingly. Write the detection category car into the dataset category file.

[0074] S3. Basic training: Use the darknet version of YOLOv4 to train the pre-trained weights of the coco dataset. Use the mosiac data enhancement strategy to scale and randomly crop every four images in the dataset, and place them in four directions and splice them together to form a training image. Input the pre-processed training image into the pre-trained model, extract and fuse features through CSPDarknet, SPP and PANet, and generate a series of convolutional feature maps. After the convolutional feature map is downsampled by 8, 16, and 32 times, it generates one-dimensional vectors of sizes 52×52×18, 26×26×18, and 13×13×18 in three yolo layers. Each yolo layer has 3 predicted bounding boxes, and each predicted bounding box regresses confidence, location and category information. Then, the final predicted box is selected through DIoU-NMS according to the confidence score of the predicted box, and the total loss graph is observed (its function reference Figure 6), when the total loss converges or reaches the maximum number of iterations, stop training and get the model after basic training. The application of 3D cloud detection technology and Vlov4 recognition system covered by this system is more practical and convenient than traditional radar scanning technology. A large number of sample training ensures the stability of AI recognition system and supports the complexity of handling multiple dynamic situations. Ai will not feel tired when performing position detection and can work all day, which improves the detection efficiency. AI detection is objective and will not be affected by subjective factors. It increases the stability of coping with various environments and the comprehensiveness of handling complex situations;

[0075] S4. Sparse training: Use the sparse factor of the BN layer to measure the importance of the channel during the training process, and get the scale factor of each channel and then sort them, and delete the channels with small scale factors. Set the sparse factor value to 0.005, use the undeleted channels for training operations, and get the sparsely trained model. The objective function of sparse training is: J = Floss + α∑|γ|, where J is the objective function, Floss is the loss generated by the model prediction, α represents the penalty factor (which converts the constrained optimization problem into an unconstrained optimization problem), γ is the training scale factor, and |·| represents the L1 norm;

[0076] S5. Use the slim-yolo extreme channel pruning strategy for pruning. Set the global threshold (pruning rate) to 0.8 and the local safety threshold to 0.01. The global threshold controls the pruning rate, and the local safety threshold prevents over-pruning and ensures the integrity of the network connection. During the pruning process, first construct a pruning mask for all convolutional layers based on the global threshold and the local safety threshold and use a pruning rate of 80% for pruning. For the route layer, concatenate the pruning masks of its incoming layers in sequence and use the concatenated masks as the pruning masks of the route layer. The shortcut layer in YOLOv4 plays a role similar to the residual network. Therefore, all layers connected to the shortcut layer need to have the same channel number. In order to match the feature channels of each layer connected by the shortcut layer, iterate the pruning masks of all connected layers and perform or operations on these connected layer pruning masks to generate the final pruning mask. Get the network structure diagram of the model after pruning (see Figure 5 ).

[0077] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present invention (including the claims) is limited to these examples. Under the concept of the present invention, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the present invention as described above, which are not provided in detail for the sake of simplicity.

[0078] The present invention is intended to cover all such substitutions, modifications and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A ship unloader grab anti-collision system based on three-dimensional space visual capture algorithm, characterized in that: Includes the following modules: A data acquisition module for on-site monitoring and surveying, which uses a 3D radar detector and binocular monitoring equipment; A data transmission module, which specifically includes an information transmission part between the survey equipment and the ship unloader and a data transmission part between the ship unloader and the central control server; The data modeling module includes the part of establishing the digital twin 3D model of the ship's cabin and the part of sharing the data of the central control server and the operating ship unloader.

2. The ship unloader grab bucket anti-collision system based on three-dimensional space visual capture algorithm according to claim 1 is characterized in that: The three-dimensional radar detector is used to scan the hatch data of the current operation, and to emit laser to detect the position information of the mechanical grab.

3. The ship unloader grab bucket anti-collision system based on three-dimensional space visual capture algorithm according to claim 2 is characterized in that: The specific application steps of the three-dimensional radar detector are as follows: S1: laser emission; S2: laser scanning; S3: Data processing: Calculate the three-dimensional coordinates of the target object; S4: Data output: The processed data can be output to a computer or other device through a communication interface for further processing or application.

4. The ship unloader grab bucket anti-collision system based on three-dimensional space visual capture algorithm according to claim 1 is characterized in that: The binocular monitoring device is installed on the moving part of the ship unloader, and the binocular vision is used for image acquisition and image correction to eliminate distortion and error.

5. The ship unloader grab bucket anti-collision system based on three-dimensional space visual capture algorithm according to claim 4 is characterized in that: The specific operation steps of the binocular monitoring device are as follows: S1, dual-target positioning; S2, image acquisition: obtain two images of the same scene through a binocular camera; S3, image correction: preprocessing the acquired image; S4. 3D reconstruction: obtaining 3D information of an object.

6. A ship unloader grab bucket anti-collision system based on three-dimensional space visual capture algorithm according to claim 1, characterized in that: The installation of the binocular survey equipment and the three-dimensional radar detector is simultaneously equipped with an outdoor client terminal device for realizing fixed-point communication.

7. The ship unloader grab bucket anti-collision system based on three-dimensional space visual capture algorithm according to claim 1 is characterized in that: The channel for data communication between the ship unloader and the central server arranged in the port central control command center adopts optical fiber to ensure the validity and accuracy of the data.

8. The ship unloader grab bucket anti-collision system based on three-dimensional space visual capture algorithm according to claim 1 is characterized in that: The establishment of the digital twin 3D model of the ship cabin includes the following steps: S1, select the aerial image dataset; S2, configure model parameters; S3, basic training; S4, sparse training; S5. Channel pruning.

Images