Intelligent Grab Control Method and System Based on Multi-Sensor Fusion

Abstract

The present invention relates to the technical field of mechanical intelligence, and particularly relates to an intelligent grab control method and system based on multi-sensor fusion. The method of the present invention includes the following steps: obtaining point cloud data of multiple sensors; performing rough stitching on the point cloud data to obtain a rough stitching rigid body transformation matrix; performing fine registration and fusion on the point cloud data according to the rough stitching rigid body transformation matrix to obtain fine registration and fusion data; constructing a three-dimensional solid model of the material by using the fine registration and fusion data, and intelligently controlling the grab to work through the three-dimensional solid model. By fusing the point cloud data of multiple sensors, then reducing and reconstructing the data to form a material model, and further intelligently controlling the grab to work according to the material model, the present invention reduces labor costs, solves the problem of realizing intelligent control of the grab to work, and is beneficial to improving the material handling efficiency and safety.

Description

Technical Field

[0001] The present invention relates to the technical field of mechanical intelligence, and in particular to an intelligent grab control method and system based on multi-sensor fusion. Background Art

[0002] The bridge grab ship unloader has been increasingly favored by users among the many types of port loading and unloading machinery because of its wide adaptability to a wide range of cargoes, especially for bulk cargoes, moderate price, low operating cost, low probability of mechanical failure during operation, and ability to reduce physical damage to the machinery itself caused by the turbulence of the ship caused by sea waves. Its main purpose is to complete the loading and unloading of cargoes from ships and vehicles at the port, as well as the lifting and transportation of cargo stacking and transshipment. The bridge grab ship unloader is good at loading and unloading bulk cargoes, making it the main mechanical equipment for bulk cargo loading and unloading at present. However, the manual control or semi-automatic control system with low intelligence of the current bridge grab ship unloader can no longer meet the high frequency of cargo loading and unloading under modern large-scale operations. Therefore, it is an urgent problem to realize the intelligent control of the ship unloader grab to reduce labor costs and improve material handling efficiency and safety. Summary of the invention

[0003] In view of the shortcomings of existing methods and the needs of practical applications, in order to improve the operating efficiency of the loading and unloading system and the safe and reliable loading and unloading performance capabilities, the problem of intelligently controlling the grab bucket is solved. On the one hand, the present invention provides an intelligent grab bucket control method based on multi-sensor fusion, comprising the following steps: acquiring point cloud data from multiple sensors; roughly splicing the point cloud data to obtain a roughly spliced ​​rigid body transformation matrix; accurately aligning and fusing the point cloud data according to the roughly spliced ​​rigid body transformation matrix to obtain precisely aligned fused data; using the precisely aligned fused data to construct a three-dimensional model of the material, and intelligently controlling the grab bucket to work through the three-dimensional model. The present invention fuses the point cloud data from multiple sensors, simplifies the data and reconstructs the material model, and then intelligently controls the grab bucket to work according to the material model, thereby reducing labor costs, solving the problem of realizing intelligent control of the grab bucket, and is conducive to improving material handling efficiency and safety.

[0004] Optionally, the intelligent grab control method based on multi-sensor fusion further includes the following steps:

[0005] A data screening model is constructed, and the point cloud data is screened and processed using the data screening model; geometric features of the screened point cloud data are extracted, the point cloud data is divided according to the geometric features, and differential downsampling is performed based on the division results. The present invention effectively reduces the size of data used for subsequent steps by screening and differential downsampling the point cloud data, further improving the use efficiency of the present invention.

[0006] Optionally, the data screening model satisfies the following formula: , where represents the number of other point clouds in the neighborhood, represents the th position of the point cloud, represents the th position of the point cloud, represents the Euclidean distance, represents the neighborhood radius, represents the base of the natural logarithm.

[0007] Optionally, the differential downsampling based on the partitioning result satisfies the following formula:

[0008] ,

[0009] where represents the target sampling quantity, represents the highest distance from the grab bucket to the deck, represents the effective grabbing area of the grab bucket, represents the depth of the cabin, represents the discharge port area of the cabin, represents the number of feature points, represents the number of point clouds. According to the size of the grab bucket, the present invention determines the target sampling quantity, realizes the reasonable matching of the grab bucket with the material and the cabin, and further improves the reliability of obtaining the three-dimensional solid model of the material in the subsequent steps.

[0010] Optionally, the rough stitching of the point cloud data to obtain the rough stitching rigid body transformation matrix includes the following steps:

[0011] Taking the centroid of the point cloud data as the origin, constructing the covariance matrix of the point cloud data; performing eigenvalue decomposition on the covariance matrix, and obtaining the main direction coordinate system transformation matrix according to the decomposition result; using the feature descriptor to identify the feature points of the point cloud data, and obtaining the feature points of the point cloud to be stitched and the feature points of the target point cloud; combining the feature points of the point cloud to be stitched and the main direction coordinate system transformation matrix to obtain the transformed feature points of the target point cloud; comparing the feature points of the target point cloud and the transformed feature points of the target point cloud to obtain the effective stitched point cloud pairs; using the effective stitched point cloud pairs to obtain the rough stitching rigid body transformation matrix. By comparing the feature points of the target point cloud and the transformed feature points of the target point cloud, the present invention identifies the effective stitched point cloud pairs, and then solves to obtain the rough stitching rigid body transformation matrix, completing the preliminary stitching of the point cloud, and providing a basis for further effective and accurate fusion.

[0012] Optionally, the fine registration and fusion of the point cloud data according to the rough stitching rigid body transformation matrix to obtain the fine registration and fusion data includes the following steps:

[0013] Construct a fine registration fusion objective function; use the rough stitching rigid body transformation matrix to transform the point cloud to be fused, perform subset sampling from the transformed point cloud to be fused, and based on the sampling result, obtain the fine registration rigid body transformation parameters according to the fine registration fusion objective function; through the fine registration rigid body transformation parameters, perform fine registration fusion on the point cloud data to obtain fine registration fusion data. The present invention performs fine registration again based on the sampling result, further effectively improving the accuracy of point cloud data fusion.

[0014] Optionally, the fine registration fusion objective function satisfies the following formula:

[0015] , where represents the fine registration fusion objective function value, represents the number of corresponding point pairs, represents the th point in the target point cloud, represents the th point in the transformed point cloud to be fused, represents the rotation transformation matrix, represents the translation transformation matrix. Evaluating the fine registration rigid body transformation parameters effectively through the fine registration fusion objective function is beneficial to improving the accuracy of the present invention.

[0016] Optionally, the obtaining of the fine registration rigid body transformation parameters according to the fine registration fusion objective function based on the sampling result satisfies the following formula:

[0017] , where represents the number of corresponding point pairs, represents the th point in the target point cloud, represents the th point in the point cloud to be fused after the th transformation, represents the th point in the point cloud to be fused after the th transformation, represents the fine registration threshold. Determining the final fine registration rigid body transformation parameters by comparing the differences after two transformations is further beneficial to improving the accuracy of the present invention.

[0018] Optionally, the intelligent control of the grab by the three-dimensional solid model includes the following steps:

[0019] Set the grab control strategy; combine the grab control strategy and the three-dimensional solid model to complete the grab control. The present invention is beneficial to realizing the intelligent control of the grab by setting a reasonable grab control strategy.

[0020] In a second aspect, to efficiently execute an intelligent grab control method based on multi-sensor fusion provided by the present invention, the present invention also provides an intelligent grab control system based on multi-sensor fusion, including a processor, an input device, an output device, and a memory. The processor, the input device, the output device, and the memory are interconnected. Among them, the memory is used to store a computer program, and the computer program contains program instructions. The processor is configured to call the program instructions to execute an intelligent grab control method based on multi-sensor fusion as described in the first aspect of the present invention. The intelligent grab control system based on multi-sensor fusion of the present invention has a compact structure and stable performance, and can stably execute an intelligent grab control method based on multi-sensor fusion provided by the present invention, further improving the overall applicability and practical application ability of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a flowchart of an intelligent grab control method based on multi-sensor fusion provided by an embodiment of the present invention;

[0022] Figure 2 It is a framework diagram of an intelligent grab control system based on multi-sensor fusion provided by an embodiment of the present invention;

[0023] Figure 3 It is a schematic structural diagram of an intelligent grab control device based on multi-sensor fusion provided by an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0024] The specific embodiments of the present invention will be described in detail below. It should be noted that the embodiments described here are only for illustrative purposes and are not used to limit the present invention. In the following description, in order to provide a thorough understanding of the present invention, a large number of specific details are set forth. However, it will be apparent to those of ordinary skill in the art that: the present invention does not have to be practiced with these specific details. In other instances, well-known circuits, software, or methods have not been specifically described to avoid obscuring the present invention.

[0025] Throughout the specification, the reference to "an embodiment", "embodiment", "an example" or "example" means that a particular feature, structure, or characteristic described in connection with the embodiment or example is included in at least one embodiment of the present invention. Thus, the phrases "in an embodiment", "in an embodiment", "an example" or "example" appearing throughout the specification do not necessarily all refer to the same embodiment or example. In addition, the specific features, structures, or characteristics may be combined in any suitable combination and / or sub-combination in one or more embodiments or examples. In addition, those of ordinary skill in the art should understand that the diagrams provided herein are for illustrative purposes only and are not necessarily drawn to scale.

[0026] Please refer to Figure 1 , in order to improve the operating efficiency of the loading and unloading system and the safe and reliable loading and unloading performance, and to solve the problem of intelligent control of the grab, the present invention provides an intelligent grab control method based on multi-sensor fusion. As Figure 1 shown, in one embodiment, the method includes the following steps:

[0027] S1. Obtain the point cloud data of multiple sensors.

[0028] In the embodiment, relevant point cloud data of the material is obtained through multiple lidar sensors arranged on the trolley, the carriage of the grab ship unloader and other places convenient for scanning the material.

[0029] The lidar sensor measures the characteristics of the reflected echo signal after the laser pulse hits the nearby object, calculates the distance of each object accurately, and further analyzes information such as the magnitude of the reflected energy on the surface of the target object, the amplitude, frequency and phase of the reflected wave spectrum, so as to present the accurate three-dimensional structure information of the target object. The working principle is based on the emission, reception and processing of laser; it mainly consists of a laser emitter, a receiver, a scanner (or rotating mechanism), a lens antenna and a signal processing circuit, etc. When working, the laser emitter emits a laser beam with a specific wavelength, these laser beams change direction through the scanner or rotating mechanism and irradiate the target object, the reflected laser beam is captured by the receiver and converted into an electrical signal, and the signal processing circuit processes these electrical signals to calculate the distance, azimuth, height and other information of the target object.

[0030] The main components of the lidar sensor include:

[0031] Laser, which is the laser emission device in the lidar. Commonly used lasers include laser diodes and fiber lasers. The selection of the laser needs to be determined according to the usage scenario, such as requirements in terms of emission wavelength, emission energy, stability, compactness, eye safety, etc.;

[0032] Rotating mechanism, which is used to change the measurement direction of the laser and realize rapid laser scanning of a large range of areas. The rotation speed of the rotating mechanism and the repetition frequency of the laser together determine the point cloud density;

[0033] Filter, which is used to filter background noise in space and wavelength to improve the signal quality. Filters usually include two types: spatial filters and spectral filters;

[0034] Photoelectric detector, which converts the received optical signal into an electrical signal for subsequent signal processing and analysis;

[0035] The lidar sensor has the following remarkable characteristics:

[0036] High resolution, whether it is angular resolution, range resolution or velocity resolution, lidar can meet the needs of fine characterization of target objects;

[0037] Three-dimensional imaging ability. By emitting laser pulses and receiving the reflected signals, lidar can accurately calculate the position of each point in three-dimensional space, thus creating high-quality three-dimensional point cloud data;

[0038] Accurate ranging and velocity measurement. Using the principle of laser time of flight and Doppler effect, lidar can measure the distance and motion speed of the target very accurately;

[0039] All-weather working ability. Lidar can still work stably at night or under low light conditions, and the laser has relatively strong ability to penetrate haze and smoke;

[0040] Stability and reliability. Solid-state lidar has a compact structure and no mechanical rotating parts, so it has high reliability and a long service life.

[0041] Use a non-repetitive multi-line lidar to collect and identify point cloud data of multiple hulls, cabins and materials in the cabins. Through the high resolution and wide field of view of the multi-line lidar, ensure the comprehensive coverage of complex scenes.

[0042] Furthermore, when obtaining data, it is also necessary to achieve time synchronization between lidars to ensure that the data collected by different lidars at the same time can be correctly fused, and avoid data errors caused by time delay.

[0043] Even further, after obtaining the point cloud data of multiple sensors, the point cloud data is also preprocessed, including the following steps:

[0044] S11. Construct a data screening model, and use the data screening model to screen and process the point cloud data.

[0045] Specifically, the data screening model satisfies the following formula:

[0046] , where, represents the number of other point clouds in the neighborhood, represents the position of the th point cloud, represents the position of the th point cloud, represents the Euclidean distance, represents the neighborhood radius, represents the base of the natural logarithm.

[0047] If the selection of the point cloud neighborhood is too large, it is easy to cause the situation of incorrect elimination of the point cloud. Generally, the neighborhood radius can be set to 0.2m.

[0048] The point cloud data is screened using the data screening model, and the point cloud data that meets the data screening model is removed, effectively solving the problems of point cloud displacement, blurred edge information of the point cloud, point cloud adhesion, etc. caused by natural environmental factors, camera's own errors, etc., thus solving the problem that a large number of material points are misidentified as cabin equipment.

[0049] S12. Extract the geometric features of the screened point cloud data, divide the point cloud data according to the geometric features, and perform differential downsampling based on the division result.

[0050] Since the number of points in the point cloud is large and the density is high, in order to improve the calculation efficiency and save the point cloud processing time, it is necessary to perform spatial downsampling on the original point cloud data.

[0051] Specifically, taking the geometric features of the point cloud such as point cloud curvature as an index, the number of feature points is judged according to the size of the index value, and then the point cloud data is divided.

[0052] Point cloud curvature refers to a measure that describes the curvature change of the point cloud surface in three-dimensional space. It can be estimated by calculating the change rate of the surface normal, usually measured using a curvature tensor. The curvature tensor is a second-order tensor composed of the direction of the surface normal and the magnitude of the curvature. In the point cloud, there is a local surface that approximates the point cloud at any point. Therefore, the curvature of this point can be calculated by fitting this local surface.

[0053] Specific calculation methods may include quadratic surface fitting, calculating curvature using the normal vectors of adjacent points, etc. These methods all require determining the neighborhood of the point first, and then performing surface fitting or normal vector calculation based on the points within the neighborhood to obtain the curvature value.

[0054] Furthermore, after obtaining the curvature value of each point in the point cloud, a curvature threshold can be set to judge the feature points. The points with curvature values greater than the threshold are considered feature points, and these points are usually located at the edges, corners or regions with large surface changes of the point cloud. By counting the number of feature points, the density of the feature distribution in the point cloud can be understood.

[0055] Even further, the differential downsampling based on the division result satisfies the following formula:

[0056] ,

[0057] where, represents the target sampling quantity, represents the highest distance from the grab to the deck, represents the effective grabbing area of the grab, represents the depth of the cabin, represents the discharge port area of the cabin, represents the number of feature points, represents the number of point clouds.

[0058] Distinguish between regions with obvious and unobvious features, and perform independent uniform sampling. The sampling numbers are respectively and , represents the sampling uniformity parameter.

[0059] S2. Coarsely splice the point cloud data to obtain a coarse splicing rigid body transformation matrix.

[0060] Specifically, the step of coarsely splicing the point cloud data to obtain a coarse splicing rigid body transformation matrix includes the following steps:

[0061] S21. With the centroid of the point cloud data as the origin, construct the covariance matrix of the point cloud data.

[0062] The centroid is the average position of the point cloud data and is the reference point for constructing the covariance matrix. Before calculating the covariance matrix, it is usually necessary to de - centroid the point cloud data, that is, subtract the centroid coordinates from each point's coordinates, so that the new point cloud data has the centroid as the origin.

[0063] The covariance matrix is used to describe the linear relationship between each dimension (usually the three coordinates x, y, z) in the point cloud data. Specifically, the covariance matrix is a 3×3 matrix. When calculating, the de - centroid point coordinates can be written in vector form, and then calculated according to the definition of the covariance matrix.

[0064] S22. Perform eigenvalue decomposition on the covariance matrix, and obtain the main direction coordinate system transformation matrix according to the decomposition result.

[0065] Performing eigenvalue decomposition on the covariance matrix gives the eigenvalues and corresponding eigenvectors. The magnitude of the eigenvalue reflects the data variance in the direction of the eigenvector, and the eigenvector represents the main direction of the data.

[0066] According to the magnitude of the eigenvalues, select the corresponding eigenvectors as the main directions. Usually, select the three eigenvectors with the largest eigenvalues as the main directions of the point cloud, which respectively correspond to the longest, second - longest, and shortest directions of the point cloud. Form a matrix with these three eigenvectors as column vectors, which is the main direction coordinate system transformation matrix.

[0067] S23. Use feature descriptors to identify the feature points of the point cloud data to obtain the feature points of the point cloud to be spliced and the feature points of the target point cloud.

[0068] Feature descriptors are vectors or sets of feature values used to describe the features in the area around feature points. Common feature descriptors include PFH (Point Feature Histograms), FPFH (Fast Point Feature Histograms), VFH (Viewpoint Feature Histogram), CVFH (Clustered Viewpoint Feature Histogram), and NARF (Normal Aligned Radial Feature), etc.

[0069] Use the feature descriptor to identify the feature points of the point cloud to be stitched and the target point cloud respectively, and obtain the feature points of the point cloud to be stitched and the feature points of the target point cloud.

[0070] S24. Combine the feature points of the point cloud to be stitched and the main direction coordinate system transformation matrix to obtain the transformed feature points of the target point cloud.

[0071] Use the main direction coordinate system transformation matrix to transform the feature points of the point cloud to be stitched, and the transformation result is the transformed feature points of the target point cloud.

[0072] S25. Compare the feature points of the target point cloud and the transformed feature points of the target point cloud to obtain valid stitched point cloud pairs.

[0073] Specifically, calculate the Euclidean distance between the feature points of the target point cloud and the transformed feature points of the target point cloud. If the corresponding Euclidean distance is less than the preset threshold, then the point to be stitched and the corresponding target point are used as valid stitched point cloud pairs.

[0074] S26. Use the valid stitched point cloud pairs to obtain the rough stitching rigid body transformation matrix.

[0075] In the embodiment, based on the valid stitched point cloud pairs, the rough stitching rigid body transformation matrix is obtained by the singular value decomposition method or the quaternion method.

[0076] S3. Perform fine registration and fusion on the point cloud data according to the rough stitching rigid body transformation matrix to obtain fine registration and fusion data.

[0077] Specifically, the performing fine registration and fusion on the point cloud data according to the rough stitching rigid body transformation matrix to obtain fine registration and fusion data includes the following steps:

[0078] S31. Construct a fine registration and fusion objective function.

[0079] In the embodiment, the fine registration and fusion objective function satisfies the following formula:

[0080] , where represents the value of the fine registration fusion objective function represents the number of corresponding point pairs represents the th point in the target point cloud represents the th point in the transformed point cloud to be fused represents the rotation transformation matrix represents the translation transformation matrix

[0081] S32. Transform the point cloud to be fused using the rough stitching rigid body transformation matrix, perform subset sampling from the transformed point cloud to be fused, and based on the sampling result, obtain the fine registration rigid body transformation parameters according to the fine registration fusion objective function

[0082] Specifically, transform the point cloud to be fused using the rough stitching rigid body transformation matrix, perform subset sampling from the transformed point cloud to be fused, and the sampling methods include

[0083] Random sampling. Random sampling is the simplest and most direct method. It randomly selects points from the original point cloud as sampling points according to a certain probability. This method is simple and easy to implement, but the distribution of the sampling points may be uneven and it cannot well retain the features of the point cloud

[0084] Uniform sampling. Uniform sampling aims to make the sampling points evenly distributed in the point cloud. A commonly used uniform sampling method is farthest point sampling (FPS). It iteratively selects the point farthest from the set of sampled points as the new sampling point. This method can ensure that the sampling points are relatively evenly distributed, but the computational complexity is relatively high

[0085] Lattice sampling (grid sampling). Lattice sampling divides the three-dimensional space into multiple small grids and selects a point in each grid as the sampling point. Usually, the point closest to the center point of the grid is selected as the sampling point. This method is efficient and the sampling points are relatively evenly distributed, but the uniformity may not be as good as uniform sampling and the number of sampling points cannot be precisely controlled

[0086] Geometric sampling. Geometric sampling samples according to the local geometric features of the point cloud. In regions with large curvature, the number of sampling points is relatively large; in flat regions, the number of sampling points is relatively small. This method can well retain the geometric features of the point cloud, but the computational complexity is relatively high

[0087] In the embodiment, the fine registration rigid body transformation parameters obtained according to the fine registration fusion objective function based on the sampling result satisfy the following formula

[0088] , where represents the number of corresponding point pairs represents the point in the target point cloud, represents the point in the point cloud to be fused after the th transformation, represents the point in the point cloud to be fused after the th transformation, represents the fine registration threshold.

[0089] S33. Through the fine registration rigid body transformation parameters, perform fine registration and fusion on the point cloud data to obtain fine registration fusion data.

[0090] The fine registration rigid body transformation parameters are the rotation matrix and the translation matrix. By performing rotation and translation transformations on all points in the source point cloud, their spatial positions in the target point cloud coordinate system are obtained, and the final stitching and fusion are completed to obtain fine registration fusion data.

[0091] S4. Use the fine registration fusion data to construct a three-dimensional solid model of the material, and intelligently control the grab through the three-dimensional solid model.

[0092] Based on the fine registration fusion data, construct a three-dimensional solid model of the material through the triangulation algorithm or the Poisson reconstruction algorithm, and intelligently control the grab through the three-dimensional solid model.

[0093] Specifically, the intelligent control of the grab through the three-dimensional solid model includes the following steps:

[0094] S41. Set the grab control strategy.

[0095] According to factors such as the type of material and the draft of the ship's hold, set the grab control strategy, and the grab control strategy includes: the maximum grab amount of the grab and the path planning principle.

[0096] S42. Combine the grab control strategy and the three-dimensional solid model to complete the grab control.

[0097] According to the three-dimensional solid model, identify all material grab points. The material grab points refer to the peak points in the three-dimensional solid model, and then evaluate the optimal peak point as the optimal grab point according to the grab control strategy to achieve intelligent grab control.

[0098] Please refer to Figure 2, in an embodiment, to efficiently execute an intelligent grab control method provided by the present invention based on multi-sensor fusion, the present invention further provides an intelligent grab control system based on multi-sensor fusion, including: an input device, an output device, a processor, and a memory. The input device, output device, processor, and memory are interconnected. The memory contains program instructions for the steps of the intelligent grab control method based on multi-sensor fusion. The intelligent grab control system based on multi-sensor fusion of the present invention has a compact structure and stable performance, and can stably execute the intelligent grab control method based on multi-sensor fusion of the present invention, further improving the overall applicability and practical application ability of the present invention.

[0099] In an embodiment, the so-called processor may be a central processing unit (CPU), and this processor may also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or this processor may also be any conventional processor, etc. The input device can be used to obtain data information. The output device can be used to output the result obtained from the program instructions included in the computer program stored in the memory provided by the present invention. The memory may include a read-only memory and a random access memory, and provide instructions and data to the processor. A part of the memory may also include a non-volatile random access memory.

[0100] In another alternative embodiment, please refer to Figure 3 , to efficiently execute an intelligent grab control method provided by the present invention based on multi-sensor fusion, this embodiment further provides an intelligent grab control device based on multi-sensor fusion, as Figure 3 shown, including:

[0101] A memory 10 for storing a computer program; a processor 20 for executing the computer program to implement the above-mentioned intelligent grab control method based on multi-sensor fusion. The memory 10, the processor 20, a communication interface 31, and a communication bus 32. The memory 10, the processor 20, and the communication interface 31 are all connected to each other through the communication bus 32.

[0102] In an embodiment, the memory 10 is used to store one or more program instructions. The memory 10 may store program instructions for implementing the following functions:

[0103] Obtain the point cloud data of multiple sensors; perform rough stitching on the point cloud data to obtain a rough stitching rigid body transformation matrix; perform fine registration and fusion on the point cloud data according to the rough stitching rigid body transformation matrix to obtain fine registration and fusion data; use the fine registration and fusion data to construct a three-dimensional solid model of the material, and intelligently control the grab operation through the three-dimensional solid model.

[0104] In a possible implementation, the memory 10 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system and application programs required for at least one function, etc.; the data storage area may store the data created during use. In addition, the memory 10 may include a read-only memory and a random access memory, and provide instructions and data to the processor. A part of the memory may also include NVRAM. The memory stores an operating system, operation instructions, executable modules or data structures, or subsets thereof, or extended sets thereof. Among them, the operation instructions may include various operation instructions for implementing various operations. The operating system may include various system programs for implementing various basic tasks and processing hardware-based tasks.

[0105] The processor 20 may be a central processing unit (CPU), an application specific integrated circuit, a digital signal processor, a field programmable gate array or other programmable logic devices. The processor 20 may be a microprocessor or any conventional processor, etc. The processor 20 may call the programs stored in the memory 10. The communication interface 31 may be an interface of a communication module for connecting to other devices or systems.

[0106] Of course, it should be noted that Figure 3 The structure shown does not constitute a limitation on the intelligent grab control device based on multi-sensor fusion in this embodiment. In actual applications, the intelligent grab control device based on multi-sensor fusion may include more or fewer components than Figure 3 the structure shown, or combine certain components.

[0107] An embodiment also provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned intelligent grab control method based on multi-sensor fusion are implemented.

[0108] The storage medium may include: various media that can store program codes, such as USB flash drives, external hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical discs.

[0109] In summary, the present invention fuses the point cloud data of multiple sensors, then simplifies and reconstructs the material model for the data, and further intelligently controls the grab to work according to the material model, reducing labor costs, solving the problem of realizing the intelligent control of the grab, and being beneficial to improving the material handling efficiency and safety.

[0110] Therefore, the present invention effectively overcomes various disadvantages in the prior art and has high industrial utilization value.

[0111] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be covered by the scope recorded in the present invention.

Claims

1. An intelligent grab control method based on multi-sensor fusion, characterized in that: The intelligent grab control method based on multi-sensor fusion comprises the following steps: Acquire point cloud data from multiple sensors; Roughly splicing the point cloud data to obtain a rough splicing rigid body transformation matrix; Performing precise registration and fusion on the point cloud data according to the coarse splicing rigid body transformation matrix to obtain precise registration and fusion data; Using the precise registration fusion data to construct a three-dimensional model of the material, and intelligently controlling the operation of the grab bucket through the three-dimensional model; Constructing a data screening model, and using the data screening model to perform screening processing on the point cloud data; Extracting geometric features of the filtered point cloud data, dividing the point cloud data according to the geometric features, and performing differential downsampling based on the division results; The differential downsampling based on the division result satisfies the following formula: in, represents the target sampling number, Indicates the maximum distance between the grab and the deck. It represents the effective grabbing area of ​​the grab bucket. Indicates the depth of the cabin. Indicates the discharge port area of ​​the ship's hold, represents the number of feature points, Indicates the number of point clouds.

2. According to claim 1, the intelligent grab control method based on multi-sensor fusion is characterized in that: The data screening model satisfies the following formula: in, represents the number of other point clouds in the neighborhood, Indicates The location of the point cloud, Indicates The location of the point cloud, represents the Euclidean distance, represents the neighborhood radius, Represents the base of natural logarithms.

3. The intelligent grab control method based on multi-sensor fusion according to claim 1 is characterized in that: The step of roughly splicing the point cloud data to obtain a roughly spliced ​​rigid body transformation matrix comprises the following steps: Taking the centroid of the point cloud data as the origin, constructing a covariance matrix of the point cloud data; Performing eigenvalue decomposition on the covariance matrix, and obtaining a main direction coordinate system transformation matrix according to the decomposition result; Using feature descriptors to identify feature points of the point cloud data, and obtaining feature points of the point cloud to be spliced ​​and feature points of the target point cloud; Combining the feature points of the point cloud to be spliced ​​and the transformation matrix of the main direction coordinate system, obtaining the conversion feature points of the target point cloud; Comparing the target point cloud feature points with the target point cloud conversion feature points to obtain a valid spliced ​​point cloud pair; The coarse splicing rigid body transformation matrix is ​​obtained by using the effective splicing point cloud pairs.

4. The intelligent grab control method based on multi-sensor fusion according to claim 1 is characterized in that: The step of performing precise registration and fusion on the point cloud data according to the coarse splicing rigid body transformation matrix to obtain precise registration and fusion data comprises the following steps: Construct precise registration and fusion objective function; The rough splicing rigid body transformation matrix is ​​used to transform the point cloud to be fused, and a subset sampling is performed from the transformed point cloud to be fused. Based on the sampling result, the precise registration rigid body transformation parameters are obtained according to the precise registration fusion objective function; The point cloud data is precisely registered and fused by using the precisely registered rigid body transformation parameters to obtain precisely registered fused data.

5. The intelligent grab control method based on multi-sensor fusion according to claim 4 is characterized in that: The precise registration fusion objective function satisfies the following formula: in, represents the value of the precise registration fusion objective function, represents the number of corresponding point pairs, Indicates the first point, Represents the first point in the transformed point cloud to be fused point, represents the rotation transformation matrix, Represents the translation transformation matrix.

6. The intelligent grab control method based on multi-sensor fusion according to claim 4 is characterized in that: Based on the sampling result, the precise registration rigid body transformation parameters are obtained according to the precise registration fusion objective function, satisfying the following formula: in, represents the number of corresponding point pairs, Indicates the first point, Indicates After the first transformation, the point cloud to be fused point, Indicates After the first transformation, the point cloud to be fused point, Represents the fine registration threshold.

7. The intelligent grab control method based on multi-sensor fusion according to claim 1 is characterized in that: The method of intelligently controlling the grab bucket operation through the three-dimensional model comprises the following steps: Set grab control strategy; The grab control strategy and the three-dimensional model are combined to complete the grab control.

8. An intelligent grab control system based on multi-sensor fusion, characterized in that: The intelligent grab control system based on multi-sensor fusion includes: an input device, an output device, a processor, and a memory. The input device, output device, processor, and memory are interconnected. The memory includes program instructions, and the program instructions are used to execute the intelligent grab control method based on multi-sensor fusion according to any one of claims 1 to 7.

Images