Intelligent grab bucket control method and system based on multi-sensor fusion

Abstract

The invention relates to the technical field of mechanical intellectualization, in particular to an intelligent grab bucket control method and system based on multi-sensor fusion. The method comprises the following steps: acquiring point cloud data of a plurality of sensors; performing coarse splicing on the point cloud data to obtain a coarse splicing rigid body transformation matrix; performing fine registration fusion on the point cloud data according to the coarse splicing rigid body transformation matrix to obtain fine registration fusion data; and constructing a three-dimensional model of the material by using the fine registration fusion data, and intelligently controlling the grab bucket to work through the three-dimensional model. According to the invention, by fusing the point cloud data of the plurality of sensors, simplifying the data and reconstructing the material model, the grab bucket is intelligently controlled to work according to the material model, so that the labor cost is reduced, the problem of intelligently controlling the work of the grab bucket is solved, and the material carrying efficiency and safety are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of mechanical intelligence, and in particular to an intelligent grab control method and system based on multi-sensor fusion. Background Art

[0002] The bridge grab ship unloader has been increasingly favored by users among the many types of port loading and unloading machinery because of its wide adaptability to a wide range of cargoes, especially for bulk cargoes, moderate price, low operating cost, low probability of mechanical failure during operation, and ability to reduce physical damage to the machinery itself caused by the turbulence of the ship caused by sea waves. Its main purpose is to complete the loading and unloading of cargoes from ships and vehicles at the port, as well as the lifting and transportation of cargo stacking and transshipment. The bridge grab ship unloader is good at loading and unloading bulk cargoes, making it the main mechanical equipment for bulk cargo loading and unloading at present. However, the manual control or semi-automatic control system with low intelligence of the current bridge grab ship unloader can no longer meet the high frequency of cargo loading and unloading under modern large-scale operations. Therefore, it is an urgent problem to realize the intelligent control of the ship unloader grab to reduce labor costs and improve material handling efficiency and safety. Summary of the invention

[0003] In view of the shortcomings of existing methods and the needs of practical applications, in order to improve the operating efficiency of the loading and unloading system and the safe and reliable loading and unloading performance capabilities, the problem of intelligently controlling the grab bucket is solved. On the one hand, the present invention provides an intelligent grab bucket control method based on multi-sensor fusion, comprising the following steps: acquiring point cloud data from multiple sensors; roughly splicing the point cloud data to obtain a roughly spliced ​​rigid body transformation matrix; accurately aligning and fusing the point cloud data according to the roughly spliced ​​rigid body transformation matrix to obtain precisely aligned fused data; using the precisely aligned fused data to construct a three-dimensional model of the material, and intelligently controlling the grab bucket to work through the three-dimensional model. The present invention fuses the point cloud data from multiple sensors, simplifies the data and reconstructs the material model, and then intelligently controls the grab bucket to work according to the material model, thereby reducing labor costs, solving the problem of realizing intelligent control of the grab bucket, and is conducive to improving material handling efficiency and safety.

[0004] Optionally, the intelligent grab control method based on multi-sensor fusion further includes the following steps: A data screening model is constructed, and the point cloud data is screened and processed using the data screening model; geometric features of the screened point cloud data are extracted, the point cloud data is divided according to the geometric features, and differential downsampling is performed based on the division results. The present invention effectively reduces the size of data used for subsequent steps by screening and differential downsampling the point cloud data, further improving the use efficiency of the present invention.

[0005] Optionally, the data screening model satisfies the following formula: ,in, represents the number of other point clouds in the neighborhood, Indicates The location of the point cloud, Indicates The location of the point cloud, represents the Euclidean distance, represents the neighborhood radius, Represents the base of natural logarithms.

[0006] Optionally, the differentiated downsampling is performed based on the division result, satisfying the following formula: , in, represents the target sampling number, Indicates the maximum distance between the grab and the deck. It represents the effective grabbing area of ​​the grab bucket. Indicates the depth of the cabin. Indicates the discharge port area of ​​the ship's hold, represents the number of feature points, The present invention determines the target sampling quantity according to the size of the grab bucket, realizes the reasonable matching of the grab bucket with the material and the cabin, and further improves the reliability of the three-dimensional model of the material obtained in the subsequent steps.

[0007] Optionally, the step of roughly splicing the point cloud data to obtain a roughly spliced ​​rigid body transformation matrix comprises the following steps: Taking the center of mass of the point cloud data as the origin, construct the covariance matrix of the point cloud data; perform eigenvalue decomposition on the covariance matrix, and obtain the main direction coordinate system transformation matrix according to the decomposition result; use feature descriptors to identify the feature points of the point cloud data, and obtain the feature points of the point cloud to be spliced ​​and the feature points of the target point cloud; combine the feature points of the point cloud to be spliced ​​and the main direction coordinate system transformation matrix to obtain the target point cloud conversion feature points; compare the target point cloud feature points and the target point cloud conversion feature points to obtain a valid splicing point cloud pair; use the valid splicing point cloud pair to obtain the coarse splicing rigid body transformation matrix. The present invention identifies the valid splicing point cloud pair by comparing the target point cloud feature points and the target point cloud conversion feature points, and then solves to obtain the coarse splicing rigid body transformation matrix, completes the preliminary point cloud splicing, and provides a basis for further effective and accurate fusion.

[0008] Optionally, the step of performing precise registration and fusion on the point cloud data according to the coarse splicing rigid body transformation matrix to obtain precise registration and fusion data comprises the following steps: Construct a precise registration fusion objective function; transform the point cloud to be fused using the coarse splicing rigid body transformation matrix, perform subset sampling from the transformed point cloud to be fused, obtain precise registration rigid body transformation parameters based on the sampling results and the precise registration fusion objective function; perform precise registration fusion on the point cloud data through the precise registration rigid body transformation parameters to obtain precise registration fusion data. The present invention performs precise registration again based on the sampling results, further effectively improving the accuracy of point cloud data fusion.

[0009] Optionally, the precise registration fusion objective function satisfies the following formula: ,in, represents the value of the precise registration fusion objective function, represents the number of corresponding point pairs, Indicates the first point, Represents the first point in the transformed point cloud to be fused point, represents the rotation transformation matrix, Represents the translation transformation matrix. Effectively evaluating the precise registration rigid body transformation parameters through the precise registration fusion objective function is beneficial to improving the accuracy of the present invention.

[0010] Optionally, based on the sampling result, obtaining the precise registration rigid body transformation parameters according to the precise registration fusion objective function satisfies the following formula: ,in, represents the number of corresponding point pairs, Indicates the first point, Indicates After the first transformation, the point cloud to be fused point, Indicates After the first transformation, the point cloud to be fused point, Denotes the precision registration threshold. By comparing the difference between the two transformations, the final precision registration rigid body transformation parameters are determined, which is further beneficial to improve the accuracy of the present invention.

[0011] Optionally, the intelligently controlling the grab bucket operation through the three-dimensional model comprises the following steps: The grab control strategy is set; the grab control strategy is combined with the three-dimensional model to complete the grab control. The present invention is conducive to realizing intelligent control of the grab by setting a reasonable grab control strategy.

[0012] In the second aspect, in order to efficiently execute the intelligent grab control method based on multi-sensor fusion provided by the present invention, the present invention also provides an intelligent grab control system based on multi-sensor fusion, including a processor, an input device, an output device and a memory, wherein the processor, the input device, the output device and the memory are interconnected, wherein the memory is used to store a computer program, the computer program includes program instructions, and the processor is configured to call the program instructions to execute the intelligent grab control method based on multi-sensor fusion as described in the first aspect of the present invention. The intelligent grab control system based on multi-sensor fusion of the present invention has a compact structure and stable performance, and can stably execute the intelligent grab control method based on multi-sensor fusion provided by the present invention, further improving the overall applicability and practical application ability of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 A flow chart of an intelligent grab control method based on multi-sensor fusion provided by an embodiment of the present invention; Figure 2 A framework diagram of an intelligent grab control system based on multi-sensor fusion provided by an embodiment of the present invention; Figure 3 A schematic diagram of the structure of an intelligent grab control device based on multi-sensor fusion provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0014] The specific embodiments of the present invention will be described in detail below. It should be noted that the embodiments described herein are only for illustration and are not intended to limit the present invention. In the following description, a large number of specific details are set forth in order to provide a thorough understanding of the present invention. However, it is obvious to those of ordinary skill in the art that these specific details do not need to be adopted to implement the present invention. In other examples, in order to avoid confusing the present invention, known circuits, software or methods are not specifically described.

[0015] Throughout the specification, references to "one embodiment," "an embodiment," "an example," or "an example" mean that a particular feature, structure, or characteristic described in conjunction with the embodiment or example is included in at least one embodiment of the present invention. Therefore, the phrases "in one embodiment," "in an embodiment," "an example," or "an example" appearing in various places throughout the specification do not necessarily all refer to the same embodiment or example. In addition, particular features, structures, or characteristics may be combined in one or more embodiments or examples in any suitable combination and / or subcombination. In addition, it should be understood by those of ordinary skill in the art that the figures provided herein are for illustrative purposes and that the figures are not necessarily drawn to scale.

[0016] See also Figure 1In order to improve the operating efficiency and safe and reliable loading and unloading performance of the loading and unloading system and solve the problem of intelligent control of grab bucket, the present invention provides an intelligent grab bucket control method based on multi-sensor fusion, such as Figure 1 As shown, in one embodiment, the method includes the following steps: S1. Obtain point cloud data from multiple sensors.

[0017] In an embodiment, relevant point cloud data of the material is obtained by deploying multiple laser radar sensors on the trolley, small trolley and other places convenient for scanning materials of the grab ship unloader.

[0018] The laser radar sensor accurately calculates the distance of each object by emitting laser pulses and then measuring the characteristics of the reflected echo signal after hitting nearby objects. It also further analyzes the reflected energy on the surface of the target object, the amplitude, frequency and phase of the reflected spectrum, thereby presenting accurate three-dimensional structural information of the target object. The working principle is based on the emission, reception and processing of lasers. It is mainly composed of a laser transmitter, a receiver, a scanner (or a rotating mechanism), a lens antenna and a signal processing circuit. When working, the laser transmitter emits a laser beam of a specific wavelength. These laser beams change direction through a scanner or a rotating mechanism and irradiate the target object. The reflected laser beam is captured by the receiver and converted into an electrical signal. The signal processing circuit processes these electrical signals to calculate the distance, direction, height and other information of the target object.

[0019] The main components of a LiDAR sensor include: Laser is the laser emitting device in LiDAR. Commonly used lasers include laser diodes and fiber lasers. The choice of laser should be determined according to the usage scenario, such as emission wavelength, emission energy, stability, compactness, eye safety and other requirements; The rotating mechanism is used to change the measurement direction of the laser to achieve rapid laser scanning of a large area. The rotation speed of the rotating mechanism and the repetition frequency of the laser jointly determine the point cloud density; Filters are used to filter out background noise in space and wavelength to improve signal quality. Filters usually include spatial filters and spectral filters; Photodetector, which converts the received optical signal into an electrical signal for subsequent signal processing and analysis; LiDAR sensors have the following notable features: High resolution: whether it is angle resolution, distance resolution or speed resolution, LiDAR can meet the requirements for fine characterization of target objects; 3D imaging capability: By emitting laser pulses and receiving reflected signals, LiDAR can accurately calculate the position of each point in 3D space, thereby creating high-quality 3D point cloud data; Precise distance and speed measurement: Using the principle of laser flight time and the Doppler effect, LiDAR can measure the distance and speed of the target very accurately. All-weather working capability: LiDAR can still work stably at night or in low light conditions, and the laser has a relatively strong ability to penetrate haze and smoke; Stability and reliability: Solid-state laser radar has a compact structure and no mechanical rotating parts, so it has high reliability and long service life.

[0020] Use non-repetitive multi-line LiDAR to collect and identify point cloud data of multiple hulls, cabins and materials in the cabins. The high resolution and wide field of view of the multi-line LiDAR ensure full coverage of complex scenes.

[0021] Furthermore, when acquiring data, time synchronization between lidars is also required to ensure that data collected by different lidars at the same time can be correctly integrated to avoid data errors caused by time delays.

[0022] Furthermore, after acquiring the point cloud data of multiple sensors, the point cloud data is also preprocessed, including the following steps: S11, constructing a data screening model, and using the data screening model to perform screening processing on the point cloud data.

[0023] Specifically, the data screening model satisfies the following formula: ,in, represents the number of other point clouds in the neighborhood, Indicates The location of the point cloud, Indicates The location of the point cloud, represents the Euclidean distance, represents the neighborhood radius, Represents the base of natural logarithms.

[0024] If the point cloud neighborhood is too large, it is easy to cause point clouds to be mistakenly removed. Generally, the neighborhood radius can be set to 0.2m.

[0025] The point cloud data is filtered and processed using the data screening model, and the point cloud data that meets the data screening model is eliminated, which effectively solves the problem of a large number of material points being misidentified as cabin equipment due to point cloud displacement, blurred information between point cloud edges, point cloud adhesion and other phenomena caused by the existence of natural environmental factors, camera errors and other factors.

[0026] S12, extracting geometric features of the filtered point cloud data, dividing the point cloud data according to the geometric features, and performing differential downsampling based on the division results.

[0027] The number of point clouds is large and the density is high. In order to improve the computing efficiency and save the point cloud processing time, the original point cloud data needs to be spatially downsampled.

[0028] Specifically, the geometric features of the point cloud, such as the curvature of the point cloud, are used as indicators, and the number of feature points is determined according to the value of the indicator, and then the point cloud data is divided.

[0029] Point cloud curvature refers to a measure that describes the change in curvature of a point cloud surface in three-dimensional space. It can be estimated by calculating the rate of change of the surface normal and is usually measured using a curvature tensor, which is a second-order tensor consisting of the direction of the surface normal and the magnitude of the curvature. For any point in a point cloud, there is a surface of a neighborhood point cloud that approximates the point, so the curvature of the point can be calculated by fitting this local surface.

[0030] Specific calculation methods may include quadratic surface fitting, using normal vectors of adjacent points to calculate curvature, etc. These methods all require first determining the neighborhood of the point, and then performing surface fitting or normal vector calculation based on the points in the neighborhood to obtain the curvature value.

[0031] Furthermore, after obtaining the curvature value of each point in the point cloud, a curvature threshold can be set to determine the feature points. Points with curvature values ​​greater than the threshold are considered feature points. These points are usually located at the edge, corner, or area with large surface changes in the point cloud. By counting the number of feature points, the density of feature distribution in the point cloud can be understood.

[0032] Furthermore, the differentiated downsampling is performed based on the division result, satisfying the following formula: , in, represents the target sampling number, Indicates the maximum distance between the grab and the deck. It represents the effective grabbing area of ​​the grab bucket. Indicates the depth of the cabin. Indicates the discharge port area of ​​the ship's hold, represents the number of feature points, Indicates the number of point clouds.

[0033] The regions with obvious features and those with unclear features are distinguished and independently sampled evenly. The sampling numbers are and , Represents the sampling uniformity parameter.

[0034] S2. Roughly stitch the point cloud data to obtain a rough stitching rigid body transformation matrix.

[0035] Specifically, the step of roughly splicing the point cloud data to obtain a roughly spliced ​​rigid body transformation matrix includes the following steps: S21. Constructing a covariance matrix of the point cloud data with the centroid of the point cloud data as the origin.

[0036] The centroid is the average position of the point cloud data and is the reference point for constructing the covariance matrix. Before calculating the covariance matrix, it is usually necessary to de-centroid the point cloud data, that is, the coordinates of each point are subtracted from the centroid coordinates so that the new point cloud data takes the centroid as the origin.

[0037] The covariance matrix is ​​used to describe the linear relationship between the dimensions (usually the three coordinates of x, y, and z) in the point cloud data. Specifically, the covariance matrix is ​​a 3×3 matrix. When calculating, the de-centroided point coordinates can be written in vector form and then calculated according to the definition of the covariance matrix.

[0038] S22, performing eigenvalue decomposition on the covariance matrix, and obtaining a main direction coordinate system transformation matrix according to the decomposition result.

[0039] Perform eigenvalue decomposition on the covariance matrix to obtain eigenvalues ​​and corresponding eigenvectors. The size of the eigenvalue reflects the size of the data variance in the direction of the eigenvector, while the eigenvector represents the main direction of the data.

[0040] According to the size of the eigenvalue, the corresponding eigenvector is selected as the main direction. Usually, the three eigenvectors with the largest eigenvalues ​​are selected as the main directions of the point cloud, which correspond to the longest, second longest and shortest directions of the point cloud respectively. These three eigenvectors are used as column vectors to form a matrix, which is the main direction coordinate system transformation matrix.

[0041] S23. Using feature descriptors to identify feature points of the point cloud data, and obtaining feature points of the point cloud to be spliced ​​and feature points of the target point cloud.

[0042] A feature descriptor is a vector or a set of eigenvalues ​​used to describe the features of the area around a feature point. Common feature descriptors include PFH (Point Feature Histograms), FPFH (Fast Point Feature Histograms), VFH (Viewpoint Feature Histogram), CVFH (Clustered Viewpoint Feature Histogram), and NARF (Normal Aligned Radial Feature).

[0043] Feature descriptors are used to identify feature points of the point cloud to be spliced ​​and the target point cloud respectively, so as to obtain feature points of the point cloud to be spliced ​​and feature points of the target point cloud.

[0044] S24. Combining the feature points of the point cloud to be spliced ​​and the transformation matrix of the main direction coordinate system, obtain the conversion feature points of the target point cloud.

[0045] The feature points of the point cloud to be spliced ​​are transformed using the transformation matrix of the main direction coordinate system, and the transformation result is the transformation feature point of the target point cloud.

[0046] S25. Compare the target point cloud feature points with the target point cloud conversion feature points to obtain a valid spliced ​​point cloud pair.

[0047] Specifically, the Euclidean distance between the target point cloud feature point and the target point cloud conversion feature point is calculated. If the corresponding Euclidean distance is less than a preset threshold, the point to be spliced ​​and the corresponding target point are taken as a valid splicing point cloud pair.

[0048] S26. Obtain the coarse splicing rigid body transformation matrix using the valid splicing point cloud pairs.

[0049] In an embodiment, based on the valid splicing point cloud pairs, the coarse splicing rigid body transformation matrix is ​​obtained by singular value decomposition method or quaternion method.

[0050] S3. Perform precise registration and fusion on the point cloud data according to the coarse splicing rigid body transformation matrix to obtain precise registration and fusion data.

[0051] Specifically, the step of performing precise registration and fusion on the point cloud data according to the coarse splicing rigid body transformation matrix to obtain precise registration and fusion data includes the following steps: S31. Construct a precise registration and fusion objective function.

[0052] In an embodiment, the precise registration fusion objective function satisfies the following formula: ,in, represents the value of the precise registration fusion objective function, represents the number of corresponding point pairs, Indicates the first point, Represents the first point in the transformed point cloud to be fused point, represents the rotation transformation matrix, Represents the translation transformation matrix.

[0053] S32, using the coarse stitching rigid body transformation matrix to transform the point cloud to be fused, performing subset sampling from the transformed point cloud to be fused, and obtaining the precise registration rigid body transformation parameters based on the sampling result and the precise registration fusion objective function.

[0054] Specifically, the rough splicing rigid body transformation matrix is ​​used to transform the point cloud to be fused, and subset sampling is performed from the transformed point cloud to be fused. The sampling method includes: Random sampling,Random sampling is the simplest and most direct method. It randomly selects points from the original point cloud as sampling points according to a certain probability. This method is simple and easy, but the distribution of sampling points may be uneven and cannot preserve the characteristics of the point cloud well.

[0055] Uniform sampling aims to make the sampling points evenly distributed in the point cloud. A commonly used uniform sampling method is the farthest point sampling (FPS), which iteratively selects the point farthest from the sampled point set as the new sampling point. This method can ensure that the sampling points are relatively evenly distributed, but the computational complexity is high; Grid sampling (grid sampling), grid sampling divides the three-dimensional space into multiple small grids, and selects a point in each grid as a sampling point. Usually, the point closest to the center of the grid is selected as the sampling point. This method is efficient and the sampling points are relatively evenly distributed, but the uniformity may not be as good as uniform sampling, and the number of sampling points cannot be accurately controlled.

[0056] Geometric sampling, geometric sampling is based on the local geometric features of the point cloud. In areas with large curvature, the number of sampling points is large; in flat areas, the number of sampling points is small. This method can well preserve the geometric features of the point cloud, but the computational complexity is high.

[0057] In an embodiment, the precise registration rigid body transformation parameters are obtained based on the sampling results and the precise registration fusion objective function, satisfying the following formula: ,in, represents the number of corresponding point pairs, Indicates the first point, Indicates After the first transformation, the point cloud to be fused point, Indicates After the first transformation, the point cloud to be fused point, Represents the fine registration threshold.

[0058] S33, performing precise registration and fusion on the point cloud data through the precise registration rigid body transformation parameters to obtain precise registration and fusion data.

[0059] The rigid body transformation parameters, namely the rotation matrix and translation matrix, are precisely aligned. By performing rotation and translation transformations on all points in the source point cloud, their spatial positions in the target point cloud coordinate system are obtained, and the final splicing and fusion are completed to obtain precisely aligned fusion data.

[0060] S4. Use the precise registration and fusion data to construct a three-dimensional model of the material, and intelligently control the operation of the grab bucket through the three-dimensional model.

[0061] Based on the precise registration and fusion data, a three-dimensional model of the material is constructed by a triangulation algorithm or a Poisson reconstruction algorithm, and the grab bucket is intelligently controlled by the three-dimensional model.

[0062] Specifically, the intelligent control of the grab bucket operation through the three-dimensional model includes the following steps: S41. Set grab control strategy.

[0063] According to factors such as material type and cabin draft, a grab control strategy is set, which includes: maximum grabbing capacity of the grab and path planning principle.

[0064] S42, combining the grab control strategy and the three-dimensional model to complete the grab control.

[0065] According to the three-dimensional model, all material grabbing points are identified. The material grabbing points refer to the peak points in the three-dimensional model. Then, according to the grab control strategy, the optimal peak point is evaluated as the optimal grabbing point to realize the intelligent control of the grab.

[0066] See also Figure 2In an embodiment, in order to efficiently execute the intelligent grab control method based on multi-sensor fusion provided by the present invention, the present invention also provides an intelligent grab control system based on multi-sensor fusion, including: an input device, an output device, a processor, and a memory, wherein the input device, the output device, the processor, and the memory are interconnected, and the memory contains program instructions, and the program instructions are used for the steps of the intelligent grab control method based on multi-sensor fusion. The intelligent grab control system based on multi-sensor fusion of the present invention has a compact structure and stable performance, and can stably execute the intelligent grab control method based on multi-sensor fusion of the present invention, further improving the overall applicability and practical application ability of the present invention.

[0067] In an embodiment, the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. An input device may be used to obtain data information. An output device may be used to output the results obtained by storing program instructions contained in a computer program in a memory provided by the present invention. The memory may include a read-only memory and a random access memory, and provide instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory.

[0068] In yet another alternative embodiment, see Figure 3 In order to efficiently execute the intelligent grab control method based on multi-sensor fusion provided by the present invention, this embodiment also provides an intelligent grab control device based on multi-sensor fusion, such as Figure 3 As shown, including: The memory 10 is used to store computer programs; the processor 20 is used to execute the computer programs to implement the above-mentioned intelligent grab control method based on multi-sensor fusion. The memory 10, the processor 20, the communication interface 31 and the communication bus 32. The memory 10, the processor 20 and the communication interface 31 all communicate with each other through the communication bus 32.

[0069] In an embodiment, the memory 10 is used to store one or more program instructions. The memory 10 may store program instructions for implementing the following functions: Acquire point cloud data from multiple sensors; roughly splice the point cloud data to obtain a roughly spliced ​​rigid body transformation matrix; accurately align and fuse the point cloud data according to the roughly spliced ​​rigid body transformation matrix to obtain accurately aligned and fused data; use the accurately aligned and fused data to construct a three-dimensional model of the material, and intelligently control the grab bucket through the three-dimensional model.

[0070] In one possible implementation, the memory 10 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, and at least one application required for a function, etc.; the data storage area may store data created during use. In addition, the memory 10 may include a read-only memory and a random access memory, and provide instructions and data to the processor. A portion of the memory may also include NVRAM. The memory stores an operating system and operating instructions, executable modules or data structures, or a subset thereof, or an extended set thereof, wherein the operating instructions may include various operating instructions for implementing various operations. The operating system may include various system programs for implementing various basic tasks and processing hardware-based tasks.

[0071] The processor 20 may be a central processing unit (CPU), an application specific integrated circuit, a digital signal processor, a field programmable gate array or other programmable logic device, and the processor 20 may be a microprocessor or any conventional processor, etc. The processor 20 may call a program stored in the memory 10. The communication interface 31 may be an interface of a communication module, and is used to connect with other devices or systems.

[0072] Of course, it should be noted that Figure 3 The structure shown does not constitute a limitation on the intelligent grab control device based on multi-sensor fusion in this embodiment. In actual applications, the intelligent grab control device based on multi-sensor fusion may include: Figure 3 More or fewer components than shown, or combinations of certain components.

[0073] The embodiment further provides a storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the above-mentioned intelligent grab control method based on multi-sensor fusion are implemented.

[0074] The storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0075] To sum up, the present invention integrates the point cloud data of multiple sensors, simplifies the data and reconstructs the material model, and then intelligently controls the grab to work according to the material model, thereby reducing labor costs, solving the problem of realizing the operation of the intelligent control grab, and is conducive to improving material handling efficiency and safety.

[0076] Therefore, the present invention effectively overcomes various shortcomings in the prior art and has high industrial utilization value.

[0077] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present invention, and they should all be included in the scope recorded in the present invention.

Claims

1. An intelligent grab control method based on multi-sensor fusion, characterized in that: The intelligent grab control method based on multi-sensor fusion comprises the following steps: Acquire point cloud data from multiple sensors; Roughly splicing the point cloud data to obtain a rough splicing rigid body transformation matrix; Performing precise registration and fusion on the point cloud data according to the coarse splicing rigid body transformation matrix to obtain precise registration and fusion data; The precise registration fusion data is used to construct a three-dimensional model of the material, and the grab bucket is intelligently controlled through the three-dimensional model.

2. According to claim 1, the intelligent grab control method based on multi-sensor fusion is characterized in that: The following steps are also included: Constructing a data screening model, and using the data screening model to perform screening processing on the point cloud data; The geometric features of the filtered point cloud data are extracted, the point cloud data are divided according to the geometric features, and differential downsampling is performed based on the division results.

3. The intelligent grab control method based on multi-sensor fusion according to claim 2 is characterized in that: The data screening model satisfies the following formula: ,in, represents the number of other point clouds in the neighborhood, Indicates The location of the point cloud, Indicates The location of the point cloud, represents the Euclidean distance, represents the neighborhood radius, Represents the base of natural logarithms.

4. According to claim 2, the intelligent grab control method based on multi-sensor fusion is characterized in that: The differential downsampling based on the division result satisfies the following formula: ,in, represents the target sampling number, Indicates the maximum distance between the grab and the deck. It represents the effective grabbing area of ​​the grab bucket. Indicates the depth of the cabin. Indicates the discharge port area of ​​the ship's hold, represents the number of feature points, Indicates the number of point clouds.

5. The intelligent grab control method based on multi-sensor fusion according to claim 1 is characterized in that: The step of roughly splicing the point cloud data to obtain a roughly spliced ​​rigid body transformation matrix comprises the following steps: Taking the centroid of the point cloud data as the origin, constructing a covariance matrix of the point cloud data; Performing eigenvalue decomposition on the covariance matrix, and obtaining a main direction coordinate system transformation matrix according to the decomposition result; Using feature descriptors to identify feature points of the point cloud data, and obtaining feature points of the point cloud to be spliced ​​and feature points of the target point cloud; Combining the feature points of the point cloud to be spliced ​​and the transformation matrix of the main direction coordinate system, obtaining the conversion feature points of the target point cloud; Comparing the target point cloud feature points with the target point cloud conversion feature points to obtain a valid spliced ​​point cloud pair; The coarse splicing rigid body transformation matrix is ​​obtained by using the effective splicing point cloud pairs.

6. The intelligent grab control method based on multi-sensor fusion according to claim 1 is characterized in that: The step of performing precise registration and fusion on the point cloud data according to the coarse splicing rigid body transformation matrix to obtain precise registration and fusion data comprises the following steps: Construct precise registration and fusion objective function; The rough splicing rigid body transformation matrix is ​​used to transform the point cloud to be fused, and a subset sampling is performed from the transformed point cloud to be fused. Based on the sampling result, the precise registration rigid body transformation parameters are obtained according to the precise registration fusion objective function; The point cloud data is precisely registered and fused by using the precisely registered rigid body transformation parameters to obtain precisely registered fused data.

7. The intelligent grab control method based on multi-sensor fusion according to claim 6 is characterized in that: The precise registration fusion objective function satisfies the following formula: ,in, represents the value of the precise registration fusion objective function, represents the number of corresponding point pairs, Indicates the first point, Represents the first point in the transformed point cloud to be fused point, represents the rotation transformation matrix, Represents the translation transformation matrix.

8. The intelligent grab control method based on multi-sensor fusion according to claim 6 is characterized in that: Based on the sampling result, the precise registration rigid body transformation parameters are obtained according to the precise registration fusion objective function, satisfying the following formula: ,in, represents the number of corresponding point pairs, Indicates the first point, Indicates After the first transformation, the point cloud to be fused point, Indicates After the first transformation, the point cloud to be fused point, Represents the fine registration threshold.

9. The intelligent grab control method based on multi-sensor fusion according to claim 1 is characterized in that: The method of intelligently controlling the grab bucket operation through the three-dimensional model comprises the following steps: Set grab control strategy; The grab control strategy and the three-dimensional model are combined to complete the grab control.

10. An intelligent grab control system based on multi-sensor fusion, characterized in that: The intelligent grab control system based on multi-sensor fusion includes: an input device, an output device, a processor, and a memory. The input device, output device, processor, and memory are interconnected. The memory includes program instructions, and the program instructions are used to execute the intelligent grab control method based on multi-sensor fusion according to any one of claims 1 to 9.

Images