Multi-sensor fusion cabin loading and unloading equipment cooperative positioning method and system

Abstract

The invention relates to the technical field of cabin loading and unloading automation, and provides a multi-sensor fusion cabin loading and unloading equipment cooperative positioning method and system. A laser radar is used for scanning a cabin, laser radar point cloud data are generated, and a laser radar coordinate system and a point cloud map are constructed; a monocular camera is adopted to obtain an operation area image of loading and unloading equipment, camera image data is generated, and a camera coordinate system is constructed; performing anti-interference processing on the laser radar point cloud data; performing illumination change influence resistance processing on the camera image data; fusing the laser radar point cloud data with the camera image data to obtain fused positioning information; calculating the distance and angle of loading and unloading equipment according to the fused positioning information, generating a moving instruction, and driving the loading and unloading equipment to perform cooperative positioning; a dynamic object in a cabin is detected through a laser radar and a monocular camera, multi-modal verification is carried out on a detection result, the dynamic object is filtered, and a point cloud map is updated in real time. The problems of insufficient positioning precision, GPS failure and large dynamic environment interference of a single sensor are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of cabin loading and unloading automation, and in particular to a multi-sensor fusion cabin loading and unloading equipment collaborative positioning method and system. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] With the rapid growth of global trade, the efficiency of cargo loading and unloading at ports and on ships has become a key factor influencing the speed of the logistics chain. Traditional ship loading and unloading relies primarily on manual operations or semi-automated equipment, which suffers from low efficiency, high costs, and insufficient safety. In recent years, advances in automatic identification technologies (such as computer vision and RFID) have provided new solutions for intelligent ship loading and unloading.

[0004] In cargo loading and unloading operations at ports and ship holds, accurately locating the positions of loaders and gantry cranes is key to achieving automated loading and unloading. However, existing technologies, which mainly rely on traditional methods of manual operation or GPS-based positioning systems, have the following shortcomings, resulting in insufficient positioning accuracy.

[0005] First, when the loading and unloading equipment loader and gantry crane work together, the positioning system needs to complete multi-source data fusion in a short period of time, otherwise there will be a risk of collision between the loader and the bulkhead. However, manual or single sensors (such as lidar or GPS) are difficult to meet the high-precision and high-robustness positioning requirements.

[0006] Secondly, the problem of signal blocking is prominent. The fully enclosed metal structure of the cabin will greatly block the GPS signal, and the positioning is prone to failure. As a result, when the loader is running, it is necessary to manually observe the position of the loader and operate the gantry crane to load and unload cargo, which requires high manual operation; and wireless signals such as ultra-wideband technology are easily affected by reflection interference from multiple decks and stacked cargo, resulting in multi-path effect.

[0007] Finally, the dynamic environment on the cabin is complex, and dynamic objects can affect sensor accuracy. For example, the point cloud loss rate of lidar in a dusty environment is as high as 40%. Summary of the Invention

[0008] The present invention provides a multi-sensor fusion coordinated positioning method and system for ship loading and unloading equipment, wherein a laser radar and a monocular camera are installed inside the ship cabin to collect data, and the specific position of the loader in the ship cabin is accurately determined through data fusion; a sensor that can receive information collected by the laser radar and the monocular camera is installed on the gantry crane, so that the gantry crane can accurately move to the position of the loader; dynamic objects are processed by multi-sensor dynamic detection, and a multi-sensor anti-interference solution is added; the technical problems in the prior art such as insufficient positioning accuracy of a single sensor, GPS failure, and large interference from the dynamic environment are solved.

[0009] To achieve the above object, the present invention adopts the following technical solutions: A first aspect of the present invention provides a multi-sensor fusion coordinated positioning method for ship loading and unloading equipment, comprising: Use LiDAR to scan the cabin, generate LiDAR point cloud data, and build a LiDAR coordinate system and point cloud map. Use a monocular camera to obtain images of the loading and unloading equipment operating area, generate camera image data, and build a camera coordinate system. Perform anti-interference processing on LiDAR point cloud data; perform anti-illumination change processing on camera image data; Fuse the lidar point cloud data with the camera image data to obtain fused positioning information; Calculate the distance and angle of loading and unloading equipment based on the fused positioning information, generate movement instructions, and drive the collaborative positioning of loading and unloading equipment; Dynamic objects in the cabin are detected using lidar and a monocular camera, the detection results are verified multimodally, dynamic objects are filtered, and the point cloud map is updated in real time.

[0010] As a further technical solution, the loading and unloading equipment includes a gantry crane and a loader, and the gantry crane is provided with sensors for receiving data collected by a laser radar and a monocular camera respectively.

[0011] As a further technical solution, anti-interference processing is performed on the lidar point cloud data, specifically: Through reflectivity threshold processing, the valid point set of LiDAR point cloud data with reflectivity within the set range is screened out, and noise points and metal reflection points with abnormal reflectivity are eliminated; Perform multipath effect suppression by comparing the lidar measurement timestamp with the theoretical predicted time to determine whether the time deviation is within the system noise range and mark the valid point; Based on the point cloud plane model, the average distance deviation from the point to the plane is calculated, and outliers that do not conform to the plane geometric structure are eliminated to obtain geometrically consistent lidar point cloud data.

[0012] As a further technical solution, the camera image data is processed to resist the influence of light changes, specifically: Adjust the exposure time according to the average grayscale of the camera image data, shorten the exposure time in strong light, and extend the exposure time in weak light to obtain image data with balanced lighting; The threshold of feature extraction is dynamically adjusted according to the grayscale variance of local area image data. The threshold is lowered in high variance areas and increased in low variance areas. At the same time, feature points with a depth greater than the set value are retained to obtain a stable feature point set of camera image data.

[0013] As a further technical solution, the fusion of the lidar point cloud data and the camera image data specifically includes: Align the lidar point cloud data and camera image data to the same coordinate system; The original LiDAR point cloud data is voxelized and convolved to obtain LiDAR voxel features; the data is input into a multi-layer perceptron to extract LiDAR global features; The original camera image data is input into the CNN network to extract camera features, and the camera features are projected into the bird's-eye view space to obtain the bird's-eye view features; Splice lidar voxel features and bird's-eye view features along the channel dimension; use attention fusion to fuse lidar global features and bird's-eye view local features; use Transformer fusion to perform cross-attention calculation and output fused features; The fusion features are used to establish the global coordinate system of the cabin and obtain the fusion positioning information.

[0014] As a further technical solution, the lidar point cloud data and the camera image data are aligned to the same coordinate system, specifically: the points in the lidar coordinate system are converted to the camera coordinate system through external parameter calibration; the points in the camera coordinate system are projected to the image pixel coordinates through internal parameter calibration; and the lidar point cloud after external parameter calibration is projected to the image pixel coordinates through point cloud projection.

[0015] As a further technical solution, the specific process of updating the point cloud map in real time is as follows: The monocular camera acquires continuous frame images; The laser radar point cloud data is detected through point cloud clustering algorithm and motion state judgment to obtain dynamic object clusters, and the dynamic object clusters are projected onto continuous frame images to calculate the laser radar detection area; The optical flow motion vector and its corresponding dynamic area are calculated by the optical flow method to obtain the dynamic mask, and the monocular camera visual optical flow area corresponding to the dynamic mask is calculated; The intersection-over-union ratio of the lidar detection area and the monocular camera visual optical flow area is used as the confidence level. If the confidence level is greater than the set value, the object is confirmed to be a dynamic obstacle. Remove the confirmed dynamic objects from the current point cloud map to obtain an updated point cloud map.

[0016] A second aspect of the present invention provides a multi-sensor fusion coordinated positioning system for ship loading and unloading equipment, comprising: The data acquisition module is configured to: use a lidar to scan the cabin, generate lidar point cloud data, and construct a lidar coordinate system and point cloud map; use a monocular camera to acquire images of the loading and unloading equipment operating area, generate camera image data, and construct a camera coordinate system; The data processing module is configured to: perform anti-interference processing on the lidar point cloud data; perform anti-illumination change processing on the camera image data; The data fusion module is configured to: fuse the lidar point cloud data with the camera image data to obtain fused positioning information; The equipment coordination module is configured to calculate the distance and angle of the loading and unloading equipment based on the fused positioning information, generate movement instructions, and drive the loading and unloading equipment to coordinate positioning; The data update module is configured to detect dynamic objects in the cabin through lidar and monocular camera, perform multimodal verification on the detection results, filter dynamic objects and update the point cloud map in real time.

[0017] The third aspect of the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps of the multi-sensor fusion ship loading and unloading equipment collaborative positioning method as described in the first aspect of the present invention.

[0018] The fourth aspect of the present invention provides an electronic device, including a memory, a processor, and a program stored in the memory and runnable on the processor. When the processor executes the program, it implements the steps in the multi-sensor fusion ship loading and unloading equipment collaborative positioning method as described in the first aspect of the present invention.

[0019] The technical solution of the present invention has the following beneficial effects: The present invention's cargo hold loading and unloading equipment positioning method based on lidar and monocular cameras differs from existing manual detection methods in that it is based on the recognition and data fusion of lidar and monocular cameras, enabling precise collaborative positioning. This solves the problem that manual or single sensors (such as lidar or GPS) are unable to meet the high-precision and high-robustness positioning requirements.

[0020] The present invention respectively processes the laser radar's anti-interference and the monocular camera's anti-illumination change effects. In a cabin without a GPS signal, the present invention can adapt to the complex environment inside the cabin and locate the positions of the loader and gantry crane in real time with high precision.

[0021] The present invention uses clustering, optical flow and other methods to detect dynamic objects (such as vehicles and pedestrians) in lidar point clouds and camera images, eliminates the influence of dynamic objects, filters static backgrounds, and updates the point cloud map in real time. During the movement of the loader, the stacking status of the cargo is dynamically detected to provide precise target positions for gantry crane grasping.

[0022] Advantages of additional aspects of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0024] Figure 1 This is a flow chart of the collaborative positioning method for ship loading and unloading equipment in Example 1 of the present invention; Figure 2 This is a front cross-sectional view of the sensor installation position in Example 1 of the present invention; Figure 3 This is a side sectional view of the sensor installation position in Example 1 of the present invention; Figure 4 This is a flow chart of sensor feature fusion in Example 1 of the present invention. DETAILED DESCRIPTION

[0025] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used in the present invention have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0026] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0027] In the absence of conflict, the embodiments of the present invention and the features thereof may be combined with each other.

[0028] Explanation of terms: LiDAR: A sensor that measures the distance and shape of objects by emitting laser beams and receiving reflected signals, generating highly accurate three-dimensional point cloud data.

[0029] Monocular camera: A single-camera visual sensor used to capture images of the environment and extract feature points.

[0030] Loader: A mechanical device used to move and stack goods.

[0031] Gantry crane: A lifting device used for loading and unloading cargo in ports or ship holds, usually with moving and grabbing functions.

[0032] Example 1 This embodiment discloses a multi-sensor fusion method for collaborative positioning of ship loading and unloading equipment. Figure 1 Shown, including: S1: Use LiDAR to scan the cabin, generate LiDAR point cloud data, and build a LiDAR coordinate system and point cloud map. Use a monocular camera to obtain images of the loading and unloading equipment operating area, generate camera image data, and build a camera coordinate system. S2: Anti-interference processing of lidar point cloud data; Anti-illumination change processing of camera image data; S3: Fuse the lidar point cloud data with the camera image data to obtain fused positioning information; S4: Calculate the distance and angle of the loading and unloading equipment based on the fused positioning information, generate movement instructions, and drive the loading and unloading equipment to coordinate positioning; S5: Detect dynamic objects in the cabin using LiDAR and a monocular camera, perform multimodal verification on the detection results, filter dynamic objects, and update the point cloud map in real time.

[0033] Among them, the loading and unloading equipment includes a gantry crane and a loader. The gantry crane is equipped with sensors that receive data collected by laser radar and monocular camera respectively.

[0034] In S1, a lidar and a monocular camera are installed inside the cabin. The lidar is used to scan the cabin environment, generate three-dimensional lidar point cloud data, and construct a point cloud map. The size of the entire cabin and the current position of the loader are determined by scanning the edge environment of the entire cabin. The monocular camera captures images of the loading and unloading equipment operating area and generates camera image data.

[0035] Before the loader moves, the LiDAR coordinate system and real-time map construction are used to determine the loader's absolute position and its relative position to the cargo in the hold. Because the accuracy of a single sensor is limited and distances within the hold are long, a monocular camera is added to capture images and estimate the loader's position and distance to more accurately determine its specific location within the hold. This generates camera image data and a two-dimensional coordinate system. Simultaneously, a sensor installed on the gantry crane that receives data from the LiDAR and monocular camera enables the crane to precisely move to the loader's location and more efficiently coordinate with the loader to load and unload cargo.

[0036] In S2, while LiDAR can construct highly accurate 3D point cloud maps, it can lead to misjudgments or data loss when encountering highly reflective surfaces or transparent objects. While monocular cameras can capture rich image information, image quality can degrade significantly in low or high light conditions, impacting positioning accuracy. Therefore, before data fusion, the data collected by LiDAR and monocular cameras must undergo anti-interference processing to reduce the impact of environmental interference on positioning accuracy.

[0037] S2-1: To eliminate the noise points caused by the high reflectivity of metal surfaces or multipath effects in the LiDAR and improve the quality of point cloud data, anti-interference processing is performed on the LiDAR point cloud data: S2-1-1: In the LiDAR coordinate system, each point contains reflectivity (p). Thresholding by reflectivity:

[0038] Filter out the valid point set of lidar point cloud data with reflectivity within the set range , filter out points with abnormal reflectivity, remove noise points with abnormal reflectivity (reflectivity is too low) and metal reflection points (reflectivity is too high).

[0039] S2-1-2: By comparing the measurement timestamps of the lidar (The actual time from the emission of the laser pulse to the reception of the reflected signal) and the theoretically predicted time (Expected time calculated based on physical distance and speed of light) to determine whether the time deviation is within the system noise range, thereby suppressing multipath effects:

[0040] Measure noise for system time and mark the effective points , eliminating false signals caused by multipath reflections (such as glass or mirror reflections).

[0041] S2-1-3: An ideal geometric plane is fitted through the local LiDAR point cloud data, namely the point cloud plane model, which is used to describe the geometric structure of the surface of objects in the cabin environment (such as walls and floors). Based on the point cloud plane model, the vector is , point on the laser radar coordinate system The distance to the plane is d, calculate the point Mean distance deviation from the plane:

[0042] For the The coordinates of points, N is the total number of points.

[0043] Outliers that do not conform to the plane geometric structure (such as suspension noise) are eliminated to obtain geometrically consistent lidar point cloud data.

[0044] It should be noted that reflectivity filtering (S2-1-1) and timestamp verification (S2-1-2) can be performed simultaneously, because both are based on single-point attributes (reflectivity, time deviation) and do not require global point cloud information; geometric consistency verification (S2-1-3) requires a local point cloud set (such as plane fitting) for outlier detection, so it must be performed after the first two steps.

[0045] S2-2: To address the overexposure / underexposure issues caused by illumination changes in monocular cameras and ensure feature extraction stability, camera image data is processed to resist the effects of illumination changes: S2-2-1: Real-time analysis of the average grayscale value of camera image data. The monocular camera's automatic exposure control (AutoExposure, AE) technology automatically adjusts the exposure time based on the average grayscale value of the camera image data. , without relying on multi-eye or depth information:

[0046] is the average grayscale of the image, To dynamically adjust parameters (decrease in strong light, increase in weak light), The limit range is: 2ms-30ms.

[0047] Shorten the exposure time in bright light (where the image's average grayscale is above the grayscale threshold), and extend it in dim light (where the image's average grayscale is below the grayscale threshold) to obtain image data with balanced lighting. The grayscale threshold can be preset and adjusted based on the actual application scenario and camera characteristics to ensure accurate and stable exposure time adjustment.

[0048] S2-2-2: In the data obtained by the monocular camera, the monocular camera recognizes and locates the target object by capturing feature points in the image. Feature points are important elements in the image. The stability and number of feature points directly affect the subsequent image processing algorithm.

[0049] Dynamically adjust the feature extraction threshold T based on the grayscale variance of the camera image data in the local area (local grayscale variance), lower the threshold T in high variance areas, and increase the threshold T in low variance areas, while retaining feature points with depth Z(f)>1m:

[0050] Retain feature point set

[0051] A stable camera image data feature point set f is obtained; among them, the local grayscale variance is calculated based on the original pixel grayscale value and reflects the richness of the image texture (for example, the variance in the edge and corner areas is higher).

[0052] After the above processing, the anti-interference ability of the two sensors (lidar and monocular camera) in acquiring data is improved, achieving stable and high-precision data collection in the complex, long-distance and possibly unevenly illuminated scene of the cabin.

[0053] In S3, the lidar point cloud data is fused with the camera image data to establish the cabin global coordinate system and obtain fused positioning information. The fusion process specifically includes: S3-1: Align the anti-interference processed lidar point cloud data and camera image data to the same coordinate system to ensure spatial consistency in subsequent feature extraction and fusion.

[0054] The data alignment process includes: S3-1-1: Obtain the rigid body transformation matrix from the lidar to the monocular camera through external parameter calibration:

[0055] in is the rotation matrix, is the translation vector.

[0056] Through the rigid body transformation matrix Convert the point in the lidar coordinate system to the camera coordinate system: Points in the lidar coordinate system (Homogeneous coordinates) Convert the point to the camera coordinate system:

[0057] Through homogeneous coordinates, it is expanded into a non-homogeneous matrix form:

[0058] S3-1-2: Obtain camera intrinsic parameter matrix through intrinsic parameter calibration , use the camera intrinsic parameter matrix K to transform the point in the camera coordinate system Projection to image pixel coordinates :

[0059] The internal parameter matrix K is usually:

[0060] in, 、 is the camera focal length (in pixels), 、 is the principal point of the image (optical center pixel coordinates).

[0061] After expansion, we get the pixel coordinate formula:

[0062] S3-1-3: Project the laser radar point cloud after external parameter calibration Projection to image pixel coordinates :

[0063]

[0064] in, The depth of the point in the camera coordinate system in the camera coordinate system.

[0065] S3-2: Extract high-dimensional features from lidar point clouds and camera images respectively to prepare for multimodal fusion.

[0066] S3-2-1: a. Through voxelization function Aligning LiDAR point cloud data Convert to voxel grid , and then use 3D convolution (Conv3D) to extract lidar voxel features :

[0067]

[0068] N is the number of points in the point cloud, H, W, D are the height, width, and depth of the voxel grid, respectively, and C is the number of feature channels.

[0069] b. Extraction through Multi-layer Perceptron (MLP) The point features are then aggregated into global features through the maximum pooling (MaxPool):

[0070] Extracting global features of lidar .

[0071] S3-2-2: a. The original camera image data Input ResNet and other CNN networks to extract camera features :

[0072] H′ and W′ are the height and width of the feature map, and C′ is the number of feature channels.

[0073] b. Set the camera feature Projecting to the bird's-eye view (BEV) space to obtain bird's-eye view features ;

[0074] IPM() stands for Inverse Perspective Transform.

[0075] S3-3: Fuse the features of LiDAR and camera, combining their geometric and texture information to improve object recognition performance.

[0076] S3-3-1: Combine the LiDAR voxel features and the bird’s-eye view features along the channel dimension to obtain fusion features :

[0077] S3-3-2: Local feature fusion using attention fusion Through the learnable weight matrix 、 、 The global features of the lidar and the bird's-eye view features are linearly transformed to generate the query matrix Q, key matrix K and value matrix V.

[0078] in, , represents the query intent learned from the lidar features, such as "which geometric structures need attention"; , represents the key matching rules learned from the monocular camera features, such as "which textures are related to geometric structures"; , represents the importance of the value learned from the monocular camera features, such as "how to weight useful texture information".

[0079] Weight calculation and fusion formula:

[0080] The residual connection helps to retain the original lidar feature information while incorporating relevant information obtained from the monocular camera features, thereby enhancing the expressive power of the features.

[0081] S3-3-3: Use Transformer fusion to refine global features and output fused features .

[0082] a. Initialize BEV query: BEV query is a learnable parameter matrix, ,in and are the height and width of the bird's-eye view feature map, and C is the number of feature channels.

[0083] b. Calculate cross attention: calculate BEV query separately With camera features Cross attention between, and BEV query With LiDAR voxel features The calculation process of cross attention obtains two attention results through the interaction of query, key and value.

[0084]

[0085] The two attention results represent the BEV query extracted from the bird's eye view features and the lidar voxel features. Finally, add the two attention results to get the fusion feature .

[0086] S3-3-4: Use the fusion features to establish the global coordinate system of the cabin and obtain the fusion positioning information.

[0087] To unify the spatial description of different sensor data, a cabin global coordinate system needs to be established. This coordinate system takes a fixed point in the cabin as its origin and sets the directions of the X-axis, Y-axis, and Z-axis according to actual needs.

[0088] In this embodiment, let the cabin be L in length and W in width. If the loader coordinates are (x, y, z), with the lower left corner of the cabin as the origin (0, 0, 0), then the shortest distance from the loader to each cabin wall is: The distance to port is x; the distance to starboard is ( ); the distance to the bow is y; the distance to the stern is ( ), the distance to the ground is z.

[0089] Through coordinate transformation, the data of all sensors such as camera features, lidar voxel features and BEV features are converted to the global coordinate system of the cabin. Comparison and analysis are carried out in this unified coordinate system to obtain fused positioning information after the fusion of image and position data, providing a basis for subsequent equipment collaborative positioning.

[0090] In S3-3, the three fusion steps (feature concatenation, attention fusion, and Transformer fusion) are progressively complementary, forming a multi-level feature fusion framework from local to global. Feature-level concatenation is a low-level fusion that directly concatenates the channels of the lidar voxel features and the bird's-eye view features, preserving the original geometry and texture information. Attention fusion is a mid-level fusion that dynamically weights the bird's-eye view features to supplement the lidar global features through an attention mechanism. Transformer fusion is a high-level fusion that integrates global contextual information through cross-attention in the BEV space. The final output is the result of the Transformer fusion. The outputs of the first two steps serve as intermediate features and are fed into the Transformer fusion, which is then directly used for subsequent tasks.

[0091] The fusion of lidar and monocular images enables dual geometric and semantic verification: the lidar confirms structural outlines, while the monocular camera identifies container numbers and handling labels. This also improves recognition's anti-interference capabilities, allowing the lidar to penetrate dust while the monocular camera compensates for the lidar's metal reflection errors. The fusion of lidar and camera data essentially complements visual ambiguity (camera) with spatial accuracy (lidar) and geometric monotony (lidar) with texture information (camera), enabling stable, high-precision positioning and collaborative operation.

[0092] In S4, the fused data can be used to accurately determine the position and posture of the gantry crane and loader in the global coordinate system of the cabin.

[0093] Assume the coordinates of the gantry crane are ( , , ), the loader coordinates are ( , ),but The horizontal distance is: ; The spatial distance is: ; The direction angle is:

[0094] The pitch angle is:

[0095] Indicates the image depth after the fusion of lidar and monocular camera data.

[0096] Subsequently, movement instructions can be generated according to the requirements of the loading and unloading tasks, such as the need to transport from a cargo pile to a specified location in the cabin, etc. The optimal gantry crane movement path can be planned based on the distance and angle, and the loading and unloading equipment can be driven to coordinate positioning to complete the loading and unloading work.

[0097] In S5, the process of detecting dynamic objects (such as vehicles and pedestrians) in the LiDAR point cloud and camera images and filtering out static backgrounds to eliminate the influence of dynamic objects and update the point cloud map in real time is as follows: S5-1: Monocular camera acquires continuous frame images ; S5-2: Detecting dynamic objects in LiDAR point clouds.

[0098] S5-2-1: Clustering and motion state judgment: a. In the original lidar point cloud data Select Two points are clustered using the point cloud clustering DBSCAN algorithm when the Euclidean distance between the two points is less than 0.3m and the number of neighborhood points is greater than or equal to 5:

[0099] Get dynamic object clusters , each cluster represents a potential dynamic object.

[0100] b. Through the formula:

[0101] Determine the state of motion, is marked as a dynamic object. and Indicates that the Euclidean distance (straight-line distance in three-dimensional space) between two points is detected.

[0102] S5-2-2: Combine the dynamic object cluster point cloud with the camera calibration parameters (internal parameter matrix , external parameter matrix ) is projected onto continuous frame images to generate a mask and count the number of pixels covered by the projected point on the image. Projection formula:

[0103] S5-2-3: Calculate the lidar detection area.

[0104] The actual physical size of a single pixel:

[0105] in is the focal length, 、 is the pixel size, and z is the depth (vertical distance) from the target object (dynamic object cluster) to the camera.

[0106] The pixel area is converted into a physical area to represent the lidar detection area:

[0107] is the pixel area covered by the dynamic cluster on the image.

[0108] S5-3: Detecting dynamic objects in camera images.

[0109] S5-3-1: Using the optical flow method, through the spatial gradient and time gradient Calculate the optical flow motion vector (u, v), the optical flow constraint equation is:

[0110] This equation is based on the assumption of constant brightness (the brightness of the same object point in adjacent frames remains unchanged) and infers pixel motion through gradient changes to express the relationship between pixel changes in space and time.

[0111] S5-3-2: Generate a dynamic mask Mask(x,y) based on the optical flow amplitude. The result is a binary mask that marks the dynamic pixel area. Calculate the monocular camera visual optical flow area corresponding to the dynamic mask, that is, the area of ​​the dynamic pixel area. If the optical flow amplitude is greater than the threshold, it is marked as 1 (dynamic pixel), otherwise it is marked as 0 (static background). The formula is:

[0112]

[0113]

[0114] Where W×H is the image resolution (width×height), is the area of ​​the dynamic pixel region.

[0115] S5-4: To combine the detection results of the LiDAR and camera and improve the robustness of dynamic object detection, the intersection-over-union ratio of the LiDAR detection area and the monocular camera visual optical flow area is used as the confidence level. If the confidence level is greater than the set value, the object is confirmed to be a dynamic obstacle to avoid false detection:

[0116] In this embodiment, if the confidence level is greater than 0.7, it is confirmed to be a real dynamic obstacle.

[0117] S5-5: Remove the confirmed dynamic objects from the current point cloud map to obtain an updated point cloud map, thereby ensuring the static consistency of the point cloud map.

[0118] Dynamic point filtering:

[0119] in, Represents the current point cloud map, represents a dynamic object cluster, Represents the updated static point cloud map.

[0120] In addition, the original distance from the loader to the bulkhead is measured using LiDAR. , determine the system error through calibration experiments in a static environment , that is, the average deviation between the measured value and the true value, and the accurate distance data is calculated:

[0121] The depth estimation formula of a monocular camera is:

[0122] where f is the focal length of the camera, v is the speed of the loader, and Δt is the time interval.

[0123] When the loader moves quickly, the image becomes blurred due to displacement during the exposure time. It is necessary to correct the pixel displacement Δp of the loader in the image. The steps are as follows: Assuming exposure time Internal uniform motion, blur kernel length pixels) is:

[0124] Actual observed is the true displacement Superposition with blur effect, restored by deconvolution:

[0125] Corrected depth formula:

[0126] Analyze the errors of lidar and monocular camera respectively: The error of the lidar is the systematic error + random noise:

[0127] in, is the systematic error, measured through calibration experiments (e.g., the average deviation between the lidar measurement and the true value in a static environment); It is random noise caused by sensor hardware noise or environmental interference (such as dust); The standard deviation of the total error of the LiDAR measurement is quantified, providing a weight basis for subsequent multi-sensor weighted fusion.

[0128] The error of the monocular camera is pixel error + blur correction residual:

[0129] Where Z is the target depth, f is the focal length of the camera, v is the speed of the loader, ∆t is the time interval between adjacent frames, is the measurement error of pixel displacement Δp (caused by image resolution), is the camera exposure time, is the standard deviation of the total error of the monocular camera. It comprehensively evaluates the error of the camera depth estimation, including pixel displacement error and residual error caused by motion blur.

[0130] The measurement results of the lidar and monocular camera are fused weighted by the error covariance:

[0131] in, is the depth value measured by the lidar (system error has been corrected), The depth value estimated by the monocular camera. The purpose of fusion is to improve the final depth accuracy by fusing the depth measurement results of the lidar and camera based on the sensor confidence (the smaller the error, the greater the weight).

[0132] The complementary advantages of LiDAR and cameras directly determine the accuracy of positioning, obstacle avoidance, and grasping. This depth calculation is reflected in the angle calculation in S4, used for real-time positioning of the loader and gantry crane, and for calculating the loader's 3D coordinates in the global coordinate system of the cabin. During map updates in S5, it is used to update static structures in the global map (such as bulkheads and cargo stack heights) and filter dynamic noise.

[0133] Example 2 This embodiment discloses a multi-sensor fusion coordinated positioning system for ship loading and unloading equipment, including: The data acquisition module is configured to: use a lidar to scan the cabin, generate lidar point cloud data, and construct a lidar coordinate system and point cloud map; use a monocular camera to acquire images of the loading and unloading equipment operating area, generate camera image data, and construct a camera coordinate system; The data processing module is configured to: perform anti-interference processing on the lidar point cloud data; perform anti-illumination change processing on the camera image data; The data fusion module is configured to: fuse the lidar point cloud data with the camera image data to obtain fused positioning information; The equipment coordination module is configured to calculate the distance and angle of the loading and unloading equipment based on the fused positioning information, generate movement instructions, and drive the loading and unloading equipment to coordinate positioning; The data update module is configured to detect dynamic objects in the cabin through lidar and monocular camera, perform multimodal verification on the detection results, filter dynamic objects and update the point cloud map in real time.

[0134] Example 3 The purpose of this embodiment is to provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the multi-sensor fusion ship loading and unloading equipment collaborative positioning method as described in the first embodiment of the present disclosure.

[0135] Example 4 The purpose of this embodiment is to provide an electronic device. The electronic device includes a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the steps of the multi-sensor fusion collaborative positioning method for ship loading and unloading equipment as described in the first embodiment of the present disclosure.

[0136] The steps involved in the apparatuses of Examples 2, 3, and 4 above correspond to those of Method Example 1. For detailed implementations, please refer to the relevant description of Example 1. The term "computer-readable storage medium" should be understood to mean a single medium or multiple media containing one or more instruction sets; it should also be understood to include any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and causing the processor to perform any method of the present invention.

[0137] Those skilled in the art will appreciate that the modules or steps of the present invention described above can be implemented using a general-purpose computer device. Alternatively, they can be implemented using program code executable by a computing device, which can then be stored in a storage device and executed by the computing device. Alternatively, they can be fabricated into separate integrated circuit modules, or multiple modules or steps can be fabricated into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.

[0138] Although the above describes the specific embodiments of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without any creative work are still within the scope of protection of the present invention.

Claims

1. A multi-sensor fusion collaborative positioning method for ship loading and unloading equipment, characterized in that: include: Use LiDAR to scan the cabin, generate LiDAR point cloud data, and build a LiDAR coordinate system and point cloud map. Use a monocular camera to obtain images of the loading and unloading equipment operating area, generate camera image data, and build a camera coordinate system. Perform anti-interference processing on LiDAR point cloud data; perform anti-illumination change processing on camera image data; Fuse the lidar point cloud data with the camera image data to obtain fused positioning information; Calculate the distance and angle of loading and unloading equipment based on the fused positioning information, generate movement instructions, and drive the collaborative positioning of loading and unloading equipment; Dynamic objects in the cabin are detected using lidar and a monocular camera, the detection results are verified multimodally, dynamic objects are filtered, and the point cloud map is updated in real time.

2. The multi-sensor fusion collaborative positioning method for ship loading and unloading equipment according to claim 1, characterized in that: The loading and unloading equipment includes a gantry crane and a loader, and the gantry crane is provided with sensors that respectively receive data collected by a laser radar and a monocular camera.

3. The multi-sensor fusion collaborative positioning method for ship loading and unloading equipment according to claim 1, characterized in that: Perform anti-interference processing on the lidar point cloud data, specifically: Through reflectivity threshold processing, the valid point set of LiDAR point cloud data with reflectivity within the set range is screened out, and noise points and metal reflection points with abnormal reflectivity are eliminated; Perform multipath effect suppression by comparing the lidar measurement timestamp with the theoretical predicted time to determine whether the time deviation is within the system noise range and mark the valid point; Based on the point cloud plane model, the average distance deviation from the point to the plane is calculated, and outliers that do not conform to the plane geometric structure are eliminated to obtain geometrically consistent lidar point cloud data.

4. The multi-sensor fusion collaborative positioning method for ship loading and unloading equipment according to claim 1, characterized in that: The camera image data is processed to resist the influence of lighting changes, specifically: Adjust the exposure time according to the average grayscale of the camera image data, shorten the exposure time in strong light, and extend the exposure time in weak light to obtain image data with balanced lighting; The threshold of feature extraction is dynamically adjusted according to the grayscale variance of local area image data. The threshold is lowered in high variance areas and increased in low variance areas. At the same time, feature points with a depth greater than the set value are retained to obtain a stable feature point set of camera image data.

5. The multi-sensor fusion collaborative positioning method for ship loading and unloading equipment according to claim 1, characterized in that: The fusion of the lidar point cloud data and the camera image data specifically includes: Align the lidar point cloud data and camera image data to the same coordinate system; The original LiDAR point cloud data is voxelized and convolved to obtain LiDAR voxel features; the data is input into a multi-layer perceptron to extract LiDAR global features; The original camera image data is input into the CNN network to extract camera features, and the camera features are projected into the bird's-eye view space to obtain the bird's-eye view features; Splice lidar voxel features and bird's-eye view features along the channel dimension; use attention fusion to fuse lidar global features and bird's-eye view local features; use Transformer fusion to perform cross-attention calculation and output fused features; The fusion features are used to establish the global coordinate system of the cabin and obtain the fusion positioning information.

6. The multi-sensor fusion collaborative positioning method for ship loading and unloading equipment according to claim 5, characterized in that: The laser radar point cloud data and the camera image data are aligned to the same coordinate system, specifically: the points in the laser radar coordinate system are converted to the camera coordinate system through external parameter calibration; the points in the camera coordinate system are projected to the image pixel coordinates through internal parameter calibration; and the laser radar point cloud after external parameter calibration is projected to the image pixel coordinates through point cloud projection.

7. The multi-sensor fusion coordinated positioning method for ship loading and unloading equipment according to claim 1, characterized in that: The specific process of updating the point cloud map in real time is as follows: The monocular camera acquires continuous frame images; The laser radar point cloud data is detected through point cloud clustering algorithm and motion state judgment to obtain dynamic object clusters, and the dynamic object clusters are projected onto continuous frame images to calculate the laser radar detection area; The optical flow motion vector and its corresponding dynamic area are calculated by the optical flow method to obtain the dynamic mask, and the monocular camera visual optical flow area corresponding to the dynamic mask is calculated; The intersection-over-union ratio of the lidar detection area and the monocular camera visual optical flow area is used as the confidence level. If the confidence level is greater than the set value, the object is confirmed to be a dynamic obstacle. Remove the confirmed dynamic objects from the current point cloud map to obtain an updated point cloud map.

8. A multi-sensor fusion coordinated positioning system for ship loading and unloading equipment, characterized in that: include: The data acquisition module is configured to: use a lidar to scan the cabin, generate lidar point cloud data, and construct a lidar coordinate system and point cloud map; use a monocular camera to acquire images of the loading and unloading equipment operating area, generate camera image data, and construct a camera coordinate system; The data processing module is configured to: perform anti-interference processing on the lidar point cloud data; perform anti-illumination change processing on the camera image data; The data fusion module is configured to: fuse the lidar point cloud data with the camera image data to obtain fused positioning information; The equipment coordination module is configured to calculate the distance and angle of the loading and unloading equipment based on the fused positioning information, generate movement instructions, and drive the loading and unloading equipment to coordinate positioning; The data update module is configured to detect dynamic objects in the cabin through lidar and monocular camera, perform multimodal verification on the detection results, filter dynamic objects and update the point cloud map in real time.

9. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, the steps of the multi-sensor fusion ship loading and unloading equipment collaborative positioning method as described in any one of claims 1 to 7 are implemented.

10. An electronic device comprising a memory, a processor, and a program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps in the multi-sensor fusion ship loading and unloading equipment collaborative positioning method as described in any one of claims 1 to 7 are implemented.

Images