Method and system for detecting number of shipping containers based on Leiyu fusion algorithm

Abstract

The invention discloses a method and a system for intelligently detecting the number of shipping containers based on a Leiyu fusion algorithm. According to the method, the ship appearance in a camera image is recognized through a deep learning model, the ship profile is measured in combination with a laser radar, three-dimensional point cloud data is obtained and enhanced and complemented, and high-precision measurement of the ship size is achieved. The multi-modal data fusion and space alignment technology is adopted, visual and radar information is analyzed in a combined mode, and the detection accuracy of the ship position, the boundary and the cargo hold area is improved. Through improved boundary fitting and a self-adaptive algorithm, the number of containers is automatically calculated, the detection efficiency and accuracy are remarkably improved, the method is suitable for complex scenes such as simultaneous passing of multiple ships, and the intelligent shipping management requirement is met.

Description

Technical Field

[0001] The present invention relates to the technical field of container identification, and in particular to a method and system for detecting the number of shipping containers based on a radar-visual fusion algorithm. Background Art

[0002] Traditional methods for measuring the number of shipping containers rely primarily on manual verification or single sensors (such as video surveillance systems, vessel name recognition systems, or radar systems). Manual verification is not only inefficient and error-prone, but also significantly affected by environmental factors such as field visibility and weather, making it difficult to ensure accuracy and hindering subsequent supervision and review. While existing video surveillance systems can capture images of ships, they often require manual counting of vessel identity information and are unable to obtain precise dimensional data. Vessel name recognition systems only provide information about the vessel's appearance, making it difficult to accurately identify the vessel type and obtain detailed identity information. While radar systems can measure a vessel's size, speed, and direction, they still rely on other information for identity verification and struggle to efficiently match visual information.

[0003] Furthermore, existing container quantity detection algorithms (such as backtracking and dynamic programming) suffer from low computational efficiency and high time complexity when processing large amounts of data, making them difficult to meet practical application requirements. The accuracy and consistency of sensor data also face challenges under complex operating conditions. Traditional timestamp-based methods for matching visual and radar information require extremely high synchronization accuracy, and are prone to mismatching identity and size information due to time delays or interference from multiple targets, limiting their practicality and robustness.

[0004] In summary, existing technologies have prominent problems such as low accuracy, low efficiency, and poor adaptability in the automatic identification, dimension measurement, matching of identity and dimension information, and automatic detection of container quantities of container ships. There is an urgent need for an intelligent detection method and system that can integrate multi-source sensor data, improve detection accuracy, and improve matching robustness. Summary of the Invention

[0005] In order to solve the above problems, the present invention provides a method for intelligent detection of the number of shipping containers based on a radar-visual fusion algorithm, which is characterized by comprising the following steps:

[0006] S1: Use deep learning models to identify the appearance of ships in images from visual acquisition equipment and determine the type of ship;

[0007] S2: Use high-precision LiDAR to measure the ship profile, obtain hull data and restore the three-dimensional effect, and perform data enhancement and completion on the original point cloud;

[0008] S3: Obtain radar point cloud and measure ship size information to enhance the completion effect of ship edge points;

[0009] S4: Combining lidar and camera data, determining the ship's position and bounding box through computer vision and radar point cloud algorithms, correlating detection results, and fusing information;

[0010] S5: Measure the length and width of the ship's cargo hold and the area below the deck, use an improved algorithm to fit the cargo hold outline, and calculate the number of containers carried.

[0011] Furthermore, in step S1, ship images or sensor data are acquired through monitoring cameras and radars, and ships are classified and identified through pre-trained models or deep learning algorithms. Surveillance camera image and video data are collected, and multimodal data fusion is used to improve recognition robustness. An adaptive region proposal network (RPN) is designed to dynamically adjust the generation strategy of the proposal box according to the size and shape of the ship, thereby improving the accuracy of target detection.

[0012] Furthermore, in step S2, data enhancement is performed on the original point cloud by rotating and scaling to improve the robustness of the subsequent model.

[0013] Furthermore, in step S3, the ship size information is measured using the point cloud scale calculation algorithm, and the edge point weight loss function is used to enhance the completion effect of the completion training network on the ship edge points.

[0014] Furthermore, in step S4, by calibrating the internal and external parameters of the lidar and camera, a projection matrix from the point cloud to the image is constructed to achieve spatial alignment of the point cloud and the image; a semantic segmentation network and an edge detection algorithm are used to extract visual boundary information, and the visual boundary features are fused with the point cloud features in the early or mid-term, and three-dimensional segmentation is performed through a three-dimensional point cloud deep neural network to improve the accuracy and robustness of cargo hold area detection.

[0015] Furthermore, in step S5, an improved boundary fitting algorithm is used to more accurately fit the contour of the cargo hold. High-precision fitting improves the accuracy of length and width measurement. Through the adaptive fitting algorithm, the fitting parameters can be automatically adjusted according to the shape and size of the cargo hold, which is suitable for different types of ships.

[0016] Furthermore, in the process of matching ship identity information and size information, the radar point cloud and visual high-level semantic features are fused, and through spatial calibration and dynamic alignment, the joint reasoning and matching fusion of multimodal features are realized. Based on spatial consistency constraints and size association rules, the accuracy of identity-size matching in scenarios where multiple ships pass simultaneously is improved.

[0017] An intelligent shipping container quantity detection system based on a radar-visual fusion algorithm includes:

[0018] a ship identification module for executing step S1 of claim 1, identifying the appearance of a ship in an image captured by a visual acquisition device using a deep learning model, and determining the type of the ship;

[0019] a laser radar measurement module, configured to execute step S2 of claim 1, obtain hull point cloud data and restore the three-dimensional effect, and perform data enhancement and completion on the original point cloud;

[0020] a size calculation module, configured to execute step S3 of claim 1, obtain radar point clouds and use relevant algorithms to measure ship size information, thereby enhancing the completion effect of ship edge points;

[0021] a multimodal data fusion and 3D segmentation module for performing step S4 of claim 1, combining lidar and camera data, determining the vessel's position and bounding box through spatial alignment, feature fusion, and 3D segmentation algorithms, correlating detection results, and fusing the information;

[0022] The container quantity calculation module is used to execute step S5 in claim 1, measure the length and width of the ship's cargo hold and the area below the deck, use an improved algorithm to fit the cargo hold contour, and calculate the number of containers carried.

[0023] Furthermore, the ship identification module obtains ship images or sensor data through monitoring cameras and radars, and classifies and identifies ships through pre-trained models or deep learning algorithms. It collects surveillance camera image and video data, improves recognition robustness through multimodal data fusion, and designs an adaptive region proposal network (RPN) to dynamically adjust the generation strategy of the proposal box according to the size and shape of the ship, thereby improving the accuracy of target detection.

[0024] Furthermore, the multimodal data fusion and three-dimensional segmentation module includes:

[0025] Data preprocessing and alignment unit, used to achieve spatiotemporal synchronization and spatial alignment of lidar point clouds and visual images;

[0026] Visual boundary information extraction unit, used to extract image boundary features through semantic segmentation network and edge detection algorithm;

[0027] The feature fusion and 3D segmentation unit is used to fuse visual boundary features with point cloud features, and realize 3D segmentation of the cargo hold area through a 3D point cloud deep neural network.

[0028] Furthermore, the container quantity calculation module adopts a vision-guided boundary optimization method and an adaptive fitting algorithm, which can automatically adjust the fitting parameters according to the shape and size of the cargo hold, is suitable for different types of ships, and realizes high-precision container quantity calculation.

[0029] In summary, the present invention has at least one of the following beneficial effects:

[0030] The present invention proposes a method and system for intelligent detection of shipping container quantity based on a radar-visual fusion algorithm, which has the following beneficial effects:

[0031] This method achieves high-precision recognition and measurement of ship appearance, type, size, and other information by deeply fusing LiDAR point clouds with visual image data. The use of multimodal feature fusion and spatial alignment technology significantly improves the accuracy and robustness of ship detection and container quantity calculation.

[0032] To address the challenge of identity-size information matching in complex scenarios with multiple ships passing simultaneously, this invention innovatively fuses radar point clouds with high-level visual semantic features. Through spatial calibration, dynamic alignment, and joint reasoning of multimodal features, it effectively overcomes the limitations of traditional timestamp-based matching and significantly improves the accuracy and fault tolerance of identity and size information matching.

[0033] The present invention adopts a vision-guided boundary optimization method and an adaptive fitting algorithm, which can automatically adjust the fitting parameters according to the actual shape and size of the cargo hold, realize high-precision fitting and measurement of the cargo hold contours of different types of ships, and further improve the accuracy of container quantity calculation.

[0034] The present invention performs enhancement processing such as rotation and scaling on point cloud data, and optimizes the point cloud completion effect in combination with the edge point weight loss function, effectively avoiding dimensional measurement errors caused by data missing and improving the adaptability and reliability of the system in complex environments.

[0035] The present invention can realize automatic detection and real-time recording of the number of containers during the passage of container ships through locks, providing accurate and timely data support for lock management and subsequent supervision and inspection, greatly improving the automation and intelligent level of shipping management, and reducing manual intervention and error rates. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0037] Figure 1 Flow chart of the method of the present invention;

[0038] Figure 2 This is an example of capturing a container ship;

[0039] Figure 3Identify steps for vessels;

[0040] Figure 4 It is the step of LiDAR measurement and point cloud processing;

[0041] Figure 5 This is an example of a point cloud image after edge enhancement;

[0042] Figure 6 Multimodal data fusion for dissemination data;

[0043] Figure 7 Schematic diagram of the 3D segmentation process with multimodal data fusion and boundary optimization of the present invention;

[0044] Figure 8 This is a schematic diagram of container heights in the visible area of ​​a container ship;

[0045] Figure 9 This is a schematic diagram of container heights in the non-visible area of ​​a container ship;

[0046] Figure 10 This is a schematic diagram for calculating the number of containers on a container ship;

[0047] Figure 11 This is a schematic diagram of the height division of the container top surface under irregular loading conditions;

[0048] Figure 12 This is a schematic diagram of calculating the number of container layers in each area under irregular loading conditions;

[0049] Figure 13 This is a flowchart for multimodal feature fusion and matching of container ship identity information and size information;

[0050] Figure 14 Schematic diagram of the cross-attention fusion mechanism of radar point cloud and visual features. DETAILED DESCRIPTION

[0051] The present invention will be further described in detail below with reference to the accompanying drawings.

[0052] Example 1:

[0053] This embodiment provides a method for intelligently detecting the number of shipping containers based on a radar-visual fusion algorithm, comprising the following steps:

[0054] S1: Use deep learning models to identify the appearance of ships in images from visual acquisition equipment and determine the type of ship.

[0055] For example Figure 2 The example ship uses a deep learning algorithm model to identify the appearance of the ship in the image connected to the visual acquisition device. The ship identification process is as follows: Figure 3As shown, the system outputs the vessel's exterior color and type (e.g., hazardous chemical carrier, passenger and cargo ship, container ship, oil tanker, etc.). This system can acquire ship images or sensor data from surveillance cameras, radar, and other devices, and classify and identify ships using pre-trained models or deep learning algorithms. Video data from surveillance cameras is collected and multimodally fused (visual data and lidar data) to improve recognition robustness. An adaptive RPN is designed to dynamically adjust the proposed box generation strategy based on the ship's size and shape, improving object detection accuracy. This system accurately identifies different types of ships, particularly container ships.

[0056] S2: Use high-precision lidar to measure the ship profile, obtain hull data and restore the three-dimensional effect, and perform data enhancement and completion on the original point cloud.

[0057] The overall steps are as follows Figure 4 As shown, a high-precision lidar is used as the core equipment for ship size detection. The sensor obtains cross-sectional image data of the ship through ship profile measurement. Upon receiving a trigger request from the video analysis module, the high-precision lidar immediately starts the measurement scan. After hundreds of cycles, it reads the large amount of metadata returned and uses algorithms to restore the three-dimensional effect of the ship in the scan area. The original point cloud is enhanced by applying random rotation, translation, and scale transformation to the point cloud to improve model robustness, or performing color dithering and random cropping on the image to enhance adaptability to lighting changes. The collected point cloud of one side of the ship is then input into the point cloud completion model, which outputs a complete point cloud of the ship.

[0058] S3: Obtain radar point cloud and measure ship size information to enhance the completion effect of ship edge points.

[0059] The radar point cloud of the ship is acquired for data analysis, and the ship's dimensions are measured using a point cloud scale calculation algorithm. The edge point weight loss function is used to enhance the completion of the ship's edge points in the completion training network. The completed point cloud is used to calculate the ship's dimensions (the length, width, and height of the ship on the water surface), avoiding measurement errors caused by missing point clouds and further improving the accuracy of dimension measurement. Figure 5 Given Figure 2 Enhanced point cloud image after processing the above steps.

[0060] S4: Combining lidar and camera data, determining the ship's position and bounding box through computer vision and radar point cloud algorithms, correlating detection results, and fusing information;

[0061] like Figure 6As shown in the figure, laser radar and camera are used to obtain real-time video images and laser radar scanning data of the ship. Computer vision technology and radar point cloud algorithm are used to detect and segment the ship in the camera image, determine the position and bounding box of the ship, and obtain the number of container layers in the visible area. Multimodal data fusion, by building a projection relationship between the laser radar and camera, associates the image detection results with the point cloud detection results, thereby improving the detection accuracy. Combining Bayesian theory to fuse multi-source perception information further improves the reliability of the detection results. Figure 7 The specific process is as follows:

[0062] First, the LiDAR point cloud and camera image are synchronized in time and space to ensure that the two types of data are collected at the same time. Through hardware triggering or timestamp interpolation, and motion compensation for dynamic scenes, the misalignment caused by sensor delay is eliminated. Then, coordinate alignment is performed. By calibrating the internal and external parameters of the LiDAR and camera, a projection matrix from the point cloud to the image is constructed, and the point cloud data is projected onto the image plane to achieve the spatial correspondence between the point cloud and the image, laying the foundation for subsequent feature fusion. The coordinate alignment method is as follows:

[0063] Calibrate the internal and external parameters of the lidar and camera, and construct the projection matrix from the point cloud to the image pixels:

[0064]

[0065] Where K is the camera intrinsic parameter, R,t is the rotation and translation matrix, and Tlidar2cam is the coordinate transformation from the lidar to the camera. The point cloud is projected onto the image plane, and point-by-point or region-by-region correspondence is established to generate point cloud data with RGB information.

[0066] After preprocessing, the image data is segmented in real time using a semantic segmentation network (such as DeepLabv3+ or HRNet) to obtain pixel-by-pixel class probability maps and extract fine boundaries of objects (such as ships and cargo holds). Edge detection algorithms such as Canny or HED are then used to further extract pixel-level geometric edges. Semantic boundaries are fused with low-level edge information to generate multi-scale boundary confidence maps, providing high-resolution boundary guidance for point cloud segmentation. Specifically:

[0067] This example uses a lightweight model (such as DeepLabv3+ or HRNet) to perform real-time semantic segmentation on the image and output a pixel-by-pixel category probability map. It also outputs fine boundaries containing object outlines (such as building edges and vehicle outlines). It also uses Canny edge detection or a learning-based edge detector (such as HED) to extract pixel-level edge maps and perform edge enhancement. The semantic boundaries are fused with low-level edges to generate a multi-scale boundary confidence map E. vis .

[0068]

[0069] Among them, α is the weight coefficient, which balances the importance of semantic and geometric edges. is the value of the fused visual boundary confidence map at pixel (x, y), is the category probability (i.e., semantic boundary information) at the pixel (x, y) output by the semantic segmentation network, is the edge response (i.e., low-level geometric edge information) of the Canny edge detector at pixel (x, y).

[0070] The point cloud is then fused with visual features. In this embodiment, a sparse convolutional network (such as MinkowskiNet) or a point cloud transformer is used as the backbone network to process large-scale point clouds and extract multi-scale features. The specific steps include:

[0071] Early fusion: The visual boundary features projected onto the point cloud are used as additional channels and spliced ​​with the coordinates, reflectivity and other attributes of the point cloud to form an enhanced point cloud input:

[0072]

[0073] in, is the input feature set of the point cloud-vision fusion network, is the 3D coordinate of the point cloud. is the point cloud reflection intensity, It is a three-dimensional feature that integrates visual boundary information.

[0074] Mid-term fusion: In the middle layer of the point cloud 3D segmentation network (such as the sparse convolutional network MinkowskiNet or the point cloud Transformer), the image features and point cloud features are dynamically aligned through the cross-modal attention mechanism to achieve deeper information fusion:

[0075]

[0076] Among them, Q point is the point cloud feature query, K image is the image feature key, V image is the image eigenvalue, and d is the feature dimension.

[0077] After fusion, the multimodal features are processed by a three-dimensional segmentation network: the fused multimodal features are input into the three-dimensional point cloud segmentation network, and the network's powerful feature extraction and expression capabilities are used to perform three-dimensional segmentation on targets such as ships and cargo holds, output the semantic label of each point, and achieve accurate recognition of ship structures and cargo hold areas.

[0078] Vision-guided boundary optimization and label correction: Based on the initial 3D segmentation results, a vision-guided boundary optimization strategy is used to further improve segmentation accuracy. Specifically:

[0079] First, the 3D segmentation result is back-projected into image space and aligned with the high-resolution boundary information extracted from the image. Methods such as conditional random fields (CRFs) and boundary-aware loss are used to carefully optimize the segmentation boundaries using visual boundary information, making the segmentation results more accurate at object boundaries. For boundary regions, a boundary-weighted loss function can also be used to emphasize the network's focus on boundary pixels, further reducing segmentation error. The optimized 2D mask is then projected back into 3D space, and the point cloud labels are corrected to achieve refined and consistent 3D segmentation boundaries.

[0080] This embodiment provides a specific process of the above-mentioned projection-optimization iteration:

[0081] (1) Back-projection of 3D segmentation results into image space

[0082] The point cloud segmentation results are projected onto the camera's image plane using sensor calibration parameters. Each 3D point is mapped to a pixel or small region on the image, forming a 2D segmentation mask. The goal is to directly align and compare the 3D segmentation results with the original image, especially the high-resolution boundary information in the image.

[0083] (2) Image space boundary optimization

[0084] Utilizing the image's high-resolution semantic segmentation and edge detection results (such as boundaries obtained using Canny or HED, and semantic masks obtained using DeepLabv3+ / HRNet), we apply conditional random fields (CRFs) and boundary-aware losses (such as Diceloss, Boundary loss, and Edge loss) to optimize this 2D mask, correcting and refining boundaries and improving consistency with real-world object edges. We also enforce stronger constraints and supervision on segmentation boundaries to ensure that optimization errors are primarily concentrated in the boundary regions.

[0085] (3) The optimization results are projected back into three-dimensional space

[0086] The optimized 2D segmentation mask is then back-projected back into the point cloud space using the previous projection mapping relationship, updating and correcting the original 3D segmentation labels. The labels of some point clouds are updated based on the visual optimization results, especially at the boundaries, where semantic information is more reliably and precisely annotated.

[0087] Design a joint optimization loss function: During network training, a joint optimization loss function is designed to consider both segmentation accuracy and boundary alignment, thereby improving the accuracy and robustness of the overall 3D segmentation.

[0088]

[0089] in, is the total loss function, is the cross entropy loss (Cross Entropy Loss), To predict the segmentation results is the true label for segmentation, and λ is the balance factor is the edge-weighted loss, E pred is the prediction boundary, E vis The visual boundary.

[0090] In summary, by high-precision alignment and fusion of multimodal features of LiDAR point clouds and visual images, a deep learning network was used to achieve fine segmentation in three-dimensional space. In particular, visual information was used to guide boundary regions, significantly improving segmentation accuracy and adaptability. This provided a solid data foundation and technical support for subsequent ship dimension measurement, cargo hold contour fitting, and automatic container count calculation.

[0091] S5: Measure the length and width of the ship's cargo hold and the area below the deck, use an improved algorithm to fit the cargo hold outline, and calculate the number of containers carried.

[0092] Cargo hold measurement and container quantity calculation: Use LiDAR data to measure the length and width of the cargo hold and below deck, and determine the actual length and width of the cargo hold and the number of container layers in the non-visible area below deck. The improved boundary fitting algorithm can more accurately fit the contour of the cargo hold, and high-precision fitting improves the accuracy of length and width measurement. The adaptive fitting algorithm can automatically adjust the fitting parameters according to the shape and size of the cargo hold, which is suitable for different types of ships. Figure 8-10 , divide the measured total length of the ship's cargo hold by the length of a standard container, divide the total width of the ship's cargo hold by the width of a standard container, add the number of container layers below the deck and the number of container layers in the visible area, and finally multiply the three together to get the number of containers carried.

[0093] like Figure 11-12As shown in the figure, for irregular loading of containers, the vertical height hi from the top of the container to the lidar is calculated using the lidar point cloud algorithm. According to the vertical height hi from the top of the container to the lidar, the container is divided into different areas (area 1, area 2, area 3...). The total height Hi of the loaded containers in each area is obtained by adding the deck to the lidar cross-sectional height plus the molded depth D (the ship's molded depth is used to approximate the cargo hold depth, and the height difference between the two is small relative to the container height) and subtracting the vertical height hi from the container surface in each area when irregularly loaded. The total height Hi of the containers in each area with irregular loading is divided by the standard height h of the container to obtain the number of layers of loaded containers in each area. Layer. Calculate the number of containers in each area based on the calculated number of container layers in each area.

[0094] Example 2:

[0095] In response to the problem that the traditional approach of matching and binding visual recognition information (such as ship name) and radar measurement dimensions at the same time by timestamps requires extremely high time synchronization accuracy, is prone to information matching errors due to delays or the passage of multiple ships at the same time, and has stringent requirements on hardware synchronization conditions, and has limited practicality, this embodiment provides a method that uses radar point cloud and visual high-level semantic features to achieve a highly robust matching method for container ship identity information (such as ship name, appearance features) and size information (such as length, width, height, and structural parameters). Figure 13 The specific process is as follows:

[0096] (1) Radar point cloud processing branch

[0097] Point cloud semantic segmentation: Using deep networks such as 3D U-Net or PointTransformer based on sparse convolution, the original point cloud (including coordinates and reflection intensity) is semantically segmented point by point to distinguish the ship body, structural components, background, etc., and generate 3D bounding box parameters.

[0098] Key feature extraction: Extract structural features (such as length, width, height, and main axis direction) and reflection features (such as highly reflective areas of metal parts), and introduce local geometric descriptors (such as FPFH features) to enhance the ability to recognize detailed parts.

[0099] (2) Visual processing branch

[0100] Ship key point detection: Use improved HRNet and other networks to detect ship key points (such as bow, stern, and cockpit vertex) and predict the geometric relationship between key points.

[0101] Visual semantic feature extraction: Use networks such as ResNet-50 to extract features such as the ship's appearance texture and color. Use methods such as Mask R-CNN to locate and extract text features in the ship name area.

[0102] Feature encoding: Concatenate key point coordinates and visual features into a unified vector for subsequent fusion.

[0103] (3) Multimodal feature fusion

[0104] Spatial calibration and alignment: Offline calibration is used to obtain the extrinsic parameter matrices of the radar and camera, enabling projection of 3D point clouds onto image pixels. For moving targets, the ICP algorithm is used to optimize the projected coordinates in real time, eliminating misalignment caused by vessel movement.

[0105] Feature fusion strategy: Use the cross-attention mechanism to calculate the association weights of radar and visual features and output the fused joint features.

[0106] (4) Joint reasoning and matching fusion

[0107] Object detection and size estimation: A multi-task network is trained based on fused features. Branch 1 predicts the center point of the ship, and branch 2 regresses the three-dimensional size (length, width, and height).

[0108] Ship name recognition: The visual branch is used to locate the text area, and CRNN (CNN+BiLSTM+CTC) is used for end-to-end recognition. The point cloud projection coordinates are combined to filter out false detections in non-ship hull areas.

[0109] Identity-Dimension Matching Fusion: This approach leverages spatial consistency constraints (e.g., the center of the ship's name area must be within the ship's bounding box as detected by radar) and dimension association rules (e.g., comparison with a ship type database) to achieve highly reliable matching of identity and dimension information. Finally, a confidence fusion strategy dynamically adjusts the weighting of each sensor's information to improve matching accuracy.

[0110] The following is a specific example of achieving highly robust matching of container ship identity and size information by fusing radar point clouds with high-level visual semantic features:

[0111] (1) Radar point cloud processing branch:

[0112] (11) Point cloud semantic segmentation

[0113] The network architecture uses 3D U-Net or PointTransformer based on sparse convolution.

[0114] Input raw point cloud (coordinate + reflection intensity);

[0115] Output: point-by-point semantic labels (ship body, hull structural components, background, etc.), generating the ship's three-dimensional bounding box parameters Bradar = (xc, yc, zc, l, w, h, θ).

[0116] This method enhances the recognition ability of hull details (masts, decks) by introducing local geometric descriptors (such as FPFH features) as additional input channels.

[0117] (12) Point cloud key feature extraction

[0118] Feature types include:

[0119] Structural characteristics: hull length, width, height, symmetry axis direction, principal component analysis (PCA) direction;

[0120] Reflection characteristics: distribution of high-reflection areas (corresponding to metal parts);

[0121] Mathematical expression: Calculate the segmented ship point cloud subset Pship:

[0122]

[0123] (2) Visual processing branch

[0124] (21) Ship key point detection

[0125] Network Design: Using an improved HRNet, we output heatmaps of key points of the ship (bow, stern, cockpit vertex, etc.) and simultaneously predict the geometric constraint relationships between key points.

[0126] Loss function:

[0127]

[0128] in The relative distance and angle of the constraint key points.

[0129] (22) Visual semantic feature extraction

[0130] Feature Type:

[0131] Appearance features: Ship texture and color distribution extracted by ResNet-50;

[0132] Text features: ROI features of the ship name area (using Mask R-CNN to locate the text area);

[0133] Feature encoding: concatenate keypoint coordinates and visual features into a unified vector:

[0134] .

[0135] (3) Multimodal feature fusion

[0136] (31) Spatial calibration and alignment

[0137] Calibration matrix calculation:

[0138] Obtain the radar-camera extrinsic matrix Tradar2cam through offline calibration to realize the projection of 3D point cloud to image pixels:

[0139]

[0140] Dynamic alignment compensation: The ICP algorithm is used to optimize the projection coordinates of moving targets (ships) in real time to eliminate misalignment caused by ship displacement.

[0141] Feature fusion strategies such as Figure 14 As shown, the cross attention mechanism adopted is as follows:

[0142] Calculate the association weight of radar and visual features: αij, and output fusion features:

[0143]

[0144] (4) Joint reasoning and matching fusion

[0145] (41) Object detection and size estimation

[0146] Detection network: Multi-task head network training based on fusion features:

[0147] Branch 1: Ship center point prediction (Heatmap regression);

[0148] Branch 2: 3D size regression (length, width, and height prediction);

[0149] Loss function:

[0150]

[0151] (42) Ship name identification

[0152] Text recognition process: Use the visual branch to locate the text ROI area; use CRNN (CNN+BiLSTM+CTC) for end-to-end recognition; combine point cloud projection coordinates to filter out false detections in non-hull areas.

[0153] (43) Identity-Size Matching Fusion

[0154] Matching strategies include:

[0155] Spatial consistency constraint: The center of the ship name area must be within the ship bounding box detected by the radar.

[0156] Size association rules: Validate the rationality of predicted sizes against a database of vessel types (e.g., standard sizes for tankers / cargo ships).

[0157] Using confidence fusion:

[0158]

[0159] Where γ is dynamically adjusted according to the sensor reliability.

[0160] The method disclosed in this embodiment has a greater fault tolerance for time synchronization through feature fusion, and can also achieve good information matching when multiple ships pass through at the same time.

[0161] Example 3: This embodiment provides an intelligent shipping container quantity detection system based on a radar-visual fusion algorithm, including: A ship identification module, configured to execute step S1 in Example 1, using a deep learning model to identify the appearance of a ship in an image captured by a visual acquisition device and determine the type of the ship; A laser radar measurement module is used to execute step S2 in Example 1, obtain the hull point cloud data and restore the three-dimensional effect, and perform data enhancement and completion on the original point cloud; A size calculation module, configured to execute step S3 in Example 1, obtain radar point clouds and use relevant algorithms to measure ship size information, thereby enhancing the completion effect of ship edge points; A multimodal data fusion and 3D segmentation module, configured to execute step S4 in Example 1, combine the lidar and camera data, determine the vessel's position and bounding box through spatial alignment, feature fusion, and 3D segmentation algorithms, correlate the detection results, and fuse the information; The container quantity calculation module is used to execute step S5 in Example 1, measure the length and width of the ship's cargo hold and the area below the deck, use an improved algorithm to fit the cargo hold contour, and calculate the number of carried containers.

[0162] This embodiment provides a method for acquiring ship images or sensor data through monitoring cameras and radars, classifying and identifying ships using pre-trained models or deep learning algorithms, collecting surveillance camera image and video data, improving recognition robustness through multimodal data fusion, and designing an adaptive region proposal network (RPN). The strategy for generating proposal boxes is dynamically adjusted based on the size and shape of the ship, thereby improving target detection accuracy.

[0163] The multimodal data fusion and 3D segmentation module provided in this embodiment includes: Data preprocessing and alignment unit, used to achieve spatiotemporal synchronization and spatial alignment of lidar point clouds and visual images; Visual boundary information extraction unit, used to extract image boundary features through semantic segmentation network and edge detection algorithm; The feature fusion and 3D segmentation unit is used to fuse visual boundary features with point cloud features, and realize 3D segmentation of the cargo hold area through a 3D point cloud deep neural network.

[0164] The container quantity calculation module provided in this embodiment adopts a vision-guided boundary optimization method and an adaptive fitting algorithm, which can automatically adjust the fitting parameters according to the shape and size of the cargo hold. It is suitable for different types of ships and can achieve high-precision container quantity calculation.

[0165] The above are all preferred embodiments of the present invention, and are not intended to limit the scope of protection of the present invention. Therefore, any equivalent changes made based on the structure, shape, and principle of the present invention should be included in the scope of protection of the present invention.

Claims

1. An intelligent detection method for the number of shipping containers based on a radar-visual fusion algorithm, characterized in that: The following steps are involved: S1: Use deep learning models to identify the appearance of ships in images from visual acquisition equipment and determine the type of ship; S2: Use high-precision LiDAR to measure the ship profile, obtain hull data and restore the three-dimensional effect, and perform data enhancement and completion on the original point cloud; S3: Obtain radar point cloud and measure ship size information to enhance the completion effect of ship edge points; S4: Combining lidar and camera data, determining the ship's position and bounding box through computer vision and radar point cloud algorithms, correlating detection results, and fusing information; S5: Measure the length and width of the ship's cargo hold and the area below the deck, use an improved algorithm to fit the cargo hold outline, and calculate the number of containers carried.

2. The method for intelligently detecting the number of shipping containers based on a radar and visual fusion algorithm according to claim 1 is characterized by: In step S1, ship images or sensor data are acquired through monitoring cameras and radars, and ships are classified and identified using pre-trained models or deep learning algorithms. Surveillance camera image and video data are collected, and multimodal data fusion is used to improve recognition robustness. An adaptive region proposal network (RPN) is designed to dynamically adjust the generation strategy of the proposal box according to the size and shape of the ship, thereby improving the accuracy of target detection.

3. The method for intelligently detecting the number of shipping containers based on a radar and visual fusion algorithm according to claim 1 is characterized by: In step S2, data enhancement is performed on the original point cloud by rotating and scaling to improve the robustness of the subsequent model.

4. The method for intelligently detecting the number of shipping containers based on a radar and visual fusion algorithm according to claim 1 is characterized by: In step S3, the ship size information is measured using the point cloud scale calculation algorithm, and the edge point weight loss function is used to enhance the completion effect of the completion training network on the ship edge points.

5. The method for intelligent detection of shipping container quantity based on radar and visual fusion algorithm according to claim 1 is characterized in that In step S4, the internal and external parameters of the lidar and camera are calibrated to construct a projection matrix from the point cloud to the image, and spatial alignment of the point cloud and the image is achieved. The semantic segmentation network and edge detection algorithm are used to extract visual boundary information, and the visual boundary features are fused with the point cloud features in the early or mid-term. The 3D point cloud deep neural network is used to perform 3D segmentation to improve the accuracy and robustness of cargo hold area detection.

6. The method for intelligently detecting the number of shipping containers based on a radar and visual fusion algorithm according to claim 1 is characterized by: In step S5, an improved boundary fitting algorithm is used to fit the outline of the cargo hold to improve the accuracy of length and width measurement. The adaptive fitting algorithm automatically adjusts the fitting parameters according to the shape and size of the cargo hold, making it suitable for different types of ships.

7. The method for intelligently detecting the number of shipping containers based on a radar and visual fusion algorithm according to claim 1 is characterized by: In the process of matching ship identity information and size information, the radar point cloud and visual high-level semantic features are fused, and through spatial calibration and dynamic alignment, the joint reasoning and matching fusion of multimodal features are realized. Based on spatial consistency constraints and size association rules, the accuracy of identity-size matching in scenarios where multiple ships pass simultaneously is improved.

8. An intelligent detection system for the number of shipping containers based on radar and visual fusion algorithm, characterized by: include: a ship identification module for executing step S1 of claim 1, identifying the appearance of a ship in an image captured by a visual acquisition device using a deep learning model, and determining the type of the ship; a laser radar measurement module, configured to execute step S2 of claim 1, obtain hull point cloud data and restore the three-dimensional effect, and perform data enhancement and completion on the original point cloud; a size calculation module, configured to execute step S3 of claim 1, obtain radar point clouds and use relevant algorithms to measure ship size information, thereby enhancing the completion effect of ship edge points; a multimodal data fusion and 3D segmentation module for performing step S4 of claim 1, combining lidar and camera data, determining the vessel's position and bounding box through spatial alignment, feature fusion, and 3D segmentation algorithms, correlating detection results, and fusing the information; The container quantity calculation module is used to execute step S5 in claim 1, measure the length and width of the ship's cargo hold and the area below the deck, use an improved algorithm to fit the cargo hold contour, and calculate the number of containers carried.

9. The intelligent shipping container quantity detection system based on radar and visual fusion algorithm according to claim 7 is characterized by: The ship identification module acquires ship images or sensor data through monitoring cameras and radars, and classifies and identifies ships using pre-trained models or deep learning algorithms. It collects surveillance camera image and video data, improves recognition robustness through multimodal data fusion, and designs an adaptive region proposal network (RPN) to dynamically adjust the generation strategy of the proposal box according to the size and shape of the ship, thereby improving the accuracy of target detection.

Images