Ship freeboard height detection method and device and electronic equipment
By using cross-modal alignment parameters and a heuristic attention network, the freeboard height of ships is automatically detected, solving the problem of low efficiency in manual measurement and achieving efficient and accurate freeboard height detection.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-09
- Publication Date
- 2026-03-27
AI Technical Summary
Current technologies rely on manual measurement for ship freeboard height, which is inefficient and cannot meet the requirements for high efficiency.
By using point cloud features and visual features of the target ship, cross-modal alignment parameters are determined. By utilizing the spatial transformation relationship between lidar and image acquisition equipment, combined with a heuristic attention network, the freeboard weight distribution is automatically detected, and the freeboard height is accurately determined.
It has achieved automated freeboard height detection, which improves measurement efficiency, ensures measurement accuracy, and reduces the need for human resources.
Smart Images
Figure CN121739900A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of image processing, in particular to a ship freeboard height detection method and device and electronic equipment. BACKGROUND
[0002] The ship freeboard refers to the distance between the upper surface of the deck of the ship and the waterline of the ship. If the ship is overloaded and the freeboard is less than the specified height, the ship cannot float safely on the water surface. Therefore, relevant personnel need to detect the freeboard height of the ship during navigation to determine whether the ship has safety hazards.
[0003] Currently, the height measurement of the ship freeboard mainly relies on manual measurement. For example, relevant personnel measure and record the height of the freeboard by boarding the ship in a designated water area before the ship lock. This method requires a large amount of human resources and cannot meet the requirement of high efficiency. SUMMARY
[0004] Therefore, the present application provides a ship freeboard height detection method, device and electronic equipment to improve the freeboard height detection efficiency while ensuring the measurement accuracy.
[0005] The technical scheme provided by the present application is as follows: According to the embodiment of the first aspect of the present application, a ship freeboard height detection method is provided, which comprises: Based on the point cloud features of the target ship and the visual features of the target ship, a cross-modal alignment parameter is determined; the point cloud features are obtained based on the point cloud data collected by the laser radar, the visual features are obtained based on the image data collected by the image collection device, and the cross-modal alignment parameter is used to represent the spatial transformation relationship between the first coordinate system corresponding to the laser radar and the second coordinate system corresponding to the image collection device; The cross-modal alignment parameter, the point cloud features and the visual features are used to determine a freeboard weight distribution; the freeboard weight distribution is used to represent whether the position corresponding to each feature point in the point cloud features belongs to the freeboard area on the target ship; The freeboard height of the target ship is determined according to the freeboard weight distribution.
[0006] Optionally, the cross-modal alignment parameter is determined based on the point cloud features of the target ship and the visual features of the target ship, comprising: The point cloud features and the visual features are input into an unsupervised alignment network based on geometric consistency constraints to obtain the cross-modal alignment parameter; wherein the unsupervised alignment network determines the cross-modal alignment parameter by optimizing a total loss function.
[0007] Optionally, the total loss function includes geometric projection consistency constraint parameters and depth consistency constraint parameters; the step of inputting the point cloud features and the visual features into an unsupervised alignment network based on geometric consistency constraints to obtain the cross-modal alignment parameters includes: Based on the point cloud features and the visual features, geometric projection consistency constraint parameters and depth consistency constraint parameters are determined. The geometric projection consistency constraint parameters indicate the difference between the projection position of a point cloud feature point in the point cloud features after transformation from the first coordinate system to the second coordinate system, projected onto a two-dimensional plane corresponding to the visual feature, and the position of a visual feature point matching that point cloud feature point in the visual feature on the two-dimensional plane. The depth consistency constraint parameters indicate the difference between the first depth information of a point cloud feature point in the point cloud features after transformation from the first coordinate system to the second coordinate system, and the second depth information estimated by the visual features at the same spatial point. The total loss function is constructed based on the geometric projection consistency constraint and the depth consistency constraint, and the unsupervised alignment network is optimized by minimizing the total loss function to obtain the cross-modal alignment parameters.
[0008] Optionally, determining the freeboard weight distribution using the cross-modal alignment parameters, the point cloud features, and the visual features includes: The cross-modal alignment parameters, the visual features, and the point cloud features are input into a heuristic attention network to obtain a freeboard weight distribution; the freeboard weight distribution includes the probability that the position of each feature point in each point cloud feature on the target ship belongs to the freeboard region.
[0009] Optionally, the heuristic attention network includes a freeboard region segmentation network and a cross-modal heuristic attention network; the step of inputting the cross-modal alignment parameters, the visual features, and the point cloud features into the heuristic attention network to obtain the freeboard weight distribution includes: Based on the freeboard region segmentation network, the visual features are semantically segmented to obtain a visual mask of the freeboard region of the target ship. Based on the cross-modal alignment parameters and the point cloud features, the point cloud features are transformed from the first coordinate system to the second coordinate system; The visual mask and the point cloud features in the second coordinate system are input into the cross-modal heuristic attention network to obtain the freeboard weight distribution.
[0010] Optionally, determining the freeboard height of the target vessel based on the freeboard weight distribution includes: The freeboard is determined from the point cloud data of the target vessel based on the freeboard weight distribution, and the freeboard height of the target vessel is determined based on the point cloud data corresponding to the freeboard.
[0011] Optionally, determining the freeboard height of the target vessel based on the point cloud data corresponding to the freeboard includes: Determine the vertical coordinates of the point cloud data of the freeboard area, and determine the freeboard height of the target vessel based on the maximum and minimum values of the vertical coordinates of the point cloud data of the freeboard area; or, input the point cloud data of the freeboard area into a trained regression network to obtain the freeboard height of the target vessel.
[0012] According to an embodiment of the second aspect of this application, a ship freeboard height detection device is provided, the device comprising: The alignment parameter determination unit is used to determine cross-modal alignment parameters based on the point cloud features and visual features of the target ship; the point cloud features are obtained based on point cloud data collected by lidar, the visual features are obtained based on image data collected by image acquisition device, and the cross-modal alignment parameters are used to characterize the spatial transformation relationship between the first coordinate system corresponding to the lidar and the second coordinate system corresponding to the image acquisition device. The weight distribution determination unit is used to determine the freeboard weight distribution using the cross-modal alignment parameters, the point cloud features, and the visual features; the freeboard weight distribution is used to characterize whether the position of each feature point in the point cloud features on the target ship belongs to the freeboard area; A freeboard height detection unit is used to determine the freeboard height of the target vessel based on the freeboard weight distribution.
[0013] Optionally, the alignment parameter determination unit is specifically used for: The point cloud features and the visual features are input into an unsupervised alignment network based on geometric consistency constraints to obtain the cross-modal alignment parameters; wherein, the unsupervised alignment network determines the cross-modal alignment parameters by optimizing the total loss function; And / or, the total loss function includes geometric projection consistency constraint parameters and depth consistency constraint parameters; the alignment parameter determination unit is specifically used for: Based on the point cloud features and the visual features, geometric projection consistency constraint parameters and depth consistency constraint parameters are determined. The geometric projection consistency constraint parameters indicate the difference between the projection position of a point cloud feature point in the point cloud features after transformation from the first coordinate system to the second coordinate system, projected onto a two-dimensional plane corresponding to the visual feature, and the position of a visual feature point matching that point cloud feature point in the visual feature on the two-dimensional plane. The depth consistency constraint parameters indicate the difference between the first depth information of a point cloud feature point in the point cloud features after transformation from the first coordinate system to the second coordinate system, and the second depth information estimated by the visual features at the same spatial point. The total loss function is constructed based on the geometric projection consistency constraint and the depth consistency constraint, and the unsupervised alignment network is optimized by minimizing the total loss function to obtain the cross-modal alignment parameters. And / or, the weight distribution determination unit is specifically used for: The cross-modal alignment parameters, the visual features, and the point cloud features are input into a heuristic attention network to obtain a freeboard weight distribution; the freeboard weight distribution includes the probability that the position of each feature point in each point cloud feature on the target ship belongs to the freeboard region; And / or, the heuristic attention network includes a freeboard region segmentation network and a cross-modal heuristic attention network; the weight distribution determination unit is specifically used for: Based on the freeboard region segmentation network, the visual features are semantically segmented to obtain a visual mask of the freeboard region of the target ship. Based on the cross-modal alignment parameters and the point cloud features, the point cloud features are transformed from the first coordinate system to the second coordinate system; The visual mask and the point cloud features in the second coordinate system are input into the cross-modal heuristic attention network to obtain the freeboard weight distribution; And / or, the freeboard height detection unit is specifically used for: The freeboard is determined from the point cloud data of the target vessel based on the freeboard weight distribution, and the freeboard height of the target vessel is determined based on the point cloud data corresponding to the freeboard. And / or, the freeboard height detection unit is specifically used for: Determine the vertical coordinates of the point cloud data of the freeboard area, and determine the freeboard height of the target vessel based on the maximum and minimum values of the vertical coordinates of the point cloud data of the freeboard area; or, input the point cloud data of the freeboard area into a trained regression network to obtain the freeboard height of the target vessel.
[0014] According to an embodiment of a third aspect of this application, an electronic device is provided, comprising: a processor and a machine-readable storage medium storing machine-executable instructions executable by the processor; the processor is configured to execute the machine-executable instructions to implement the method described in the first aspect.
[0015] As can be seen from the above technical solutions, this application determines cross-modal alignment parameters based on the point cloud features and visual features of the target ship to characterize the spatial transformation relationship between the image acquisition device and the lidar that acquires the point cloud data. Furthermore, by utilizing the cross-modal alignment parameters, point cloud features, and visual features, it determines the freeboard weight distribution to characterize whether the position of each feature point in the point cloud features corresponds to a freeboard area on the target ship. This allows for precise focusing on the freeboard area to be detected, fully utilizing the clear freeboard semantic features in the visual features and the scale information of the point cloud data. Furthermore, based on the freeboard weight distribution, the freeboard height of the target ship is determined, achieving automated freeboard height detection. This improves the efficiency of freeboard height detection while ensuring measurement accuracy and controlling costs. Attached Figure Description
[0016] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the principles of this application.
[0017] Figure 1 This is a schematic diagram of the ship freeboard height detection method provided in the embodiments of this application; Figure 2 A schematic diagram of a geometrically consistent unsupervised network provided in an embodiment of this application; Figure 3 A schematic diagram of a heuristic attention network provided in an embodiment of this application; Figure 4 This is a schematic diagram of the overall network architecture for ship freeboard height detection provided in an embodiment of this application; Figure 5 A structural diagram of the ship freeboard height detection device provided in the embodiments of this application; Figure 6 This is a schematic diagram of an electronic device structure provided in an embodiment of this application. Detailed Implementation
[0018] To enable those skilled in the art to better understand the technical solutions provided in the embodiments of this application, and to make the above-mentioned objectives, features and advantages of the embodiments of this application more apparent and understandable, the technical solutions in the embodiments of this application will be further described in detail below with reference to the accompanying drawings.
[0019] Freeboard refers to the distance between the upper surface of a ship's deck and its waterline. If a ship is overloaded, causing its freeboard to be less than the prescribed height, the ship cannot safely float on the water. Therefore, relevant personnel need to inspect the freeboard height of ships during navigation to determine if there are any safety hazards.
[0020] Specifically, the minimum freeboard of a vessel is the minimum freeboard height required to ensure its safe buoyancy. All vessels adhere to the minimum freeboard height specified by their designated load line. If a vessel is overloaded and its freeboard decreases below the prescribed limit, it cannot safely float. When a vessel is overloaded, its draft increases, resulting in insufficient freeboard height and significant safety hazards, such as the possibility of grounding or collisions. By measuring the vessel's freeboard height during navigation and comparing it to the minimum freeboard, it is possible to determine whether the vessel is overloaded.
[0021] Currently, the measurement of ship freeboard height mainly relies on human vision and manual measurement. For example, in designated waters, relevant personnel are dispatched to board the ship to measure and record the freeboard height. This method requires a lot of human resources and cannot meet the requirements of high efficiency.
[0022] Based on this, this application proposes a method for detecting the freeboard height of a ship to achieve automated freeboard height detection and improve measurement efficiency while ensuring measurement accuracy.
[0023] Please refer to Figure 1 , Figure 1 This is a schematic flowchart of the freeboard height detection method provided in the embodiments of this application; like Figure 1 As shown, the method includes the following steps: Step 101: Determine cross-modal alignment parameters based on the point cloud features of the target ship and the visual features of the target ship.
[0024] The point cloud features are obtained based on point cloud data collected by the lidar, and the visual features are obtained based on image data collected by the image acquisition device. The cross-modal alignment parameter is used to characterize the spatial transformation relationship between the first coordinate system corresponding to the lidar and the second coordinate system corresponding to the image acquisition device.
[0025] In this embodiment, to save costs, the image acquisition device can be a monocular camera, and the lidar can be a low-beam lidar (such as 32 beams, 64 beams, etc.). This application does not impose any restrictions on this.
[0026] In this embodiment, since the image acquisition device and the lidar are not integrated, their relative poses are difficult to obtain through factory calibration when they are photographing the same target ship. Therefore, it is necessary to align the data collected by the image acquisition device and the lidar for the same target ship to determine the spatial transformation relationship between the lidar coordinate system (denoted as the first coordinate system) and the image acquisition device coordinate system (denoted as the second coordinate system).
[0027] Specifically, image data acquired by the camera and point cloud data acquired by the lidar at the current moment can be obtained; feature extraction is performed on the image data to obtain the visual feature matrix of the image data; feature extraction is performed on the point cloud data to obtain the point cloud feature matrix of the point cloud data; based on the visual feature matrix of the image data and / or the point cloud feature matrix of the point cloud data, it can be determined whether the target ship exists at the current moment. If so, the visual feature matrix of the image data is determined as the visual feature matrix of the target ship, and the point cloud feature matrix of the point cloud data is determined as the point cloud feature matrix of the target ship.
[0028] In this embodiment, the method for feature extraction of image data acquired by the camera and point cloud data acquired by the lidar can employ convolutional neural networks, such as ResNet or PointPillar, and this application does not impose any limitations on this method.
[0029] After feature extraction is completed, visual features and point cloud features are obtained. Further, it can be determined whether there is a ship in the current image based on the visual features and / or point cloud features. If it is determined that there is no ship based on either the visual features or the point cloud features, the point cloud data and image data are discarded, and the subsequent freeboard detection process is not performed.
[0030] If a ship is determined to exist in the current image based on visual features and / or point cloud features, the ship is designated as the target ship, and the visual features and point cloud features are determined as the visual features and point cloud features of the target ship, respectively.
[0031] After obtaining the visual features and point cloud features of the target ship, the spatial transformation relationship between the first coordinate system corresponding to the lidar and the second coordinate system corresponding to the image acquisition device can be determined based on the visual features and point cloud features of the target ship.
[0032] As an example, a specific method for determining cross-modal alignment parameters based on the point cloud features and visual features of the target vessel may include: The point cloud features and the visual features are input into an unsupervised alignment network based on geometric consistency constraints to obtain the cross-modal alignment parameters; wherein, the unsupervised alignment network determines the cross-modal alignment parameters by optimizing the total loss function.
[0033] Specifically, the total loss function may include geometric projection consistency constraint parameters and depth consistency constraint parameters. A specific method for inputting the point cloud features and the visual features into an unsupervised alignment network based on geometric consistency constraints to obtain the cross-modal alignment parameters may include: Based on the point cloud features and the visual features, geometric projection consistency constraint parameters and depth consistency constraint parameters are determined. The geometric projection consistency constraint parameters indicate the difference between the projection position of a point cloud feature point in the point cloud features after transformation from the first coordinate system to the second coordinate system, projected onto a two-dimensional plane corresponding to the visual feature, and the position of a visual feature point matching that point cloud feature point in the visual feature on the two-dimensional plane. The depth consistency constraint parameters indicate the difference between the first depth information of a point cloud feature point in the point cloud features after transformation from the first coordinate system to the second coordinate system, and the second depth information estimated by the visual features at the same spatial point. The total loss function is constructed based on the geometric projection consistency constraint and the depth consistency constraint, and the unsupervised alignment network is optimized by minimizing the total loss function to obtain the cross-modal alignment parameters.
[0034] Please refer to Figure 2 , Figure 2 A schematic diagram of a geometrically consistent unsupervised network provided for an embodiment of this application.
[0035] like Figure 2 As shown, the input to this geometric consistency unsupervised network is the visual features obtained after feature extraction from image data acquired by a monocular camera and the point cloud features obtained after feature extraction from point cloud data acquired by a lidar. Through this geometric consistency unsupervised network, cross-modal alignment parameters are obtained.
[0036] Among them, the geometric projection consistency constraint parameters ( Figure 2 Projection loss and depth consistency constraint parameters (in the context) Figure 2 The depth loss in the model is used to optimize the feature matching loss between visual features and point cloud features, and the regularization constraint parameters can be further adjusted. Figure 2 The regularized loss in the model prevents overfitting and ensures the reasonableness of the obtained cross-modal alignment parameters.
[0037] Specifically, the geometric projection consistency constraint can be expressed by the following formula: in, Represents the geometric projection consistency constraint parameters. Let N represent the point cloud feature of the i-th point cloud feature point, and N represent the number of point cloud feature points. For projection function, This means transforming the point cloud features from the first coordinate system to the second coordinate system and projecting them onto the two-dimensional plane containing the visual features; Represents visual features and point cloud feature points Matched two-dimensional visual feature points; These are weighting coefficients used to balance the effects of the projection error term and the interior point mask term. This is an interior point masking function used to remove outlier matching point pairs, suppress the negative impact of mismatches on the loss function, and improve robustness. The square of the Euclidean distance is used to measure the projection point. With visual feature points Positional deviation on the two-dimensional image plane.
[0038] In the above formula, the geometric projection consistency constraint calculates the sum of squared Euclidean distances between the positions of all point cloud feature points projected onto the 2D image plane and the positions of their matched visual feature points, and adds an interior point mask term. By minimizing this geometric projection consistency constraint parameter, the point cloud projection positions are aligned with the visual feature point positions, thereby optimizing the cross-modal alignment parameters, making the spatial transformation relationship between the first and second coordinate systems more accurate. At the same time, the interior point mask mechanism eliminates abnormally matched point pairs, improving optimization stability.
[0039] The depth consistency constraint can be expressed by the following formula: in, This refers to the depth consistency constraint parameter. This refers to the depth information obtained from point cloud features, such as the depth value corresponding to each feature point in the point cloud. It refers to depth information estimated from visual features, such as a depth map predicted from an image by a depth estimation network, where each pixel location (u, v) corresponds to a depth value; Let be the spatial transformation matrix from the first coordinate system to the second coordinate system, which is the cross-modal alignment parameter to be solved. This is a projection function used to project three-dimensional points onto a two-dimensional image plane; This is a depth query function. This indicates that based on the image pixel coordinates (u, v), the depth information from the point cloud... Obtain the corresponding depth value from the data; This represents the set of valid pixel regions, which are usually pixel regions in the image with valid depth values (such as non-occluded regions with high prediction confidence). Generally, the entire image can be used.
[0040] As an example, the total loss function may include geometric projection consistency constraint parameters and depth consistency constraint parameters, and may also include regularization constraint parameters.
[0041] For example, the total loss function is determined by the following formula: in, This is the total loss function; , These are weighting coefficients, which can be set as updatable parameters; For regularization constraint parameters ( Figure 2 The regularized loss can be set as a smoothing constraint. This is the scaling factor, which is set to 0.5 by default.
[0042] Based on the above total loss function, the feature matching loss can be obtained. The network parameters can be updated by deep learning network training methods such as backpropagation gradient descent to minimize the total loss function and obtain the cross-modal alignment parameters, which will not be elaborated here.
[0043] This concludes the description of step 101. We will now proceed to step 102.
[0044] Step 102: Determine the freeboard weight distribution using the cross-modal alignment parameters, the point cloud features, and the visual features.
[0045] The freeboard weight distribution is used to characterize whether the position of each feature point in the point cloud corresponding to the target vessel belongs to the freeboard region. Specifically, the freeboard weight distribution can include the probability that the position of each feature point in each point cloud corresponding to the target vessel belongs to the freeboard region. In this embodiment, the process of determining the freeboard weight distribution can also be implemented through a trained network. For example, the cross-modal alignment parameters, the visual features, and the point cloud features can be input into a heuristic attention network to obtain the freeboard weight distribution.
[0046] As one embodiment, the heuristic attention network includes a freeboard region segmentation network and a cross-modal heuristic attention network; the specific method for inputting the cross-modal alignment parameters, the visual features, and the point cloud features into the heuristic attention network to obtain the freeboard weight distribution may include: Based on the freeboard region segmentation network, the visual features are semantically segmented to obtain a visual mask of the freeboard region of the target ship; according to the cross-modal alignment parameters and the point cloud features, the point cloud features are transformed from the first coordinate system to the second coordinate system; the visual mask and the point cloud features in the second coordinate system are input into the cross-modal heuristic attention network to obtain the freeboard weight distribution.
[0047] Considering the sparse freeboard point cloud detected by low-beam lidar and the difficulty in labeling it, this embodiment of the application uses a heuristic attention network mechanism to add a semantic segmentation module, extract the visual freeboard segmentation region, and generate freeboard attention weights by combining the cross-modal sensor alignment parameters obtained in step 101, thereby reducing the difficulty of cross-modal freeboard feature learning. Furthermore, since the visual freeboard edge is clear and the feature recognition is high, making full use of visual features can effectively solve the problem of low measurement accuracy caused by a large amount of noise in the water surface ripple point cloud.
[0048] For details, please refer to Figure 3 , Figure 3 A schematic diagram of a heuristic attention network provided in an embodiment of this application.
[0049] like Figure 3 As shown, the heuristic attention network can include two parts: a freeboard region segmentation network and a cross-modal heuristic attention network. The input of the heuristic attention network is visual features and point cloud features, as well as cross-modal alignment parameters obtained through step 101.
[0050] First, a freeboard region mask based on visual features can be extracted using a freeboard region segmentation network (such as a semantic segmentation module like U-Net). Then, the point cloud features are transformed from the first coordinate system to the second coordinate system through cross-modal homogeneity parameters to obtain the point cloud features in the second coordinate system. .
[0051] Furthermore, mask the freeboard area. and point cloud features in the second coordinate system The input is fed into a cross-modal heuristic attention network to obtain the freeboard weight distribution.
[0052] Specifically, the freeboard weight distribution can be obtained using the following cross-modal heuristic attention network: in, For freeboard weight distribution, For the Sigmoid function; These are learnable parameters.
[0053] This concludes the description of step 102. We will now proceed to step 103.
[0054] Step 103: Determine the freeboard height of the target vessel based on the freeboard weight distribution.
[0055] In this embodiment, since the freeboard weight distribution includes the probability that the position of each feature point in the point cloud corresponds to the freeboard region on the target ship, the point cloud data corresponding to the freeboard can be determined from the point cloud data of the target ship first according to the freeboard weight distribution, and then the freeboard height of the target ship can be determined according to the point cloud data corresponding to the freeboard.
[0056] Specifically, the method for determining the point cloud data corresponding to the freeboard from the point cloud data of the target vessel based on the freeboard weight distribution may include: The probability value of each feature point in the point cloud data of the target vessel belonging to the freeboard region is determined according to the freeboard weight distribution; the set of point cloud data with probability values greater than a preset threshold is determined as the point cloud data corresponding to the freeboard.
[0057] Furthermore, the method for determining the freeboard height of the target vessel based on the point cloud data corresponding to the freeboard may include: Determine the vertical coordinates of the point cloud data of the freeboard area, and determine the freeboard height of the target vessel based on the maximum and minimum values of the vertical coordinates of the point cloud data of the freeboard area; or, input the point cloud data of the freeboard area into a trained regression network to obtain the freeboard height of the target vessel.
[0058] In this embodiment, the probability value of each feature point in the point cloud data belonging to the freeboard area can be determined based on the probability that the position of each feature point on the target ship corresponds to the freeboard area. The set of point cloud data with probability values greater than a preset threshold is determined as the point cloud data corresponding to the freeboard area.
[0059] After obtaining the point cloud data corresponding to the freeboard area, the coordinates of the point cloud data in the vertical direction of the freeboard area can be determined. Based on the maximum and minimum values of the coordinates of the point cloud data in the vertical direction of the freeboard area, the freeboard height of the target vessel can be determined. For example, the difference between the maximum and minimum values of the coordinates of the point cloud data in the vertical direction of the freeboard area can be determined as the freeboard height of the target vessel.
[0060] As an example, the point cloud data of the freeboard area can also be input into a trained regression network to directly obtain the freeboard height of the target vessel through the regression network.
[0061] This concludes the description of step 103.
[0062] This concludes the discussion. Figure 1 The description.
[0063] This application determines cross-modal alignment parameters based on the point cloud features and visual features of the target vessel to characterize the spatial transformation relationship between the image acquisition device and the lidar that acquires the point cloud data. Furthermore, by utilizing the cross-modal alignment parameters, point cloud features, and visual features, it determines the freeboard weight distribution to characterize whether the position of each feature point in the point cloud on the target vessel belongs to the freeboard area. This allows for precise focusing on the freeboard area to be detected, fully utilizing the clear freeboard semantic features in the visual features and the scale information of the point cloud data. Subsequently, the freeboard height of the target vessel is determined based on the freeboard weight distribution, achieving automated freeboard height detection. This improves freeboard height detection efficiency while ensuring measurement accuracy and controlling costs.
[0064] The following is through Figure 4 This application provides an overall description of the ship freeboard height detection method proposed in this application.
[0065] Please refer to Figure 4 , Figure 4 This is a schematic diagram of the overall network architecture for ship freeboard height detection provided in an embodiment of this application.
[0066] like Figure 4 As shown, visual features are obtained by extracting features from RGB images acquired by a monocular camera, and point cloud features are obtained by extracting features from point cloud data acquired by LiDAR.
[0067] Visual features and point cloud features are input into a geometrically consistent unsupervised network to obtain cross-modal alignment parameters that characterize the spatial transformation relationship between the first coordinate system corresponding to the lidar and the second coordinate system corresponding to the image acquisition device.
[0068] Cross-modal alignment parameters, visual features, and point cloud features are all input into a heuristic attention network to obtain a freeboard weight distribution that characterizes whether the position of each feature point in the point cloud features on the target ship belongs to the freeboard area.
[0069] The obtained freeboard weight distribution is input into the dimensional measurement unit to determine the freeboard height of the target vessel based on the freeboard weight distribution. This dimensional measurement unit can be a trained regression network, etc. The specific method for determining the freeboard height based on the freeboard weight distribution has been detailed above and will not be repeated here.
[0070] This concludes the discussion. Figure 4 The description.
[0071] In this embodiment, to address the cross-modal sensor feature alignment problem, the ship's length, width, and height dimensions in three-dimensional space exhibit significant projection and depth consistency. An unsupervised network based on geometric consistency is employed to project visual features and point cloud features into a virtual space. The significant length-width-height ratio and depth consistency are used as geometric constraints to force the point cloud projection and visual edge alignment, achieving centimeter-level alignment of the modal information of the two sensors.
[0072] To address the problem of sparse and difficult-to-label freeboard point clouds in low-beam lidar, a heuristic attention network mechanism is adopted, adding a semantic segmentation module to extract the visual freeboard segmentation region. The freeboard attention weights are generated by combining cross-modal sensor alignment parameters, reducing the learning difficulty of cross-modal freeboard features. Furthermore, by leveraging the advantages of clear visual freeboard edges and high feature recognition, the problem of low measurement accuracy caused by noise in a large number of water surface ripple point clouds is solved.
[0073] Please refer to Figure 5 , Figure 5 This is a structural diagram of a ship freeboard height detection device proposed in an embodiment of this application. Figure 5 As shown, the device may include an alignment parameter determination unit 501, a weight distribution determination unit 502, and a freeboard height detection unit 503. Specifically, the device includes: Alignment parameter determination unit 501 is used to determine cross-modal alignment parameters based on the point cloud features of the target ship and the visual features of the target ship; the point cloud features are obtained based on the point cloud data collected by the lidar, the visual features are obtained based on the image data collected by the image acquisition device, and the cross-modal alignment parameters are used to characterize the spatial transformation relationship between the first coordinate system corresponding to the lidar and the second coordinate system corresponding to the image acquisition device. The weight distribution determination unit 502 is used to determine the freeboard weight distribution using the cross-modal alignment parameters, the point cloud features, and the visual features; the freeboard weight distribution is used to characterize whether the position of each feature point in the point cloud features on the target ship belongs to the freeboard area. The freeboard height detection unit 503 is used to determine the freeboard height of the target vessel based on the freeboard weight distribution.
[0074] Optionally, the alignment parameter determination unit 501 is specifically used for: The point cloud features and the visual features are input into an unsupervised alignment network based on geometric consistency constraints to obtain the cross-modal alignment parameters; wherein, the unsupervised alignment network determines the cross-modal alignment parameters by optimizing the total loss function; And / or, the total loss function includes geometric projection consistency constraint parameters and depth consistency constraint parameters; the alignment parameter determination unit 501 is specifically used for: Based on the point cloud features and the visual features, geometric projection consistency constraint parameters and depth consistency constraint parameters are determined. The geometric projection consistency constraint parameters indicate the difference between the projection position of a point cloud feature point in the point cloud features after transformation from the first coordinate system to the second coordinate system, projected onto a two-dimensional plane corresponding to the visual feature, and the position of a visual feature point matching that point cloud feature point in the visual feature on the two-dimensional plane. The depth consistency constraint parameters indicate the difference between the first depth information of a point cloud feature point in the point cloud features after transformation from the first coordinate system to the second coordinate system, and the second depth information estimated by the visual features at the same spatial point. The total loss function is constructed based on the geometric projection consistency constraint and the depth consistency constraint, and the unsupervised alignment network is optimized by minimizing the total loss function to obtain the cross-modal alignment parameters. And / or, the weight distribution determination unit 502 is specifically used for: The cross-modal alignment parameters, the visual features, and the point cloud features are input into a heuristic attention network to obtain a freeboard weight distribution; the freeboard weight distribution includes the probability that the position of each feature point in each point cloud feature on the target ship belongs to the freeboard region; And / or, the heuristic attention network includes a freeboard region segmentation network and a cross-modal heuristic attention network; the weight distribution determination unit 502 is specifically used for: Based on the freeboard region segmentation network, the visual features are semantically segmented to obtain a visual mask of the freeboard region of the target ship. Based on the cross-modal alignment parameters and the point cloud features, the point cloud features are transformed from the first coordinate system to the second coordinate system; The visual mask and the point cloud features in the second coordinate system are input into the cross-modal heuristic attention network to obtain the freeboard weight distribution; And / or, the freeboard height detection unit 503 is specifically used for: The freeboard is determined from the point cloud data of the target vessel based on the freeboard weight distribution, and the freeboard height of the target vessel is determined based on the point cloud data corresponding to the freeboard. And / or, the freeboard height detection unit 503 is specifically used for: Determine the vertical coordinates of the point cloud data of the freeboard area, and determine the freeboard height of the target vessel based on the maximum and minimum values of the vertical coordinates of the point cloud data of the freeboard area; or, input the point cloud data of the freeboard area into a trained regression network to obtain the freeboard height of the target vessel.
[0075] This concludes the process. Figure 5 Description of the freeboard height detection device for Chinese ships.
[0076] This application also provides embodiments that... Figure 5 Hardware structure description of the illustrated device. This hardware structure is... Figure 6 The structure in the illustrated electronic device. Please refer to [link / reference]. Figure 6 , Figure 6 This is a structural diagram of an electronic device provided in an embodiment of this application. Figure 6 As shown, the hardware structure may include: a processor and a machine-readable storage medium, the machine-readable storage medium storing machine-executable instructions that can be executed by the processor; the processor is used to execute the machine-executable instructions to implement the method disclosed in the above example of this application.
[0077] Based on the same concept as the above method, this application also provides a machine-readable storage medium storing a plurality of computer instructions, which, when executed by a processor, can implement the method disclosed in the above examples of this application.
[0078] For example, the aforementioned machine-readable storage medium can be any electronic, magnetic, optical, or other physical storage device that can contain or store information such as executable instructions, data, etc. For instance, machine-readable storage media can be: RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, storage drives (such as hard disk drives), solid-state drives, any type of storage disk (such as optical discs, DVDs, etc.), or similar storage media, or combinations thereof.
[0079] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of this application should be included within the scope of the claims of this application. The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.
Claims
1. A method for detecting the freeboard height of a ship, characterized in that, The method includes: Based on the point cloud features and visual features of the target vessel, cross-modal alignment parameters are determined; the point cloud features are obtained based on point cloud data collected by lidar, and the visual features are obtained based on image data collected by image acquisition equipment; the cross-modal alignment parameters are used to characterize the spatial transformation relationship between the first coordinate system corresponding to the lidar and the second coordinate system corresponding to the image acquisition equipment. The freeboard weight distribution is determined using the cross-modal alignment parameters, the point cloud features, and the visual features; the freeboard weight distribution is used to characterize whether the position of each feature point in the point cloud features on the target ship belongs to the freeboard area. The freeboard height of the target vessel is determined based on the freeboard weight distribution.
2. The method according to claim 1, characterized in that, The determination of cross-modal alignment parameters based on the point cloud features and visual features of the target ship includes: The point cloud features and the visual features are input into an unsupervised alignment network based on geometric consistency constraints to obtain the cross-modal alignment parameters; wherein, the unsupervised alignment network determines the cross-modal alignment parameters by optimizing the total loss function.
3. The method according to claim 2, characterized in that, The total loss function includes geometric projection consistency constraint parameters and depth consistency constraint parameters; the step of inputting the point cloud features and the visual features into an unsupervised alignment network based on geometric consistency constraints to obtain the cross-modal alignment parameters includes: Based on the point cloud features and the visual features, geometric projection consistency constraint parameters and depth consistency constraint parameters are determined. The geometric projection consistency constraint parameters indicate the difference between the projection position of a point cloud feature point in the point cloud features after transformation from the first coordinate system to the second coordinate system, projected onto a two-dimensional plane corresponding to the visual feature, and the position of a visual feature point matching that point cloud feature point in the visual feature on the two-dimensional plane. The depth consistency constraint parameters indicate the difference between the first depth information of a point cloud feature point in the point cloud features after transformation from the first coordinate system to the second coordinate system, and the second depth information estimated by the visual features at the same spatial point. The total loss function is constructed based on the geometric projection consistency constraint and the depth consistency constraint, and the unsupervised alignment network is optimized by minimizing the total loss function to obtain the cross-modal alignment parameters.
4. The method according to claim 1, characterized in that, The step of determining the freeboard weight distribution using the cross-modal alignment parameters, the point cloud features, and the visual features includes: The cross-modal alignment parameters, the visual features, and the point cloud features are input into a heuristic attention network to obtain a freeboard weight distribution; the freeboard weight distribution includes the probability that the position of each feature point in each point cloud feature on the target ship belongs to the freeboard region.
5. The method according to claim 4, characterized in that, The heuristic attention network includes a freeboard region segmentation network and a cross-modal heuristic attention network; the step of inputting the cross-modal alignment parameters, the visual features, and the point cloud features into the heuristic attention network to obtain the freeboard weight distribution includes: Based on the freeboard region segmentation network, the visual features are semantically segmented to obtain a visual mask of the freeboard region of the target ship. Based on the cross-modal alignment parameters and the point cloud features, the point cloud features are transformed from the first coordinate system to the second coordinate system; The visual mask and the point cloud features in the second coordinate system are input into the cross-modal heuristic attention network to obtain the freeboard weight distribution.
6. The method according to claim 1, characterized in that, Determining the freeboard height of the target vessel based on the freeboard weight distribution includes: The freeboard is determined from the point cloud data of the target vessel based on the freeboard weight distribution, and the freeboard height of the target vessel is determined based on the point cloud data corresponding to the freeboard.
7. The method according to claim 6, characterized in that, Determining the freeboard height of the target vessel based on the point cloud data corresponding to the freeboard includes: Determine the vertical coordinates of the point cloud data of the freeboard area, and determine the freeboard height of the target vessel based on the maximum and minimum values of the vertical coordinates of the point cloud data of the freeboard area; or, input the point cloud data of the freeboard area into a trained regression network to obtain the freeboard height of the target vessel.
8. A device for detecting the freeboard height of a ship, characterized in that, The device includes: The alignment parameter determination unit is used to determine cross-modal alignment parameters based on the point cloud features and visual features of the target ship; the point cloud features are obtained based on point cloud data collected by lidar, the visual features are obtained based on image data collected by image acquisition device, and the cross-modal alignment parameters are used to characterize the spatial transformation relationship between the first coordinate system corresponding to the lidar and the second coordinate system corresponding to the image acquisition device. The weight distribution determination unit is used to determine the freeboard weight distribution using the cross-modal alignment parameters, the point cloud features, and the visual features; the freeboard weight distribution is used to characterize whether the position of each feature point in the point cloud features on the target ship belongs to the freeboard area; A freeboard height detection unit is used to determine the freeboard height of the target vessel based on the freeboard weight distribution.
9. The apparatus according to claim 8, characterized in that, The alignment parameter determination unit is specifically used for: The point cloud features and the visual features are input into an unsupervised alignment network based on geometric consistency constraints to obtain the cross-modal alignment parameters; wherein, the unsupervised alignment network determines the cross-modal alignment parameters by optimizing the total loss function; And / or, the total loss function includes geometric projection consistency constraint parameters and depth consistency constraint parameters; the alignment parameter determination unit is specifically used for: Based on the point cloud features and the visual features, geometric projection consistency constraint parameters and depth consistency constraint parameters are determined. The geometric projection consistency constraint parameters indicate the difference between the projection position of a point cloud feature point in the point cloud features after transformation from the first coordinate system to the second coordinate system, projected onto a two-dimensional plane corresponding to the visual feature, and the position of a visual feature point matching that point cloud feature point in the visual feature on the two-dimensional plane. The depth consistency constraint parameters indicate the difference between the first depth information of a point cloud feature point in the point cloud features after transformation from the first coordinate system to the second coordinate system, and the second depth information estimated by the visual features at the same spatial point. The total loss function is constructed based on the geometric projection consistency constraint and the depth consistency constraint, and the unsupervised alignment network is optimized by minimizing the total loss function to obtain the cross-modal alignment parameters. And / or, the weight distribution determination unit is specifically used for: The cross-modal alignment parameters, the visual features, and the point cloud features are input into a heuristic attention network to obtain a freeboard weight distribution; the freeboard weight distribution includes the probability that the position of each feature point in each point cloud feature on the target ship belongs to the freeboard region; And / or, the heuristic attention network includes a freeboard region segmentation network and a cross-modal heuristic attention network; the weight distribution determination unit is specifically used for: Based on the freeboard region segmentation network, the visual features are semantically segmented to obtain a visual mask of the freeboard region of the target ship. Based on the cross-modal alignment parameters and the point cloud features, the point cloud features are transformed from the first coordinate system to the second coordinate system; The visual mask and the point cloud features in the second coordinate system are input into the cross-modal heuristic attention network to obtain the freeboard weight distribution; And / or, the freeboard height detection unit is specifically used for: The freeboard is determined from the point cloud data of the target vessel based on the freeboard weight distribution, and the freeboard height of the target vessel is determined based on the point cloud data corresponding to the freeboard. And / or, the freeboard height detection unit is specifically used for: Determine the vertical coordinates of the point cloud data of the freeboard area, and determine the freeboard height of the target vessel based on the maximum and minimum values of the vertical coordinates of the point cloud data of the freeboard area; or, input the point cloud data of the freeboard area into a trained regression network to obtain the freeboard height of the target vessel.
10. An electronic device, characterized in that, include: A processor and a machine-readable storage medium, the machine-readable storage medium storing machine-executable instructions that can be executed by the processor; The processor is configured to execute machine-executable instructions to implement the method according to any one of claims 1 to 7.