Intelligent labeling method and system for marine image, electronic equipment and storage medium

Abstract

The invention relates to the technical field of maritime image labeling, and discloses an intelligent labeling method and system for a maritime image, electronic equipment and a storage medium, and the method comprises the steps: firstly, obtaining clear maritime target point cloud data through removing outliers in laser radar point cloud data; clustering the point cloud data by using a DBSCAN algorithm, and identifying and distinguishing different marine targets; then geometric features and intensity features of the target objects are extracted, a target object classification index TCL is calculated, and an unsupervised classification algorithm is adopted to classify the marine target objects; and finally, projecting the point cloud data with semantic information to an image plane, generating a prompt mask of a semantic segmentation model, and inputting the prompt mask and the marine image into a semantic segmentation large model SAM for semantic annotation. According to the method, the laser radar data and the semantic segmentation large model are combined, so that efficient and accurate marking of the marine image is realized, and reliable data support is provided for a marine ship automatic driving system.

Description

Technical Field

[0001] The present invention relates to the technical field of marine image annotation, and in particular to an intelligent annotation method, system, electronic device and storage medium for marine images. Background Art

[0002] In the development of autonomous driving algorithms for marine vessels, data-driven methods play a key role in improving algorithm performance and robustness, especially in terms of the accuracy of image annotation. However, marine image data is easily affected by changes in lighting conditions and weather changes, resulting in reduced annotation accuracy, which in turn affects the perception ability of the autonomous driving system. In contrast, lidar data shows significant advantages in marine environments. Due to the absorption of lidar rays by seawater, most laser rays will not be reflected on the sea surface, so there are almost no background interference points in the point cloud data. This makes the background of the point cloud data simpler, which is conducive to the high-precision extraction and annotation of marine targets such as ships and buoys. In addition, lidar is not affected by changes in lighting and weather, and its three-dimensional position information provides a reliable reference for image annotation.

[0003] The semantic segmentation model (SAM) has acquired the ability of semantic segmentation of images at the object level by training on massive data sets. The model can achieve refined extraction of zero-sample targets based on different prompts, without the need to train re-labeled data for specific tasks. Therefore, in the segmentation task of marine image data, SAM shows excellent potential performance and can efficiently handle target recognition in complex backgrounds. By clustering and projecting the lidar point cloud data onto the image, generating a prompt mask, and providing high-precision prompt words for SAM to use, the problem that the segmentation results output by the SAM model lack the necessary semantic information can be solved, further improving the efficiency and accuracy of intelligent labeling.

[0004] Traditional manual annotation methods are time-consuming and laborious when processing large amounts of marine image data, and accuracy and consistency are difficult to guarantee due to the need to deal with variable factors in the marine environment. In contrast, the intelligent annotation method we proposed not only greatly reduces labor costs and annotation time, but also significantly improves the annotation quality through automated processing, providing more efficient and accurate data support for marine ship autonomous driving systems. Summary of the invention

[0005] The purpose of the present invention is to provide a method, system, electronic device and storage medium for intelligent annotation of marine images to solve the problems raised in the above-mentioned background technology.

[0006] To achieve the above object, the present invention provides the following technical solution: an intelligent annotation method for marine images, the method comprising: Step a: Remove outliers from the marine lidar point cloud data, determine and remove noise points based on the local density statistics of the point cloud, and obtain denoised marine target point cloud data; Step b: cluster the denoised marine target point cloud data, use the DBSCAN algorithm to divide the point cloud data into different clusters, and identify and distinguish different marine targets; Step c: extracting the geometric features and intensity features of the clustered marine target point cloud, including aspect ratio, average height, main direction and average intensity, and calculating the target classification index TCL based on these features; Step d: According to the calculated target classification index TCL, an unsupervised classification algorithm is used to classify the marine targets and divide the point cloud data into different categories; Step e: Project the classified point cloud data with semantic information onto the image plane to generate a hint mask for the semantic segmentation model. Then input the marine image and hint mask into the semantic segmentation model SAM for semantic annotation.

[0007] Preferably, the method for removing outliers in step a is: Step a1: define a neighborhood for each point and use the k-nearest neighbor algorithm to determine the points around it; Step a2: Calculate the number of points in the neighborhood of each point to obtain the local density of each point; Step a3: Set a density threshold and remove points whose local density is lower than the threshold.

[0008] Preferably, the step of clustering marine objects using the DBSCAN algorithm includes: Step b1: define the neighborhood range of a point. Given any point, its neighborhood is defined as the set of points whose distances between the point and other points are less than a preset neighborhood range value. Step b2: Determine the core point and define the hyperparameter MinPts as the minimum number of points in the neighborhood required to determine whether a point is a core point. If the number of points in the neighborhood of a point is greater than or equal to MinPts, then the point is a core point. Step b3: Find boundary points. For non-core points, if its neighborhood contains at least one core point, then the point is a boundary point. Step b4: Determine the noise points. If a point is neither a core point nor a boundary point, it is a noise point and is ignored in the clustering process.

[0009] Preferably, the method for extracting the geometric features and intensity features of the target object point cloud in step c is: Step c1: Project the clustered point cloud to the BEV perspective, calculate the width and height of the bounding rectangle of the target object, and obtain the aspect ratio of the target object; Step c2: Calculate the average height value of the target point cloud; Step c3: extract the main directions of the target object using principal component analysis; Step c4: Calculate the average intensity of the target point cloud.

[0010] Preferably, the unsupervised classification algorithm in step d is a K-means algorithm, which divides the point cloud data into different categories according to the target classification index TCL.

[0011] Preferably, the step of generating a hint mask of the semantic segmentation model comprises: Step e1: Determine the projection transformation relationship from point cloud to image, including rotation matrix, translation vector and camera intrinsic matrix; Step e2: According to the projection transformation relationship, the point cloud with semantic information is projected into the image to generate a hint mask with the same size as the image.

[0012] Preferably, after generating the hint mask of the semantic segmentation model, the generated hint mask is preprocessed, including adjusting the size, resolution and format of the mask.

[0013] Preferably, an intelligent annotation system for marine images comprises: The point cloud data preprocessing module is used to remove outliers from the marine lidar point cloud data, determine and remove noise points based on the local density statistics of the point cloud, and obtain the denoised marine target point cloud data; The point cloud clustering processing module is used to perform clustering processing on the denoised marine target point cloud data. The DBSCAN algorithm is used to divide the point cloud data into different clusters to identify and distinguish different marine targets. The feature extraction module is used to extract the geometric features and intensity features of the clustered marine target point cloud, including aspect ratio, average height, main direction and average intensity, and calculate the target classification index TCL based on these features; The target classification module is used to classify the marine targets according to the calculated target classification index TCL using an unsupervised classification algorithm to divide the point cloud data into different categories; The semantic annotation module is used to project the classified point cloud data with semantic information onto the image plane, generate the prompt mask of the semantic segmentation model, and input the marine image and the prompt mask into the semantic segmentation model SAM for semantic annotation.

[0014] Preferably, an electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned intelligent annotation method for marine images when executing the computer program.

[0015] Preferably, a storage medium stores a computer program thereon, characterized in that the program, when executed by a processor, implements the above-mentioned intelligent annotation method for marine images.

[0016] Compared with the prior art, the present invention has the following beneficial effects: The present invention utilizes the high precision and three-dimensional position information of LiDAR point cloud data to effectively reduce the interference of changes in lighting conditions and weather changes on image annotation. By removing outliers, clustering processing and feature extraction, it can accurately identify and distinguish different targets at sea, providing a reliable reference for image annotation and significantly improving the accuracy and robustness of annotation.

[0017] The present invention realizes intelligent annotation of marine images through the application of automated point cloud data processing and semantic segmentation large model, which greatly reduces labor costs and annotation time and improves work efficiency.

[0018] The semantic segmentation model (SAM) has a powerful ability to segment semantic images at the object level, but it may lack the necessary semantic information when directly applied to marine images. The present invention generates a prompt mask by projecting point cloud data with semantic information onto the image plane, providing the SAM model with high-precision prompt words, solving the problem of insufficient semantic information in its output segmentation results, and making the annotation results richer and more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a flow chart of a method for intelligently labeling marine images according to the present invention; Figure 2 Visualize the resulting graph for the hint mask. DETAILED DESCRIPTION

[0020] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0021] See also Figure 1-2The present invention proposes an intelligent annotation method for marine images. By clustering the marine lidar point cloud data, the semantic information of the point cloud of the main targets (such as ships and buoys) is extracted, and the prompt mask of the SAM semantic segmentation model is generated according to the projection transformation relationship between the camera and the radar coordinate system, so as to realize the intelligent semantic annotation of the marine images, reduce the workload of manual annotation, and improve the annotation accuracy. The process of this method is shown in the attached figure. Figure 1 The method comprises the following steps: Step a: Removal of outliers from offshore point clouds Since the wavelength range of LiDAR is usually concentrated in the near-infrared band, and the near-infrared band is easily soluble in water, seawater will absorb most of the LiDAR rays, and there will be fewer background interference point clouds. The remaining point clouds are mainly concentrated on marine targets (such as ships, buoys, etc.). Although the absorption of laser signals by seawater can effectively remove most background points, a small number of noise points will still be introduced due to factors such as obstacles, waves and diffuse reflection of the water surface.

[0022] To solve this problem, outlier removal is performed based on local density statistics of point clouds. This method determines whether it is noise by analyzing the neighborhood density of each point. The specific steps are as follows: First, define a neighborhood for each point, and often use a k-nearest neighbor or radius search algorithm to determine the points around it. Then, calculate the number of points in the neighborhood to obtain the local density of each point. Next, set a density threshold, usually based on experience or statistical analysis, to distinguish between normal points and noise points. Finally, remove those points whose local density is lower than the threshold to obtain the denoised point cloud data.

[0023] Step b: Clustering of marine target point clouds For the denoised marine scene point cloud, DBSCAN (density-based spatial clustering algorithm) is used to cluster the preprocessed point cloud. The process of clustering marine objects using the DBSCAN algorithm is as follows: Define the neighborhood of a point. Given any point , whose neighborhood is defined as: in, It is the target point With point The distance between them, the hyperparameter Defined as the neighborhood range.

[0024] Determine the core points. Define hyperparameters To determine whether a point is a core point The minimum number of points required in the neighborhood, therefore, the judgment condition for point p to be a core point is: Find boundary points. For point q, if its neighborhood contains at least one core point, but it itself is not a core point, then point q is a boundary point: Determine the noise points. If any point is neither a core point nor a boundary point, it is considered a noise point.

[0025] DBSCAN can automatically identify high-density areas in point clouds and divide them into different clusters, avoiding the prior assumption on the number of clusters and is suitable for handling the diversity of targets in complex marine environments.

[0026] Step c: Classification of maritime targets based on geometric and intensity features In the marine environment, different targets (such as ships, buoys and lighthouses) have different shape characteristics and intensity characteristics. Shape characteristics (such as aspect ratio and main axis direction) can effectively distinguish different types of targets, while intensity characteristics (point intensity value) can reflect the material and surface characteristics of the object. This patent combines these two characteristics for unsupervised classification, which can improve the accuracy and robustness of classification. This step is the core step of the patent, which specifically includes: Point cloud feature extraction of marine targets: (1) First, the clustered point cloud is projected to the BEV perspective, and the aspect ratio (AR) of the target object is obtained by calculating the width and height of the bounding rectangle of the target object: in, and are the width and height of the target object’s circumscribed rectangle, respectively.

[0027] (2) Then calculate the average height value (Z) of the target point cloud: (3) The principal component analysis (PCA) method is used to extract the main direction of the target object. By calculating the covariance matrix of the point cloud, the eigenvector of the first principal component of the target point cloud is obtained, and the direction parameter (P) of the target point cloud is calculated: Calculate the mean of the target point cloud: Calculate the normalized point cloud data: Calculate the target point cloud covariance matrix: Use eigenvalue decomposition to calculate the eigenvalues ​​of the point cloud covariance matrix and the eigenvector : Project the original point cloud data onto the first principal component and calculate the projection value of each target point cloud on the principal component. and the first principal component For example, the direction parameter of the point cloud is calculated as: (4) Calculate the average intensity of the target point cloud: Calculation of target classification index: Based on the extracted features, a target classification index (TCL) is constructed: in, , , , is the feature weight.

[0028] Step d: Object classification based on unsupervised algorithms According to the calculated target classification index (TCL), the unsupervised classification algorithm K-means is used to classify the target obstacles, and the point cloud data are divided into different categories according to similar features.

[0029] Semantic segmentation mask generation: The projection transformation of point cloud data to image usually includes the rotation matrix R and the translation vector t, as well as the camera's intrinsic parameter matrix K. These parameters can be obtained through the Zhang Zhengyou calibration method and the hand-eye calibration method. Projection onto the image plane The conversion relationship is as follows: Through the projection relationship between the point cloud and the image, the point cloud with semantic information is projected onto the image to generate the mask of the semantic segmentation SAM model. The mask generation result is shown in the attached figure. Figure 2 shown.

[0030] Step e: Automatic labeling of maritime targets based on SAM In order to solve the problem that the segmentation results output by the SAM model lack the necessary semantic information, we use a prompt mask with semantic information to improve the effect of automatic target extraction. The specific method is: according to the point cloud projection results, different mask values ​​are assigned to different categories of targets in the mask, ensuring that the size of the mask is the same as the image, and the marine image and the prompt mask are input into the SAM large model together to accurately and automatically extract the targets in the marine image (such as ships, lighthouses, etc.).

[0031] The present invention also includes an intelligent annotation system for marine images, which realizes accurate semantic annotation of marine images by combining laser radar point cloud data and image processing technology. The system includes: Point cloud data preprocessing module: This module first preprocesses the acquired marine lidar point cloud data. The preprocessing process mainly includes the removal of outliers. The method based on the local density statistics of the point cloud is used to perform density analysis on each point and the points in its neighborhood. If the local density of a point is significantly lower than the average density of its surrounding points, the point is regarded as a noise point and removed. Through this process, the denoised marine target point cloud data can be obtained, providing accurate basic data for subsequent processing.

[0032] Point cloud clustering processing module: The denoised point cloud data is sent to the point cloud clustering processing module. This module uses the DBSCAN (Density-Based Spatial Clustering of Applications with Noise) algorithm to cluster the point cloud data. The DBSCAN algorithm can automatically divide the clusters according to the density distribution of the points, thereby effectively identifying and distinguishing different targets at sea, such as ships, buoys, waves, etc. The clustering results are represented by different clusters, and each cluster represents an independent target at sea.

[0033] Feature extraction module: For each point cloud of marine targets after clustering, the feature extraction module is responsible for extracting its geometric features and intensity features. Geometric features include aspect ratio, average height and main direction, which reflect the shape and size of the target. Intensity features focus on the average intensity of the point cloud, which reflects the reflective characteristics of the target surface. Based on these features, the target classification index TCL (Target Classification Index) is calculated, which is used for subsequent target classification.

[0034] Object classification module: The object classification module uses an unsupervised classification algorithm to classify marine objects based on the object classification index TCL calculated by the feature extraction module. The unsupervised classification algorithm can automatically discover potential patterns in the data and divide the point cloud data into different categories, such as ship class, buoy class, wave class, etc. The classification results provide accurate category information for subsequent semantic annotation.

[0035] Semantic annotation module: Finally, the semantic annotation module projects the classified point cloud data with semantic information onto the image plane. This step is achieved through coordinate transformation and projection algorithm to ensure the accurate correspondence between the point cloud data and the image pixels. After projection, the hint mask of the semantic segmentation model is generated, which contains the category information of the target object. Subsequently, the sea image and the hint mask are input into the pre-trained semantic segmentation large model SAM (Segment Anything Model) for semantic annotation. The SAM model uses the hint mask as prior information to perform fine semantic segmentation on the image, and finally outputs a sea image with semantic annotations.

[0036] In addition, the present invention also includes an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the above-mentioned intelligent annotation method for marine images is implemented. Similarly, the present invention also includes a storage medium, on which a computer program is stored, and when the program is executed by the processor, the above-mentioned intelligent annotation method for marine images is implemented.

[0037] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0038] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. An intelligent annotation method for marine images, characterized in that: The method comprises: Step a: Remove outliers from the marine lidar point cloud data, determine and remove noise points based on the local density statistics of the point cloud, and obtain denoised marine target point cloud data; Step b: cluster the denoised marine target point cloud data, use the DBSCAN algorithm to divide the point cloud data into different clusters, and identify and distinguish different marine targets; Step c: extracting the geometric features and intensity features of the clustered marine target point cloud, including aspect ratio, average height, main direction and average intensity, and calculating the target classification index TCL based on these features; Step d: According to the calculated target classification index TCL, an unsupervised classification algorithm is used to classify the marine targets and divide the point cloud data into different categories; Step e: Project the classified point cloud data with semantic information onto the image plane to generate a hint mask for the semantic segmentation model. Then input the marine image and hint mask into the semantic segmentation model SAM for semantic annotation.

2. The intelligent annotation method for marine images according to claim 1, characterized in that: The method for removing outliers in step a is: Step a1: define a neighborhood for each point and use the k-nearest neighbor algorithm to determine the points around it; Step a2: Calculate the number of points in the neighborhood of each point to obtain the local density of each point; Step a3: Set a density threshold and remove points whose local density is lower than the threshold.

3. The intelligent annotation method for marine images according to claim 1, characterized in that: The steps of clustering marine objects using the DBSCAN algorithm include: Step b1: define the neighborhood range of a point. Given any point, its neighborhood is defined as the set of points whose distances between the point and other points are less than a preset neighborhood range value. Step b2: Determine the core point and define the hyperparameter MinPts as the minimum number of points in the neighborhood required to determine whether a point is a core point. If the number of points in the neighborhood of a point is greater than or equal to MinPts, then the point is a core point. Step b3: Find boundary points. For non-core points, if its neighborhood contains at least one core point, then the point is a boundary point. Step b4: Determine the noise points. If a point is neither a core point nor a boundary point, it is a noise point and is ignored in the clustering process.

4. The intelligent annotation method for marine images according to claim 1, characterized in that: The method for extracting the geometric features and intensity features of the target point cloud in step c is: Step c1: Project the clustered point cloud to the BEV perspective, calculate the width and height of the bounding rectangle of the target object, and obtain the aspect ratio of the target object; Step c2: Calculate the average height value of the target point cloud; Step c3: extract the main directions of the target object using principal component analysis; Step c4: Calculate the average intensity of the target point cloud.

5. The intelligent annotation method for marine images according to claim 1, characterized in that: The unsupervised classification algorithm in step d is a K-means algorithm, which divides the point cloud data into different categories according to the target classification index TCL.

6. The intelligent annotation method for marine images according to claim 1, characterized in that: The step of generating a hint mask of a semantic segmentation model comprises: Step e1: Determine the projection transformation relationship from point cloud to image, including rotation matrix, translation vector and camera intrinsic matrix; Step e2: According to the projection transformation relationship, the point cloud with semantic information is projected into the image to generate a hint mask with the same size as the image.

7. The intelligent annotation method for marine images according to claim 6, characterized in that: After generating the hint mask for the semantic segmentation model, the generated hint mask is preprocessed, including adjusting the size, resolution, and format of the mask.

8. An intelligent annotation system for marine images, characterized in that: The system comprises: The point cloud data preprocessing module is used to remove outliers from the marine lidar point cloud data, determine and remove noise points based on the local density statistics of the point cloud, and obtain the denoised marine target point cloud data; The point cloud clustering processing module is used to perform clustering processing on the denoised marine target point cloud data. The DBSCAN algorithm is used to divide the point cloud data into different clusters to identify and distinguish different marine targets. The feature extraction module is used to extract the geometric features and intensity features of the clustered marine target point cloud, including aspect ratio, average height, main direction and average intensity, and calculate the target classification index TCL based on these features; The target classification module is used to classify the marine targets according to the calculated target classification index TCL using an unsupervised classification algorithm to divide the point cloud data into different categories; The semantic annotation module is used to project the classified point cloud data with semantic information onto the image plane, generate the prompt mask of the semantic segmentation model, and input the marine image and the prompt mask into the semantic segmentation model SAM for semantic annotation.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the intelligent annotation method for marine images according to any one of claims 1 to 7 is implemented.

10. A storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the intelligent annotation method for marine images as described in any one of claims 1 to 7 is implemented.

Images