An underwater three-dimensional moving and static target sonar data labeling method and system

Abstract

The application discloses a kind of underwater three-dimensional moving and static target sonar data labeling method and system, wherein method includes: the sonar data of collection is converted, and underwater point cloud data is obtained;According to the three-dimensional space coordinates and the reflected wave intensity, the underwater point cloud data containing the target to be labeled is identified and data superposition is carried out according to the ping serial number, and target superposition data is obtained;According to the pre-set target category, the target superposition data is carried out batch target segmentation and class label is added, and target segmentation data is obtained;Based on the class label, the three-dimensional space coordinates are automatically generated in the target segmentation data corresponding to each ping serial number Corresponding stereoscopic frame is generated.The underwater data containing the target to be labeled is superimposed into an inlay data for one-time target segmentation, and then a stereoscopic labeling frame is automatically generated for the single ping data after segmentation, which can greatly improve the target labeling efficiency and accuracy.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of underwater sonar data processing, and particularly relates to a method and system for labeling underwater three-dimensional moving and static target sonar data. BACKGROUND

[0002] Sonar is the most widely used device in underwater target recognition, and new sonar technologies provide necessary technical support for timely and accurate detection of underwater targets. Target recognition, as an important part of post-processing of sonar data, provides necessary data support for classification and identification of multiple targets.

[0003] For underwater three-dimensional data obtained by sonar, existing three-dimensional target labeling methods use laser radar, depth camera or radar point cloud data. The labeling method is mainly manual, similar to two-dimensional image labeling method, and three-dimensional target labeling mainly uses manual pulling of a three-dimensional frame that can envelope the target. Due to the influence of the labeling personnel's working ability, the existing labeling method relying on manual pulling of the target frame has large spatial position error, low efficiency, large engineering quantity, and high cost. Moreover, there is currently a lack of labeling specification for underwater sonar data. SUMMARY

[0004] The application provides a method and system for labeling underwater three-dimensional moving and static target sonar data. The underwater data containing the target to be labeled are superimposed as a mosaic data for one-time target segmentation, and a three-dimensional labeling frame is automatically generated after the segmented single ping data, which can greatly improve the target labeling efficiency and accuracy.

[0005] The first aspect of the application provides a method for labeling underwater three-dimensional moving and static target sonar data, which comprises:

[0006] Converting the collected sonar data to obtain underwater point cloud data; wherein the underwater point cloud data contains three-dimensional spatial coordinates, reflected wave intensity and ping sequence number;

[0007] According to the three-dimensional spatial coordinates and the reflected wave intensity, the underwater point cloud data containing the target to be labeled is identified, and the data is superimposed according to the ping sequence number to obtain target superimposed data;

[0008] According to a preset target category, the target superimposed data is subjected to target segmentation and addition of category labels to obtain target segmented data;

[0009] Based on the category label, a corresponding three-dimensional frame is automatically generated in the target segmented data corresponding to each ping sequence number through the three-dimensional spatial coordinates; wherein the spatial position of the target to be labeled under water can be determined according to the three-dimensional frame.

[0010] The above scheme first identifies the target to be labeled from the underwater point cloud data according to the three-dimensional space coordinates and the reflected wave intensity, and stacks the underwater point cloud data containing the target to be labeled into a single mosaic data according to frames, that is, target superposition data, which provides data support for subsequent simultaneous labeling of the target superposition data as a whole. Then, the target superposition data is added with a category label, and the target superposition data containing the target to be labeled is simultaneously segmented for each frame of data through point cloud labeling, greatly improving the data labeling rate. Then, according to the category label and the three-dimensional space coordinates, a cube of the target to be labeled is generated, completing the labeling of the underwater data and realizing accurate labeling of the underwater target.

[0011] In a possible implementation method of the first aspect, the underwater point cloud data containing the target to be labeled is identified according to the three-dimensional space coordinates and the reflected wave intensity, and data stacking is performed according to the ping sequence number to obtain target superposition data, specifically as follows:

[0012] If the target to be labeled is a static target, the underwater point cloud data is compared with preset background environment information according to the three-dimensional space coordinates and the reflected wave intensity to obtain all ping sequence numbers containing the target to be labeled in the underwater point cloud data.

[0013] If the target to be labeled is a dynamic target, the underwater point cloud data is played back frame by frame, and all ping sequence numbers containing the target to be labeled are identified according to the inter-frame change relationship of the underwater point cloud data.

[0014] The underwater point cloud data corresponding to each ping sequence number is stacked to obtain a mosaic data, denoted as target superposition data.

[0015] In the above scheme, for a static target, the target to be labeled is distinguished from the surrounding background environment according to the three-dimensional space coordinates and the reflected wave intensity, and the target to be labeled in the underwater point cloud data is identified and the corresponding ping sequence number is obtained. For a dynamic target, the dynamic change of the target to be labeled in each frame is identified by playing back the underwater point cloud data frame by frame, so as to obtain the underwater point cloud data containing the target to be labeled and the corresponding ping sequence number. Finally, the corresponding underwater point cloud data is stacked into a mosaic data through the ping sequence number, and the mosaic data is taken as a whole of target superposition data. Because the target superposition data all contain the target to be labeled, the data does not need to be screened and can be directly labeled, which provides data support for subsequent simultaneous processing of the target superposition data.

[0016] In a possible implementation method of the first aspect, the target superposition data is segmented and added with a category label according to a preset target category to obtain target segmentation data, specifically as follows:

[0017] According to the preset target category, confirm the category label of the target to be marked;

[0018] Performing point cloud annotation of the target to be annotated on the target overlay data, and segmenting all the point cloud annotations at one time to complete target segmentation;

[0019] The category label is added to the target superposition data corresponding to each ping number after the target segmentation to obtain target segmentation data; wherein the target segmentation data includes three-dimensional space coordinates, reflection wave intensity, ping number and category label.

[0020] The above solution treats the target overlay data of multiple pings as a whole and performs target segmentation at the same time, which greatly improves the data processing rate and adds category labels to the segmented data to provide data support for subsequent cube generation.

[0021] In a possible implementation method of the first aspect, based on the category label, a corresponding stereoscopic frame is automatically generated in the target segmentation data corresponding to each ping sequence number using the three-dimensional spatial coordinates, specifically:

[0022] Split the target segmentation data according to each ping sequence number to obtain several single ping target data;

[0023] Based on the three-dimensional spatial coordinates, a spatial matrix is ​​generated for the point cloud annotation of the target to be annotated in each single ping target data;

[0024] Generate a corresponding stereo frame according to the spatial matrix;

[0025] The center point of the stereoscopic frame is determined according to the length, width and height information of the stereoscopic frame.

[0026] The above scheme splits the data that has been preliminarily point cloud annotated into individual data according to the ping sequence number, and then generates the corresponding cubic box and center point of each data through the three-dimensional spatial coordinates for each individual data, thereby determining the spatial position of the annotated target in each underwater point cloud data, and realizing the accurate annotation of underwater targets.

[0027] In a possible implementation method of the first aspect, based on the three-dimensional spatial coordinates, a spatial matrix is ​​generated for the point cloud annotation of the target to be annotated in each ping target data, specifically:

[0028] Filter data with the same category label from all single ping target data to obtain the first data;

[0029] According to the point cloud annotation of the target to be annotated in the first data, the maximum and minimum values ​​of the three-dimensional space coordinates of the point cloud annotation are calculated, and a corresponding space matrix is ​​generated according to the maximum and minimum values.

[0030] A second aspect of the present application provides a sonar data annotation system for underwater three-dimensional moving and static targets, the system comprising: a point cloud data acquisition module, a data superposition module, a target segmentation module, and a target annotation module;

[0031] The point cloud data acquisition module is used to convert the format of the collected sonar data to obtain underwater point cloud data; wherein the underwater point cloud data includes three-dimensional spatial coordinates, reflection wave intensity and ping sequence number;

[0032] The data superposition module is used to identify the underwater point cloud data containing the target to be marked according to the three-dimensional spatial coordinates and the reflected wave intensity and to perform data superposition according to the ping sequence number to obtain target superposition data;

[0033] The target segmentation module is used to perform target segmentation and add category labels to the target superposition data according to preset target categories to obtain target segmentation data;

[0034] The target labeling module is used to automatically generate a corresponding stereo frame in the target segmentation data corresponding to each ping sequence number based on the category label and the three-dimensional spatial coordinates; wherein the spatial position of the target to be labeled underwater can be confirmed according to the stereo frame.

[0035] In a possible implementation of the second aspect, the data superposition module includes: a dynamic and static target display unit;

[0036] Among them, the dynamic and static target display unit is used to compare the underwater point cloud data with the preset background environment information according to the three-dimensional spatial coordinates and the reflected wave intensity if the target to be marked is a static target, so as to obtain all ping numbers of the target to be marked in the underwater point cloud data; if the target to be marked is a dynamic target, the underwater point cloud data is replayed frame by frame, and all ping numbers containing the target to be marked are identified according to the inter-frame change relationship of the underwater point cloud data; the underwater point cloud data corresponding to each ping number are superimposed to obtain a mosaic data, which is recorded as target superposition data.

[0037] In a possible implementation of the second aspect, the object segmentation module includes: an object segmentation unit;

[0038] Among them, the target segmentation unit is used to confirm the category label of the target to be labeled according to the preset target category; perform point cloud labeling of the target to be labeled on the target overlay data, and segment all the point cloud labels at one time to complete target segmentation; add the category label to the target overlay data corresponding to each ping number after target segmentation to obtain target segmentation data; wherein, the target segmentation data includes three-dimensional space coordinates, reflection wave intensity, ping number and category label.

[0039] In a possible implementation of the second aspect, the target annotation module includes: a stereo frame generation unit;

[0040] Among them, the stereo frame generation unit is used to split the target segmentation data according to each ping sequence number to obtain a plurality of single-ping target data; based on the three-dimensional spatial coordinates, a spatial matrix is ​​generated for the point cloud annotation of the target to be annotated in each single-ping target data; according to the spatial matrix, a corresponding stereo frame is generated; and according to the length, width and height information of the stereo frame, the center point of the stereo frame is determined.

[0041] In a possible implementation of the second aspect, the target labeling module includes: a space matrix generating unit;

[0042] Among them, the spatial matrix generation unit is used to filter out data with the same category label from all single ping target data to obtain first data; according to the point cloud annotation of the target to be labeled in the first data, calculate the maximum and minimum values ​​of the three-dimensional spatial coordinates of the point cloud annotation, and generate a corresponding spatial matrix according to the maximum and minimum values. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solution of the present application, the following is a brief introduction to the drawings required for use in the implementation. Obviously, the drawings described below are only some implementation methods of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0044] Figure 1 This is a schematic diagram of a specific process of a sonar data annotation method for underwater three-dimensional moving and static targets provided by a certain embodiment of the present application;

[0045] Figure 2 This is a schematic diagram of segmentation and labeling of static targets in a sonar data labeling method for underwater three-dimensional dynamic and static targets provided by a certain embodiment of the present application;

[0046] Figure 3 This is a schematic diagram of segmentation and labeling of dynamic targets in a sonar data labeling method for underwater three-dimensional dynamic and static targets provided by a certain embodiment of the present application;

[0047] Figure 4 This is a structural diagram of a sonar data annotation system for underwater three-dimensional moving and static targets provided by a certain embodiment of the present application. DETAILED DESCRIPTION

[0048] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0049] It should be understood that the step numbers used herein are only for convenience of description and are not intended to limit the order in which the steps are to be executed.

[0050] First embodiment

[0051] Current underwater sonar data labeling mainly relies on manual processing, which often requires experienced technicians to accurately label the data. In addition, manual operation often takes a lot of time compared to computer operation. When encountering large-scale data labeling projects, it is often time-consuming and labor-intensive. Therefore, compared to manually labeling underwater targets frame by frame, how to quickly and accurately perform batch target labeling through computers is one of the main research directions of this application.

[0052] like Figure 1 As shown, Figure 1 A specific flow chart of a method for annotating sonar data of underwater three-dimensional moving and static targets is provided for one embodiment of the present application. The method for annotating sonar data of underwater three-dimensional moving and static targets in this embodiment includes steps S1 to S4, which are described in detail as follows:

[0053] Step S1: convert the collected sonar data to obtain underwater point cloud data.

[0054] In an embodiment of the present application, underwater data is first collected through sonar, and the collected underwater point cloud data is saved in the form of a point cloud; wherein the underwater point cloud data includes three-dimensional spatial coordinates, reflection wave intensity and ping number.

[0055] Specifically, the underwater point cloud data consists of five columns: X-coordinate, Y-coordinate, Z-coordinate, reflected wave intensity I, and ping number. The X-, Y-, and Z-coordinates form the three-dimensional spatial coordinates. The reflected wave intensity is affected by factors such as the incident frequency, angle of incidence, plate thickness, side length, and target material type, and can be used to distinguish the target from other background objects. The ping number is similar to the frame number in LiDAR.

[0056] Step S2: identifying the underwater point cloud data containing the target to be marked according to the three-dimensional spatial coordinates and the reflected wave intensity, and superimposing the data according to the ping sequence number to obtain target superimposed data.

[0057] In this embodiment, the target recognition method is first determined based on the type of target to be labeled. Targets to be labeled are mainly divided into static targets and dynamic targets. Static targets are often measured by a moving sonar platform, so the identification of static targets is related to the sonar platform's speed and scanning area.

[0058] Since the target is stationary, for underwater static targets, it is necessary to distinguish the background environment information from the static targets in the underwater point cloud data based on the sonar platform's speed, scanning area, size of the static target, etc., and obtain the ping sequence number of all underwater point cloud data containing static targets.

[0059] Specifically, in terms of the distribution of three-dimensional spatial coordinates and reflected wave intensity, static targets generally have a clear contrast with the background environmental information and can be distinguished by the naked eye. Therefore, the embodiment of the present application identifies whether the underwater point cloud data contains static targets based on the three-dimensional spatial coordinates and reflected wave intensity. In terms of three-dimensional spatial coordinates, static targets generally protrude from the seabed topography, showing regular spatial protrusions. In terms of reflected wave intensity, the reflected wave intensity of static targets is generally higher than that of the background environmental information, and is highlighted in the underwater point cloud data. Therefore, through the contrast between the three-dimensional spatial coordinates and the reflected wave intensity, underwater point cloud data containing static targets can be identified.

[0060] Data collection for dynamic targets is often more complex than for static targets because they are moving, but identifying and distinguishing them is simpler than for static targets. If the target to be labeled is a dynamic target, it is only necessary to replay the underwater point cloud data frame by frame. The changes between frames can be clearly seen by the naked eye, the movement of the dynamic target in the water can be observed, the underwater point cloud data containing the dynamic target is identified, and the ping sequence number of all underwater point cloud data containing the dynamic target is obtained. In particular, since the underwater environmental noise and the underwater background basically do not change much, the dynamic target can be seen to move frame by frame with the naked eye.

[0061] Finally, based on the obtained ping sequence number containing the target to be annotated, the corresponding underwater point cloud data is found and superimposed to generate a mosaic data, which is recorded as the target superimposed data. The target superimposed data is a part of the data separated from the underwater point cloud data, but stored as a separate whole, including the process from the appearance to the disappearance of the target to be annotated.

[0062] Furthermore, the target overlay data combines underwater background information, underwater noise information, and target information to be annotated. This overlayed data can be used as a whole to segment the target, significantly improving data annotation efficiency. Mosaic data can be used to manage raster-format image data, where the image data does not need to be adjacent or overlapping and can exist as unconnected, discontinuous datasets.

[0063] Step S3: According to a preset target category, target segmentation and category labels are performed on the target superposition data to obtain target segmentation data.

[0064] In an embodiment of the present application, target overlay data containing multiple ping numbers is segmented to achieve point cloud annotation of the target to be labeled, and the target to be labeled is segmented and displayed from the target overlay data.

[0065] Optionally, the embodiment of the present application uses CloudCompare open source software for target segmentation.

[0066] Because the target segmentation process does not require identifying static or dynamic targets, it only requires mapping the entire point cloud of the target to be labeled within the target overlay data. Therefore, the annotation operator's experience requirement is lower than that of existing technologies. Furthermore, the overlaid targets are more distinct and easier to distinguish, improving the accuracy of data annotation.

[0067] Furthermore, because the target overlay data is a mosaic of multiple pings, batch point cloud segmentation can be performed simultaneously, significantly improving the annotation rate of single-ping point clouds within the overlay data. In the present embodiment, since the target overlay data is a single entity, target segmentation can be completed all at once. By then splitting the segmented data into single-ping data, multiple data sets with segmented targets can be obtained simultaneously. This allows for rapid stereo box annotation of the target to be annotated within the single-ping data.

[0068] At the same time, we also combine prior knowledge from the data collection process to determine the category labels corresponding to the target to be annotated from the preset target categories. We then add category labels to the target overlay data after the point cloud annotation is completed frame by frame, and add the category labels as numerical codes to the sixth column of the data to obtain the target segmentation data. Therefore, the target segmentation data includes 3D spatial coordinates, reflection wave intensity, ping number, and category label.

[0069] Step S4: Based on the category label, a corresponding stereoscopic frame is automatically generated in the target segmentation data corresponding to each ping number using the three-dimensional space coordinates.

[0070] In the embodiment of the present application, the target segmentation data containing multiple pings stacked together are all split into single-ping data according to the ping sequence number of the target segmentation data, to obtain multiple single-ping target data. Among them, all the split single-ping target data contain point clouds of the target to be labeled and the category label corresponding to the target to be labeled.

[0071] Compared with the existing manual frame-by-frame target labeling, the target labeling method for stacked data provided in the embodiment of the present application can identify targets by naked eye, and can also label data in large quantities, greatly improving the efficiency and accuracy of data labeling.

[0072] Then, based on the three-dimensional space coordinates, a space matrix is generated for the point clouds of the target to be labeled in the single-ping target data of the same category label. Then, a corresponding cubic space frame, referred to as a three-dimensional frame, is generated according to the space matrix. Among them, the three-dimensional frame can represent the spatial position of the target to be labeled in the underwater environment.

[0073] In addition, in order to better show the results of static targets in target segmentation and target labeling, Figure 2 A segmentation-labeling diagram of static targets is provided. As shown in the figure, taking the target to be labeled as a cylinder as an example, the left side is the result of one-time segmentation of the target stacked data of the cylinder, in which the orange is the point cloud of the background environment information in the space coordinate system, and the blue is the point cloud of the cylinder in the space coordinate system. According to the difference between the two kinds of point clouds, the point cloud of the cylinder can be labeled and segmented from the target stacked data at one time to obtain the target segmentation data of the cylinder; the right side is the result of target labeling in the single-ping target data obtained after the target segmentation data of the cylinder is completed. Among them, the orange is the point cloud of the background environment information in the space coordinate system, and the blue is the point cloud of the cylinder in the space coordinate system. According to the blue point cloud data, a corresponding cubic space frame can be generated, and in Figure 2 the cubic frame is green, and the labeling of the cylinder is completed accordingly.

[0074] In order to show the results of dynamic targets in target segmentation and target labeling, Figure 3A schematic diagram of segmentation and labeling of dynamic targets is provided. As shown in the figure, taking the human model as an example of the target to be labeled, the left side is the result of a one-time segmentation of the target data of the human model, where the blue is the point cloud of the background environment information in the spatial coordinate system, the orange is the point cloud of the environmental noise in the spatial coordinate system, and the green is the point cloud of the human model in the spatial coordinate system. According to the difference between the three point clouds, the point cloud labeling of the human model can be segmented out from the target superposition data at one time to obtain the target segmentation data of the human model. The right side is the result of target labeling in the single ping target data obtained after the splitting of the target segmentation data of the human model, where the blue is the point cloud of the background environment information in the spatial coordinate system, the orange is the point cloud of the environmental noise in the spatial coordinate system, and the green is the point cloud of the human model in the spatial coordinate system. According to the green point cloud data, the corresponding cubic space frame can be generated. Figure 3 The box in the middle is the cube with a red border, which completes the labeling of the human model.

[0075] Furthermore, the extreme values ​​of the three-dimensional space coordinates of the point cloud of the target to be labeled are obtained, and then the space matrix is ​​determined according to the obtained six extreme values.

[0076] Exemplarily, the three-dimensional spatial coordinates of the point cloud of the target to be labeled are (X, Y, Z), and the six extreme values ​​obtained are Xmin, Xmax, Ymin, Ymax, Zmin and Zmax, where the subscript min represents the minimum value and the subscript max represents the maximum value.

[0077] Then, based on the provided historical annotations, the length, width, and height of the 3D frame are calculated: Xmax-Xmin, Ymax-Ymin, and Zmax-Zmin. This information can be used to calculate the coordinates of the 3D frame's center point, which is used to fix the 3D frame in the coordinate system and confirm its uniqueness. The center point coordinates are (Xmax-Xmin / 2, Ymax-Ymin / 2, Zmax-Zmin / 2).

[0078] Optionally, the embodiment of the present application selects the open source KITTI dataset as the historical annotation situation.

[0079] In summary, the single ping target data of the remaining ping numbers can be used to obtain the corresponding stereo frames and center points according to the above method, marking the spatial position of the target to be marked underwater.

[0080] Furthermore, the distance from the center point to the coordinate origin and the direction angle of the center point, as well as the average reflected wave intensity of the target to be marked, can be obtained as the spatial position information of the target to be marked underwater based on the length, width and height information of the stereo frame and the coordinates of the center point.

[0081] The implementation of the embodiments of the present application has the following beneficial effects:

[0082] The embodiment of the application identifies the target to be labeled from the underwater point cloud data according to the three-dimensional space coordinates and the reflected wave intensity, and stacks the underwater point cloud data containing the target to be labeled into a single mosaic data, i.e., target superposition data, to provide data support for subsequent target labeling of the target superposition data as a whole. Then, the target superposition data is added with a category label, and the target superposition data containing the target to be labeled is simultaneously segmented for each frame of data through point cloud labeling, so that the data labeling rate is greatly improved. Then, a cube of the target to be labeled is generated according to the category label and the three-dimensional space coordinates, the labeling of the underwater data is completed, and the accurate labeling of the underwater target is realized.

[0083] Second embodiment

[0084] Further, in order to execute the underwater three-dimensional dynamic and static target sonar data labeling system corresponding to the above-mentioned method embodiment, to realize the corresponding functions and technical effects, Figure 4 A structural diagram of an underwater three-dimensional dynamic and static target sonar data labeling system is provided. For ease of illustration, only the part related to the embodiment is shown, and the underwater three-dimensional dynamic and static target sonar data labeling system provided by the embodiment includes:

[0085] The point cloud data acquisition module 201 is configured to convert the collected sonar data to obtain underwater point cloud data, wherein the underwater point cloud data contains three-dimensional space coordinates, reflected wave intensity, and ping sequence numbers.

[0086] The data superposition module 202 is configured to identify the underwater point cloud data containing the target to be labeled according to the three-dimensional space coordinates and the reflected wave intensity, and perform data superposition according to the ping sequence numbers to obtain target superposition data.

[0087] In the embodiment of the application, if the target to be labeled is a static target, the underwater point cloud data is compared with preset background environment information according to the three-dimensional space coordinates and the reflected wave intensity to obtain all ping sequence numbers containing the target to be labeled in the underwater point cloud data.

[0088] If the target to be labeled is a dynamic target, the underwater point cloud data is played back frame by frame, and all ping sequence numbers containing the target to be labeled are identified according to the inter-frame change relationship of the underwater point cloud data.

[0089] The underwater point cloud data corresponding to each ping sequence number is superimposed to obtain a mosaic data, which is recorded as target superposition data.

[0090] The target segmentation module 203 is used to perform target segmentation and add category labels to the target superposition data according to preset target categories to obtain target segmentation data.

[0091] In the embodiment of the present application, the category label of the target to be marked is confirmed according to the preset target category;

[0092] Performing point cloud annotation of the target to be annotated on the target overlay data to complete target segmentation;

[0093] The category label is added to the target superposition data corresponding to each ping number after the target segmentation to obtain target segmentation data; wherein the target segmentation data includes three-dimensional space coordinates, reflection wave intensity, ping number and category label.

[0094] The target labeling module 204 is used to label the target to be labeled in the target segmentation data based on the category label and the three-dimensional spatial coordinates; wherein the spatial position of the target to be labeled underwater can be confirmed according to the three-dimensional frame.

[0095] In the embodiment of the present application, the target segmentation data is split according to the ping sequence number to obtain a plurality of single ping target data;

[0096] Based on the three-dimensional spatial coordinates, a spatial matrix is ​​generated for the point cloud annotation of the target to be annotated in each single ping target data;

[0097] Generate a corresponding stereo frame according to the spatial matrix;

[0098] The center point of the stereoscopic frame is determined according to the length, width and height information of the stereoscopic frame.

[0099] In some embodiments, the point cloud data acquisition module 201 further includes:

[0100] First, underwater data is collected through sonar, and the collected underwater point cloud data is saved in the form of point cloud; wherein, the underwater point cloud data includes three-dimensional spatial coordinates, reflection wave intensity and ping sequence number.

[0101] Specifically, the underwater point cloud data consists of five columns: X-coordinate, Y-coordinate, Z-coordinate, reflected wave intensity I, and ping number. The X-, Y-, and Z-coordinates form the three-dimensional spatial coordinates. The reflected wave intensity is affected by factors such as the incident frequency, angle of incidence, plate thickness, side length, and target material type, and can be used to distinguish the target from other background objects. The ping number is similar to the frame number in LiDAR.

[0102] In some embodiments, the data overlay module 202 further includes:

[0103] First, determine the corresponding target recognition method based on the type of target to be labeled. Targets to be labeled are mainly divided into static targets and dynamic targets. Static targets are often measured by the sonar platform's movement, so the recognition of static targets is related to the sonar platform's speed and scanning area.

[0104] Since the target is stationary, for underwater static targets, it is necessary to distinguish the background environment information from the static targets in the underwater point cloud data based on the sonar platform's speed, scanning area, size of the static target, etc., and obtain the ping sequence number of all underwater point cloud data containing static targets.

[0105] Specifically, in terms of the distribution of three-dimensional spatial coordinates and reflected wave intensity, static targets generally have a clear contrast with the background environmental information and can be distinguished by the naked eye. Therefore, the embodiment of the present application identifies whether the underwater point cloud data contains static targets based on the three-dimensional spatial coordinates and reflected wave intensity. In terms of three-dimensional spatial coordinates, static targets generally protrude from the seabed topography, showing regular spatial protrusions. In terms of reflected wave intensity, the reflected wave intensity of static targets is generally higher than that of the background environmental information, and is highlighted in the underwater point cloud data. Therefore, through the contrast between the three-dimensional spatial coordinates and the reflected wave intensity, underwater point cloud data containing static targets can be identified.

[0106] Data collection for dynamic targets is often more complex than for static targets because they are moving, but identifying and distinguishing them is simpler than for static targets. If the target to be labeled is a dynamic target, it is only necessary to replay the underwater point cloud data frame by frame. The changes between frames can be clearly seen by the naked eye, the movement of the dynamic target in the water can be observed, the underwater point cloud data containing the dynamic target is identified, and the ping sequence number of all underwater point cloud data containing the dynamic target is obtained. In particular, since the underwater environmental noise and the underwater background basically do not change much, the dynamic target can be seen to move frame by frame with the naked eye.

[0107] Finally, based on the obtained ping sequence number containing the target to be annotated, the corresponding underwater point cloud data is found and superimposed to generate a mosaic data, which is recorded as the target superimposed data. The target superimposed data is a part of the data separated from the underwater point cloud data, but stored as a separate whole, including the process from the appearance to the disappearance of the target to be annotated.

[0108] Furthermore, the target overlay data combines underwater background information, underwater noise information, and target information to be annotated. This overlayed data can be used as a whole to segment the target, significantly improving data annotation efficiency through batch processing. Mosaic data can be used to manage raster-format image data, where the image data does not need to be adjacent or overlapping and can exist as unconnected, discontinuous datasets.

[0109] In some embodiments, the object segmentation module 203 further includes:

[0110] The target overlay data containing multiple ping numbers is segmented to realize point cloud annotation of the target to be labeled, and the target to be labeled is segmented and displayed from the target overlay data.

[0111] Optionally, the embodiment of the present application uses CloudCompare open source software for target segmentation.

[0112] Because the target segmentation process does not require identifying static or dynamic targets, it only requires mapping the entire point cloud of the target to be labeled within the target overlay data. Therefore, the annotation operator's experience requirement is lower than that of existing technologies. Furthermore, the overlaid targets are more distinct and easier to distinguish, improving the accuracy of data annotation.

[0113] In addition, since the target overlay data is a separate, superimposed mosaic data, batch point cloud annotation can be performed simultaneously, thus greatly improving the data processing rate.

[0114] At the same time, we also combine prior knowledge from the data collection process to determine the category labels corresponding to the target to be annotated from the preset target categories. We then add category labels to the target overlay data after the point cloud annotation is completed frame by frame, and add the category labels as numerical codes to the sixth column of the data to obtain the target segmentation data. Therefore, the target segmentation data includes 3D spatial coordinates, reflection wave intensity, ping number, and category label.

[0115] In some embodiments, the target labeling module 204 further includes:

[0116] First, based on the ping sequence number of the target segmentation data, all the superimposed target segmentation data containing multiple pings are split into single ping data to obtain multiple single ping target data. Among them, all the split single ping target data contain the point cloud of the target to be labeled and the corresponding category label of the target to be labeled.

[0117] Compared with the existing manual frame-by-frame target labeling, the target labeling method for superimposed data provided in the embodiment of the present application can identify targets with the naked eye and can also label data in large quantities, greatly improving the efficiency and accuracy of data labeling.

[0118] Based on the three-dimensional spatial coordinates, a spatial matrix is ​​generated for the point cloud of the target to be annotated in the single-ping target data with the same category label. A corresponding cubic spatial frame, referred to as a stereo frame, is then generated based on the spatial matrix. The stereo frame can represent the spatial position of the target to be annotated in the underwater environment.

[0119] Furthermore, the extreme values ​​of the three-dimensional space coordinates of the point cloud of the target to be labeled are obtained, and then the space matrix is ​​determined according to the obtained six extreme values.

[0120] Exemplarily, the three-dimensional spatial coordinates of the point cloud of the target to be labeled are (X, Y, Z), and the six extreme values ​​obtained are Xmin, Xmax, Ymin, Ymax, Zmin and Zmax, where the subscript min represents the minimum value and the subscript max represents the maximum value.

[0121] Then, based on the provided historical annotations, the length, width, and height of the 3D frame are calculated: Xmax-Xmin, Ymax-Ymin, and Zmax-Zmin. This information can be used to calculate the coordinates of the 3D frame's center point, which is used to fix the 3D frame in the coordinate system and confirm its uniqueness. The center point coordinates are (Xmax-Xmin / 2, Ymax-Ymin / 2, Zmax-Zmin / 2).

[0122] Optionally, the embodiment of the present application selects the open source KITTI dataset as the historical annotation situation.

[0123] In summary, the single ping target data of the remaining ping numbers can be used to obtain the corresponding stereo frames and center points according to the above method, marking the spatial position of the target to be marked underwater.

[0124] Furthermore, the distance from the center point to the coordinate origin and the direction angle of the center point, as well as the average reflected wave intensity of the target to be marked, can be obtained as the spatial position information of the target to be marked underwater based on the length, width and height information of the stereo frame and the coordinates of the center point.

[0125] The implementation of the embodiments of the present application has the following beneficial effects:

[0126] The embodiment of the present application first identifies the target to be labeled from the underwater point cloud data based on the three-dimensional spatial coordinates and the reflected wave intensity, and superimposes the underwater point cloud data containing the target to be labeled into a separate mosaic data by frame, that is, the target superposition data, to provide data support for the subsequent target superposition data to be labeled as a whole. Then, a category label is added to the target superposition data and the target superposition data containing the target to be labeled is treated as a whole in the data through point cloud annotation, and the target segmentation is performed on each frame of data at the same time, which greatly improves the data annotation rate. Then, based on the category label and the three-dimensional spatial coordinates, a cube of the target to be labeled is generated to complete the labeling of the underwater data and achieve accurate labeling of the underwater target.

[0127] The specific embodiments described above further illustrate the purpose, technical solutions, and beneficial effects of this application. It should be understood that the above description is merely a specific embodiment of this application and is not intended to limit the scope of protection of this application. In particular, it should be noted that for those skilled in the art, any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of this application should be included in the scope of protection of this application.

Claims

1. A sonar data annotation method for underwater three-dimensional dynamic and static targets, characterized in that: include: Converting the collected sonar data to obtain underwater point cloud data; wherein the underwater point cloud data includes three-dimensional spatial coordinates, reflection wave intensity and ping sequence number; identifying the underwater point cloud data containing the target to be marked according to the three-dimensional spatial coordinates and the reflected wave intensity and superimposing the data according to the ping sequence number to obtain target superimposed data; According to the preset target categories, batch target segmentation and category labels are performed on the target superposition data to obtain target segmentation data; Based on the category label, a corresponding stereo frame is automatically generated in the target segmentation data corresponding to each ping number using the three-dimensional spatial coordinates; wherein the spatial position of the target to be labeled underwater can be confirmed based on the stereo frame.

2. The sonar data annotation method for underwater three-dimensional dynamic and static targets according to claim 1, characterized in that: The underwater point cloud data containing the target to be marked is identified based on the three-dimensional spatial coordinates and the reflected wave intensity, and data is superimposed according to the ping sequence number to obtain target superimposed data, specifically: If the target to be marked is a static target, the underwater point cloud data is compared with the preset background environment information according to the three-dimensional spatial coordinates and the reflected wave intensity to obtain all ping sequence numbers of the target to be marked in the underwater point cloud data; If the target to be marked is a dynamic target, the underwater point cloud data is replayed frame by frame, and all ping sequence numbers containing the target to be marked are identified based on the inter-frame change relationship of the underwater point cloud data; The underwater point cloud data corresponding to each ping number are superimposed to obtain mosaic data, which is recorded as target superimposed data.

3. The sonar data annotation method for underwater three-dimensional dynamic and static targets according to claim 1, characterized in that: The target superposition data is subjected to batch target segmentation and category labeling according to the preset target category to obtain target segmentation data, specifically: According to the preset target category, confirm the category label of the target to be marked; Performing point cloud annotation of the target to be annotated on the target overlay data, and segmenting all the point cloud annotations at one time to complete target segmentation; The category label is added to the target superposition data corresponding to each ping number after the target segmentation to obtain target segmentation data; wherein the target segmentation data includes three-dimensional space coordinates, reflection wave intensity, ping number and category label.

4. The sonar data annotation method for underwater three-dimensional dynamic and static targets according to claim 1, characterized in that: The method of automatically generating a corresponding stereoscopic frame in the target segmentation data corresponding to each ping sequence number based on the category label and the three-dimensional space coordinates is specifically as follows: Split the target segmentation data according to each ping sequence number to obtain several single ping target data; Based on the three-dimensional spatial coordinates, a spatial matrix is ​​generated for the point cloud annotation of the target to be annotated in each single ping target data; Generate a corresponding stereo frame according to the spatial matrix; The center point of the stereoscopic frame is determined according to the length, width and height information of the stereoscopic frame.

5. The sonar data annotation method for underwater three-dimensional dynamic and static targets according to claim 4, characterized in that: Based on the three-dimensional space coordinates, a spatial matrix is ​​generated for the point cloud annotation of the target to be annotated in each single ping target data, specifically: Filter data with the same category label from all single ping target data to obtain the first data; According to the point cloud annotation of the target to be annotated in the first data, the maximum and minimum values ​​of the three-dimensional space coordinates of the point cloud annotation are calculated, and a corresponding space matrix is ​​generated according to the maximum and minimum values.

6. A sonar data annotation system for underwater three-dimensional moving and static targets, characterized by: include: Point cloud data acquisition module, data overlay module, target segmentation module and target annotation module; The point cloud data acquisition module is used to convert the collected sonar data to obtain underwater point cloud data; wherein the underwater point cloud data includes three-dimensional spatial coordinates, reflection wave intensity and ping sequence number; The data superposition module is used to identify the underwater point cloud data containing the target to be marked according to the three-dimensional spatial coordinates and the reflected wave intensity and to perform data superposition according to the ping sequence number to obtain target superposition data; The target segmentation module is used to perform target segmentation and add category labels to the target superposition data according to preset target categories to obtain target segmentation data; The target labeling module is used to automatically generate a corresponding stereo frame in the target segmentation data corresponding to each ping sequence number based on the category label and the three-dimensional spatial coordinates; wherein the spatial position of the target to be labeled underwater can be confirmed according to the stereo frame.

7. The underwater three-dimensional dynamic and static target sonar data annotation system according to claim 6, characterized in that: The data superposition module includes: a dynamic and static target display unit; Among them, the dynamic and static target display unit is used to compare the underwater point cloud data with the preset background environment information according to the three-dimensional spatial coordinates and the reflected wave intensity if the target to be marked is a static target, so as to obtain all ping numbers of the target to be marked in the underwater point cloud data; if the target to be marked is a dynamic target, the underwater point cloud data is replayed frame by frame, and all ping numbers containing the target to be marked are identified according to the inter-frame change relationship of the underwater point cloud data; the underwater point cloud data corresponding to each ping number are superimposed to obtain a mosaic data, which is recorded as target superposition data.

8. The underwater three-dimensional dynamic and static target sonar data annotation system according to claim 6, characterized in that: The target segmentation module includes: a target segmentation unit; Among them, the target segmentation unit is used to confirm the category label of the target to be labeled according to the preset target category; perform point cloud labeling of the target to be labeled on the target overlay data, and segment all the point cloud labels at one time to complete target segmentation; add the category label to the target overlay data corresponding to each ping number after target segmentation to obtain target segmentation data; wherein, the target segmentation data includes three-dimensional space coordinates, reflection wave intensity, ping number and category label.

9. The underwater three-dimensional dynamic and static target sonar data annotation system according to claim 6, characterized in that: The target annotation module includes: a stereo frame generation unit; Among them, the stereo frame generation unit is used to split the target segmentation data according to each ping sequence number to obtain a plurality of single-ping target data; based on the three-dimensional spatial coordinates, a spatial matrix is ​​generated for the point cloud annotation of the target to be annotated in each single-ping target data; according to the spatial matrix, a corresponding stereo frame is generated; and according to the length, width and height information of the stereo frame, the center point of the stereo frame is determined.

10. The underwater three-dimensional dynamic and static target sonar data annotation system according to claim 9, characterized in that: The target marking module includes: a space matrix generating unit; Among them, the spatial matrix generation unit is used to filter out data with the same category label from all single ping target data to obtain first data; according to the point cloud annotation of the target to be labeled in the first data, calculate the maximum and minimum values ​​of the three-dimensional spatial coordinates of the point cloud annotation, and generate a corresponding spatial matrix according to the maximum and minimum values.

Images