Sonar data labeling method and system for underwater three-dimensional moving and static target

Abstract

The invention discloses a sonar data labeling method and system for an underwater three-dimensional dynamic and static target, and the method comprises the steps: carrying out the conversion of collected sonar data, and obtaining underwater point cloud data; according to the three-dimensional space coordinates and the reflected wave intensity, identifying the underwater point cloud data containing a to-be-labeled target, and performing data superposition according to the ping serial number to obtain target superposition data; according to a preset target category, performing batch target segmentation and category label addition on the target superposition data to obtain target segmentation data; and based on the category label, automatically generating a corresponding three-dimensional frame in the target segmentation data corresponding to each ping serial number through the three-dimensional space coordinates. According to the method and the device, the underwater data containing the to-be-labeled target is superposed into the mosaic data to perform one-time target segmentation, and then the three-dimensional labeling box is automatically generated for the segmented single ping data, so that the target labeling efficiency and accuracy can be greatly improved.

Description

Technical Field

[0001] The present application belongs to the field of underwater sonar data processing, and specifically relates to a sonar data labeling method and system for underwater three-dimensional dynamic and static targets. Background Art

[0002] Sonar is the most widely used device for underwater target identification. New sonar technologies provide the necessary technical support for timely and accurate detection of underwater targets. Target identification, as an important part of sonar post-data processing, provides the necessary data support for classifying multiple targets and identifying true targets.

[0003] For underwater three-dimensional data obtained by sonar, the existing three-dimensional target annotation method is to use laser radar, depth camera or radar point cloud data. Usually, the annotation means are mainly manual means, similar to the two-dimensional image annotation method. The three-dimensional target annotation is mainly based on manually pulling a stereo frame that can envelop the target. Due to the influence of the annotation personnel's professional ability, the existing annotation method that relies on manually pulling the target frame has a large spatial position error, low efficiency, large engineering workload and high cost. In addition, there is currently a lack of annotation standards for underwater sonar data. Summary of the invention

[0004] The present application proposes a sonar data labeling method and system for underwater three-dimensional dynamic and static targets, which superimposes the underwater data containing the target to be labeled into a mosaic data for a one-time target segmentation, and then automatically generates a three-dimensional labeling frame for the segmented single ping data, which can greatly improve the target labeling efficiency and accuracy.

[0005] A first aspect of the present application provides a sonar data annotation method for underwater three-dimensional dynamic and static targets, the method comprising:

[0006] Converting the collected sonar data to obtain underwater point cloud data; wherein the underwater point cloud data includes three-dimensional spatial coordinates, reflection wave intensity and ping number;

[0007] According to the three-dimensional space coordinates and the reflected wave intensity, the underwater point cloud data containing the target to be marked is identified and data is superimposed according to the ping sequence number to obtain target superimposed data;

[0008] According to a preset target category, target segmentation and category labeling are performed on the target superposition data to obtain target segmentation data;

[0009] Based on the category label, a corresponding stereoscopic frame is automatically generated in the target segmentation data corresponding to each ping number through the three-dimensional spatial coordinates; wherein the spatial position of the target to be marked underwater can be confirmed according to the stereoscopic frame.

[0010] The above scheme first identifies the target to be labeled from the underwater point cloud data according to the three-dimensional spatial coordinates and the reflected wave intensity, and superimposes the underwater point cloud data containing the target to be labeled into a separate mosaic data by frame, that is, the target superposition data, which provides data support for the subsequent target superposition data to be labeled as a whole. Then, category labels are added to the target superposition data, and the target superposition data containing the target to be labeled is treated as a whole in the data through point cloud annotation, and the target is segmented for each frame of data at the same time, which greatly improves the data annotation rate. Then, based on the category label and the three-dimensional spatial coordinates, a cube of the target to be labeled is generated to complete the labeling of the underwater data and achieve accurate labeling of underwater targets.

[0011] In a possible implementation method of the first aspect, the underwater point cloud data containing the target to be marked is identified according to the three-dimensional space coordinates and the reflected wave intensity, and data is superimposed according to the ping sequence number to obtain target superimposed data, specifically:

[0012] If the target to be marked is a static target, the underwater point cloud data is compared with the preset background environment information according to the three-dimensional space coordinates and the reflected wave intensity to obtain all ping sequence numbers of the target to be marked in the underwater point cloud data;

[0013] If the target to be marked is a dynamic target, the underwater point cloud data is played back frame by frame, and all ping numbers containing the target to be marked are identified according to the inter-frame change relationship of the underwater point cloud data;

[0014] The underwater point cloud data corresponding to each ping number are superimposed to obtain mosaic data, which is recorded as target superimposed data.

[0015] In the above scheme, for static targets, the target to be marked is distinguished from the surrounding background environment according to the difference in three-dimensional spatial coordinates and the intensity of the reflected wave, the target to be marked in the underwater point cloud data is identified and the corresponding ping number is obtained; for dynamic targets, the underwater point cloud data is played back frame by frame to identify the dynamic changes of the target to be marked in each frame, so as to obtain the underwater point cloud data containing the target to be marked and the corresponding ping number; finally, the corresponding underwater point cloud data is superimposed into a mosaic data through the ping number, and the mosaic data is used as a whole target superimposed data. Because the target superimposed data contains the target to be marked, it can be directly marked without screening the data, providing data support for the subsequent simultaneous processing of the target superimposed data.

[0016] In a possible implementation method of the first aspect, target segmentation and category labeling are performed on the target superposition data according to a preset target category to obtain target segmentation data, specifically:

[0017] According to the preset target category, confirm the category label of the target to be marked;

[0018] Performing point cloud annotation of the target to be annotated on the target overlay data, and segmenting all the point cloud annotations at one time to complete target segmentation;

[0019] The category label is added to the target superposition data corresponding to each ping number after the target is segmented to obtain target segmentation data; wherein the target segmentation data includes three-dimensional space coordinates, reflection wave intensity, ping number and category label.

[0020] The above scheme treats the target overlay data of multiple pings as a whole and performs target segmentation at the same time, which greatly improves the data processing rate and adds category labels to the segmented data to provide data support for subsequent cube generation.

[0021] In a possible implementation method of the first aspect, based on the category label, a corresponding stereoscopic frame is automatically generated in the target segmentation data corresponding to each ping sequence number by using the three-dimensional space coordinates, specifically:

[0022] Split the target segmentation data according to each ping sequence number to obtain several single ping target data;

[0023] Based on the three-dimensional spatial coordinates, a spatial matrix is ​​generated for the point cloud annotation of the target to be annotated in each single ping target data;

[0024] According to the space matrix, a corresponding stereo frame is generated;

[0025] The center point of the stereoscopic frame is determined according to the length, width and height information of the stereoscopic frame.

[0026] The above scheme splits the data that has been preliminarily point cloud annotated into single data according to the ping sequence number, and then generates the corresponding cubic box and center point of each data through the three-dimensional space coordinates for the single data, thereby determining the spatial position of the annotated target in each underwater point cloud data, and realizing the accurate annotation of underwater targets.

[0027] In a possible implementation method of the first aspect, based on the three-dimensional space coordinates, a space matrix is ​​generated for the point cloud annotation of the target to be annotated in each ping target data, specifically:

[0028] Filter data with the same category label from all single ping target data to obtain first data;

[0029] According to the point cloud annotation of the target to be annotated in the first data, the maximum and minimum values ​​of the three-dimensional space coordinates of the point cloud annotation are calculated, and the corresponding space matrix is ​​generated according to the maximum and minimum values.

[0030] A second aspect of the present application provides a sonar data annotation system for underwater three-dimensional dynamic and static targets, the system comprising: a point cloud data acquisition module, a data superposition module, a target segmentation module and a target annotation module;

[0031] The point cloud data acquisition module is used to convert the format of the collected sonar data to obtain underwater point cloud data; wherein the underwater point cloud data includes three-dimensional space coordinates, reflection wave intensity and ping number;

[0032] The data superposition module is used to identify the underwater point cloud data containing the target to be marked according to the three-dimensional space coordinates and the reflected wave intensity and to perform data superposition according to the ping sequence number to obtain target superposition data;

[0033] The target segmentation module is used to perform target segmentation and add category labels to the target superposition data according to a preset target category to obtain target segmentation data;

[0034] The target labeling module is used to automatically generate a corresponding stereo frame in the target segmentation data corresponding to each ping number based on the category label and the three-dimensional spatial coordinates; wherein the spatial position of the target to be labeled underwater can be confirmed according to the stereo frame.

[0035] In a possible implementation of the second aspect, the data superposition module includes: a dynamic and static target display unit;

[0036] Among them, the dynamic and static target display unit is used to compare the underwater point cloud data with the preset background environment information according to the three-dimensional space coordinates and the reflected wave intensity if the target to be marked is a static target, so as to obtain all ping numbers of the target to be marked in the underwater point cloud data; if the target to be marked is a dynamic target, the underwater point cloud data is played back frame by frame, and all ping numbers containing the target to be marked are identified according to the inter-frame change relationship of the underwater point cloud data; the underwater point cloud data corresponding to each ping number are superimposed to obtain a mosaic data, which is recorded as target superposition data.

[0037] In a possible implementation manner of the second aspect, the target segmentation module includes: a target segmentation unit;

[0038] The target segmentation unit is used to confirm the category label of the target to be labeled according to the preset target category; perform point cloud labeling of the target to be labeled on the target overlay data, and segment all the point cloud labeling at one time to complete the target segmentation; add the category label to the target overlay data corresponding to each ping number after the target segmentation to obtain the target segmentation data; wherein the target segmentation data includes three-dimensional space coordinates, reflection wave intensity, ping number and category label.

[0039] In a possible implementation of the second aspect, the target annotation module includes: a stereo frame generation unit;

[0040] Among them, the stereo frame generation unit is used to split the target segmentation data according to each ping sequence number to obtain a number of single ping target data; based on the three-dimensional space coordinates, a spatial matrix is ​​generated for the point cloud annotation of the target to be annotated in each single ping target data; according to the spatial matrix, a corresponding stereo frame is generated; according to the length, width and height information of the stereo frame, the center point of the stereo frame is determined.

[0041] In a possible implementation manner of the second aspect, the target labeling module includes: a space matrix generating unit;

[0042] Among them, the spatial matrix generation unit is used to filter out data with the same category label from all single ping target data to obtain first data; according to the point cloud annotation of the target to be labeled in the first data, calculate the maximum and minimum values ​​of the three-dimensional spatial coordinates of the point cloud annotation, and generate the corresponding spatial matrix according to the maximum and minimum values. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the implementation manner will be briefly introduced below. Obviously, the drawings described below are only some implementation manners of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0044] Figure 1 It is a specific flow chart of a sonar data labeling method for underwater three-dimensional dynamic and static targets provided by a certain embodiment of the present application;

[0045] Figure 2 It is a schematic diagram of segmentation and labeling of static targets in a sonar data labeling method for underwater three-dimensional dynamic and static targets provided by a certain embodiment of the present application;

[0046] Figure 3 It is a schematic diagram of segmentation and labeling of dynamic targets in a sonar data labeling method for underwater three-dimensional dynamic and static targets provided by a certain embodiment of the present application;

[0047] Figure 4 It is a structural diagram of a sonar data annotation system for underwater three-dimensional dynamic and static targets provided in a certain embodiment of the present application. DETAILED DESCRIPTION

[0048] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0049] It should be understood that the step numbers used in this article are only for the convenience of description and are not intended to limit the order in which the steps are executed.

[0050] First embodiment

[0051] Currently, the labeling of underwater sonar data mainly relies on manual processing, which often requires experienced technicians to accurately label the data. In addition, manual operation often takes a lot of time compared to computer operation. When encountering a large amount of data labeling, it is often time-consuming and laborious. Therefore, compared to manually labeling underwater targets frame by frame, how to quickly and accurately perform batch target labeling through a computer is one of the main research directions of this application.

[0052] like Figure 1 As shown, Figure 1 A specific flow chart of a method for labeling underwater three-dimensional moving and static targets sonar data is provided for a certain embodiment of the present application. The method for labeling underwater three-dimensional moving and static targets sonar data of this embodiment includes steps S1 to S4, which are described in detail as follows:

[0053] Step S1, converting the collected sonar data to obtain underwater point cloud data.

[0054] In an embodiment of the present application, underwater data is first collected through sonar, and the collected underwater point cloud data is saved in the form of a point cloud; wherein the underwater point cloud data includes three-dimensional spatial coordinates, reflection wave intensity and ping number.

[0055] Specifically, the underwater point cloud data mainly has 5 columns, namely X-direction coordinates, Y-direction coordinates, Z-direction coordinates, reflection wave intensity I and ping number. The X-direction coordinates, Y-direction coordinates, and Z-direction coordinates constitute three-dimensional space coordinates. The reflection wave intensity is affected by the incident frequency, incident angle, plate thickness, side length, and target material type, and can be used to distinguish the target to be marked from other background objects. The ping number is similar to the frame number in the laser radar.

[0056] Step S2: identifying the underwater point cloud data containing the target to be marked according to the three-dimensional space coordinates and the reflected wave intensity, and superimposing the data according to the ping sequence number to obtain target superimposed data.

[0057] In the embodiment of the present application, the corresponding target recognition method is first determined according to the type of the target to be marked. The targets to be marked are mainly divided into static targets and dynamic targets. Static targets are often obtained by moving the sonar platform for measurement, so the recognition of static targets is related to the speed and scanning area of ​​the sonar platform.

[0058] Since the target is stationary, for underwater static targets, it is necessary to distinguish the background environment information from the static targets in the underwater point cloud data based on the sonar platform's speed, scanning area, size of the static targets, etc., and obtain the ping sequence number of all underwater point cloud data containing static targets.

[0059] Specifically, in terms of the distribution of three-dimensional spatial coordinates and reflected wave intensity, static targets generally have a clear contrast with the background environmental information and can be distinguished by the naked eye. Therefore, the embodiment of the present application identifies whether the underwater point cloud data contains static targets based on the three-dimensional spatial coordinates and reflected wave intensity. In terms of three-dimensional spatial coordinates, static targets generally protrude from the seabed topography, showing regular spatial protrusions. In terms of reflected wave intensity, the reflected wave intensity of static targets is generally higher than the background environmental information, and is highlighted in the underwater point cloud data. Therefore, through the contrast between the three-dimensional spatial coordinates and the reflected wave intensity, underwater point cloud data containing static targets can be identified.

[0060] The data collection of dynamic targets is often more complicated than that of static targets because the targets are moving, but the recognition and distinction of the targets are simpler than those of static targets. If the target to be marked is a dynamic target, it is only necessary to replay the underwater point cloud data frame by frame. The naked eye can clearly see the change relationship between frames, observe the movement of the dynamic target in the water, identify the underwater point cloud data containing the dynamic target, and obtain the ping sequence number of all underwater point cloud data containing the dynamic target. Among them, since the underwater environmental noise and the underwater background basically do not change much, it can be seen by the naked eye that the dynamic target is moving frame by frame.

[0061] Finally, according to each ping sequence number obtained containing the target to be labeled, the corresponding underwater point cloud data is found, and these data are superimposed to generate a mosaic data, which is recorded as the target superposition data. The target superposition data is part of the data separated from the underwater point cloud data, but is stored as a separate whole, including the process from the appearance to the disappearance of the target to be labeled.

[0062] Furthermore, the target overlay data overlays underwater background environment information, underwater noise information and target information to be annotated, and the overlayed data can be used as a whole to segment the target, greatly improving the efficiency of data annotation. The mosaic data can be used to manage image data in raster form, in which the image data does not have to be adjacent or overlapped, and can exist in the form of unconnected discontinuous data sets.

[0063] Step S3, according to a preset target category, target segmentation and adding category labels to the target superposition data to obtain target segmentation data.

[0064] In an embodiment of the present application, target overlay data containing multiple ping numbers is segmented to achieve point cloud annotation of the target to be labeled, and the target to be labeled is segmented and displayed from the target overlay data.

[0065] Optionally, the embodiment of the present application uses CloudCompare open source software to perform target segmentation.

[0066] Because it is not necessary to identify whether the target is static or dynamic during the target segmentation process, it is only necessary to plot all the point clouds of the target to be labeled in the target superposition data, so the experience requirements for the labeling operator are lower than that of the existing technology. In addition, the superimposed targets to be labeled will be more obvious and easier to distinguish, which improves the accuracy of data labeling.

[0067] In addition, since the target overlay data is a mosaic of multiple pings superimposed together, batch point cloud segmentation can be performed simultaneously, further greatly improving the annotation rate of single ping point clouds in the overlay data. In the embodiment of the present application, since the target overlay data is a whole, the target segmentation of the superimposed target overlay data can be completed at one time, and then the segmented data can be split into single ping data, so that multiple data that have completed target segmentation can be obtained at the same time. On this basis, the stereoscopic frame annotation of the target to be annotated in the single ping data can be quickly realized.

[0068] At the same time, combined with the prior knowledge in the data collection process, the category label corresponding to the target to be annotated is determined from the preset target category, and the category label is added frame by frame to the target superposition data after the point cloud annotation is completed, and the category label is added to the sixth column of the data in the form of a digital code to obtain the target segmentation data. Therefore, the target segmentation data includes three-dimensional space coordinates, reflection wave intensity, ping sequence number and category label.

[0069] Step S4, based on the category label, automatically generate a corresponding stereoscopic frame in the target segmentation data corresponding to each ping number through the three-dimensional space coordinates.

[0070] In the embodiment of the present application, first, according to the ping sequence number of the target segmentation data, all the superimposed target segmentation data containing multiple pings are split into single ping data to obtain multiple single ping target data. Among them, all the split single ping target data contain the point cloud of the target to be labeled and the category label corresponding to the target to be labeled.

[0071] Compared with the existing manual frame-by-frame target labeling, the target labeling method for superimposed data provided in the embodiment of the present application can identify targets by naked eyes and can also label data in large quantities, which greatly improves the efficiency and accuracy of data labeling.

[0072] Then, based on the three-dimensional spatial coordinates, a spatial matrix is ​​generated for the point cloud of the target to be labeled in the single ping target data of the same category label. Then, a corresponding cubic spatial frame, referred to as a stereo frame, is generated according to the spatial matrix. The stereo frame can represent the spatial position of the target to be labeled in the underwater environment.

[0073] In addition, in order to better show the results of static target segmentation and target labeling, Figure 2 A schematic diagram of the segmentation and annotation of static targets is provided. As shown in the figure, taking the target to be annotated as a cylinder as an example, the left side shows the result of a one-time segmentation of the target superposition data of the cylinder, where the orange is the point cloud of the background environment information in the spatial coordinate system, and the blue is the point cloud of the cylinder in the spatial coordinate system. According to the difference between the two point clouds, the point cloud annotation of the cylinder can be segmented out from the target superposition data at one time to obtain the target segmentation data of the cylinder; the right side shows the result of target annotation in the single ping target data obtained after the completion of the splitting of the target segmentation data of the cylinder, where the orange is the point cloud of the background environment information in the spatial coordinate system, and the blue is the point cloud of the cylinder in the spatial coordinate system. According to the blue point cloud data, the corresponding cubic space frame can be generated. Figure 2 The cyan cube in the middle is used to complete the labeling of the cylinder.

[0074] In order to show the results of target segmentation and target labeling of dynamic targets, Figure 3A schematic diagram of the segmentation and labeling of dynamic targets is provided. As shown in the figure, taking the human model as an example of the target to be labeled, the left side is the result of a one-time segmentation of the target data of the human model, where blue is the point cloud of the background environmental information in the spatial coordinate system, orange is the point cloud of the environmental noise in the spatial coordinate system, and green is the point cloud of the human model in the spatial coordinate system. According to the difference between the three point clouds, the point cloud labeling of the human model can be segmented out from the target superposition data at one time to obtain the target segmentation data of the human model. The right side is the result of target labeling in the single ping target data obtained after the splitting of the target segmentation data of the human model, where blue is the point cloud of the background environmental information in the spatial coordinate system, orange is the point cloud of the environmental noise in the spatial coordinate system, and green is the point cloud of the human model in the spatial coordinate system. According to the green point cloud data, the corresponding cubic space frame can be generated. Figure 3 The box in the middle is the cube with a red border, which is used to complete the labeling of the human model.

[0075] Furthermore, the extreme values ​​of the three-dimensional space coordinates of the point cloud of the target to be labeled are calculated, and then the space matrix is ​​determined according to the obtained six extreme values.

[0076] Exemplarily, the three-dimensional spatial coordinates of the point cloud of the target to be labeled are (X, Y, Z), and the six extreme values ​​obtained are Xmin, Xmax, Ymin, Ymax, Zmin and Zmax, where the subscript min represents the minimum value and the subscript max represents the maximum value.

[0077] Then, according to the historical annotation information provided, the length, width and height information of the stereoscopic frame, namely Xmax-Xmin, Ymax-Ymin and Zmax-Zmin, is calculated. According to the length, width and height information, the coordinates of the center point of the stereoscopic frame can be calculated, which is used to fix the stereoscopic frame in the coordinate system to confirm the uniqueness of the stereoscopic frame. Among them, the coordinates of the center point are (Xmax-Xmin / 2, Ymax-Ymin / 2, Zmax-Zmin / 2).

[0078] Optionally, the embodiment of the present application selects the open source KITTI dataset as the historical annotation situation.

[0079] In summary, the single ping target data of the remaining ping numbers can obtain the corresponding stereoscopic frames and center points according to the above method, and mark the spatial position of the target to be marked underwater.

[0080] Furthermore, the distance between the center point and the origin of the coordinates, the direction angle of the center point, and the average reflected wave intensity of the target to be marked can be obtained as the spatial position information of the target to be marked underwater according to the length, width, and height information of the stereo frame and the coordinates of the center point.

[0081] Implementing the embodiments of the present application has the following beneficial effects:

[0082] The embodiment of the present application first identifies the target to be labeled from the underwater point cloud data according to the three-dimensional spatial coordinates and the reflected wave intensity, and superimposes the underwater point cloud data containing the target to be labeled into a separate mosaic data by frame, that is, the target superposition data, to provide data support for the subsequent target superposition data as a whole and target labeling. Then, a category label is added to the target superposition data, and the target superposition data containing the target to be labeled is treated as a whole in the data through point cloud labeling, and each frame of data is segmented at the same time, which greatly improves the data labeling rate. Then, based on the category label and the three-dimensional spatial coordinates, a cube of the target to be labeled is generated to complete the labeling of the underwater data and achieve accurate labeling of the underwater target.

[0083] Second embodiment

[0084] Furthermore, in order to implement the underwater three-dimensional dynamic and static target sonar data annotation system corresponding to the above method embodiment to achieve corresponding functions and technical effects, Figure 4 A structural diagram of a sonar data annotation system for underwater three-dimensional moving and static targets is provided. For ease of description, only the parts related to this embodiment are shown. The sonar data annotation system for underwater three-dimensional moving and static targets provided in the embodiment of the present application includes:

[0085] The point cloud data acquisition module 201 is used to convert the collected sonar data to obtain underwater point cloud data; wherein the underwater point cloud data includes three-dimensional space coordinates, reflection wave intensity and ping number.

[0086] The data superposition module 202 is used to identify the underwater point cloud data containing the target to be marked according to the three-dimensional space coordinates and the reflected wave intensity and to perform data superposition according to the ping sequence number to obtain target superposition data.

[0087] In an embodiment of the present application, if the target to be marked is a static target, the underwater point cloud data is compared with the preset background environment information according to the three-dimensional space coordinates and the reflected wave intensity to obtain all ping sequence numbers of the target to be marked in the underwater point cloud data;

[0088] If the target to be marked is a dynamic target, the underwater point cloud data is played back frame by frame, and all ping numbers containing the target to be marked are identified according to the inter-frame change relationship of the underwater point cloud data;

[0089] The underwater point cloud data corresponding to each ping number are superimposed to obtain mosaic data, which is recorded as target superimposed data.

[0090] The target segmentation module 203 is used to perform target segmentation and add category labels to the target superposition data according to a preset target category to obtain target segmentation data.

[0091] In the embodiment of the present application, the category label of the target to be marked is confirmed according to the preset target category;

[0092] Performing point cloud annotation of the target to be annotated on the target overlay data to complete target segmentation;

[0093] The category label is added to the target superposition data corresponding to each ping number after the target is segmented to obtain target segmentation data; wherein the target segmentation data includes three-dimensional space coordinates, reflection wave intensity, ping number and category label.

[0094] The target labeling module 204 is used to label the stereoscopic frame corresponding to the target to be labeled in the target segmentation data through the three-dimensional spatial coordinates based on the category label; wherein the spatial position of the target to be labeled underwater can be confirmed according to the stereoscopic frame.

[0095] In the embodiment of the present application, the target segmentation data is split according to the ping sequence number to obtain a plurality of single ping target data;

[0096] Based on the three-dimensional spatial coordinates, a spatial matrix is ​​generated for the point cloud annotation of the target to be annotated in each single ping target data;

[0097] According to the space matrix, a corresponding stereo frame is generated;

[0098] The center point of the stereoscopic frame is determined according to the length, width and height information of the stereoscopic frame.

[0099] In some embodiments, the point cloud data acquisition module 201 further includes:

[0100] First, underwater data is collected through sonar, and the collected underwater point cloud data is saved in the form of point cloud; wherein the underwater point cloud data includes three-dimensional space coordinates, reflection wave intensity and ping number.

[0101] Specifically, the underwater point cloud data mainly has 5 columns, namely X-direction coordinates, Y-direction coordinates, Z-direction coordinates, reflection wave intensity I and ping number. The X-direction coordinates, Y-direction coordinates, and Z-direction coordinates constitute three-dimensional space coordinates. The reflection wave intensity is affected by the incident frequency, incident angle, plate thickness, side length, and target material type, and can be used to distinguish the target to be marked from other background objects. The ping number is similar to the frame number in the laser radar.

[0102] In some embodiments, the data overlay module 202 further includes:

[0103] First, determine the corresponding target recognition method according to the type of target to be marked. The targets to be marked are mainly divided into static targets and dynamic targets. Static targets are often obtained by moving the sonar platform for measurement, so the recognition of static targets is related to the speed of the sonar platform and the scanning area.

[0104] Since the target is stationary, for underwater static targets, it is necessary to distinguish the background environment information from the static targets in the underwater point cloud data based on the sonar platform's speed, scanning area, size of the static targets, etc., and obtain the ping sequence number of all underwater point cloud data containing static targets.

[0105] Specifically, in terms of the distribution of three-dimensional spatial coordinates and reflected wave intensity, static targets generally have a clear contrast with the background environmental information and can be distinguished by the naked eye. Therefore, the embodiment of the present application identifies whether the underwater point cloud data contains static targets based on the three-dimensional spatial coordinates and reflected wave intensity. In terms of three-dimensional spatial coordinates, static targets generally protrude from the seabed topography, showing regular spatial protrusions. In terms of reflected wave intensity, the reflected wave intensity of static targets is generally higher than the background environmental information, and is highlighted in the underwater point cloud data. Therefore, through the contrast between the three-dimensional spatial coordinates and the reflected wave intensity, underwater point cloud data containing static targets can be identified.

[0106] The data collection of dynamic targets is often more complicated than that of static targets because the targets are moving, but the recognition and distinction of the targets are simpler than those of static targets. If the target to be marked is a dynamic target, it is only necessary to replay the underwater point cloud data frame by frame. The naked eye can clearly see the change relationship between frames, observe the movement of the dynamic target in the water, identify the underwater point cloud data containing the dynamic target, and obtain the ping sequence number of all underwater point cloud data containing the dynamic target. Among them, since the underwater environmental noise and the underwater background basically do not change much, it can be seen by the naked eye that the dynamic target is moving frame by frame.

[0107] Finally, according to each ping sequence number obtained containing the target to be labeled, the corresponding underwater point cloud data is found, and these data are superimposed to generate a mosaic data, which is recorded as the target superposition data. The target superposition data is part of the data separated from the underwater point cloud data, but is stored as a separate whole, including the process from the appearance to the disappearance of the target to be labeled.

[0108] Furthermore, the target overlay data overlays underwater background environment information, underwater noise information and target information to be annotated. The overlayed data can be used as a whole to segment the target, which can greatly improve the efficiency of data annotation through batch processing of data. The mosaic data can be used to manage image data in raster form, in which the image data does not have to be adjacent or overlapped, and can exist in the form of unconnected discontinuous data sets.

[0109] In some embodiments, the object segmentation module 203 further includes:

[0110] The target overlay data containing multiple ping numbers is segmented to realize point cloud annotation of the target to be labeled, and the target to be labeled is segmented and displayed from the target overlay data.

[0111] Optionally, the embodiment of the present application uses CloudCompare open source software to perform target segmentation.

[0112] Because it is not necessary to identify whether the target is static or dynamic during the target segmentation process, it is only necessary to plot all the point clouds of the target to be labeled in the target superposition data, so the experience requirements for the labeling operator are lower than that of the existing technology. In addition, the superimposed targets to be labeled will be more obvious and easier to distinguish, which improves the accuracy of data labeling.

[0113] In addition, since the target overlay data is a separate, superimposed mosaic data, batch point cloud annotation can be performed simultaneously, thus greatly improving the data processing rate.

[0114] At the same time, combined with the prior knowledge in the data collection process, the category label corresponding to the target to be annotated is determined from the preset target category, and the category label is added frame by frame to the target superposition data after the point cloud annotation is completed, and the category label is added to the sixth column of the data in the form of a digital code to obtain the target segmentation data. Therefore, the target segmentation data includes three-dimensional space coordinates, reflection wave intensity, ping sequence number and category label.

[0115] In some embodiments, the target marking module 204 further includes:

[0116] First, according to the ping sequence number of the target segmentation data, all the superimposed target segmentation data containing multiple pings are split into single ping data to obtain multiple single ping target data. Among them, all the split single ping target data contain the point cloud of the target to be labeled and the category label corresponding to the target to be labeled.

[0117] Compared with the existing manual frame-by-frame target labeling, the target labeling method for superimposed data provided in the embodiment of the present application can identify targets by naked eyes and can also label data in large quantities, which greatly improves the efficiency and accuracy of data labeling.

[0118] Then, based on the three-dimensional spatial coordinates, a spatial matrix is ​​generated for the point cloud of the target to be labeled in the single ping target data of the same category label. Then, a corresponding cubic spatial frame, referred to as a stereo frame, is generated according to the spatial matrix. The stereo frame can represent the spatial position of the target to be labeled in the underwater environment.

[0119] Furthermore, the extreme values ​​of the three-dimensional space coordinates of the point cloud of the target to be labeled are calculated, and then the space matrix is ​​determined according to the obtained six extreme values.

[0120] Exemplarily, the three-dimensional spatial coordinates of the point cloud of the target to be labeled are (X, Y, Z), and the six extreme values ​​obtained are Xmin, Xmax, Ymin, Ymax, Zmin and Zmax, where the subscript min represents the minimum value and the subscript max represents the maximum value.

[0121] Then, according to the historical annotation information provided, the length, width and height information of the stereoscopic frame, namely Xmax-Xmin, Ymax-Ymin and Zmax-Zmin, is calculated. According to the length, width and height information, the coordinates of the center point of the stereoscopic frame can be calculated, which is used to fix the stereoscopic frame in the coordinate system to confirm the uniqueness of the stereoscopic frame. Among them, the coordinates of the center point are (Xmax-Xmin / 2, Ymax-Ymin / 2, Zmax-Zmin / 2).

[0122] Optionally, the embodiment of the present application selects the open source KITTI dataset as the historical annotation situation.

[0123] In summary, the single ping target data of the remaining ping numbers can obtain the corresponding stereoscopic frames and center points according to the above method, and mark the spatial position of the target to be marked underwater.

[0124] Furthermore, the distance between the center point and the origin of the coordinates, the direction angle of the center point, and the average reflected wave intensity of the target to be marked can be obtained as the spatial position information of the target to be marked underwater according to the length, width, and height information of the stereo frame and the coordinates of the center point.

[0125] Implementing the embodiments of the present application has the following beneficial effects:

[0126] The embodiment of the present application first identifies the target to be labeled from the underwater point cloud data based on the three-dimensional spatial coordinates and the reflected wave intensity, and superimposes the underwater point cloud data containing the target to be labeled into a separate mosaic data by frame, that is, the target superposition data, to provide data support for the subsequent target superposition data as a whole and target labeling. Then, a category label is added to the target superposition data, and the target superposition data containing the target to be labeled is treated as a whole in the data through point cloud labeling, and the target segmentation is performed on each frame of data at the same time, which greatly improves the data labeling rate. Then, based on the category label and the three-dimensional spatial coordinates, a cube of the target to be labeled is generated to complete the labeling of the underwater data and achieve accurate labeling of the underwater target.

[0127] The specific embodiments described above further describe the purpose, technical solutions and beneficial effects of the present application in detail. It should be understood that the above description is only a specific embodiment of the present application and is not intended to limit the scope of protection of the present application. It is particularly pointed out that for those skilled in the art, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of protection of the present application.

Claims

1. A sonar data labeling method for underwater three-dimensional dynamic and static targets, characterized in that: include: Converting the collected sonar data to obtain underwater point cloud data; wherein the underwater point cloud data includes three-dimensional spatial coordinates, reflection wave intensity and ping number; According to the three-dimensional space coordinates and the reflected wave intensity, the underwater point cloud data containing the target to be marked is identified and data is superimposed according to the ping sequence number to obtain target superimposed data; According to the preset target categories, batch target segmentation and category labels are performed on the target superposition data to obtain target segmentation data; Based on the category label, a corresponding stereoscopic frame is automatically generated in the target segmentation data corresponding to each ping number through the three-dimensional spatial coordinates; wherein the spatial position of the target to be marked underwater can be confirmed according to the stereoscopic frame.

2. The sonar data labeling method for underwater three-dimensional dynamic and static targets according to claim 1 is characterized in that: The underwater point cloud data containing the target to be marked is identified according to the three-dimensional space coordinates and the reflected wave intensity, and data is superimposed according to the ping sequence number to obtain target superimposed data, specifically: If the target to be marked is a static target, the underwater point cloud data is compared with the preset background environment information according to the three-dimensional space coordinates and the reflected wave intensity to obtain all ping sequence numbers of the target to be marked in the underwater point cloud data; If the target to be marked is a dynamic target, the underwater point cloud data is played back frame by frame, and all ping numbers containing the target to be marked are identified according to the inter-frame change relationship of the underwater point cloud data; The underwater point cloud data corresponding to each ping number are superimposed to obtain mosaic data, which is recorded as target superimposed data.

3. The sonar data labeling method for underwater three-dimensional dynamic and static targets according to claim 1 is characterized in that: According to the preset target category, the target superposition data is batch segmented and a category label is added to obtain the target segmentation data, specifically: According to the preset target category, confirm the category label of the target to be marked; Performing point cloud annotation of the target to be annotated on the target overlay data, and segmenting all the point cloud annotations at one time to complete target segmentation; The category label is added to the target superposition data corresponding to each ping number after the target is segmented to obtain target segmentation data; wherein the target segmentation data includes three-dimensional space coordinates, reflection wave intensity, ping number and category label.

4. The sonar data labeling method for underwater three-dimensional dynamic and static targets according to claim 1 is characterized in that: Based on the category label, the corresponding stereo frame is automatically generated in the target segmentation data corresponding to each ping number through the three-dimensional space coordinates, specifically: Split the target segmentation data according to each ping sequence number to obtain several single ping target data; Based on the three-dimensional spatial coordinates, a spatial matrix is ​​generated for the point cloud annotation of the target to be annotated in each single ping target data; According to the space matrix, a corresponding stereo frame is generated; The center point of the stereoscopic frame is determined according to the length, width and height information of the stereoscopic frame.

5. The sonar data labeling method for underwater three-dimensional dynamic and static targets according to claim 4 is characterized in that: Based on the three-dimensional space coordinates, a space matrix is ​​generated for the point cloud annotation of the target to be annotated in each single ping target data, specifically: Filter data with the same category label from all single ping target data to obtain first data; According to the point cloud annotation of the target to be annotated in the first data, the maximum and minimum values ​​of the three-dimensional space coordinates of the point cloud annotation are calculated, and the corresponding space matrix is ​​generated according to the maximum and minimum values.

6. A sonar data annotation system for underwater three-dimensional dynamic and static targets, characterized in that: include: Point cloud data acquisition module, data overlay module, target segmentation module and target annotation module; The point cloud data acquisition module is used to convert the collected sonar data to obtain underwater point cloud data; wherein the underwater point cloud data includes three-dimensional space coordinates, reflection wave intensity and ping number; The data superposition module is used to identify the underwater point cloud data containing the target to be marked according to the three-dimensional space coordinates and the reflected wave intensity and to perform data superposition according to the ping sequence number to obtain target superposition data; The target segmentation module is used to perform target segmentation and add category labels to the target superposition data according to a preset target category to obtain target segmentation data; The target labeling module is used to automatically generate a corresponding stereo frame in the target segmentation data corresponding to each ping number based on the category label and the three-dimensional spatial coordinates; wherein the spatial position of the target to be labeled underwater can be confirmed according to the stereo frame.

7. The underwater three-dimensional dynamic and static target sonar data annotation system according to claim 6, characterized in that: The data superposition module includes: a dynamic and static target display unit; Among them, the dynamic and static target display unit is used to compare the underwater point cloud data with the preset background environment information according to the three-dimensional space coordinates and the reflected wave intensity if the target to be marked is a static target, so as to obtain all ping numbers of the target to be marked in the underwater point cloud data; if the target to be marked is a dynamic target, the underwater point cloud data is played back frame by frame, and all ping numbers containing the target to be marked are identified according to the inter-frame change relationship of the underwater point cloud data; the underwater point cloud data corresponding to each ping number are superimposed to obtain a mosaic data, which is recorded as target superposition data.

8. The underwater three-dimensional dynamic and static target sonar data annotation system according to claim 6, characterized in that: The target segmentation module includes: a target segmentation unit; The target segmentation unit is used to confirm the category label of the target to be labeled according to the preset target category; perform point cloud labeling of the target to be labeled on the target overlay data, and segment all the point cloud labeling at one time to complete the target segmentation; add the category label to the target overlay data corresponding to each ping number after the target segmentation to obtain the target segmentation data; wherein the target segmentation data includes three-dimensional space coordinates, reflection wave intensity, ping number and category label.

9. The underwater three-dimensional dynamic and static target sonar data annotation system according to claim 6, characterized in that: The target labeling module includes: a stereo frame generating unit; Among them, the stereo frame generation unit is used to split the target segmentation data according to each ping sequence number to obtain a number of single ping target data; based on the three-dimensional space coordinates, a spatial matrix is ​​generated for the point cloud annotation of the target to be annotated in each single ping target data; according to the spatial matrix, a corresponding stereo frame is generated; according to the length, width and height information of the stereo frame, the center point of the stereo frame is determined.

10. The underwater three-dimensional dynamic and static target sonar data annotation system according to claim 9, characterized in that: The target marking module includes: a space matrix generating unit; Among them, the spatial matrix generation unit is used to filter out data with the same category label from all single ping target data to obtain first data; according to the point cloud annotation of the target to be labeled in the first data, calculate the maximum and minimum values ​​of the three-dimensional spatial coordinates of the point cloud annotation, and generate the corresponding spatial matrix according to the maximum and minimum values.

Images