Underwater three-dimensional reconstruction method and device for single sonar image, electronic equipment and storage medium

Abstract

The invention provides an underwater three-dimensional reconstruction method for a single sonar image. The underwater three-dimensional reconstruction method can be applied to the technical field of image sonar perception. According to the method, a point cloud structure of a target object is reconstructed by using a pseudo forward-looking depth map of a single sonar image, a sonar image segmentation network is trained by using a sonar image obtained in a simulation environment and a corresponding depth map true value, and the precision of pseudo forward-looking image estimation is improved by fully using the true value of inverse depth and the characteristics of a sonar imaging principle; and meanwhile, a weight loss function is constructed by using a sonar imaging principle, and a logarithmic loss function with an invariable scale is used for training the sonar image segmentation network. The invention also provides an underwater three-dimensional reconstruction device and equipment for the single sonar image, and a storage medium.

Description

Technical Field

[0001] The present invention relates to the field of image sonar perception technology, and more specifically to an underwater three-dimensional reconstruction method, device, electronic equipment and storage medium for a single sonar image. Background Art

[0002] Underwater 3D reconstruction technology has become an indispensable tool in many fields such as marine biology, underwater archaeology and environmental monitoring. Accurate and reliable 3D underwater models are essential for a deep understanding of the complexity of these environments. However, traditional underwater mapping techniques, such as methods that rely on optical cameras and structured light, are often limited in turbid or deep water environments. These limitations mainly stem from the attenuation and scattering of light in underwater environments, which can significantly reduce the quality and accuracy of 3D reconstructed models.

[0003] Image sonar, as an underwater acoustic sensing tool, has shown its unique advantages in 3D reconstruction in challenging underwater environments. Compared with optical cameras, sonar can transmit information over longer distances and is not limited by underwater lighting conditions. Therefore, it has become an ideal choice for environmental perception in various water conditions, including turbid and deep water environments. Sonar systems are mainly divided into two types: single-beam sonar and multi-beam sonar. Single-beam sonar only emits one sound beam at a time, with a small detection area, so the information obtained is limited and the resolution is relatively low. Multi-beam sonar can emit sound waves from multiple angles, providing more comprehensive perception information and generating more detailed images through echolocation technology. Sonar 3D reconstruction methods can be divided into geometry-based methods and deep learning-based methods, both of which aim to restore the lost elevation information of sonar images. However, geometry-based methods strongly rely on accurate model construction, while deep learning-based methods lack the combination of sonar-specific imaging principles in network construction or loss function calculation, resulting in a disconnect between 3D reconstruction and sonar sensor characteristics. Summary of the invention

[0004] In view of the above problems, the present invention provides an underwater three-dimensional reconstruction method, device, electronic device and storage medium for a single sonar image.

[0005] According to a first aspect of the present invention, there is provided an underwater 3D reconstruction method for a single sonar image, comprising:

[0006] The sonar images in the sonar image sample set are subjected to channel splitting and image size adjustment to obtain a multi-channel sonar image sample set;

[0007] A sonar image segmentation model is constructed based on the skip connection mechanism and feature fusion mechanism, and a training total loss function based on the sonar weight loss function and the scale-invariant logarithmic loss function is constructed using a predefined balance coefficient.

[0008] The sonar image segmentation model is used to process the multi-channel sonar image sample set to obtain the pseudo front-view depth map sample set, and the training total loss function is used to process the true depth value of the multi-channel sonar image sample set and the predicted depth value of the pseudo front-view depth map sample set to obtain the training total loss value;

[0009] The total training loss value is used to optimize the parameters of the sonar image segmentation model, and the parameter optimization operation is iterated until the preset conditions are met to obtain a trained sonar image segmentation model;

[0010] Perform multi-channel splitting on a single sonar image of an underwater target object to obtain a multi-channel sonar image of the underwater target object;

[0011] The trained sonar image segmentation model is used to perform image segmentation and depth prediction on the multi-channel sonar images of underwater target objects to obtain a pseudo forward-looking depth map of the underwater target objects.

[0012] Based on the pseudo front-view depth map, the point cloud of the underwater target object is reconstructed through coordinate transformation operation to obtain the three-dimensional reconstruction result of the underwater target object.

[0013] According to an embodiment of the present invention, the sonar image segmentation model constructed based on the skip connection mechanism and the feature fusion mechanism includes:

[0014] An encoder is constructed based on a plurality of encoding convolutional layers and pooling layers for extracting features from a target image, and a decoder is constructed based on a plurality of decoding convolutional layers and upsampling layers for size mapping the extracted features;

[0015] Determine a skip connection mechanism for connecting features of different levels extracted by the encoder, and determine a feature fusion mechanism for fusing output results of different levels of the encoder and decoder based on upsampling and downsampling strategies;

[0016] Based on the skip connection mechanism and feature fusion mechanism, a sonar image segmentation model including an encoder and a decoder is constructed.

[0017] According to an embodiment of the present invention, the above-mentioned construction of a training total loss function based on a sonar weight loss function and a scale-invariant logarithmic loss function using a predefined balance coefficient includes:

[0018] Based on the echo intensity model of the pixel points in the sonar image, the transmittance that represents the sonar echo reflection intensity is determined, and the sonar weight loss function is constructed using the transmittance.

[0019] A scale-invariant logarithmic loss function is constructed based on the inverse depth representation, and based on the predefined balance coefficient, the sonar weight loss function and the scale-invariant logarithmic loss function are used to obtain the total training loss function.

[0020] According to an embodiment of the present invention, the echo intensity model of the pixel points in the sonar image is as shown in formula (1):

[0021] (1),

[0022] in, Indicates the beam strength emitted by the sonar. represents the transmittance, Indicates The particle concentration in the water at the point, Indicates The distance between the point location and the sonar, represents the displacement of a point, Indicates The azimuth of the point, Indicates The elevation angle of the point in the sonar coordinate system, Indicates the echo reflection intensity of a point;

[0023] Among them, the sonar weight loss function is shown in formula (2):

[0024] (2),

[0025] in, represents the distance weight, is the inverse depth, represents the training error weight, Indicates the weight used to balance the distance weight and the training error weight;

[0026] Among them, the scale-invariant logarithmic loss function is shown in formula (3):

[0027] (3),

[0028] in, is the number of pixels, and represents the preset coefficient, .

[0029] According to an embodiment of the present invention, the above training total loss function is shown in formula (4):

[0030] (4),

[0031] in, Represents a predefined balance factor.

[0032] According to an embodiment of the present invention, the above-mentioned point cloud reconstruction of the underwater target object is performed through a coordinate transformation operation based on the pseudo front-view depth map to obtain a three-dimensional reconstruction result of the underwater target object, including:

[0033] Perform image analysis on the pseudo-forward-view depth map to obtain the azimuth, elevation, and pixel points representing the distance of the underwater target object;

[0034] Based on the predefined coordinate transformation relationship, the azimuth angle, elevation angle and pixel points representing the distance of the underwater target object are used for coordinate transformation to obtain the three-dimensional point cloud reconstruction result of the underwater target object.

[0035] According to an embodiment of the present invention, the above predefined coordinate transformation relationship is shown in formula (5):

[0036] (5),

[0037] in, represents the coordinate position of the underwater target object in the Euclidean space, represents the elevation angle of the pixel in the pseudo front view, Represents the azimuth of the pixel in the pseudo front view, Represents the value of the pixel in the pseudo front view.

[0038] According to a second aspect of the present invention, there is provided an underwater three-dimensional reconstruction device for a single sonar image, comprising:

[0039] A sample preprocessing module is used to perform channel splitting and image size adjustment on the sonar images in the sonar image sample set to obtain a multi-channel sonar image sample set;

[0040] The model and loss function construction module is used to construct a sonar image segmentation model based on a skip connection mechanism and a feature fusion mechanism, and to construct a training total loss function based on a sonar weight loss function and a scale-invariant logarithmic loss function using a predefined balance coefficient;

[0041] The model processing and loss calculation module is used to process the multi-channel sonar image sample set using the sonar image segmentation model to obtain a pseudo front-view depth map sample set, and use the training total loss function to process the true depth value of the multi-channel sonar image sample set and the predicted depth value of the pseudo front-view depth map sample set to obtain a training total loss value;

[0042] The model parameter iterative optimization module is used to optimize the parameters of the sonar image segmentation model using the total training loss value, and iteratively perform parameter optimization operations until the preset conditions are met to obtain a trained sonar image segmentation model;

[0043] A target image splitting module is used to perform multi-channel splitting on a single sonar image of an underwater target object to obtain a multi-channel sonar image of the underwater target object;

[0044] A pseudo forward-looking depth map acquisition module is used to perform image segmentation and depth prediction on multi-channel sonar images of underwater target objects using the trained sonar image segmentation model to obtain a pseudo forward-looking depth map of the underwater target objects;

[0045] The point cloud reconstruction module is used to reconstruct the point cloud of the underwater target object based on the pseudo front-view depth map through coordinate transformation operations to obtain the three-dimensional reconstruction result of the underwater target object.

[0046] A third aspect of the present invention provides an electronic device, comprising: one or more processors; a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the above method.

[0047] The fourth aspect of the present invention further provides a computer-readable storage medium on which a computer program or instruction is stored, and the steps of the above method are implemented when the above computer program or instruction is executed by a processor.

[0048] The underwater 3D reconstruction method for a single sonar image provided by the present invention trains the sonar image segmentation model by constructing a loss function for the sonar image, and uses the trained sonar image segmentation model to segment the obtained multi-channel sonar image to obtain a pseudo front view, and uses the pseudo front view to perform coordinate transformation to complete the point cloud reconstruction of the underwater target object. The above-mentioned 3D reconstruction method for underwater target objects provided by the present invention improves the depth estimation accuracy, thereby improving the point cloud reconstruction accuracy, and enhancing the underwater 3D reconstruction capability of the underwater sonar system. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The above contents and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0050] Figure 1 is a flow chart of an underwater 3D reconstruction method for a single sonar image according to an embodiment of the present invention;

[0051] Figure 2 is a diagram of a 3D reconstruction architecture of an underwater target object based on a U-Net++ network according to an embodiment of the present invention;

[0052] Figure 3 is an image sonar imaging model according to an embodiment of the present invention;

[0053] Figure 4is an elevation blur and pseudo front-view depth map according to an embodiment of the present invention;

[0054] Figure 5 is a depth estimation effect of a pseudo front view in a water tank data set according to an embodiment of the present invention;

[0055] Figure 6 is the training depth error in the water tank dataset according to an embodiment of the present invention;

[0056] Figure 7 is a point cloud constructed according to a water tank data set according to an embodiment of the present invention;

[0057] Figure 8 is a depth estimation effect of a pseudo front view in a floating object data set according to an embodiment of the present invention;

[0058] Fig. 9 is a training depth error map in a floating object dataset implemented according to the present invention;

[0059] Fig.10 is a point cloud constructed according to a floating object data set according to an embodiment of the present invention;

[0060] Fig.11 It is a physical experiment scene implemented according to the present invention;

[0061] Fig.12 is a point cloud result reconstructed according to a physical experiment of an embodiment of the present invention;

[0062] Fig.13 is a structural diagram of an underwater three-dimensional reconstruction device for a single sonar image according to an embodiment of the present invention;

[0063] Fig.14 A block diagram of an electronic device suitable for implementing an underwater three-dimensional reconstruction method for a single sonar image according to an embodiment of the present invention is schematically shown. DETAILED DESCRIPTION

[0064] Below, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present invention. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of embodiments of the present invention. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of concepts of the present invention.

[0065] The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0066] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0067] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0068] In order to solve the technical problems faced by the existing deep learning-based sonar 3D reconstruction methods, the present invention proposes a single sonar image underwater 3D reconstruction method using a pseudo forward-looking depth map. The present invention combines an image segmentation model and uses the sonar imaging principle to construct a loss function for network training, thereby optimizing the estimation accuracy of the pseudo forward-looking depth map and the point cloud reconstruction accuracy.

[0069] To achieve the above purpose, the present invention is implemented by the following technical solutions: using an image segmentation model (such as a U-Net++ network) to generate a pseudo forward-looking depth map corresponding to the sonar image, and then reconstructing the three-dimensional point cloud using the transformation relationship between the Euclidean coordinate system and the polar coordinate system. In order to solve the problem that the true depth value is the inverse depth in network training, the present invention introduces a scale-invariant logarithmic loss function (SILog). At the same time, in order to combine the imaging principle of sonar, the present invention constructs a sonar weighted loss function (SWL) in function training to improve the estimation accuracy of depth. The present invention has been widely verified in open source data sets and physical experiments, and ablation experiments have been carried out using ResNet. The results show that compared with other methods, the present invention significantly improves the three-dimensional reconstruction accuracy of image sonar and enhances the underwater three-dimensional reconstruction capability of underwater sonar systems.

[0070] Figure 1 4 is a flow chart of an underwater 3D reconstruction method for a single sonar image according to an embodiment of the present invention.

[0071] like Figure 1As shown, the above-mentioned underwater 3D reconstruction method for a single sonar image includes operations S110 to S170.

[0072] In operation S110, the sonar images in the sonar image sample set are channel-split and image size-adjusted to obtain a multi-channel sonar image sample set.

[0073] Each pixel in the sonar image represents the depth value of that point. Channel splitting allows the size to remain consistent during subsequent image processing.

[0074] In an image sonar, the width and height information are azimuth and distance respectively, and each pixel represents the intensity of the echo reflection; while in a pseudo forward-looking depth map, the width and height information are azimuth and elevation respectively, and each pixel represents the depth estimation value. Therefore, the present invention converts a W×H sonar image into a C×W×H / C multi-channel sonar image to ensure that the input image and the output depth map are of the same size, and inputs them into the U-Net++ network. The above multi-channel splitting and image resizing are applicable to both image samples and actual sonar image processing.

[0075] In operation S120, a sonar image segmentation model is constructed based on a skip connection mechanism and a feature fusion mechanism, and a training total loss function based on a sonar weight loss function and a scale-invariant logarithmic loss function is constructed using a predefined balance coefficient.

[0076] The sonar image segmentation model may be a U-Net++ network. Those skilled in the art may select other neural network models according to actual needs.

[0077] In operation S130, the multi-channel sonar image sample set is processed using the sonar image segmentation model to obtain a pseudo front-view depth map sample set, and the true depth values ​​of the multi-channel sonar image sample set and the predicted depth values ​​of the pseudo front-view depth map sample set are processed using a training total loss function to obtain a training total loss value.

[0078] In operation S140, the total training loss value is used to optimize the parameters of the sonar image segmentation model, and the parameter optimization operation is iteratively performed until a preset condition is met to obtain a trained sonar image segmentation model.

[0079] In operation S150, a single sonar image of the underwater target object is split into multiple channels to obtain a multi-channel sonar image of the underwater target object.

[0080] In operation S160, the trained sonar image segmentation model is used to perform image segmentation and depth prediction on the multi-channel sonar image of the underwater target object to obtain a pseudo front-view depth map of the underwater target object.

[0081] In operation S170, based on the pseudo front-view depth map, a point cloud reconstruction is performed on the underwater target object through a coordinate transformation operation to obtain a three-dimensional reconstruction result of the underwater target object.

[0082] The underwater 3D reconstruction method for a single sonar image provided by the present invention trains the sonar image segmentation model by constructing a loss function for the sonar image, and uses the trained sonar image segmentation model to segment the obtained multi-channel sonar image to obtain a pseudo front view, and uses the pseudo front view to perform coordinate transformation to complete the point cloud reconstruction of the underwater target object. The above-mentioned 3D reconstruction method for underwater target objects provided by the present invention improves the depth estimation accuracy, thereby improving the point cloud reconstruction accuracy, and enhancing the underwater 3D reconstruction capability of the underwater sonar system.

[0083] The following is a sonar image segmentation model based on U-net++ as a specific embodiment, and combined with the attached Figure 2 The three-dimensional reconstruction process of the underwater target object provided by the present invention is further described in detail.

[0084] Figure 2 3D reconstruction architecture diagram of underwater target objects based on U-Net++ network according to an embodiment of the present invention.

[0085] Figure 2 The training framework of a single sonar pseudo forward-looking depth map is schematically shown. Specifically, for the input sonar image, it is first converted into multiple channels, and the image size is adjusted to match the output pseudo forward-looking depth map. Subsequently, the multi-channel sonar image is input into the U-Net++ network for training. In the design of the loss function, the present invention combines the inverse depth truth and the sonar imaging principle. In terms of the inverse depth truth, a scale-invariant logarithmic loss function is adopted to balance the influence of the long-distance and short-distance depth truth values. In addition, considering that under certain water particle concentrations and scattering conditions, the reflection intensity of the sonar is exponentially inversely proportional to the distance, a sonar weight loss function is designed. Finally, the loss function of the network consists of the above two parts, thereby improving the estimation accuracy of the forward-looking depth map. After completing the depth map estimation, the present invention uses the transformation relationship between the polar coordinate system and the Euclidean coordinate system to complete the point cloud reconstruction of the single sonar image.

[0086] According to an embodiment of the present invention, the above-mentioned construction of a sonar image segmentation model based on a skip connection mechanism and a feature fusion mechanism includes: constructing an encoder based on multiple encoding convolutional layers and pooling layers for extracting features from a target image, and constructing a decoder based on multiple decoding convolutional layers and upsampling layers for size mapping the extracted features; determining a skip connection mechanism for connecting features of different levels extracted by the encoder, and determining a feature fusion mechanism for fusing output results of different levels of the encoder and decoder based on an upsampling strategy and a downsampling strategy; constructing a sonar image segmentation model including an encoder and a decoder based on the skip connection mechanism and the feature fusion mechanism.

[0087] The above embodiment is used to construct a sonar image segmentation model, and the U-Net++ architecture is adopted. Those skilled in the art may adopt other network architectures according to actual needs.

[0088] According to an embodiment of the present invention, the above-mentioned use of a predefined balance coefficient to construct a training total loss function based on a sonar weight loss function and a scale-invariant logarithmic loss function includes: determining a transmittance characterizing the sonar echo reflection intensity based on an echo intensity model of a pixel point in a sonar image, and using the transmittance to construct a sonar weight loss function; constructing a scale-invariant logarithmic loss function based on an inverse depth representation, and based on a predefined balance coefficient, using the sonar weight loss function and the scale-invariant logarithmic loss function to obtain a training total loss function.

[0089] According to an embodiment of the present invention, the echo intensity model of the pixel points in the sonar image is as shown in formula (1):

[0090] (1),

[0091] in, Indicates the beam strength emitted by the sonar. represents the transmittance, Indicates The particle concentration in the water at the point, Indicates The distance between the point location and the sonar, represents the displacement of a point, Indicates The azimuth of the point, Indicates The elevation angle of the point in the sonar coordinate system, Indicates the echo reflection intensity of a point;

[0092] Among them, the sonar weight loss function is shown in formula (2):

[0093] (2),

[0094] in, represents the distance weight, is the inverse depth, represents the training error weight, Indicates the weight used to balance the distance weight and the training error weight;

[0095] Among them, the scale-invariant logarithmic loss function is shown in formula (3):

[0096] (3),

[0097] in, is the number of pixels, and represents the preset coefficient, .

[0098] According to an embodiment of the present invention, the above training total loss function is shown in formula (4):

[0099] (4),

[0100] in, Represents a predefined balance factor.

[0101] The construction process of the above loss function provided by the present invention is further described in detail below through specific embodiments.

[0102] First, let's explain the principle of sonar imaging. The imaging sonar sensor can emit multiple sound waves in different azimuths. For each sound wave, the receiver records the intensity and echo time of the echo. A point in the sonar coordinate system can be represented by the distance , azimuth and elevation express: The echo reflection intensity at this point is: Since the sonar image does not contain elevation information, the echo intensity of the sonar pixel can be obtained using the following formula:

[0103] (1),

[0104] in, Indicates the beam strength emitted by the sonar. represents the transmittance, which is the exponential attenuation value of the sound wave after being absorbed by the particles. It indicates the particle concentration at that location, which is an inherent property of the environment and is related to the environmental structure.

[0105] Based on the imaging model, the present invention proposes a sonar weight loss function to obtain better depth estimation. Assuming that the particle density in the environment is constant and the absorption rate of the object to the sound wave is constant, the echo reflection intensity at a specific azimuth angle depends entirely on the transmittance, so the transmittance is directly determined by the distance between the object and the sonar. Given that the transmittance is the inverse of the exponential function, the longer the distance, the smaller the transmittance value. Based on this characteristic, the present invention constructs the following loss function:

[0106] (2),

[0107] Among them, the index term represents the distance weight, is the inverse of depth, i.e. the inverse of depth, and echo reflections at shorter distances have greater weights. In order to reduce the risk of training error being amplified due to lower weights of distant pixels, the present invention introduces training error weights: . Parameters related to distance Used to balance the distance weight and the training error weight, and is set to 1.0 in the present invention.

[0108] Next, we construct a scale-invariant logarithmic loss function: SILog, which is used to evaluate the performance of the depth estimation model. This function pays special attention to the performance of the model at different scales, ensuring accurate depth estimation at different distance ranges. , construct the following SILog loss function:

[0109] (3),

[0110] in, is the number of pixels, and In the present invention, they are set to 10 and 0.85 respectively.

[0111] By taking the logarithm of the depth value, SILog reduces the impact of depth range variations and maintains consistent estimation accuracy at different scales to ensure that the model performs well when dealing with objects at both close and far distances. The inverse depth representation can highlight subtle depth changes in the image, especially for distant objects. SILog can better capture these details and improve the model's depth estimation accuracy for objects at different distances.

[0112] The overall training loss can be determined by the following formula:

[0113] (4),

[0114] in, In order to balance the two loss functions, it is set to 1.0 in the present invention.

[0115] According to an embodiment of the present invention, the above-mentioned point cloud reconstruction of the underwater target object is performed through a coordinate transformation operation based on the pseudo-forward-view depth map to obtain a three-dimensional reconstruction result of the underwater target object, including: performing image analysis on the pseudo-forward-view depth map to obtain the azimuth, elevation and pixel points representing the distance of the underwater target object; based on a predefined coordinate transformation relationship, using the azimuth, elevation and pixel points representing the distance of the underwater target object to perform coordinate transformation to obtain a three-dimensional point cloud reconstruction result of the underwater target object.

[0116] According to an embodiment of the present invention, the above predefined coordinate transformation relationship is shown in formula (5):

[0117] (5).

[0118] in, represents the coordinate position of the underwater target object in the Euclidean space, represents the elevation angle of the pixel in the pseudo front view, Represents the azimuth of the pixel in the pseudo front view, Represents the value of the pixel in the pseudo front view.

[0119] The point cloud reconstruction of a single image sonar target object using a pseudo front-view depth map and a coordinate transformation formula is further described in detail below through a specific embodiment.

[0120] First, the pseudo-forward-looking depth map is explained. The pseudo-forward-looking depth map can help solve the ambiguity problem of elevation information that is usually missing in image sonar data. In the pseudo-forward-looking depth map, the horizontal axis corresponds to the azimuth angle. , the vertical axis represents the elevation angle , each pixel represents the distance Therefore, for the sonar image, it is only necessary to reconstruct its corresponding pseudo front view. After restoring the depth map, the object scanned by the sonar is converted into a three-dimensional point cloud according to the following coordinate transformation relationship:

[0121] (5).

[0122] Using pseudo front-view depth maps, the 3D reconstruction problem of sonar data can be recast as a more understandable depth estimation task.

[0123] The following is a detailed description of the embodiments and comparative experiments. Figures 3 to 13 The advantages of the above method provided by the present invention are further described in detail.

[0124] Figure 3 is an image sonar imaging model according to an embodiment of the present invention.

[0125] in, Figure 3 (a) shows the multi-beam sonar model; Figure 3 (b) represents the sonar coordinate system; Figure 3 (c) shows single-beam sonar and sonar image.

[0126] Figure 4 is an elevation blur and pseudo front-view depth map according to an embodiment of the present invention.

[0127] in, Figure 4 (a), beam When the distances of two orange points to the sonar are the same, they correspond to the same point in the sonar image, which leads to the elevation ambiguity problem. Figure 4 (b) In the pseudo front view, there is no elevation ambiguity problem, so the one-to-one correspondence of the orange object points is similar to that of the visual camera.

[0128] Figure 5 : is the depth estimation effect of the pseudo front view in the water tank dataset according to an embodiment of the present invention.

[0129] in, Figure 5 (a) Sonar image input to the network. Figure 5 (b) Pseudo front-view depth image ground truth. Figure 5 (c) Depth estimation image obtained by the comparison method A2FNET. Figure 5 (d) Depth estimation image obtained by ResNet101 using SILog. Figure 5 (e) Depth estimation image obtained by ResNet101 using SILog and SWL. Figure 5 (f) Depth estimation image obtained by U-Net++ using SILog. Figure 5 (g) Depth estimation image obtained by U-Net++ using SILog and SWL.

[0130] Figure 6 is the training depth error in the water tank dataset according to an embodiment of the present invention, Figure 6 Darker red indicates greater error.

[0131] in, Figure 6 (a) Depth error of U-Net++ using SILog. Figure 6 (b) Depth error of U-Net++ using SILog and SWL. Figure 6 (c) Depth error of ResNet101 using SILog. Figure 6 (d) Depth error of ResNet101 using SILog and SWL.

[0132] Figure 7 is a point cloud constructed according to a water tank dataset of an embodiment of the present invention.

[0133] in, Figure 7 (a) Real point cloud of the dataset. Figure 7 (b) Point cloud generated by the comparison method A2FNET. Figure 7 (c) Point cloud generated by ResNet101 using SILog. Figure 7 (d) Point cloud generated by ResNet101 using SILog and SWL. Figure 7 (e) Point cloud generated by U-Net++ using SILog. Figure 7 (f) Point cloud generated by U-Net++ using SILog and SWL.

[0134] Figure 8 This is a depth estimation effect of a pseudo front view in a floating object data set according to an embodiment of the present invention.

[0135] in, Figure 8 (a) Sonar image input to the network. Figure 8 (b) Pseudo front-view depth image ground truth. Figure 8 (c) Depth estimation image obtained by comparing with the method A2FNET. Figure 8 (d) Depth estimation image obtained by ResNet101 using SILog. Figure 8 (e) Depth estimation image obtained by ResNet101 using SILog and SWL. Figure 8 (f) Depth estimation image obtained by U-Net++ using SILog. Figure 8 (g) Depth estimation image obtained by U-Net++ using SILog and SWL.

[0136] Fig. 9 is a training depth error map in a floating object dataset according to the present invention, wherein: Fig. 9 Darker red indicates greater error.

[0137] in, Fig. 9 (a) Depth error of ResNet101 using SILog and SWL. Fig. 9 (b) Depth error of U-Net++ using SILog and SWL.

[0138] Fig.10 It is a point cloud constructed according to the floating object data set of an embodiment of the present invention.

[0139] in, Fig.10 (a) Real point cloud of the dataset. Fig.10 (b) Point cloud generated by the comparison method A2FNET. Fig.10 (c) Point cloud generated by ResNet101 using SILog. Fig.10 (d) Point cloud generated by ResNet101 using SILog and SWL. Fig.10 (e) Point cloud generated by U-Net++ using SILog. Fig.10 (f) Point cloud generated by U-Net++ using SILog and SWL.

[0140] Fig.11 This is a physical experiment scenario implemented according to the present invention.

[0141] in, Fig.11 (a) Multibeam imaging sonar Oculus M750d. Fig.11 (a) Target object reconstructed by sonar.

[0142] Fig.12 is a point cloud result reconstructed by a physical experiment according to an embodiment of the present invention.

[0143] in, Fig.12 (a) Sonar image of the target object. Fig.12 (b) Point cloud reconstructed by U-Net++ using SILog and SWL. Fig.12 (c) Point cloud reconstructed by ResNet using SILog and SWL as a comparative experiment.

[0144] Fig.13 The figure is a structural diagram of an underwater three-dimensional reconstruction device for a single sonar image according to an embodiment of the present invention.

[0145] like Fig.13 As shown, the underwater three-dimensional reconstruction device 1300 for a single sonar image includes a sample preprocessing module 1310, a model and loss function construction module 1320, a model processing and loss calculation module 1330, a model parameter iterative optimization module 1340, a target image segmentation module 1350, a pseudo-forward-view depth map acquisition module 1360 and a point cloud reconstruction module 1370.

[0146] The sample preprocessing module 1310 is used to perform channel splitting and image size adjustment on the sonar images in the sonar image sample set to obtain a multi-channel sonar image sample set.

[0147] The model and loss function construction module 1320 is used to construct a sonar image segmentation model based on a skip connection mechanism and a feature fusion mechanism, and to construct a training total loss function based on a sonar weight loss function and a scale-invariant logarithmic loss function using a predefined balance coefficient.

[0148] The model processing and loss calculation module 1330 is used to process the multi-channel sonar image sample set using the sonar image segmentation model to obtain a pseudo front-view depth map sample set, and use the training total loss function to process the true depth value of the multi-channel sonar image sample set and the predicted depth value of the pseudo front-view depth map sample set to obtain a training total loss value.

[0149] The model parameter iterative optimization module 1340 is used to optimize the parameters of the sonar image segmentation model using the total training loss value, and iteratively perform parameter optimization operations until a preset condition is met to obtain a trained sonar image segmentation model.

[0150] The target image splitting module 1350 is used to perform multi-channel splitting on a single sonar image of an underwater target object to obtain a multi-channel sonar image of the underwater target object.

[0151] The pseudo front-view depth map acquisition module 1360 is used to perform image segmentation and depth prediction on the multi-channel sonar image of the underwater target object using the trained sonar image segmentation model to obtain a pseudo front-view depth map of the underwater target object.

[0152] The point cloud reconstruction module 1370 is used to reconstruct the point cloud of the underwater target object through coordinate transformation operation based on the pseudo front-view depth map to obtain a three-dimensional reconstruction result of the underwater target object.

[0153] According to an embodiment of the present invention, any multiple modules among the sample preprocessing module 1310, the model and loss function construction module 1320, the model processing and loss calculation module 1330, the model parameter iterative optimization module 1340, the target image segmentation module 1350, the pseudo front view depth map acquisition module 1360 and the point cloud reconstruction module 1370 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present invention, at least one of the sample preprocessing module 1310, the model and loss function construction module 1320, the model processing and loss calculation module 1330, the model parameter iterative optimization module 1340, the target image segmentation module 1350, the pseudo front-view depth map acquisition module 1360 and the point cloud reconstruction module 1370 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or can be implemented in any one of the three implementation methods of software, hardware and firmware or in an appropriate combination of any of them. Alternatively, at least one of the sample preprocessing module 1310, the model and loss function construction module 1320, the model processing and loss calculation module 1330, the model parameter iterative optimization module 1340, the target image segmentation module 1350, the pseudo front-view depth map acquisition module 1360 and the point cloud reconstruction module 1370 can be at least partially implemented as a computer program module, which can perform the corresponding function when the computer program module is executed.

[0154] Fig.14 A block diagram of an electronic device suitable for implementing an underwater three-dimensional reconstruction method for a single sonar image according to an embodiment of the present invention is schematically shown.

[0155] like Fig.14 As shown, the electronic device 1400 according to an embodiment of the present invention includes a processor 1401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1402 or a program loaded from a storage part 1408 to a random access memory (RAM) 1403. The processor 1401 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 1401 may also include an onboard memory for caching purposes. The processor 1401 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0156] In RAM 1403, various programs and data required for the operation of electronic device 1400 are stored. Processor 1401, ROM 1402 and RAM 1403 are connected to each other via bus 1404. Processor 1401 performs various operations of the method flow according to the embodiment of the present invention by executing the program in ROM 1402 and / or RAM 1403. It should be noted that the program can also be stored in one or more memories other than ROM 1402 and RAM 1403. Processor 1401 can also perform various operations of the method flow according to the embodiment of the present invention by executing the program stored in the one or more memories.

[0157] According to an embodiment of the present invention, the electronic device 1400 may further include an input / output (I / O) interface 1405, which is also connected to the bus 1404. The electronic device 1400 may further include one or more of the following components connected to the input / output (I / O) interface 1405: an input portion 1406 including a keyboard, a mouse, etc.; an output portion 1407 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 1408 including a hard disk, etc.; and a communication portion 1409 including a network interface card such as a LAN card, a modem, etc. The communication portion 1409 performs communication processing via a network such as the Internet. A drive 1410 is also connected to the input / output (I / O) interface 1405 as needed. A removable medium 1411, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1410 as needed, so that a computer program read therefrom is installed into the storage portion 1408 as needed.

[0158] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiment; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present invention is implemented.

[0159] According to an embodiment of the present invention, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, it may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present invention, the computer-readable storage medium may include the ROM 1402 and / or RAM 1403 described above and / or one or more memories other than ROM 1402 and RAM 1403.

[0160] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0161] It will be appreciated by those skilled in the art that the features described in the various embodiments of the present invention may be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in the present invention. In particular, without departing from the spirit and teachings of the present invention, the features described in the various embodiments of the present invention may be combined and / or combined in various ways. All of these combinations and / or combinations fall within the scope of the present invention.

[0162] The embodiments of the present invention are described above. However, these embodiments are only for the purpose of illustration, and are not intended to limit the scope of the present invention. Although each embodiment is described above, it does not mean that the measures in each embodiment cannot be used in combination advantageously. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.

Claims

1. An underwater 3D reconstruction method for a single sonar image, characterized in that: The method comprises: The sonar images in the sonar image sample set are subjected to channel splitting and image size adjustment to obtain a multi-channel sonar image sample set; A sonar image segmentation model is constructed based on the skip connection mechanism and feature fusion mechanism, and a training total loss function based on the sonar weight loss function and the scale-invariant logarithmic loss function is constructed using a predefined balance coefficient. Processing the multi-channel sonar image sample set using the sonar image segmentation model to obtain a pseudo front-view depth map sample set, and processing the true depth value of the multi-channel sonar image sample set and the predicted depth value of the pseudo front-view depth map sample set using the training total loss function to obtain a training total loss value; Optimizing the parameters of the sonar image segmentation model using the total training loss value, and iterating the parameter optimization operation until a pre-set condition is met to obtain a trained sonar image segmentation model; Performing multi-channel splitting on a single sonar image of an underwater target object to obtain a multi-channel sonar image of the underwater target object; Using the trained sonar image segmentation model, performing image segmentation and depth prediction on the multi-channel sonar image of the underwater target object to obtain a pseudo forward-looking depth map of the underwater target object; Based on the pseudo front-view depth map, a point cloud reconstruction is performed on the underwater target object through a coordinate transformation operation to obtain a three-dimensional reconstruction result of the underwater target object.

2. The method according to claim 1, characterized in that The sonar image segmentation model based on the skip connection mechanism and feature fusion mechanism includes: An encoder is constructed based on a plurality of encoding convolutional layers and pooling layers for extracting features from a target image, and a decoder is constructed based on a plurality of decoding convolutional layers and upsampling layers for size mapping the extracted features; Determine a skip connection mechanism for connecting features of different levels extracted by the encoder, and determine a feature fusion mechanism for fusing output results of different levels of the encoder and the decoder based on an upsampling strategy and a downsampling strategy; Based on the skip connection mechanism and the feature fusion mechanism, a sonar image segmentation model including the encoder and the decoder is constructed.

3. The method according to claim 1, characterized in that Using the predefined balance coefficient, the total training loss function based on the sonar weight loss function and the scale-invariant logarithmic loss function is constructed as follows: Determine the transmittance representing the sonar echo reflection intensity based on the echo intensity model of the pixel points in the sonar image, and construct the sonar weight loss function using the transmittance; A scale-invariant logarithmic loss function is constructed based on the inverse depth representation, and based on a predefined balance coefficient, a total training loss function is obtained by using the constructed sonar weight loss function and the scale-invariant logarithmic loss function.

4. The method according to claim 3, characterized in that The echo intensity model of the pixel points in the sonar image is shown in formula (1): （1）， in, Indicates the beam strength emitted by the sonar. represents the transmittance, Indicates The particle concentration in the water at the point, Indicates The distance between the point location and the sonar, represents the displacement of a point, Indicates The azimuth of the point, Indicates The elevation angle of the point in the coordinate system of the sonar, Indicates the echo reflection intensity of a point; Among them, the sonar weight loss function is shown in formula (2): （2）， in, represents the distance weight, is the inverse depth, represents the training error weight, Represents a weight used to balance the distance weight and the training error weight; The scale-invariant logarithmic loss function is shown in formula (3): （3）， in, is the number of pixels, and represents the preset coefficient, .

5. The method according to claim 3, characterized in that: The total training loss function is shown in formula (4): （4）， in, represents the predefined balance coefficient.

6. The method according to claim 1, characterized in that Based on the pseudo front-view depth map, point cloud reconstruction is performed on the underwater target object through a coordinate transformation operation to obtain a three-dimensional reconstruction result of the underwater target object, including: Performing image analysis on the pseudo forward-looking depth map to obtain the azimuth angle, elevation angle, and pixel points representing the distance of the underwater target object; Based on a predefined coordinate transformation relationship, coordinate transformation is performed using the azimuth angle, elevation angle and pixel points representing the distance of the underwater target object to obtain a three-dimensional point cloud reconstruction result of the underwater target object.

7. The method according to claim 6, characterized in that The predefined coordinate transformation relationship is shown in formula (5): （5）， in, represents the coordinate position of the underwater target object in the Euclidean space, represents the elevation angle of the pixel point in the pseudo front view, represents the azimuth of the pixel point in the pseudo front view, Indicates the value of the pixel in the pseudo front view.

8. An underwater 3D reconstruction device for a single sonar image, characterized in that: The device comprises: A sample preprocessing module is used to perform channel splitting and image size adjustment on the sonar images in the sonar image sample set to obtain a multi-channel sonar image sample set; The model and loss function construction module is used to construct a sonar image segmentation model based on a skip connection mechanism and a feature fusion mechanism, and to construct a training total loss function based on a sonar weight loss function and a scale-invariant logarithmic loss function using a predefined balance coefficient; A model processing and loss calculation module, used to process the multi-channel sonar image sample set using the sonar image segmentation model to obtain a pseudo front-view depth map sample set, and use the training total loss function to process the true depth value of the multi-channel sonar image sample set and the predicted depth value of the pseudo front-view depth map sample set to obtain a training total loss value; A model parameter iterative optimization module, used to optimize the parameters of the sonar image segmentation model using the total training loss value, and iteratively perform parameter optimization operations until a pre-set condition is met to obtain a trained sonar image segmentation model; A target image splitting module is used to perform multi-channel splitting on a single sonar image of an underwater target object to obtain a multi-channel sonar image of the underwater target object; A pseudo forward-looking depth map acquisition module is used to perform image segmentation and depth prediction on the multi-channel sonar image of the underwater target object using the trained sonar image segmentation model to obtain a pseudo forward-looking depth map of the underwater target object; The point cloud reconstruction module is used to perform point cloud reconstruction on the underwater target object through coordinate transformation operation based on the pseudo front-view depth map to obtain a three-dimensional reconstruction result of the underwater target object.

9. An electronic device, comprising: one or more processors; a memory for storing one or more computer programs, It is characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program or instruction stored thereon, characterized in that: When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images