Underwater three-dimensional reconstruction method and device for single sonar image, electronic equipment and storage medium

Abstract

The application provides an underwater three-dimensional reconstruction method for a single sonar image, which can be applied to the technical field of image sonar perception. The method reconstructs the point cloud structure of a target object by using a pseudo front-view depth map of a single sonar image, trains a sonar image segmentation network by using a sonar image obtained in a simulation environment and a corresponding depth map true value, fully utilizes the true value of the inverse depth and the characteristics of a sonar imaging principle to improve the estimation accuracy of the pseudo front-view depth map, simultaneously uses a weight loss function constructed according to the sonar imaging principle, and simultaneously uses a scale-invariant logarithmic loss function to train the sonar image segmentation network. The application further provides an underwater three-dimensional reconstruction device for a single sonar image, equipment and a storage medium.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image sonar perception, and more particularly to an underwater three-dimensional reconstruction method for a single sonar image, an apparatus, an electronic device and a storage medium. BACKGROUND

[0002] Underwater three-dimensional reconstruction technology has become an indispensable tool in many fields such as marine biology, underwater archaeology and environmental monitoring. Accurate and reliable three-dimensional underwater models are crucial for a deep understanding of the complexity of these environments. However, traditional underwater mapping techniques, such as methods relying on optical cameras and structured light, are often limited in turbid or deep water environments. These limitations are mainly due to the attenuation and scattering of light in underwater environments, which significantly reduces the quality and accuracy of three-dimensional reconstruction models.

[0003] Image sonar, as an underwater acoustic sensing tool, has shown its unique advantages in three-dimensional reconstruction in challenging underwater environments. Compared with optical cameras, sonar can transmit information over a longer distance and is not limited by underwater lighting conditions. Therefore, it has become an ideal choice for environmental perception in various water quality conditions, including turbid and deep water environments. Sonar systems are mainly divided into single-beam sonar and multi-beam sonar. Single-beam sonar only transmits one sound beam at a time, with a smaller detection area, so the information obtained is limited and the resolution is relatively low. Multi-beam sonar, on the other hand, can transmit sound waves from multiple angles, providing more comprehensive perception information and generating more detailed images through echo positioning technology. Sonar three-dimensional reconstruction methods can be divided into geometry-based methods and deep learning-based methods, both of which aim to restore the missing elevation information of sonar images. However, geometry-based methods strongly depend on accurate model construction, while deep learning-based methods lack the combination of sonar-specific imaging principles in network construction or loss function calculation, resulting in a disconnection between three-dimensional reconstruction and sonar sensor characteristics. SUMMARY

[0004] In view of the above problems, the present application provides an underwater three-dimensional reconstruction method for a single sonar image, an apparatus, an electronic device and a storage medium.

[0005] According to a first aspect of the present application, there is provided an underwater three-dimensional reconstruction method for a single sonar image, comprising:

[0006] Channel splitting and image size adjustment of the sonar images in the sonar image sample set to obtain a multi-channel sonar image sample set;

[0007] Constructing a sonar image segmentation model based on a skip connection mechanism and a feature fusion mechanism, and constructing a training total loss function based on a sonar weight loss function and a scale-invariant logarithmic loss function using a predefined balance coefficient;

[0008] The multi-channel sonar image sample set is processed by using the sonar image segmentation model to obtain a pseudo front-view depth map sample set, and the depth true value of the multi-channel sonar image sample set and the predicted depth value of the pseudo front-view depth map sample set are processed by using a training total loss function to obtain a training total loss value;

[0009] The sonar image segmentation model is parameter-optimized by using the training total loss value, and the parameter-optimization operation is iteratively performed until a preset condition is met to obtain a trained sonar image segmentation model;

[0010] A single sonar image of an underwater target object is multi-channel split to obtain a multi-channel sonar image of the underwater target object;

[0011] The multi-channel sonar image of the underwater target object is image segmented and depth predicted by using the trained sonar image segmentation model to obtain a pseudo front-view depth map of the underwater target object;

[0012] Based on the pseudo front-view depth map, the underwater target object is point cloud reconstructed through a coordinate transformation operation to obtain a three-dimensional reconstruction result of the underwater target object.

[0013] According to the embodiments of the present application, the sonar image segmentation model is constructed based on the skip connection mechanism and the feature fusion mechanism, and includes:

[0014] An encoder is constructed based on a plurality of encoding convolutional layers and pooling layers for feature extraction of a target image, and a decoder is constructed based on a plurality of decoding convolutional layers and up-sampling layers for size mapping of the extracted features;

[0015] A skip connection mechanism for connecting different level features extracted by the encoder is determined, and a feature fusion mechanism for fusing different level output results of the encoder and the decoder is determined based on up-sampling strategies and down-sampling strategies;

[0016] Based on the skip connection mechanism and the feature fusion mechanism, the sonar image segmentation model including the encoder and the decoder is constructed.

[0017] According to the embodiments of the present application, the training total loss function based on the sonar weight loss function and the scale-invariant logarithmic loss function is constructed by using the predefined balance coefficient, and includes:

[0018] The transmittance representing the sonar echo reflection intensity is determined based on an echo intensity model of a pixel point in the sonar image, and the sonar weight loss function is constructed by using the transmittance;

[0019] The scale-invariant logarithmic loss function is constructed based on an inverse depth representation method, and the training total loss function is obtained by using the sonar weight loss function and the scale-invariant logarithmic loss function based on the predefined balance coefficient.

[0020] According to the embodiment of the present application, the echo intensity model of the pixel point in the sonar image is shown as formula (1):

[0021] (1),

[0022] wherein, represents the beam intensity of the sonar emission, represents the transmittance, represents the particle concentration of the water body at the position of the point, represents the distance between the position of the point and the sonar, represents the displacement of the point, represents the azimuth angle of the point, represents the elevation angle of the point in the coordinate system of the sonar, represents the echo reflection intensity of the point. wherein, the sonar weight loss function is shown as formula (2): (2),

[0023] wherein,

[0024] represents the distance weight, is the inverse depth, represents the training error weight,

[0025] represents the coefficient for balancing the distance weight and the training error weight. wherein, the scale-invariant logarithmic loss function is shown as formula (3): (3),

[0026] wherein,

[0027] is the number of pixel points, and represent the preset coefficient.

[0028] According to the embodiment of the present application, the total training loss function is shown as formula (4): (4),

[0029] wherein, represents the predefined balancing coefficient.

[0030]

[0031]

[0032] ​​​​​According to an embodiment of the present invention, the above-mentioned point cloud reconstruction of an underwater target object based on a pseudo-forward depth map through coordinate transformation operations to obtain a three-dimensional reconstruction result of the underwater target object includes:

[0033] Image analysis is performed on the pseudo-forward depth map to obtain the azimuth, elevation, and distance pixels of the underwater target object;

[0034] Based on predefined coordinate transformation relationships, coordinate transformation is performed using the azimuth, elevation, and distance-representing pixels of the underwater target object to obtain the 3D point cloud reconstruction result of the underwater target object.

[0035] According to an embodiment of the present invention, the predefined coordinate transformation relationship is as shown in formula (5):

[0036] (5),

[0037] in, This represents the coordinate position of an underwater target object in Euclidean space. This represents the elevation angle of a pixel in the pseudo-forward depth map. This represents the azimuth angle of a pixel in the pseudo-forward depth map. This represents the value of a pixel in the pseudo-forward depth map.

[0038] According to a second aspect of the present invention, an underwater three-dimensional reconstruction apparatus for a single sonar image is provided, comprising:

[0039] The sample preprocessing module is used to split the sonar images in the sonar image sample set into channels and adjust the image size to obtain a multi-channel sonar image sample set.

[0040] The model and loss function construction module is used to build a sonar image segmentation model based on the skip connection mechanism and feature fusion mechanism, and to construct the total training loss function based on the sonar weight loss function and the scale-invariant log loss function using predefined balance coefficients.

[0041] The model processing and loss calculation module is used to process the multi-channel sonar image sample set using the sonar image segmentation model to obtain the pseudo-forward depth map sample set, and to process the ground truth depth value of the multi-channel sonar image sample set and the predicted depth value of the pseudo-forward depth map sample set using the total training loss function to obtain the total training loss value.

[0042] The model parameter iterative optimization module is used to optimize the parameters of the sonar image segmentation model using the total training loss value. The parameter optimization operation is performed iteratively until the preset conditions are met to obtain the trained sonar image segmentation model.

[0043] The target image segmentation module is used to perform multi-channel segmentation on a single sonar image of an underwater target object to obtain a multi-channel sonar image of the underwater target object.

[0044] The pseudo-forward depth map acquisition module is used to perform image segmentation and depth prediction on multi-channel sonar images of underwater targets using a trained sonar image segmentation model, and to obtain a pseudo-forward depth map of the underwater target.

[0045] The point cloud reconstruction module is used to reconstruct the point cloud of underwater target objects based on pseudo-forward depth maps through coordinate transformation operations, so as to obtain the three-dimensional reconstruction results of the underwater target objects.

[0046] A third aspect of the present invention provides an electronic device comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method described above.

[0047] A fourth aspect of the present invention also provides a computer-readable storage medium having a computer program or instructions stored thereon, wherein the computer program or instructions, when executed by a processor, implement the steps of the above-described method.

[0048] The underwater 3D reconstruction method for single sonar images provided by this invention trains a sonar image segmentation model by constructing a loss function specific to sonar images. The trained sonar image segmentation model is then used to segment the obtained multi-channel sonar image, generating a pseudo-forward depth map. This pseudo-forward depth map is then used for coordinate transformation to reconstruct the point cloud of the underwater target object. This 3D reconstruction method for underwater targets improves depth estimation accuracy, thereby improving point cloud reconstruction accuracy and enhancing the underwater 3D reconstruction capability of the underwater sonar system. Attached Figure Description

[0049] The above-described features, other objects, and advantages of the present invention will become clearer from the following description of embodiments of the invention with reference to the accompanying drawings, in which:

[0050] Figure 1 This is a flowchart of an underwater 3D reconstruction method for a single sonar image according to an embodiment of the present invention;

[0051] Figure 2 This is a diagram of a three-dimensional reconstruction architecture for an underwater target object based on a U-Net++ network, according to an embodiment of the present invention.

[0052] Figure 3 This is an image sonar imaging model according to an embodiment of the present invention;

[0053] Figure 4These are elevation angle blurring and pseudo-forward depth maps according to embodiments of the present invention;

[0054] Figure 5 This is the depth estimation effect of a pseudo-front view in a water tank dataset according to an embodiment of the present invention;

[0055] Figure 6 The depth error is trained on the water tank dataset according to an embodiment of the present invention.

[0056] Figure 7 The point cloud is constructed from the water tank dataset according to an embodiment of the present invention;

[0057] Figure 8 This is the depth estimation effect of a pseudo front view in a floating object dataset according to an embodiment of the present invention;

[0058] Figure 9 This is a training depth error map on a floating object dataset according to the present invention;

[0059] Figure 10 The point cloud is constructed from a floating object dataset according to an embodiment of the present invention;

[0060] Figure 11 This is a physical experimental scenario implemented according to the present invention;

[0061] Figure 12 The point cloud results are reconstructed from physical experiments according to embodiments of the present invention;

[0062] Figure 13 This is a structural diagram of an underwater three-dimensional reconstruction device for a single sonar image according to an embodiment of the present invention;

[0063] Figure 14 A block diagram of an electronic device suitable for implementing an underwater three-dimensional reconstruction method for a single sonar image, according to an embodiment of the present invention, is shown schematically. Detailed Implementation

[0064] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the invention. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the invention for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concept of the invention.

[0065] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0066] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0067] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0068] To address the technical challenges of existing deep learning-based sonar 3D reconstruction methods, this invention proposes a single-sonar image underwater 3D reconstruction method utilizing a pseudo-forward depth map. This invention combines an image segmentation model with a loss function built using sonar imaging principles to optimize the estimation accuracy of the pseudo-forward depth map and the point cloud reconstruction accuracy.

[0069] To achieve the above objectives, this invention is implemented through the following technical solution: A pseudo-forward depth map corresponding to the sonar image is generated using an image segmentation model (e.g., U-Net++ network), and then the 3D point cloud is reconstructed using the transformation relationship between Euclidean and polar coordinate systems. To address the issue that the ground truth depth is inverse depth during network training, this invention introduces a scale-invariant logarithmic loss function (SILog). Simultaneously, to align with the imaging principles of sonar, this invention constructs a sonar weighted loss function (SWL) during function training to improve the depth estimation accuracy. This invention has undergone extensive validation on open-source datasets and in physical experiments, and ablation experiments were conducted using ResNet. The results show that, compared with other methods, this invention significantly improves the 3D reconstruction accuracy of image sonar and enhances the underwater 3D reconstruction capability of underwater sonar systems.

[0070] Figure 1 This is a flowchart of an underwater 3D reconstruction method for a single sonar image according to an embodiment of the present invention.

[0071] like Figure 1As shown, the above-mentioned underwater 3D reconstruction method for a single sonar image includes operations S110 to S170.

[0072] In operation S110, the sonar images in the sonar image sample set are split into channels and their image sizes are adjusted to obtain a multi-channel sonar image sample set.

[0073] In a sonar image, each pixel represents the depth value of that point. Channel splitting is performed to ensure that the size remains consistent during subsequent image processing.

[0074] In a sonar image, the width and height information represent the azimuth and distance, respectively, with each pixel representing the intensity of the echo reflection. In a pseudo-forward-looking depth map, the width and height information represent the azimuth and elevation, respectively, with each pixel representing a depth estimate. Therefore, this invention converts a W×H sonar image into a C×W×H / C multi-channel sonar image to ensure that the input image and output depth map are the same size before inputting them into the U-Net++ network. This multi-channel splitting and image resizing method is applicable to both image samples and actual sonar image processing.

[0075] In operation S120, a sonar image segmentation model is constructed based on the skip connection mechanism and the feature fusion mechanism, and a total training loss function is constructed based on the sonar weight loss function and the scale-invariant log loss function using predefined balancing coefficients.

[0076] The sonar image segmentation model described above can optionally be a U-Net++ network. Those skilled in the art can choose other neural network models according to actual needs.

[0077] In operation S130, the sonar image segmentation model is used to process the multi-channel sonar image sample set to obtain the pseudo-forward depth map sample set. The total training loss function is then used to process the ground truth depth of the multi-channel sonar image sample set and the predicted depth value of the pseudo-forward depth map sample set to obtain the total training loss value.

[0078] In operation S140, the parameters of the sonar image segmentation model are optimized using the total training loss value. The parameter optimization operation is performed iteratively until the preset conditions are met to obtain the trained sonar image segmentation model.

[0079] In operation S150, a single sonar image of an underwater target object is split into multiple channels to obtain a multi-channel sonar image of the underwater target object.

[0080] When operating the S160, the trained sonar image segmentation model is used to perform image segmentation and depth prediction on the multi-channel sonar images of underwater targets, resulting in a pseudo-forward depth map of the underwater targets.

[0081] In operation S170, based on the pseudo-forward depth map, point cloud reconstruction of underwater target objects is performed through coordinate transformation operations to obtain the three-dimensional reconstruction results of underwater target objects.

[0082] The underwater 3D reconstruction method for single sonar images provided by this invention trains a sonar image segmentation model by constructing a loss function specific to sonar images. The trained sonar image segmentation model is then used to segment the obtained multi-channel sonar image, generating a pseudo-forward depth map. This pseudo-forward depth map is then used for coordinate transformation to reconstruct the point cloud of the underwater target object. This 3D reconstruction method for underwater targets improves depth estimation accuracy, thereby improving point cloud reconstruction accuracy and enhancing the underwater 3D reconstruction capability of the underwater sonar system.

[0083] The following uses a sonar image segmentation model based on U-net++ as a specific example, combined with the attached... Figure 2 The three-dimensional reconstruction process of the underwater target object provided by the present invention will be described in further detail.

[0084] Figure 2 This is a diagram of a three-dimensional reconstruction architecture for an underwater target object based on a U-Net++ network according to an embodiment of the present invention.

[0085] Figure 2 The training framework for a single-sonar pseudo-forward-looking depth map is illustrated schematically. Specifically, for the input sonar image, it is first converted into multiple channels, and the image size is adjusted to match the output pseudo-forward-looking depth map. Subsequently, the multi-channel sonar image is input into the U-Net++ network for training. In the design of the loss function, this invention combines inverse depth ground truth and sonar imaging principles. For inverse depth ground truth, a scale-invariant logarithmic loss function is used to balance the influence of far-range and near-range depth ground truth. In addition, considering that under certain water particle concentration and scattering conditions, the sonar reflection intensity is exponentially inversely proportional to the distance, a sonar weight loss function is designed. Finally, the network's loss function consists of the above two parts, thereby improving the estimation accuracy of the forward-looking depth map. After completing the depth map estimation, this invention utilizes the transformation relationship between polar coordinates and Euclidean coordinates to complete the point cloud reconstruction of the single sonar image.

[0086] According to an embodiment of the present invention, the sonar image segmentation model constructed based on the skip connection mechanism and the feature fusion mechanism includes: constructing an encoder based on multiple coding convolutional layers and pooling layers for feature extraction of the target image, and constructing a decoder based on multiple decoding convolutional layers and upsampling layers for size mapping of the extracted features; determining a skip connection mechanism for connecting different levels of features extracted by the encoder, and determining a feature fusion mechanism for fusing the output results of different levels of the encoder and decoder based on upsampling and downsampling strategies; and constructing a sonar image segmentation model including an encoder and a decoder based on the skip connection mechanism and the feature fusion mechanism.

[0087] The above embodiments are used to construct a sonar image segmentation model, which adopts the U-Net++ architecture. Those skilled in the art can use other network architectures according to actual needs.

[0088] According to an embodiment of the present invention, the above-described method of constructing a total training loss function based on a sonar weight loss function and a scale-invariant logarithmic loss function using predefined balance coefficients includes: determining the transmittance characterizing the sonar echo reflection intensity based on the echo intensity model of pixels in the sonar image, and constructing a sonar weight loss function using the transmittance; constructing a scale-invariant logarithmic loss function based on the inverse depth representation, and obtaining the total training loss function based on the predefined balance coefficients using the constructed sonar weight loss function and the scale-invariant logarithmic loss function.

[0089] According to an embodiment of the present invention, the echo intensity model of the pixel in the above sonar image is as shown in formula (1):

[0090] (1),

[0091] in, This indicates the intensity of the beam emitted by the sonar. Indicates transmittance. Indicates the first The concentration of water particles at the location of the point. Indicates the first The distance between the location of the point and the sonar. This represents the displacement of a point. Indicates the first The azimuth of the point, Indicates the first The elevation angle of the point in the coordinate system of the sonar. Indicates the echo reflection intensity at a point;

[0092] The sonar weight loss function is shown in formula (2):

[0093] (2),

[0094] in, This represents the distance weight. For inverse depth, Represents the training error weights. This indicates the weights used to balance distance weights and training error weights;

[0095] The scale-invariant logarithmic loss function is shown in equation (3):

[0096] (3),

[0097] in, It is the number of pixels. and This represents the preset coefficient. ;

[0098] According to an embodiment of the present invention, the total training loss function is as shown in formula (4):

[0099] (4),

[0100] in, This represents a predefined balance coefficient.

[0101] The construction process of the loss function provided by the present invention will be further described in detail below through specific embodiments.

[0102] First, let's explain the principle of sonar imaging. An image sonar sensor can emit multiple sound waves at different azimuth angles. For each sound wave, the receiver records the intensity and duration of the echo reflection. A point in the sonar coordinate system can be represented by its distance. azimuth and elevation angle express: The echo reflection intensity at this point is: Since sonar images do not contain elevation information, the echo intensity of a sonar pixel can be obtained using the following formula:

[0103] (1),

[0104] in, This indicates the intensity of the beam emitted by the sonar. Transmittance represents the exponential attenuation of sound waves after they are absorbed by particles. This indicates the particle concentration at that location, which is an inherent property of the environment and is related to the environmental structure.

[0105] Based on this imaging model, this invention proposes a sonar weighted loss function to obtain better depth estimation. Assuming a constant particle density in the environment and a constant absorption rate of sound waves by objects, the echo reflection intensity at a specific azimuth angle depends entirely on the transmittance. Therefore, the transmittance is directly determined by the distance between the object and the sonar. Given that transmittance is the reciprocal of an exponential function, the transmittance decreases with increasing distance. Based on this characteristic, this invention constructs the following loss function:

[0106] (2),

[0107] Among them, the exponent term This represents the distance weight. As the inverse of depth, i.e., the reciprocal of depth, echo reflections over shorter distances have a greater weight. To reduce the risk of training error being amplified due to the lower weight of distant pixels, this invention introduces training error weights: Parameters related to distance The distance weight and training error weight are used to balance the distance weight and training error weight, and are set to 1.0 in this invention.

[0108] Next, a scale-invariant log loss function, SILog, is constructed to evaluate the performance of the depth estimation model. This function specifically focuses on the model's performance at different scales, ensuring accurate depth estimation across various distance ranges. Construct the following SILog loss function:

[0109] (3),

[0110] in, It is the number of pixels. and In this invention, the values ​​are set to 10 and 0.85, respectively.

[0111] By taking the logarithm of the depth values, SILog reduces the impact of depth range variations, maintaining consistent estimation accuracy across different scales to ensure the model performs well when dealing with objects at varying distances. Inverse depth representation highlights subtle depth variations in images, especially for distant objects. SILog better captures these details, improving the model's depth estimation accuracy for objects at different distances.

[0112] The overall training loss can be determined by the following formula:

[0113] (4),

[0114] in, To balance the two loss functions, they are set to 1.0 in this invention.

[0115] According to an embodiment of the present invention, the above-mentioned point cloud reconstruction of an underwater target object based on a pseudo-forward depth map through coordinate transformation operation to obtain a three-dimensional reconstruction result of the underwater target object includes: performing image analysis on the pseudo-forward depth map to obtain the azimuth angle, elevation angle, and pixels representing distance of the underwater target object; and performing coordinate transformation using the azimuth angle, elevation angle, and pixels representing distance of the underwater target object based on a predefined coordinate transformation relationship to obtain a three-dimensional point cloud reconstruction result of the underwater target object.

[0116] According to an embodiment of the present invention, the predefined coordinate transformation relationship is as shown in formula (5):

[0117] (5).

[0118] in, This represents the coordinate position of an underwater target object in Euclidean space. This represents the elevation angle of a pixel in the pseudo-forward depth map. This represents the azimuth angle of a pixel in the pseudo-forward depth map. This represents the value of a pixel in the pseudo-forward depth map.

[0119] The following detailed explanation, through specific embodiments, further illustrates the point cloud reconstruction of a single-image sonar target object using pseudo-forward depth maps and coordinate transformation formulas.

[0120] First, let's explain pseudo-forward depth maps. Pseudo-forward depth maps can help resolve the ambiguity caused by the lack of elevation information in image sonar data. In pseudo-forward depth maps, the horizontal axis corresponds to the azimuth angle. The vertical axis represents the angle of elevation. Each pixel represents a distance. Therefore, for sonar images, it is only necessary to reconstruct the corresponding pseudo-forward depth map. After recovering the depth map, the objects scanned by the sonar are converted into a 3D point cloud according to the following coordinate transformation relationship:

[0121] (5).

[0122] By using pseudo-forward depth maps, the problem of 3D reconstruction of sonar data can be restructured into a more easily understood depth estimation task.

[0123] The following specific examples and comparative experiments, in conjunction with the appendix, demonstrate this approach. Figures 3-13 The advantages of the method provided by the present invention will be further explained in detail below.

[0124] Figure 3 This is an image sonar imaging model according to an embodiment of the present invention.

[0125] in, Figure 3(a) shows a multibeam sonar model; Figure 3 (b) indicates the sonar coordinate system; Figure 3 (c) shows a single-beam sonar and a sonar image.

[0126] Figure 4 This is an elevation angle blur and pseudo-forward depth map according to an embodiment of the present invention.

[0127] in, Figure 4 In (a), the beam The image propagates to the surface of the object. When two orange dots are equidistant from the sonar, these two dots correspond to the same point in the sonar image, resulting in an elevation blurring problem. Figure 4 (b) In the pseudo-forward depth map, there is no problem of elevation ambiguity, so the one-to-one correspondence of the orange object points is similar to that of a visual camera.

[0128] Figure 5 This is the depth estimation effect of a pseudo-front view in a water tank dataset according to an embodiment of the present invention.

[0129] in, Figure 5 (a) Sonar image of the input network. Figure 5 (b) True value of pseudo-forward depth image. Figure 5 (c) Contrast with depth estimation images obtained by the A2FNET method. Figure 5 (d) Depth estimation image obtained by ResNet101 using SILog. Figure 5 (e) Depth estimation image obtained by ResNet101 using SILog and SWL. Figure 5 (f) Depth estimation image obtained by U-Net++ using SILog. Figure 5 (g) Depth estimation images obtained by U-Net++ using SILog and SWL.

[0130] Figure 6 The depth error is trained on the water tank dataset according to an embodiment of the present invention. Figure 6 The darker the red, the greater the error.

[0131] in, Figure 6 (a) U-Net++ depth error using SILog. Figure 6 (b) Depth error of U-Net++ using SILog and SWL. Figure 6 (c) Depth error of ResNet101 using SILog. Figure 6 (d) Depth error of ResNet101 using SILog and SWL.

[0132] Figure 7 It is a point cloud constructed from the water tank dataset according to an embodiment of the present invention.

[0133] in, Figure 7 (a) Real point cloud of the dataset. Figure 7 (b) Comparison of point clouds generated by the A2FNET method. Figure 7 (c) Point cloud generated by ResNet101 using SILog. Figure 7 (d) Point cloud generated by ResNet101 using SILog and SWL. Figure 7 (e) Point cloud generated by U-Net++ using SILog. Figure 7 (f) Point cloud generated by U-Net++ using SILog and SWL.

[0134] Figure 8 This is the depth estimation effect of a pseudo-front view in a floating object dataset according to an embodiment of the present invention.

[0135] in, Figure 8 (a) Sonar image of the input network. Figure 8 (b) True value of pseudo-forward depth image. Figure 8 (c) The depth-estimated image obtained by the A2FNET method. Figure 8 (d) Depth estimation image obtained by ResNet101 using SILog. Figure 8 (e) Depth estimation image obtained by ResNet101 using SILog and SWL. Figure 8 (f) Depth estimation image obtained by U-Net++ using SILog. Figure 8 (g) Depth estimation images obtained by U-Net++ using SILog and SWL.

[0136] Figure 9 The depth error map is trained on the floating object dataset according to the present invention, wherein, Figure 9 The darker the red, the greater the error.

[0137] in, Figure 9 (a) Depth error of ResNet101 using SILog and SWL. Figure 9 (b) Depth error of U-Net++ using SILog and SWL.

[0138] Figure 10 It is a point cloud constructed from a floating object dataset according to an embodiment of the present invention.

[0139] in, Figure 10 (a) Real point cloud of the dataset. Figure 10 (b) Comparison of point clouds generated by the A2FNET method. Figure 10 (c) Point cloud generated by ResNet101 using SILog. Figure 10 (d) Point cloud generated by ResNet101 using SILog and SWL. Figure 10 (e) Point cloud generated by U-Net++ using SILog. Figure 10 (f) Point cloud generated by U-Net++ using SILog and SWL.

[0140] Figure 11 This is a physical experimental scenario implemented according to the present invention.

[0141] in, Figure 11 (a) Multibeam imaging sonar Oculus M750d. Figure 11 (a) The target object reconstructed by sonar.

[0142] Figure 12 The result is a point cloud reconstructed from a physical experiment according to an embodiment of the present invention.

[0143] in, Figure 12 (a) Sonar image of the target object. Figure 12 (b) Point cloud reconstructed using SILog and SWL via U-Net++. Figure 12 (c) Point clouds reconstructed by ResNet using SILog and SWL, as a comparative experiment.

[0144] Figure 13 This is a structural diagram of an underwater three-dimensional reconstruction device for a single sonar image according to an embodiment of the present invention.

[0145] like Figure 13 As shown, the underwater 3D reconstruction device 1300 for a single sonar image includes a sample preprocessing module 1310, a model and loss function construction module 1320, a model processing and loss calculation module 1330, a model parameter iterative optimization module 1340, a target image splitting module 1350, a pseudo-forward depth map acquisition module 1360, and a point cloud reconstruction module 1370.

[0146] The sample preprocessing module 1310 is used to split the sonar images in the sonar image sample set into channels and adjust the image size to obtain a multi-channel sonar image sample set.

[0147] The model and loss function construction module 1320 is used to construct a sonar image segmentation model based on the skip connection mechanism and the feature fusion mechanism, and to construct a total training loss function based on the sonar weight loss function and the scale-invariant log loss function using predefined balancing coefficients.

[0148] The model processing and loss calculation module 1330 is used to process the multi-channel sonar image sample set using the sonar image segmentation model to obtain the pseudo-forward depth map sample set, and to process the ground truth depth value of the multi-channel sonar image sample set and the predicted depth value of the pseudo-forward depth map sample set using the total training loss function to obtain the total training loss value.

[0149] The model parameter iterative optimization module 1340 is used to optimize the parameters of the sonar image segmentation model using the total training loss value, and iteratively perform parameter optimization operations until the preset conditions are met to obtain the trained sonar image segmentation model.

[0150] The target image segmentation module 1350 is used to perform multi-channel segmentation on a single sonar image of an underwater target object to obtain a multi-channel sonar image of the underwater target object.

[0151] The pseudo-forward depth map acquisition module 1360 is used to perform image segmentation and depth prediction on multi-channel sonar images of underwater target objects using a trained sonar image segmentation model to obtain pseudo-forward depth maps of underwater target objects.

[0152] The point cloud reconstruction module 1370 is used to reconstruct the point cloud of underwater target objects based on a pseudo-forward depth map through coordinate transformation operations, so as to obtain the three-dimensional reconstruction result of the underwater target objects.

[0153] According to embodiments of the present invention, any multiple modules among the sample preprocessing module 1310, model and loss function construction module 1320, model processing and loss calculation module 1330, model parameter iterative optimization module 1340, target image segmentation module 1350, pseudo-forward depth map acquisition module 1360, and point cloud reconstruction module 1370 can be merged into one module, or any one of these modules can be split into multiple modules. Alternatively, at least some of the functions of one or more of these modules can be combined with at least some of the functions of other modules and implemented in one module. According to embodiments of the present invention, at least one of the sample preprocessing module 1310, the model and loss function construction module 1320, the model processing and loss calculation module 1330, the model parameter iterative optimization module 1340, the target image segmentation module 1350, the pseudo-forward depth map acquisition module 1360, and the point cloud reconstruction module 1370 can be at least partially implemented as hardware circuits, such as field-programmable gate arrays (FPGAs), programmable logic arrays (PLAs), systems-on-a-chip, systems-on-a-substrate, systems-on-package, application-specific integrated circuits (ASICs), or any other reasonable means of integrating or packaging circuits, or implemented in hardware or firmware, or in any one of the three implementation methods of software, hardware, and firmware, or in any appropriate combination of any of them. Alternatively, at least one of the sample preprocessing module 1310, model and loss function construction module 1320, model processing and loss calculation module 1330, model parameter iterative optimization module 1340, target image segmentation module 1350, pseudo-forward depth map acquisition module 1360, and point cloud reconstruction module 1370 can be at least partially implemented as a computer program module, which can perform corresponding functions when the computer program module is run.

[0154] Figure 14 A block diagram of an electronic device suitable for implementing an underwater three-dimensional reconstruction method for a single sonar image, according to an embodiment of the present invention, is shown schematically.

[0155] like Figure 14 As shown, an electronic device 1400 according to an embodiment of the present invention includes a processor 1401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1402 or a program loaded from a storage portion 1408 into a random access memory (RAM) 1403. The processor 1401 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 1401 may also include onboard memory for caching purposes. The processor 1401 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0156] RAM 1403 stores various programs and data required for the operation of electronic device 1400. Processor 1401, ROM 1402, and RAM 1403 are interconnected via bus 1404. Processor 1401 executes various operations of the method flow according to embodiments of the present invention by executing programs in ROM 1402 and / or RAM 1403. It should be noted that the programs may also be stored in one or more memories other than ROM 1402 and RAM 1403. Processor 1401 may also execute various operations of the method flow according to embodiments of the present invention by executing programs stored in said one or more memories.

[0157] According to an embodiment of the present invention, the electronic device 1400 may further include an input / output (I / O) interface 1405, which is also connected to a bus 1404. The electronic device 1400 may also include one or more of the following components connected to the input / output (I / O) interface 1405: an input section 1406 including a keyboard, mouse, etc.; an output section 1407 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1408 including a hard disk, etc.; and a communication section 1409 including a network interface card such as a LAN card, modem, etc. The communication section 1409 performs communication processing via a network such as the Internet. A drive 1410 is also connected to the input / output (I / O) interface 1405 as needed. A removable medium 1411, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 1410 as needed so that computer programs read from it can be installed into the storage section 1408 as needed.

[0158] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of the present invention.

[0159] According to embodiments of the present invention, a computer-readable storage medium may be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of the present invention, a computer-readable storage medium may include ROM 1402 and / or RAM 1403 and / or one or more memories other than ROM 1402 and RAM 1403 described above.

[0160] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0161] Those skilled in the art will understand that the features described in the various embodiments of the present invention can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in the present invention. In particular, the features described in the various embodiments of the present invention can be combined and / or combined in various ways without departing from the spirit and teachings of the present invention. All such combinations and / or combinations fall within the scope of the present invention.

[0162] The embodiments of the present invention have been described above. However, these embodiments are merely illustrative and not intended to limit the scope of the invention. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of the invention, and all such substitutions and modifications should fall within the scope of the invention.

Claims

1. A method for underwater 3D reconstruction based on a single sonar image, characterized in that, The method includes: The sonar images in the sonar image sample set are split into channels and their image sizes are adjusted to obtain a multi-channel sonar image sample set. A sonar image segmentation model is constructed based on the skip connection mechanism and the feature fusion mechanism, and a total training loss function based on the sonar weight loss function and the scale-invariant log loss function is constructed using predefined balance coefficients. The sonar image segmentation model is used to process the multi-channel sonar image sample set to obtain a pseudo-forward-looking depth map sample set. The total training loss function is then used to process the ground truth depth value of the multi-channel sonar image sample set and the predicted depth value of the pseudo-forward-looking depth map sample set to obtain the total training loss value. The parameters of the sonar image segmentation model are optimized using the total training loss value. The parameter optimization operation is performed iteratively until the preset conditions are met to obtain the trained sonar image segmentation model. A single sonar image of an underwater target object is split into multiple channels to obtain a multi-channel sonar image of the underwater target object; The trained sonar image segmentation model is used to perform image segmentation and depth prediction on the multi-channel sonar image of the underwater target object to obtain a pseudo-forward depth map of the underwater target object. Based on the pseudo-forward depth map, the underwater target object is reconstructed by point cloud through coordinate transformation operation to obtain the three-dimensional reconstruction result of the underwater target object.

2. The method according to claim 1, characterized in that, Sonar image segmentation models based on skip connection and feature fusion mechanisms include: An encoder is constructed based on multiple encoding convolutional layers and pooling layers to extract features from the target image, and a decoder is constructed based on multiple decoding convolutional layers and upsampling layers to perform size mapping on the extracted features; A skip connection mechanism for connecting different levels of features extracted by the encoder is determined, and a feature fusion mechanism for fusing the output results of different levels of the encoder and the decoder is determined based on the upsampling and downsampling strategies. Based on the skip connection mechanism and the feature fusion mechanism, a sonar image segmentation model including the encoder and the decoder is constructed.

3. The method according to claim 1, characterized in that, Using predefined balancing coefficients, the total training loss function, based on the sonar weight loss function and the scale-invariant logarithmic loss function, is constructed as follows: Based on the echo intensity model of pixels in the sonar image, the transmittance characterizing the sonar echo reflection intensity is determined, and the sonar weight loss function is constructed using the transmittance. A scale-invariant logarithmic loss function is constructed based on the inverse depth representation, and the total training loss function is obtained using the sonar weight loss function and the scale-invariant logarithmic loss function based on the predefined balance coefficient.

4. The method according to claim 3, characterized in that, The echo intensity model of the pixels in the sonar image is shown in formula (1): （1）， in, This indicates the intensity of the beam emitted by the sonar. This represents the transmittance. Indicates the first The concentration of water particles at the location of the point. Indicates the first The distance between the location of the point and the sonar. This represents the displacement of a point. Indicates the first The azimuth of the point, Indicates the first The elevation angle of the point in the coordinate system where the sonar is located. Indicates the echo reflection intensity at a point; The sonar weight loss function is shown in formula (2): （2）， in, This represents the distance weight. For inverse depth, Represents the training error weights. This indicates that the distance weights and the training error weights are used to balance the distance weights and the training error weights. The scale-invariant logarithmic loss function is shown in formula (3): （3）， in, It is the number of pixels. and This represents the preset coefficient. .

5. The method according to claim 3, characterized in that, The total training loss function is shown in formula (4): （4）， in, This represents the predefined balance coefficient.

6. The method according to claim 1, characterized in that, Based on the pseudo-forward depth map, point cloud reconstruction of the underwater target object is performed through coordinate transformation operations to obtain the three-dimensional reconstruction result of the underwater target object, including: Image analysis is performed on the pseudo-forward depth map to obtain the azimuth angle, elevation angle, and distance-representing pixels of the underwater target object; Based on predefined coordinate transformation relationships, coordinate transformation is performed using the azimuth, elevation, and distance-representing pixels of the underwater target object to obtain the three-dimensional point cloud reconstruction result of the underwater target object.

7. The method according to claim 6, characterized in that, The predefined coordinate transformation relationship is shown in formula (5): （5）， in, This indicates the coordinate position of the underwater target object in Euclidean space. This indicates the elevation angle of the pixel in the pseudo-forward depth map. This represents the azimuth angle of a pixel in the pseudo-forward depth map. This represents the value of a pixel in the pseudo-forward depth map.

8. An underwater 3D reconstruction device for a single sonar image, characterized in that, The device includes: The sample preprocessing module is used to split the sonar images in the sonar image sample set into channels and adjust the image size to obtain a multi-channel sonar image sample set. The model and loss function construction module is used to build a sonar image segmentation model based on the skip connection mechanism and feature fusion mechanism, and to construct the total training loss function based on the sonar weight loss function and the scale-invariant log loss function using predefined balance coefficients. The model processing and loss calculation module is used to process the multi-channel sonar image sample set using the sonar image segmentation model to obtain a pseudo-forward depth map sample set, and to process the ground truth depth value of the multi-channel sonar image sample set and the predicted depth value of the pseudo-forward depth map sample set using the training total loss function to obtain the training total loss value. The model parameter iterative optimization module is used to optimize the parameters of the sonar image segmentation model using the total training loss value, and iteratively perform the parameter optimization operation until the preset conditions are met to obtain the trained sonar image segmentation model. The target image segmentation module is used to perform multi-channel segmentation on a single sonar image of an underwater target object to obtain a multi-channel sonar image of the underwater target object. The pseudo-forward depth map acquisition module is used to perform image segmentation and depth prediction on the multi-channel sonar image of the underwater target object using the trained sonar image segmentation model to obtain the pseudo-forward depth map of the underwater target object. The point cloud reconstruction module is used to reconstruct the point cloud of the underwater target object based on the pseudo forward depth map through coordinate transformation operations, so as to obtain the three-dimensional reconstruction result of the underwater target object.

9. An electronic device, comprising: One or more processors; Memory, used to store one or more computer programs. The characteristic feature is that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 7.

Images