A method for generating simulated sonar images

Abstract

The present invention relates to a method for generating simulated sonar images. The method first improves the CycleGAN network, including segmenting the input and output images of the generator G and the generator F using the Otsu algorithm and the Unet network respectively, and fusing the segmentation results. Calculate the SSIM loss based on the fused images of the input and output of the forward generator G and the reverse generator F, and add it to the generator loss of the CycleGAN network. Then, use the simple sonar simulation images and real sonar images for training, and use the trained improved CycleGAN network to generate stylized simulated sonar images. Using the improved CycleGAN network to generate simulated sonar images can reduce the differences between the simulated images and the real images in terms of structure, edge features, etc., thus ensuring that the simulated sonar images are more similar to the real images.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and more specifically, to a method for generating simulated sonar images. Background Art

[0002] Sonar images are obtained by processing sonar signals. The cost of obtaining real sonar images is high and the steps are complex, while the method for generating simulated sonar images can generate simulated sonar images of complex seabed scenes from multiple angles and directions according to user requirements, and is widely favored.

[0003] Currently, simulated sonar images are obtained by stylizing simple simulated images.

[0004] Among them, the generation of simple simulated images requires relatively complex steps. Currently, there are two common methods: one is to perform three-dimensional modeling on a specific scene. This method has a large workload. If common evaluation indicators such as SSIM and MSE are used for measurement, the position and size of the three-dimensional scene and the real scene need to correspond, further increasing the workload; the other is to use lasers to obtain accurate depth maps of real scenes, combine optical images of real scenes to train neural networks to generate depth maps, and then perform three-dimensional reconstruction on the scenes. The depth maps are highly affected by the training effect of the network, and the authenticity and versatility cannot be guaranteed.

[0005] On the other hand, although the stylization network can improve factors such as background noise that are difficult to simulate in simple simulated images, it will cause differences in target structure, edge features, etc. between the generated images and the original images.

[0006] Therefore, how to provide a new method for generating simulated sonar images to at least partially solve the current problems is an urgent problem for those skilled in the art. Summary of the Invention

[0007] In view of this, in order to simplify the workload and at the same time ensure the authenticity of simulated sonar images as much as possible, the present invention provides a method for generating simulated sonar images.

[0008] In order to achieve the above object, the present invention adopts the following technical solutions:

[0009] A method for generating simulated sonar images includes the following steps:

[0010] Obtain and utilize simple simulated sonar images and real sonar images to train an improved CycleGAN network to obtain a trained improved CycleGAN network; the CycleGAN network includes a forward generator G and a reverse generator F, and the improvement steps include:

[0011] For the input and output images of the forward generator G and the reverse generator F, fusion processing is respectively performed to obtain corresponding fused images. The fusion processing includes segmenting the images using the Otsu algorithm to obtain a first segmentation result, and segmenting the images using the Unet network to obtain a second segmentation result, and fusing the first segmentation result and the second segmentation result;

[0012] According to the fused images input and output by the forward generator G, calculate the structural similarity loss of the forward generator G;

[0013] According to the fused images input and output by the reverse generator F, calculate the structural similarity loss of the reverse generator F;

[0014] Add the structural similarity loss of the forward generator G and the structural similarity loss of the reverse generator F to the CycleGAN network generator loss to obtain the loss function of the improved CycleGAN network;

[0015] Use the trained improved CycleGAN network to generate stylized simulated sonar images in real time according to the simple simulated sonar images.

[0016] Preferably, the Unet network is trained with the real sonar images before use to facilitate the segmentation of the boundary regions of the target images;

[0017] Preferably, when segmenting using the Otsu algorithm and the Unet network, the foreground region of the image is set to 1 and the background region is set to 0, and the fusion is to retain the common foreground regions of the two segmentation results;

[0018] Preferably, the loss function of the improved CycleGAN network is:

[0019] L(G, F, D x , D y ) = L GAN (G, D y , X, Y) + L GAN (F, D x , Y, X) + λL CYC (G, F) + ωL S

[0020] where X and Y are two image domains of the CycleGAN network, G and F are respectively the forward and reverse generators of the CycleGAN network, D Y , D X are respectively the discriminators corresponding to G and F, L GAN (G, D y , X, Y) and L GAN (F, Dx , Y, X) is the adversarial loss, L CYC (G, F) is the cycle consistency loss, and the coefficients λ and ω are weights, L s is the structural similarity loss between the forward generator G and the reverse generator F;

[0021] Preferably, the L s has the following expression:

[0022] L S = L SSIM (x, G(x)) + L SSIM (y, F(y))

[0023] In the formula, L SSIM (x, G(x)) is the structural similarity loss of the generator G, where x and G(x) are the input and output images of the generator G, and L SSIM (y, F(y)) is the structural similarity loss of the generator F, where y and F(y) are the input and output images of the generator F;

[0024] Among them, the structural similarity loss:

[0025] In the formula, N is the total number of pixels in the input or output image, and p is the pixel at the same position in the input and output images.

[0026] Preferably, the calculation formula for the structural similarity is:

[0027]

[0028] Among them, a and b are the same regions obtained with p as the center in the input and output images respectively, and the parts exceeding the image boundary are set to 0; μ a is the mean of a, and μ b is the mean of b, σ ab is the covariance of a and b, and σ a 2 is the variance of a, and σ b 2 is the variance of b, and c 1 , c 2 , c 3 are constants.

[0029] Preferably, c 1 is 0.01, c 2 is 0.03, c 3 is c 2 / 2;

[0030] On the other hand, the present invention also provides a method for obtaining a simple simulated sonar image, and the steps include:

[0031] S1. Obtain the optical binocular image of the required target using a binocular camera, and crop the optical binocular image according to the sonar beam angle to obtain the cropped optical binocular image;

[0032] S2. Calculate the depth map of the scene based on the cropped optical binocular image, and perform a maximum distance limit on the depth map;

[0033] S3. Calculate the simple sonar simulation image based on the depth map with the maximum distance limit;

[0034] Preferably, the optical binocular image is cropped according to the following formula

[0035] r / L = α / θ

[0036] In the formula, r is the width of the cropped image, L is the width of the original image, α is the sonar scanning beam angle, and θ is the camera field of view angle;

[0037] Preferably, the maximum distance limit is performed on the depth map according to the maximum scanning distance of the sonar device;

[0038] Preferably, the simple sonar simulation image is calculated according to the following formula, where the echoes of each beam of the sonar at the same time are superimposed, the acoustic wave intensity is calculated, and the acoustic wave intensity is arranged in chronological order;

[0039]

[0040] where I is the superimposed acoustic wave intensity; N is the number of echoes at the same time; I 0 is a constant and can be adjusted according to experience; r is the distance from the echo point to the sonar transducer.

[0041] As can be seen from the above technical solutions, the present invention discloses a method for generating a simulated sonar image. By using the Unet network combined with the Otsu algorithm to segment the sonar image, the structure and edge features of the target in the sonar image can be better retained. By introducing the target structure feature loss based on the segmentation result into the generator loss function of the CycleGAN network, the obtained simulated image can better retain the target structure features of the original image; thus minimizing the differences between the simulated image and the real image in terms of structure, edge features, etc. In addition, the method provided in this application has a small amount of calculation and low hardware requirements while ensuring the authenticity of the simulated sonar image.

[0042] Another beneficial effect of the present invention is that binocular images are obtained through a binocular camera, the binocular images are cropped to have the same viewing angle as the sonar device, then a depth map is obtained using the cropped binocular images, and the values of the depth map are restricted to be the same as the maximum detection distance of the sonar device; finally, a simple sonar simulation image is directly calculated using the processed depth map above. Compared with the method based on 3D modeling and 3D reconstruction, the workload is small, and the accuracy is higher than the method using deep learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only the embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained according to the provided drawings.

[0044] Figure 1 Flow chart for improving the CycleGAN network;

[0045] Figure 2 Structural schematic diagram of the CycleGAN network provided by the present invention;

[0046] Figure 3 Result diagram of segmenting and fusing the image in the simulation domain (input of generator G, output of generator F) provided by the present invention;

[0047] Figure 4 Result diagram of segmenting and fusing the image in the real domain (input of generator F, output of generator G) provided by the present invention;

[0048] Figure 5 Method schematic diagram of cropping the optical binocular image provided by the present invention;

[0049] Figure 6 Image after cropping the optical binocular image provided by the present invention;

[0050] Figure 7 Depth map calculated according to the optical binocular image provided by the present invention;

[0051] Figure 8 Comparison diagram of the calculated simple simulation sonar image and the real sonar image with the same scene and the same viewing angle provided by the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0052] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0053] In view of the defects existing in the current simulation sonar image generation, an embodiment of the present invention discloses a new method for real-time generation of simulation sonar images, which specifically includes the following steps:

[0054] Obtain and utilize a simple simulation sonar image and a real sonar image to train an improved CycleGAN network to obtain a trained improved CycleGAN network; the CycleGAN network includes a forward generator G and a reverse generator F, and the improvement steps include:

[0055] For the input and output images of the forward generator G and the reverse generator F, perform fusion processing respectively to obtain corresponding fusion images. The fusion processing includes using the Otsu algorithm to segment the image to obtain the first segmentation result, and using the Unet network to segment the image to obtain the second segmentation result, and fusing the first segmentation result and the second segmentation result;

[0056] According to the fusion image of the input and output of the forward generator G, calculate the structural similarity loss of the forward generator G;

[0057] According to the fusion image of the input and output of the reverse generator F, calculate the structural similarity loss of the reverse generator F;

[0058] Add the structural similarity loss of the forward generator G and the structural similarity loss of the reverse generator F to the generator loss of the CycleGAN network to obtain the loss function of the improved CycleGAN network;

[0059] Then use the trained improved CycleGAN network to generate a stylized simulation sonar image in real time according to the simple simulation sonar image.

[0060] To make the above objects, features, and advantages of the present invention more obvious and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0061] The CycleGAN network is generally used for image style conversion, and its structure diagram is as Figure 2As shown in the figure, it mainly includes a forward generator G, a reverse generator F, discriminators Dx and Dy. X represents the simulated image domain, and Y represents the real image domain. For an image x in the simulation domain, through the generator G, y is obtained. The discriminator Dy determines whether the image y conforms to the characteristics of the real image domain Y. At the same time, in order to make the image y also have the characteristics of the simulation domain, the image y is further passed through the generator F to obtain the image x'. At this time, the image x' has the content and characteristics of both the simulated image domain X and the real image domain Y.

[0062] However, during the process of generating data, there will be differences in aspects such as the structure and edge features between the obtained simulated images and the original images, and it is not easy to guarantee their authenticity.

[0063] Therefore, this application improves the existing CycleGAN network to facilitate the stylization of simulated sonar images based on real sonar images, making the stylization results more realistic.

[0064] The specific improvement steps include:

[0065] For the input and output images of the forward generator G and the reverse generator F, fusion processing is respectively performed to obtain corresponding fused images. The fusion processing includes using the Otsu algorithm to segment the image to obtain the first segmentation result, and using the Unet network to segment the image to obtain the second segmentation result, and then fusing the first segmentation result and the second segmentation result.

[0066] Then, according to the fused images of the input and output of the forward generator G, the structural similarity loss of the forward generator G is calculated.

[0067] According to the fused images of the input and output of the reverse generator F, the structural similarity loss of the reverse generator F is calculated.

[0068] Finally, the structural similarity loss of the forward generator G and the structural similarity loss of the reverse generator F are added to the generator loss of the CycleGAN network to obtain the loss function of the improved CycleGAN network; in this way, the structure and edge features of the targets in the simulated images are retained.

[0069] In the present invention, before using the Unet network, it needs to be trained with real sonar images so that it can segment the boundary regions of the targets.

[0070] Moreover, when using the Otsu algorithm and the Unet network for segmentation, the foreground region of the image is set to 1, and the background region is set to 0.

[0071] In one embodiment, when fusing the segmentation results of the Otsu algorithm and the Unet network for the same image, the common foreground regions of the two segmentation results are retained and multiplied by the original image to further retain the corresponding foreground part of the original image, and the background part is set to black.

[0072] As shown in the figure, Figure 3 It is a schematic diagram for segmenting and fusing the simulation domain image, Figure 4 It is a schematic diagram for segmenting and fusing the real domain image.

[0073] Secondly, calculate the SSIM loss of generator G and generator F and add it to the generator loss of the CycleGAN network, so as to obtain the loss function of the improved CycleGAN network;

[0074] For SSIM, it is an index to measure the similarity between two images,

[0075] The specific calculation formula is:

[0076]

[0077] a and b are regions of the same size centered at p in the input and output images respectively. In one embodiment, the size is selected as 13×13, and the part exceeding the image boundary is set to 0, μ a is the mean of a, μ b is the mean of b, σ ab is the covariance of a and b, σ a 2 is the variance of a, σ b 2 is the variance of b, c 1 、c 2 、c 3 are constants.

[0078] In one embodiment, to prevent the denominator from being zero, take c 1 =0.01, c 2 =0.03, c 3 =c 2 / 2.

[0079] Furthermore, the structural similarity loss is:

[0080]

[0081] Among them, N is the total number of pixels in the input or output image, and P is the pixels at the same position in the two images;

[0082] Thus, the loss function of the improved CycleGAN network is:

[0083] L(G, F, D x , D y ) = L GAN (G, D y , X, Y) + L GAN (F, D x , Y, X) + λL CYC (G, F) + ωL S

[0084] where X and Y are two image domains of the CycleGAN network, G and F are the forward and backward generators of the CycleGAN network respectively, D Y , D X are discriminators corresponding to G and F respectively, L GAN (G, D y , X, Y) and L GAN (F, D x , Y, v) are adversarial losses, L CYC (G, F) is a cycle consistency loss, and the coefficients λ and ω are weights, and L S is the structural similarity loss between the forward generator G and the backward generator F;

[0085] where

[0086] L S = L SSIM (x, G(x)) + L SSIM (y, F(y)),

[0087] In the formula, L SSIM (x, G(x)) is the structural similarity loss of the generator G, x and G(x) are the input and output images of the generator G, and L SSIM (y, F(y) is the structural similarity loss of the generator F, and y and F(y) are the input and output images of the generator F;

[0088] This loss can ensure that the target structural features of the generator output image are as consistent as possible with the input image.

[0089] In one embodiment, through experiments, it is determined that when λ = 10 and ω = 20 are selected, the adversarial loss, cycle consistency loss, and SSIM loss of the network can be guaranteed to be in the same order of magnitude.

[0090] After the above improvements, an improved CycleGAN network is obtained. Further, after training with simulated sonar images and real sonar images, a trained improved CycleGAN network is obtained;

[0091] Finally, the trained improved CycleGAN network can be used to generate stylized sonar images in real time according to simple sonar simulation images.

[0092] The method provided by the present invention has a small amount of calculation and low hardware requirements, and can minimize the differences in content, features, etc. between the simulated image and the real image, ensuring the authenticity of the simulated sonar image.

[0093] In addition, for the acquisition of simple sonar simulation images, common methods generally require building a three-dimensional scene. When measured by common evaluation indicators such as SSIM and MSE, the positions and sizes of the three-dimensional scene and the real scene also need to correspond, resulting in a large amount of work.

[0094] Secondly, if the optical images of the real scene are used to train a neural network to generate a depth map for reconstructing the three-dimensional scene, the result is highly affected by the training effect of the network, and the authenticity and versatility cannot be guaranteed. At the same time, training the network requires using high-precision equipment such as lasers to obtain the accurate depth map of the real scene, resulting in a high cost.

[0095] In response to this, the present invention proposes a method for obtaining a simple simulated sonar image based on a binocular camera. After the binocular camera is calibrated, a simple depth map of the real scene can be directly generated without using a neural network, and the calculation result has a high authenticity. At the same time, after the image is Figure 2 cropped as shown, the camera view angle is the same as that of the real sonar device. By restricting the values of the depth map, it can be ensured that it is the same as the maximum detection distance of the sonar device. In addition, this method does not require reconstructing the three-dimensional scene, and directly calculates the simple simulated sonar image through the depth map, reducing the workload.

[0096] Specifically, the method steps for obtaining a simple simulated sonar image through a binocular camera are as follows:

[0097] S1. Use the binocular camera to obtain the optical binocular image of the required target, and crop the optical binocular image according to the sonar beam angle to obtain the cropped optical binocular image; since the sonar detection beam angle is often small (about 20°) and the field of view angle of the binocular camera is large, the width of the binocular image needs to be cropped so that its field of view range is consistent with the sonar detection range. The cropping method is as Figure 5 shown, and the cropping formula is:

[0098] r / L = α / θ

[0099] where r is the width of the cropped image, L is the width of the original image, α is the sonar detection beam angle, and θ is the camera field of view angle; the cropped image is as Figure 6 shown.

[0100] S2. Calculate the depth map of the scene according to the cropped optical binocular image, as Figure 7 shown, and perform a maximum distance limit on the depth map according to the maximum detection distance of the sonar device;

[0101] Specifically, first, a value A representing the maximum detection distance of the sonar device is set, and the distance from each pixel to the camera is mapped from A to 0 to 1 (black) to 0 (white) in the depth map. Pixels with a depth exceeding A are all set to 1 (black).

[0102] S3. Calculate the simple sonar simulation image according to the depth map restricted by the maximum distance.

[0103] This process is calculated according to the simplified sonar imaging principle. The simplified sonar imaging principle is to simplify the echo of each beam of the sonar into each column of pixels in the depth map. The value of the pixel in the depth map represents the distance from this point to the camera. Assuming that the sound wave speed is fixed, different values of each pixel can represent the echo of this point reaching the sonar transducer at different times. By superimposing and calculating the pixels with the same brightness in a certain column, the total intensity of the echo of a certain beam of the sonar transducer at a certain time point can be obtained.

[0104] The specific calculation process is to superimpose the echoes of each beam of the sonar at the same time, calculate the sound wave intensity, and the result of the time without echo is 0. The calculation formula is as follows.

[0105]

[0106] Where I is the superimposed sound wave intensity; N is the number of echoes at the same time; I 0 is a constant and can be adjusted according to experience; r is the distance from the echo point to the sonar transducer.

[0107] Arrange the calculation results of each column in chronological order to obtain the simple sonar simulation image. The light and dark structure of this image is similar to the real sonar image with the same viewing angle as the binocular camera. As Figure 8 shown.

[0108] In this specification, each embodiment is described in a progressive manner. The key point of each embodiment is to illustrate the differences from other embodiments. The same or similar parts among the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple. For the relevant parts, please refer to the description in the method part.

[0109] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be obvious to those skilled in the art. The general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to these embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for generating simulated sonar images, characterized in that, it includes the following steps: Obtain and utilize a simple simulated sonar image and a real sonar image to train an improved CycleGAN network, and obtain the trained improved CycleGAN network; The CycleGAN network includes a forward generator G and a reverse generator F, and the improvement steps include: Perform fusion processing on the input and output images of the forward generator G and the reverse generator F respectively to obtain corresponding fused images. The fusion processing includes using the Otsu algorithm to segment the image to obtain a first segmentation result, and using the Unet network to segment the image to obtain a second segmentation result, and fusing the first segmentation result and the second segmentation result; Calculate the structural similarity loss of the forward generator G according to the fused images input and output by the forward generator G; Calculate the structural similarity loss of the reverse generator F according to the fused images input and output by the reverse generator F; Add the structural similarity loss of the forward generator G and the structural similarity loss of the reverse generator F to the generator loss of the CycleGAN network to obtain the improved CycleGAN network; Use the trained improved CycleGAN network to generate stylized simulated sonar images in real time according to the simple simulated sonar images.

2. A method for generating simulated sonar images according to claim 1, characterized in that, The Unet network is trained with the real sonar image before use to facilitate the segmentation of the boundary region of the target image.

3. A method for generating simulated sonar images according to claim 1, characterized in that, When using the Otsu algorithm and the Unet network for segmentation, the foreground region of the image is set to 1 and the background region is set to 0, and the fusion is to retain the common foreground regions of the two segmentation results.

4. A method for generating simulated sonar images according to claim 1, characterized in that, The loss function of the improved CycleGAN network is: L(G, F, D x , D y ) = L GAN (G, D y , X, Y) + L GAN (F, D x , Y, X) + λL CYC (G, F) + ωL S Among them, X and Y are two image domains of the CycleGAN network, G and F are the forward and backward generators of the CycleGAN network respectively, and D Y , D X are discriminators corresponding to G and F respectively, L GAN (G, D y , X, Y) and L GAN (F, D x , Y, X) are adversarial losses, L CYC (G, F) is a cycle consistency loss, the coefficients λ and ω are weights, and L S is the structural similarity loss between the forward generator G and the backward generator F.

5. A method for generating simulated sonar images according to claim 4, characterized in that, The said L S has the following expression: L S = L SSIM (x, G(x)) + L SSIM (y, F(y)) where L SSIM (x, G(x)) is the structural similarity loss of the generator G, where x and G(x) are the input and output images of the generator G, and L SSIM (y, F(y)) is the structural similarity loss of the generator F, where y and F(y) are the input and output images of the generator F; wherein, the structural similarity loss: In the formula, N is the total number of pixels in the input or output image, p is the pixel at the same position of the input and output images, and SSIM(p) is the structural similarity.

6. A method for generating simulated sonar images according to claim 5, characterized in that, The calculation formula of the structural similarity is: where a and b are the same regions centered at p in the input and output images respectively, and the parts exceeding the image boundaries are set to 0; μ a is the mean of a, μ b is the mean of b, σ ab is the covariance of a and b, σ a 2 is the variance of a, σ b 2 is the variance of b, c 1 、c 2 、c 3 are constants.

7. A method for generating simulated sonar images according to claim 1, characterized in that, The simple simulated sonar image is obtained through the following steps: S1. Use a binocular camera to obtain an optical binocular image of the required target, and crop the optical binocular image according to the sonar beam angle to obtain a cropped optical binocular image; S2. Calculate the depth map of the scene according to the cropped optical binocular image, and perform a maximum distance limit on the depth map; S3. Calculate the simple simulated sonar image according to the depth map with the maximum distance limit.

8. A method for generating simulated sonar images according to claim 7, It is characterized in that the optical binocular image is cropped according to the following formula r / L = α / θ where r is the width of the cropped image, L is the width of the original image, a is the sonar scanning beam angle, and θ is the camera field of view angle.

9. A method for generating a simulated sonar image according to claim 7 It is characterized in that the maximum distance limit is imposed on the depth map according to the maximum scanning distance of the sonar device.

10. A method for generating a simulated sonar image according to claim 7 It is characterized in that the simple sonar simulation image is calculated by the following formula: the echoes of each beam of the sonar at the same time are superimposed, the acoustic wave intensity is calculated, and the acoustic wave intensity is arranged in chronological order; Among them, I is the intensity of the superimposed sound wave; N is the number of echoes at the same time; I 0 is a constant that can be adjusted according to experience; r is the distance from the echo point to the sonar transducer.

Images