Underwater side-scan sonar simulation image generation method, device and system

Abstract

The present invention relates to the field of underwater side-scan sonar simulation technology, and specifically discloses an underwater side-scan sonar simulation image generation method, device and system, including: acquiring underwater image information collected by a camera device; extracting depth features and echo intensity sensitive features of the underwater image information respectively, wherein the echo intensity sensitive features at least include hard features and edge features; superimposing an image having the depth features with an image having the echo intensity sensitive features to obtain a first superimposed image; superimposing noise simulation features on the first superimposed image to obtain a second superimposed image; intercepting the second superimposed image according to a preset side-scan sonar resolution and a preset range to obtain a plurality of sonar pixel strips; and splicing a plurality of sonar pixel strips to obtain a side-scan sonar simulation image. The underwater side-scan sonar simulation image generation method provided by the present invention can achieve the purpose of improving the fidelity of side-scan sonar images.

Description

Technical Field

[0001] The present invention relates to the technical field of underwater side-scan sonar simulation, and in particular to an underwater side-scan sonar simulation image generation method, an underwater side-scan sonar simulation image generation device and an underwater side-scan sonar simulation image generation system. Background Art

[0002] The generation of underwater side-scan sonar simulation images has a very broad application scenario, involving many application fields. For example, underwater area search, underwater bridge construction and pipeline inspection, and aquaculture automation, etc. These tasks require underwater robots to have precise control algorithms, efficient and correct perception capabilities, reliable and fast communication technologies, etc. The development and testing of various technologies often require high costs and high risks, which is not conducive to their development and optimization. Simulation has always been a powerful tool for the development and debugging of various algorithms. It can greatly increase the simulation data sets that are difficult to obtain, and provide verification methods and foundations for subsequent algorithms such as target recognition and obstacle avoidance. It can help developers better understand the performance and system behavior of underwater equipment, which is helpful for optimization, control algorithm development and task planning, and improve the optimization and progress of various technologies. However, some existing side-scan sonar images generated by simulation technologies have many disadvantages such as unreliable data and incorrect image information, which is not conducive to the development and testing of subsequent technologies.

[0003] Therefore, how to generate realistic side-scan sonar images through simulation has become a technical problem that needs to be solved urgently by technicians in this field. Summary of the invention

[0004] The present invention provides an underwater side-scan sonar simulation image generation method, an underwater side-scan sonar simulation image generation device and an underwater side-scan sonar simulation image generation system, which solve the problem of unreliable data of side-scan sonar images existing in the related art.

[0005] As a first aspect of the present invention, a method for generating underwater side-scan sonar simulation images is provided, which includes: acquiring underwater image information collected by a camera device; respectively extracting depth features and echo intensity sensitive features of the underwater image information, wherein the echo intensity sensitive features include at least hard features and edge features; superimposing an image having the depth features with an image having the echo intensity sensitive features to obtain a first superimposed image; superimposing noise simulation features on the first superimposed image to obtain a second superimposed image; intercepting the second superimposed image according to a preset side-scan sonar resolution and a preset range to obtain a plurality of sonar pixel strips; splicing the plurality of sonar pixel strips to obtain a side-scan sonar simulation image; wherein the depth features and echo intensity sensitive features of the underwater image information are respectively extracted, wherein the echo intensity sensitive features include at least hard features and edge features; superimposing an image having the depth features with an image having the echo intensity sensitive features to obtain a first superimposed image; superimposing noise simulation features on the first superimposed image to obtain a second superimposed image; The depth features and echo intensity sensitive features of underwater image information include: extracting the depth features in the underwater image information of each two adjacent cameras through a depth filter, wherein for all pixel points in the underwater image information whose depths satisfy the Gaussian distribution, obtaining the depth data of all pixel points that satisfy the convergence, and obtaining the depth features in the underwater image information; extracting all texture features from the underwater image information according to the local binary pattern, and obtaining the texture features characterizing the echo intensity sensitive features from the all texture features, wherein all texture features of the underwater image information are obtained by dividing the detection window, and the texture features characterizing the echo intensity sensitive features are obtained based on the frequency of occurrence of feature values ​​extracted from all texture features.

[0006] Further, the depth feature in the underwater image information of each two adjacent cameras is extracted by a depth filter, including: when the depth filter receives the underwater image information, determining the projection point position of each pixel in the underwater image information according to limit search and block matching, wherein the depth of all the pixels in the underwater image information satisfies the Gaussian distribution; and calculating the depth value and uncertainty of each pixel according to the geometric relationship formed by the projection point position of each pixel in the two adjacent underwater image information;

[0007] Update the distribution of the depth value of each pixel in the underwater image information, and when the uncertainty of the pixel is less than a preset uncertainty threshold, determine that the depth value of the pixel has converged; normalize the depth values ​​of the pixels in the underwater image information to obtain the depth features of each underwater image information.

[0008] Furthermore, all texture features are extracted from the underwater image information according to the local binary pattern, and texture features characterizing echo intensity sensitive features are obtained from all the texture features, including: dividing the underwater image information into multiple detection windows, and dividing each detection window into multiple detection areas; for any pixel point in each detection area, its pixel value is compared with the pixel values ​​of multiple pixels in a circular neighborhood to obtain a characteristic value of each pixel point in each detection area; the frequency of occurrence of the characteristic value of each pixel point in each detection area is calculated to obtain a histogram of the detection area; the histogram of each detection area is normalized, and the normalized histograms are connected to obtain all texture features of the underwater image information; texture features whose characteristic value occurrence frequency is greater than a preset frequency threshold are extracted from all the texture features to obtain texture features characterizing echo intensity sensitive features.

[0009] Furthermore, for any pixel point in each detection area, its pixel value is compared with the pixel values ​​of multiple pixel points in the annular neighborhood to obtain a characteristic value of each pixel point in each detection area, including: if the pixel value of the pixel point in the detection area is greater than the pixel value of the pixel point in the annular neighborhood, the pixel value of the pixel point in the annular neighborhood is assigned to 1; if the pixel value of the pixel point in the detection area is not greater than the pixel value of the pixel point in the annular neighborhood, the pixel value of the pixel point in the annular neighborhood is assigned to 0; the binary pixel value in the annular neighborhood obtained for each pixel point in each detection area is converted into decimal to obtain a characteristic value of each pixel point in each monitoring area.

[0010] Furthermore, the method further includes generating noise simulation features according to a cyclic generative adversarial network before the step of superimposing the noise simulation features on the first superimposed image.

[0011] Furthermore, generating noise simulation features according to a cyclic generative adversarial network includes: providing the depth features of the underwater image information and the real side-scan sonar image as training data sets; inputting the depth feature training data set of the underwater image information as input and the real side-scan sonar image training data set as output to a first generator, wherein the first generator is capable of converting the image style of the depth features of the underwater image information into the image style of the real side-scan sonar image; inputting the real side-scan sonar image training data set as input and the depth feature training data set of the underwater image information as output to a second generator, wherein the second generator is capable of converting the image style of the real side-scan sonar image into the image style of the depth features of the underwater image information; judging the output result of the first generator by a first discriminator to obtain a first judgment result, and judging the output result of the second generator by a second discriminator to obtain a second judgment result; constraining the first judgment result and the second judgment result according to a preset loss function to obtain noise simulation features.

[0012] Furthermore, the second superimposed image is intercepted according to a preset side scan sonar resolution and a preset range to obtain a plurality of sonar pixel strips, including: calculating the original resolution of the image with the depth feature according to the navigation information; processing the image with the depth feature according to a preset vertical track resolution and a track resolution according to a bilinear interpolation method so that the resolution of the image with the depth feature meets the preset resolution; intercepting the image with the depth feature that meets the preset resolution requirement according to the preset range, the preset vertical track resolution and the track resolution to obtain a plurality of sonar pixel strips.

[0013] As another aspect of the present invention, there is provided an underwater side-scan sonar simulation image generation device for implementing the underwater side-scan sonar simulation image generation method described above, which includes: an acquisition module for acquiring underwater image information collected by a camera device; a feature extraction module for respectively extracting depth features and echo intensity sensitive features of the underwater image information, wherein the echo intensity sensitive features include at least hard features and edge features; a first feature superposition module for superimposing an image having the depth features with an image having the echo intensity sensitive features to obtain a first superimposed image; a second feature superposition module for superimposing noise simulation features on the first superimposed image to obtain a second superimposed image; a capture module for capturing the second superimposed image according to a preset side-scan sonar resolution and a preset range to obtain a plurality of sonar pixel strips; and a splicing module for splicing a plurality of sonar pixel strips to obtain a side-scan sonar simulation image.

[0014] As another aspect of the present invention, there is provided an underwater side-scan sonar simulation image generation system, which comprises: a camera device and the underwater side-scan sonar simulation image generation device as described above, the camera device being communicatively connected to the underwater side-scan sonar simulation image generation device, the camera device being used to collect underwater image information; the underwater side-scan sonar simulation image generation device being used to perform depth feature extraction and echo intensity sensitive feature extraction according to the underwater image information, and to perform noise simulation feature superposition again after superimposing the extracted features, and to intercept and splice the superimposed images according to a preset side-scan sonar resolution and a preset range to obtain a side-scan sonar simulation image.

[0015] The underwater side-scan sonar simulation image generation method provided by the present invention extracts depth features and echo intensity sensitive features from underwater image information collected by a camera device, and then performs noise simulation feature superposition again after superimposing the extracted features, and intercepts and splices the superimposed images according to a preset side-scan sonar resolution and a preset range to obtain a side-scan sonar simulation image. Compared with field-collected underwater data, this underwater side-scan sonar simulation image generation method can simply and efficiently generate a large amount of data for the development and verification of various underwater algorithms, improve their efficiency and accuracy, improve the quality of underwater image data, greatly improve the problem of lack of underwater side-scan sonar data, provide strong visual support for underwater operations, improve the efficiency of underwater related activities, and ultimately achieve the purpose of improving the fidelity of side-scan sonar images. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. Together with the following specific embodiments, they are used to explain the present invention, but do not constitute a limitation of the present invention.

[0017] Figure 1 A flow chart of the underwater side-scan sonar simulation image generation method provided by the present invention.

[0018] Figure 2 A flow chart for extracting depth features and echo intensity sensitive features provided by the present invention.

[0019] Figure 3 A flowchart of a specific implementation method of deep feature extraction provided by the present invention.

[0020] Figure 4a A schematic diagram of the geometric relationship between the limit search and block matching provided by the present invention.

[0021] Figure 4b Another schematic diagram of the geometric relationship between limit search and block matching provided by the present invention.

[0022] Figure 5A flowchart of a specific implementation method for extracting echo intensity sensitive features provided by the present invention.

[0023] Figure 6 A schematic diagram of the network architecture of CycleGan provided by the present invention.

[0024] Figure 7 A flow chart of a method for obtaining multiple sonar pixel strips provided by the present invention.

[0025] Figure 8a A schematic diagram of the strip cutting and splicing provided by the present invention.

[0026] Figure 8b This is a schematic diagram of an actual scene after the strips provided by the present invention are cut and spliced.

[0027] Fig. 9 This is a structural block diagram of the underwater side-scan sonar simulation image generation device provided by the present invention.

[0028] Fig.10 This is a structural block diagram of the underwater side-scan sonar simulation image generation system provided by the present invention. DETAILED DESCRIPTION

[0029] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0030] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiment of the present invention. Obviously, the described embodiment is only a part of the embodiment of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work should fall within the scope of protection of the present invention.

[0031] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so as to describe the embodiments of the present invention described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0032] Sonar sensors play an indispensable role in underwater robot perception, providing information that optical cameras cannot provide in various underwater environments, and are an important component of underwater robots. Many algorithms of underwater autonomous robots use side-scan sonar images for positioning, object detection, map generation, and 3D reconstruction, while standard optical images may not be used due to turbid water. These algorithms are indispensable for many application scenarios, such as marine infrastructure inspection, marine exploration, etc. However, due to the high cost and high risk of underwater field tests, the scarcity of various underwater data, and other problems, the testing and verification efficiency of various underwater robots and underwater perception algorithms is low. The development of realistic sonar simulation images is of great significance, which can effectively solve the testing and verification dilemma of various underwater robots and underwater perception algorithms. Compared with field collection of underwater data, it can generate a large amount of data simply and efficiently, and is beneficial to the development and verification of various algorithms, improving their efficiency and accuracy.

[0033] Based on this, a method for generating an underwater side-scan sonar simulation image is provided in this embodiment. Figure 1 : is a flow chart of a method for generating underwater side-scan sonar simulation images according to an embodiment of the present invention. Figure 1 As shown, it includes: S100, obtaining underwater image information collected by a camera device; in the embodiment of the present invention, the camera device can be a camera, and the camera can collect underwater image information in real time. Therefore, the underwater image information collected by the camera can be obtained, which can be multiple underwater images.

[0034] S200, respectively extracting depth features and echo intensity sensitive features of the underwater image information, wherein the echo intensity sensitive features at least include hard features and edge features.

[0035] Depth feature extraction and echo intensity sensitive feature extraction are performed on the above underwater image information respectively. It should be understood that the size of the depth feature needs to be reflected in the pixel value, and the echo intensity is stronger for hard, rough and edge places. Therefore, it is also necessary to extract the characteristics of such objects through images and reflect them through high and low pixel values.

[0036] It is necessary to explain here the imaging principle of side-scan sonar, which is inconsistent with the imaging principle of the camera.

[0037] First of all, the basic working principle of side-scan sonar is similar to that of side-looking radar. A transducer array is installed on each side of the side-scan sonar, which emits a short sound pulse. The sound wave propagates outward in the form of a spherical wave. When it hits the seabed or objects in the water, it will be scattered. The backscattered wave (also called echo) will return to the transducer along the original propagation route and be received by the transducer, and converted into a series of electrical pulses by the transducer. In general, the hard, rough, and convex seabed has a strong echo; the soft, smooth, and concave seabed has a weak echo. The obscured seabed does not produce an echo, and the farther the distance, the weaker the echo.

[0038] Arranging the received data of each transmission cycle vertically line by line and displaying them on the monitor will form a two-dimensional acoustic map of the seabed topography. The acoustic map plane and the seabed plane are mapped point by point, and the brightness of the acoustic map includes the characteristics of the seabed.

[0039] As the underwater sonar carrier continues to move, the sonar array continuously transmits, receives and processes as it moves forward, and the records are arranged line by line. The echo data returned from each transmission is displayed line by line on each scanning line of the display. The arrival time of each echo corresponds to the position of each point, that is, the pixel coordinate, and the amplitude of the echo corresponds to the brightness of each point, that is, the pixel grayscale value.

[0040] S300: Superimpose the image having the depth feature and the image having the echo intensity sensitive feature to obtain a first superimposed image.

[0041] The image extracted by the depth feature and the image extracted by the echo intensity sensitive feature are superimposed, that is, the depth feature and the echo intensity sensitive feature are superimposed and reflected on the first superimposed image.

[0042] S400: Superimpose a noise simulation feature on the first superimposed image to obtain a second superimposed image.

[0043] For various types of noise in the underwater environment, noise simulation features are obtained through training, and the noise simulation features are superimposed on the first superimposed image to obtain a more realistic second superimposed image.

[0044] S500: intercepting the second stacked image according to a preset side scan sonar resolution and a preset range to obtain a plurality of sonar pixel strips.

[0045] In an embodiment of the present invention, the second superimposed image is intercepted according to a preset side scan sonar resolution and a preset range, that is, the second superimposed image is intercepted according to a side scan sonar imaging method and various preset side scan sonar equipment parameters to obtain a plurality of sonar pixel strips.

[0046] S600: splice multiple sonar pixel strips to obtain a side-scan sonar simulation image.

[0047] The captured multiple sonar pixel strips are arranged and spliced ​​row by row to obtain a side-scan sonar simulation image.

[0048] In summary, the underwater side-scan sonar simulation image generation method provided by the present invention extracts depth features and echo intensity sensitive features from underwater image information collected by a camera device, and then performs noise simulation feature superposition again after superimposing the extracted features, and intercepts and splices the superimposed images according to a preset side-scan sonar resolution and a preset range to obtain a side-scan sonar simulation image. Compared with field-collected underwater data, this underwater side-scan sonar simulation image generation method can generate a large amount of data simply and efficiently, which can be used for the development and verification of various underwater algorithms, improve their efficiency and accuracy, improve the quality of underwater image data, greatly improve the problem of lack of underwater side-scan sonar data, provide strong visual support for underwater operations, improve the efficiency of underwater related activities, and ultimately achieve the purpose of improving the realism of side-scan sonar images.

[0049] In the embodiment of the present invention, the depth feature and echo intensity sensitive feature of the underwater image information are extracted respectively, such as Figure 2 As shown, it includes: S210, extracting depth features in underwater image information of every two adjacent camera devices through a depth filter; in the embodiment of the present invention, a Gaussian distributed depth sensor can be used to extract depth information of adjacent camera data.

[0050] Specifically, the depth features in the underwater image information of each two adjacent cameras are extracted by a depth filter, such as Figure 3 As shown, it includes: S211, when the depth filter receives underwater image information, the projection point position of each pixel in the underwater image information is determined according to extreme search and block matching, wherein the depths of all pixels in the underwater image information satisfy the Gaussian distribution; in an embodiment of the present invention, it is assumed that the depths of all pixels satisfy a certain Gaussian distribution, and when new data is input, the projection point position is determined by extreme search and block matching.

[0051] like Figure 4a The following is a schematic diagram of extreme search and block matching. The depth value is obtained by extreme search. Assuming that the same point is observed from different perspectives, the resulting geometric relationship is as follows: Figure 4a As shown, the camera on the left observes a certain pixel , from another perspective (right side), the projection of this line segment also forms a line on the image plane, which is the epipolar line. Given the camera motion under the two perspectives, this epipolar line is also determined. Next, block matching is used to determine which point on the epipolar line corresponds to the pixel .

[0052] Get Pixels around , and also took a lot of small blocks on the polar line. The surrounding small blocks are recorded as , put the polar line Small block , . Use NCC (Normalized Cross Correlation) to calculate the correlation between two small patches :

[0053] ,

[0054] in, , respectively, indicating that the pixels are in small blocks or small pieces The coordinates in .

[0055] calculate and every The similarity measure of , the ones with the highest similarity can be considered as the same point.

[0056] When calculating the depth of a pixel, the depth restored by a single match has errors, and it is necessary to rely on the fusion of multiple depth measurements to restore the depth value with smaller errors.

[0057] Specifically, assuming that the depth value follows a Gaussian distribution, let the depth of a pixel be obey:

[0058] ,

[0059] in, Indicates depth The average expected value of Indicates depth The variance of .

[0060] Whenever a new depth is observed The data arrives, assuming that this data is also Gaussian distributed:

[0061] ,

[0062] in, Indicates depth The average expected value of Indicates depth The variance of .

[0063] You can use the new data to update the original The product of two Gaussian distributions is still a Gaussian distribution. Let The distribution of ,in, Represents the depth after fusion The average expected value of Represents the depth after fusion The variance of , then according to the product of Gaussian distribution, we have:

[0064] ,

[0065] ,

[0066] Therefore, the projection position of a certain pixel of the reference frame in the current frame is determined through limit search and block matching.

[0067] S212, calculating the depth value and uncertainty of each pixel point according to the geometric relationship formed by the projection point position of each pixel point in two adjacent underwater image information.

[0068] In the embodiment of the present invention, Figure 4b As shown, consider a certain limit search and find Corresponding point, thus observing The depth value is considered The corresponding three-dimensional point is . So we can remember for , Camera Pan , Recorded as , and let the lower two angles of this triangle be , . Polar Line There is a pixel-sized error in become ,and become , and remember the upper angle as ,right Perturbing one pixel will make Produce a change , since the camera focal length is ,then:

[0069] ,

[0070] therefore, , .

[0071] So, by the law of sines, The size of can be found:

[0072] ,

[0073] The depth uncertainty caused by the uncertainty of a single pixel is thus determined, and the error :

[0074] ,

[0075] The depth observations at the latest moment are continuously used to fuse the optimal depth value at the previous moment until the depth converges. When the uncertainty is less than a certain threshold, the depth data is considered to have converged, that is, the depth value is determined.

[0076] S213, updating the distribution of the depth value of each pixel in the underwater image information, and when the uncertainty of the pixel is less than a preset uncertainty threshold, determining that the depth value of the pixel has converged.

[0077] S214: normalize the depth values ​​of the pixels in the underwater image information to obtain the depth features of each underwater image information.

[0078] After obtaining the image depth, the image is normalized and its depth value is normalized to a pixel value between [0, 255].

[0079] S220, extracting all texture features from the underwater image information according to the local binary pattern, and obtaining texture features characterizing echo intensity sensitive features from the all texture features.

[0080] In the embodiment of the present invention, LBP (Local Binary Pattern) is specifically used to extract texture from an image.

[0081] Specifically, all texture features are extracted from the underwater image information according to the local binary pattern, and texture features representing echo intensity sensitive features are obtained from all the texture features, such as Figure 5 As shown, it includes: S221, dividing the underwater image information into multiple detection windows, and dividing each detection window into multiple detection areas; in an embodiment of the present invention, the underwater image can be divided into detection windows, and then each detection window can be divided into multiple detection areas. For example, the detection window can be divided into 16×16 small detection areas.

[0082] S222. For any pixel point in each detection area, compare its pixel value with the pixel values ​​of multiple pixel points in the annular neighborhood to obtain a feature value of each pixel point in each detection area.

[0083] It should be understood that the pixel value of any pixel point in each detection area is compared with the pixel values ​​of multiple pixel points in its annular neighborhood.

[0084] For example, the pixel value of the pixel point in the current detection area is compared with 8 points (or multiple points in the annular neighborhood) in the detection area in a clockwise or counterclockwise manner.

[0085] Specifically, for any pixel point in each detection area, its pixel value is compared with the pixel values ​​of multiple pixel points in the annular neighborhood to obtain the characteristic value of each pixel point in each detection area, including: if the pixel value of the pixel point in the detection area is greater than the pixel value of the pixel point in the annular neighborhood, the pixel value of the pixel point in the annular neighborhood is assigned to 1; if the pixel value of the pixel point in the detection area is not greater than the pixel value of the pixel point in the annular neighborhood, the pixel value of the pixel point in the annular neighborhood is assigned to 0; the binary pixel value in the annular neighborhood obtained for each pixel point in each detection area is converted into decimal to obtain the characteristic value of each pixel point in each monitoring area.

[0086] It should be understood that if the central pixel value is greater than that of the neighboring point, the neighboring point is assigned a value of 1, otherwise it is assigned a value of 0, so that each point will obtain an 8-bit binary number (usually converted to a decimal number).

[0087] S223, calculating the frequency of occurrence of the characteristic value of each pixel in each detection area, and obtaining a histogram of the detection area.

[0088] S224, normalizing the histogram of each detection area, and connecting the normalized histograms to obtain all texture features of the underwater image information.

[0089] S225. Extract texture features whose feature value occurrence frequency is greater than a preset frequency threshold from all the texture features to obtain texture features that characterize echo intensity sensitivity features.

[0090] In the embodiment of the present invention, the histogram of each detection area is calculated, that is, the frequency of occurrence of each number (assuming it is a decimal number) (that is, a binary sequence of statistics on whether each pixel point is larger than the points in the neighborhood), and then the histogram is normalized. Finally, the statistical histograms of each detection area are connected to obtain the LBP texture features of the entire image.

[0091] It should be understood that based on the working principle of side-scan sonar, the echo reflected by characteristic areas such as hard, rough, and raised bottom is strong, which is reflected in the higher pixel value of the final generated side-scan sonar image. Therefore, the pixel value of the above-mentioned extracted characteristic area is increased on the basis of the depth map to highlight the characteristic area.

[0092] In an embodiment of the present invention, in order to make the final side-scan sonar simulation image more realistic, the underwater side-scan sonar simulation image generation method also includes, before the step of superimposing the noise simulation feature on the first superimposed image: generating the noise simulation feature according to a cyclic generative adversarial network.

[0093] There will be a lot of noise in the side-scan sonar imaging process. The embodiment of the present invention uses CycleGan to transfer the style of the image, hoping to continuously learn the noise style of the real sonar based on the depth map, and simulate the generation of side-scan sonar images with various noises.

[0094] Specifically, generating noise simulation features according to a cyclic generative adversarial network includes: 1) providing depth features of the underwater image information and real side-scan sonar images as training data sets.

[0095] It should be noted that CycleGan only needs data from two domains, and does not require a strict correspondence. The embodiment of the present invention uses images with depth features and real side-scan sonar images as training sets for training, thereby obtaining target data. CycleGan is different from other style transfer algorithms in that other algorithms require paired data, while CycleGan can also be trained using unpaired data, and can achieve style conversion between different images. Moreover, the sample data does not need to be paired, and there is no need to establish a one-to-one mapping between the training data between the source domain and the target domain to achieve this migration. Because paired data sets of underwater sonar images and underwater depth images of the same location are very difficult to obtain, the embodiment of the present invention only needs to provide sonar images and underwater depth image data sets to achieve style conversion.

[0096] 2) A deep feature training dataset of underwater image information is input as input and a real side-scan sonar image training dataset is output as input to a first generator, wherein the first generator is capable of converting the image style of the deep features of the underwater image information into the image style of the real side-scan sonar image.

[0097] Specifically, CycleGan contains two parts of mapping functions: and And the adversarial discriminator and ,in and Represent the two domains of the generated image with depth features and the real side scan sonar image respectively. The generator Will Domain-specific image style transfer Domain Image Style; Generator Will Image style transfer to the domain Domain image style, such as Figure 6 shown.

[0098] 3) A real side-scan sonar image training dataset is used as input and a deep feature training dataset of underwater image information is used as output to a second generator, wherein the second generator is capable of converting the image style of the real side-scan sonar image into the image style of the deep feature of the underwater image information.

[0099] 4) Using a first discriminator to judge the output result of the first generator to obtain a first judgment result, and using a second discriminator to judge the output result of the second generator to obtain a second judgment result.

[0100] 5) Constraining the first judgment result and the second judgment result according to a preset loss function to obtain noise simulation features.

[0101] In this embodiment of the present invention, the specific expression of the preset loss function is:

[0102] ,

[0103] in:

[0104]

[0105] ,

[0106]

[0107] .

[0108] in, Represents resistance to loss, represents the expectation of the distribution function, and They represent the generated depth image and the real side scan sonar image domains respectively. Representation Domain The samples within Representation Domain The samples within and Respectively represent samples and distribution. Indicates true The samples in the discriminator The closer the score is to 1, the more real the discriminator thinks the sample is. For the generator according to Generated with Samples from the same distribution. is the score obtained by the discriminator based on the generated samples, if The more fake the generated samples are, the Rating The closer it is to 0, the The closer it is to 1.

[0109] The goal of the embodiment of the present invention is to make the pictures generated by the generator as realistic as possible, that is, the pictures generated by the generator should deceive the discriminator as much as possible. and For the generator and The closer the probability of the generated data being judged as true is to 1, the better; the purpose of the discriminator is to judge the probability distribution of real data samples as true as possible, and to judge the images generated by the generator as false as possible.

[0110] ,

[0111] in, represents the cycle consistency loss, domain Images in , through one of the generators Generate image , as another generator The image generated by the input , as close as possible to the original image Similarly, the domain Images in , through one of the generators Generate image , as another generator The image generated by the input , as close as possible to the original image near.

[0112] Preventing Generators and Contradictory, that is, the two generators can transform back after generating data.

[0113] By continuously performing gradient optimization on the model through the above loss function, the model can well learn the noise style of the real side-scan sonar, and thus generate a side-scan sonar image with very realistic noise based on the depth map generated above.

[0114] In an embodiment of the present invention, the second stacked image is intercepted according to a preset side scan sonar resolution and a preset range to obtain a plurality of sonar pixel strips, such as Figure 7 As shown, it includes: S510, calculating the original resolution of the image with the depth feature according to the navigation information.

[0115] S520: Process the image with the depth feature according to a preset vertical track resolution and a track resolution according to a bilinear interpolation method, so that the resolution of the image with the depth feature meets the preset resolution.

[0116] S530, intercepting the image with the depth feature that meets the preset resolution requirement according to the preset range, the preset vertical track resolution and the track resolution, to obtain a plurality of sonar pixel strips.

[0117] In the embodiment of the present invention, since the side scan sonar can only scan one line of information per cycle, the vertical track resolution and track resolution of the side scan sonar are selected, the original resolution of the depth map is calculated through the navigation information, and the image is processed according to the selected vertical track resolution and track resolution through bilinear interpolation, so that the image resolution meets the requirements, then the range of the simulated side scan sonar is selected, and finally the image is intercepted according to the range and vertical track resolution and track resolution to generate sonar pixel strips, and each row of pixel strips is arranged and spliced ​​row by row, and finally a simulated side scan sonar image is generated. Figure 8a The figure shows the schematic diagram of strip cutting and splicing. Figure 8b This is a schematic diagram of the actual scene after strip cutting and splicing.

[0118] In summary, the underwater side-scan sonar simulation image generation method provided by the present invention realizes the function of generating high-quality side-scan sonar simulation data based on the camera and navigation data; by introducing a depth filter, the depth information between adjacent image frames is calculated, which greatly improves the accuracy of the final simulated side-scan sonar image; the feature texture of the image is extracted by LBP, and the texture characteristics are reflected by the pixel value, which is close to the actual side-scan sonar imaging principle, so that the generated simulation image is more realistic and has richer features; by using CycleGan to realize the mapping of the depth map to the side-scan sonar, the noise characteristics and types of the side-scan sonar image in the real environment are continuously learned, and the depth map is converted into a simulation image with more realistic noise, which is more similar to the actual situation, and CycleGan does not require a one-to-one paired data set, which greatly solves the problem that the underwater corresponding data set is difficult to obtain, reduces the processing difficulty, and improves the simulation efficiency and speed; the resolution of the image is changed by bilinear interpolation, and a certain range of pixel strips are intercepted according to the set parameters, and then spliced, which is very similar to the side-scan sonar imaging principle, and a realistic side-scan sonar image with characteristics is generated.

[0119] As another embodiment of the present invention, an underwater side-scan sonar simulation image generation device 100 is provided for implementing the underwater side-scan sonar simulation image generation method described above, wherein: Fig. 9As shown, it includes: an acquisition module 110, used to acquire underwater image information collected by a camera device; a feature extraction module 120, used to respectively extract depth features and echo intensity sensitive features of the underwater image information, wherein the echo intensity sensitive features at least include hard features and edge features; a first feature superposition module 130, used to superimpose an image having the depth features with an image having the echo intensity sensitive features to obtain a first superimposed image; a second feature superposition module 140, used to superimpose a noise simulation feature on the first superimposed image to obtain a second superimposed image; a capture module 150, used to capture the second superimposed image according to a preset side scan sonar resolution and a preset range to obtain a plurality of sonar pixel strips; a splicing module 160, used to splice a plurality of sonar pixel strips to obtain a side scan sonar simulation image.

[0120] The underwater side-scan sonar simulation image generation device provided by the present invention extracts depth features and echo intensity sensitive features from underwater image information collected by a camera device, and then performs noise simulation feature superposition again after superimposing the extracted features, and intercepts and splices the superimposed images according to a preset side-scan sonar resolution and a preset range to obtain a side-scan sonar simulation image. Compared with field-collected underwater data, this underwater side-scan sonar simulation image generation device can simply and efficiently generate a large amount of data for the development and verification of various underwater algorithms, improve their efficiency and accuracy, improve the quality of underwater image data, greatly improve the problem of lack of underwater side-scan sonar data, provide strong visual support for underwater operations, improve the efficiency of underwater related activities, and ultimately achieve the purpose of improving the fidelity of side-scan sonar images.

[0121] The specific working principle of the underwater side-scan sonar simulation image generation device provided by the present invention can be referred to the description of the underwater side-scan sonar simulation image generation method mentioned above, which will not be repeated here.

[0122] As another embodiment of the present invention, there is provided an underwater side-scan sonar simulation image generation system 10, wherein: Fig.10 As shown, it includes: a camera device 200 and the underwater side-scan sonar simulation image generating device 100 mentioned above, the camera device 200 is communicatively connected with the underwater side-scan sonar simulation image generating device 100, and the camera device 200 is used to collect underwater image information; the underwater side-scan sonar simulation image generating device 100 is used to extract depth features and echo intensity sensitive features according to the underwater image information, and superimpose the extracted features and then perform noise simulation feature superposition again, and intercept and splice the superimposed image according to the preset side-scan sonar resolution and preset range to obtain a side-scan sonar simulation image.

[0123] It should be understood that in order to obtain navigation information, the underwater side-scan sonar simulation image generation system in the embodiment of the present invention also includes a navigation device. Specifically, the navigation device can be implemented by a combined navigation controller integrating one or more of INS (Inertial Navigation System), DVL (Doppler Velocity Log) and GPS (Global Positioning System).

[0124] The imaging device 200 in the embodiment of the present invention may specifically be a camera.

[0125] The underwater side-scan sonar simulation image generation system provided by the present invention extracts depth features and echo intensity sensitive features from underwater image information collected by a camera device, and then performs noise simulation feature superposition again after superimposing the extracted features, and intercepts and splices the superimposed images according to a preset side-scan sonar resolution and a preset range to obtain a side-scan sonar simulation image. Compared with field-collected underwater data, this underwater side-scan sonar simulation image generation system can simply and efficiently generate a large amount of data for the development and verification of various underwater algorithms, improve their efficiency and accuracy, improve the quality of underwater image data, greatly improve the problem of lack of underwater side-scan sonar data, provide strong visual support for underwater operations, improve the efficiency of underwater related activities, and ultimately achieve the purpose of improving the fidelity of side-scan sonar images.

[0126] The specific working principle of the underwater side-scan sonar simulation image generation system provided by the present invention can be referred to the description of the underwater side-scan sonar simulation image generation method in the previous text, and will not be repeated here.

[0127] It is to be understood that the above embodiments are merely exemplary embodiments used to illustrate the principles of the present invention, but the present invention is not limited thereto. For those of ordinary skill in the art, various modifications and improvements can be made without departing from the spirit and essence of the present invention, and these modifications and improvements are also considered to be within the scope of protection of the present invention.

Claims

1. A method for generating underwater side-scan sonar simulation images, characterized in that: include: Acquire underwater image information collected by a camera device; Respectively extracting depth features and echo intensity sensitive features of the underwater image information, wherein the echo intensity sensitive features at least include hard features and edge features; Superimposing the image having the depth feature and the image having the echo intensity sensitive feature to obtain a first superimposed image; superimposing a noise simulation feature on the first superimposed image to obtain a second superimposed image; intercepting the second stacked image according to a preset side scan sonar resolution and a preset range to obtain a plurality of sonar pixel strips; A side-scan sonar simulation image is obtained by splicing multiple sonar pixel strips; The depth feature and echo intensity sensitive feature of the underwater image information are extracted respectively, including: Extracting depth features in underwater image information of every two adjacent cameras through a depth filter, wherein for all pixel points in the underwater image information whose depths satisfy Gaussian distribution, obtaining depth data satisfying convergence of all pixel points, and obtaining depth features in the underwater image information; Extracting all texture features from the underwater image information according to the local binary pattern, and obtaining texture features characterizing echo intensity sensitive features from all texture features, wherein all texture features are obtained by dividing the underwater image information by a detection window, and texture features characterizing echo intensity sensitive features are obtained based on the frequency of occurrence of feature values ​​extracted from all texture features; The method further includes the following steps before the step of superimposing the noise simulation feature on the first superimposed image: Generate noise-like features based on a recurrent generative adversarial network; Generate noise simulation features based on the cycle generative adversarial network, including: Providing depth features of the underwater image information and real side-scan sonar images as training data sets; Inputting a deep feature training dataset of underwater image information as input and a real side-scan sonar image training dataset as output to a first generator, wherein the first generator is capable of converting an image style of the deep feature of the underwater image information into an image style of the real side-scan sonar image; Inputting a real side-scan sonar image training dataset as input and a deep feature training dataset of underwater image information as output to a second generator, wherein the second generator is capable of converting the image style of the real side-scan sonar image into the image style of the deep feature of the underwater image information; The output result of the first generator is judged by a first discriminator to obtain a first judgment result, and the output result of the second generator is judged by a second discriminator to obtain a second judgment result; Constraining the first judgment result and the second judgment result according to a preset loss function to obtain a noise simulation feature; The second stacked image is intercepted according to a preset side scan sonar resolution and a preset range to obtain a plurality of sonar pixel strips, including: Calculating an original resolution of the image having the depth feature according to the navigation information; Processing the image with the depth feature according to a preset vertical track resolution and a track resolution according to a bilinear interpolation method, so that the resolution of the image with the depth feature meets the preset resolution; According to the preset range, the preset vertical track resolution and the track resolution, an image with the depth feature that meets the preset resolution requirement is intercepted to obtain a plurality of sonar pixel strips.

2. The underwater side-scan sonar simulation image generation method according to claim 1, characterized in that: The depth feature in the underwater image information of each two adjacent cameras is extracted by a depth filter, including: When the depth filter receives underwater image information, the projection point position of each pixel in the underwater image information is determined according to limit search and block matching, wherein the depths of all pixels in the underwater image information satisfy Gaussian distribution; The depth value and uncertainty of each pixel are calculated based on the geometric relationship formed by the projection point position of each pixel in two adjacent underwater image information; Updating the distribution of the depth value of each pixel in the underwater image information, and determining that the depth value of the pixel has converged when the uncertainty of the pixel is less than a preset uncertainty threshold; The depth values ​​of the pixels in the underwater image information are normalized to obtain the depth features of each underwater image information.

3. The underwater side-scan sonar simulation image generation method according to claim 1, characterized in that: Extracting all texture features from the underwater image information according to the local binary pattern, and obtaining texture features characterizing echo intensity sensitive features from all texture features, including: Dividing the underwater image information into a plurality of detection windows, and dividing each detection window into a plurality of detection areas; For any pixel point in each detection area, its pixel value is compared with the pixel values ​​of multiple pixel points in the annular neighborhood to obtain the characteristic value of each pixel point in each detection area; Calculate the frequency of occurrence of the characteristic value of each pixel in each detection area to obtain the histogram of the detection area; The histogram of each detection area is normalized, and the normalized histograms are connected to obtain all the texture features of the underwater image information; Texture features whose feature value occurrence frequency is greater than a preset frequency threshold are extracted from all the texture features to obtain texture features that characterize echo intensity sensitivity features.

4. The underwater side-scan sonar simulation image generation method according to claim 3, characterized in that: For any pixel point in each detection area, its pixel value is compared with the pixel values ​​of multiple pixels in the annular neighborhood to obtain the feature value of each pixel point in each detection area, including: If the pixel value of the pixel point in the detection area is greater than the pixel value of the pixel point in the annular neighborhood, the pixel value of the pixel point in the annular neighborhood is assigned to 1; If the pixel value of the pixel point in the detection area is not greater than the pixel value of the pixel point in the annular neighborhood, the pixel value of the pixel point in the annular neighborhood is assigned to 0; The binary pixel values ​​in the annular neighborhood obtained by each pixel point in each detection area are converted into decimal to obtain the characteristic value of each pixel point in each monitoring area.

5. An underwater side-scan sonar simulation image generation device, used to implement the underwater side-scan sonar simulation image generation method according to any one of claims 1 to 4, characterized in that: include: An acquisition module, used to acquire underwater image information collected by a camera device; A feature extraction module, used to extract depth features and echo intensity sensitive features of the underwater image information respectively, wherein the echo intensity sensitive features at least include hard features and edge features; A first feature superposition module, used for superimposing the image having the depth feature with the image having the echo intensity sensitive feature to obtain a first superimposed image; A second feature superposition module, used for superimposing a noise simulation feature on the first superimposed image to obtain a second superimposed image; An interception module, used for intercepting the second stacked image according to a preset side scan sonar resolution and a preset range to obtain a plurality of sonar pixel strips; The stitching module is used to stitch multiple sonar pixel strips to obtain a side-scan sonar simulation image.

6. An underwater side-scan sonar simulation image generation system, characterized in that: include: A camera device and the underwater side-scan sonar simulation image generation device according to claim 5, wherein the camera device is communicatively connected to the underwater side-scan sonar simulation image generation device, The camera device is used to collect underwater image information; The underwater side-scan sonar simulation image generation device is used to extract depth features and echo intensity sensitive features according to the underwater image information, and to perform noise simulation feature superposition again after superimposing the extracted features, and to intercept and splice the superimposed image according to a preset side-scan sonar resolution and a preset range to obtain a side-scan sonar simulation image.

Images