A sonar image three-dimensional reconstruction method based on target shadow and gray scale

Abstract

The application provides a sonar image three-dimensional reconstruction method based on target shadow and gray scale, which comprises the following steps: Step 1, converting the original sonar image into a gray scale image; Step 2, carrying out denoising treatment on the image; Step 3, carrying out shadow edge calibration; Step 4, segmenting a target region; Step 5, eliminating small target regions; Step 6, solving the pitch angle of the target region; and Step 7, reconstructing the target region. The application uses the multi-beam forward-looking sonar imaging principle as a research object, analyzes the transmission, reflection and reception process of sound waves, realizes a reverse reasoning using sound propagation, and obtains the three-dimensional reconstruction of the target object in the sonar image.

Description

TECHNICAL FIELD

[0001] The present application relates to the underwater acoustic three-dimensional reconstruction technical field, specifically to a sonar image three-dimensional reconstruction method based on target shadow and gray scale. BACKGROUND

[0002] In order to overcome the problems of low imaging quality, low environmental perception ability, difficulty in detecting underwater structures, and difficulty in quantitative evaluation of optical imaging under complex water conditions, acoustic three-dimensional reconstruction can effectively solve the problems. The acoustic detection method can reflect the three-dimensional geometry of the internal object, estimate the geometric size of the object, and determine the relative position of the defect according to the position of the robot in the pipeline. The above data can effectively guarantee the information accuracy, comprehensiveness and reliability of underwater structure detection.

[0003] The purpose of denoising and segmentation processing of sonar images is to realize more accurate three-dimensional reconstruction, and accurate three-dimensional reconstruction provides qualitative and quantitative basis for health diagnosis of underwater structures or topographic survey of underwater environment. In view of the above requirements, firstly, the image forming process of multi-beam sonar is modeled, and according to the generation principle of defect target and shadow area gray scale in the image, it is concluded that the key to restoring the height of the defect is to solve the pitch angle of the target defect surface. Secondly, combining the defect shadow obtained by edge calibration and the gray scale of the target area after segmentation, the pitch angle is solved by spatial geometric analysis. Finally, combining the gray scale of the target area after segmentation and the pitch angle of the defect surface, the three-dimensional reconstruction of the defect target in the sonar image is realized.

[0004] Therefore, there is an urgent need for a sonar image three-dimensional reconstruction method based on target shadow and gray scale to solve the above problems. SUMMARY

[0005] The purpose of the present application is to overcome the above-mentioned shortcomings, a sonar image three-dimensional reconstruction method based on target shadow and gray scale to solve the problems proposed in the background art.

[0006] In order to solve the above technical problems, the technical scheme adopted by the present application is: a sonar image three-dimensional reconstruction method based on target shadow and gray scale, which comprises the following steps:

[0007] Step 1, converting the original sonar image into a gray scale image;

[0008] Step 2, denoising the image;

[0009] Step 3, shadow edge calibration;

[0010] Step 4, segmenting the target area;

[0011] Step 5, removing small target areas;

[0012] Step 6, solving the pitch angle of the target region;

[0013] Step 7, reconstructing the target region.

[0014] Preferably, the specific method of converting the original sonar image into a gray-scale image in Step 1 is to use the component method, that is, to use any one channel component of the three channels of the color image, and here the R channel is selected as the gray-scale value of the gray-scale image.

[0015] Preferably, the denoising processing of the image in Step 2 uses a guided filter with strong filtering capability. The guided filter is a local linear image filter, which has the advantages of smoothing image details and edge preservation, can keep more image details near the edge, and the single pixel k exists in the linear relationship within the window center, and the linear function can be expressed as a complex function.

[0016] Preferably, the core of the guided filter is to obtain the linear model of each pixel, that is:

[0017] ;

[0018] In the above formula, is the pixel value of the guided image, is the pixel value of the output image, is a square window with a radius of , , is the linear coefficient in the window;

[0019] In order to solve the linear coefficient , , a regularization parameter is needed to prevent , that is:

[0020] ;

[0021] wherein, wherein is the noise value in the image, is the input noise-containing image, that is:

[0022] ;

[0023] ;

[0024] wherein, and respectively represent the mean and variance of the guided image in the window , and is Number of pixels in the middle is the average value of the pixels in the window ;

[0025] As an input in the filter constraint, the output on the model is ; In the process of n (noise or redundant texture) is an unwanted component of the input , which can be removed, and its formula can be expressed as: .

[0026] Preferably, the specific steps of shadow edge calibration in Step 3 are as follows:

[0027] Step 3.1, the edge of the shadow area starts, each pixel point is the shadow trailing edge, mark the pixel point;

[0028] Step 3.2, the first pixel point of the target area in each column in front of the column where the marked shadow trailing edge is located is the target trailing edge of the region, mark the target trailing edge while retaining the shadow trailing edge;

[0029] Step 3.3, continue to mark forward, when passing through the last pixel of the target area, the boundary pixel of the background area is in front of it, mark the boundary pixel, and the region is the front edge of the target;

[0030] Step 3.4, three edges can be found, which are complete calibration, and they are retained.

[0031] Preferably, the SLIC method is used in Step 4 to segment the sonar image into superpixels, and the SLIC method specifically implements the following steps:

[0032] Step 4.1, cluster center initialization: after setting the number of superpixels, the cluster centers in the image are uniformly distributed, assuming that the number of pixel points in the image is N, the image is pre-segmented into K superpixels of the same size by pre-segmentation, at this time each superpixel contains N / K pixel points, and the distance between each adjacent cluster center can be approximated as ;

[0033] Step 4.2, the cluster center can be reselected within the range of neighborhood, first calculate the gradient value of all pixel points in the neighborhood, and place the cluster center at the minimum gradient in the neighborhood, which effectively prevents the cluster center from being placed on the boundary of the outline with large gradient, and does not affect the subsequent clustering effect;

[0034] Step4.3, assigning class labels to each pixel point in the neighborhood of each cluster center: setting the size of the superpixel to ;

[0035] Step4.4, searching each pixel point found by distance measurement, including color distance and spatial distance, the distance of each pixel point to the cluster center needs to be calculated;

[0036] Step4.5, iterative optimization is performed on the above calculation process until the error converges.

[0037] Preferably, the small target region is removed in Step 5 using a connected domain marking method, which can divide the points meeting the conditions into the same connected domain, and separate the points not adjacent to other target points, thereby achieving the effect of removing small target points.

[0038] Preferably, the specific method for solving the normal vector of the planar point of the target region in Step 6 is to first perform three-dimensional reconstruction of the defect, which needs to represent the defect in three-dimensional space and two-dimensional plane, and establish a space and plane coordinate system respectively.

[0039] Take the point P in the space coordinate system, its corresponding three-dimensional space coordinates are , and the corresponding polar coordinates are , the angle between P and the XY plane is the pitch angle of the sonar The conversion relationship between the three-dimensional space coordinates and the polar coordinates can be expressed as:

[0040] ;

[0041] The corresponding polar coordinate expression is:

[0042] ;

[0043] Where R represents the straight-line distance from point P to the center of the sonar.

[0044] From the above formula, the height expression of point P in the space scene is known, where the R values of each point are known, how to solve the pitch angle of the target surface point is the difficulty of three-dimensional reconstruction, assuming that the defect target is located in a flat pipeline, the positional relationship between the pipeline and the target needs to be determined, define the pipeline bottom plane normal vector , assuming that the P three-dimensional feature position point ball coordinate P on the flat pipeline bottom can be expressed as:

[0045] ;

[0046] Since the normal vector has the uncertainty of arbitrary scaling, The value of is a negative constant, let be a fixed value. Substituting the coordinates of P and n into the above equation and rearranging, we obtain the following equation:

[0047] ;

[0048] Point O in the three-dimensional coordinate system is the location of the sonar wave emission center, and its height relative to the bottom of the tube is [missing information]. h Draw a line OH perpendicular to the bottom plane of the tube, with the foot of the perpendicular at H. The intersection of the negative Z-axis and the bottom plane of the tube is at K. It can be obtained by analyzing data measured by sensors in the original sonar software, assuming Then we can obtain the following formula:

[0049] ;

[0050] The spatial coordinates P at the bottom of the pipe satisfy the following conditions: Therefore, the relationship between point K and point n can be:

[0051] ;

[0052] The window is set with the minimum tilt angle of the sonar at the beginning and the maximum tilt angle at the farthest point. Assuming... To represent the number of points at the minimum detection range on a sonar image, the coordinates of the points can be written as follows:

[0053] ;

[0054] ;

[0055] Substituting the coordinates of each point, we get...

[0056] ;

[0057] It can be abbreviated as In the form of, A and X can be represented as:

[0058] ;

[0059] ;

[0060] Will get After simplification, the normal vector n can be expressed as:

[0061] ;

[0062] in, for The linear regression method is used in the above solving process, and the least square method is used to fit the coordinate point plane by using the pixel point of the pipeline bottom plane point. The plane equation coefficient is the coordinate of the pixel point normal vector. The pixel in the region can be calculated to obtain the normal vector of the pipeline bottom plane point.

[0063] Preferably, the pitch angle in Step 6 is solved by The specific method for restoring the target height Z is that there is a point P in the spatial coordinate system, and the corresponding three-dimensional space coordinates of the point P are The corresponding polar coordinates are The conversion relationship between the three-dimensional space coordinates and the polar coordinates can be expressed as:

[0064] ;

[0065] The corresponding conversion is the polar coordinates, which are shown in the following formula:

[0066] ;

[0067] Wherein, R represents the straight line distance from point P to the center of the sonar;

[0068] The point P is projected on the Y-axis point of the sonar imaging XY plane , the included angle is , and the included angle between the P point and the XY plane is the pitch angle of the sonar , the projection of the P point on the XY plane intersects at a point, and the coordinates of the point are When the pitch angle of the P point relative to the sonar is zero, the corresponding relationship between the point in the sonar coordinate system and the projection point is shown in the following formula:

[0069] ;

[0070] Considering that the pitch angle is not zero, the conversion relationship between the projection point P coordinates in the XY projection plane coordinate system of the sonar and the coordinates of the space point P is shown in the following formula, that is:

[0071] ;

[0072] The sonar relies on emitting and receiving sound waves in the water environment. When the sound waves reach the target surface and the background environment of the target, the sound waves are reflected. The reflected sound waves have different intensities. After the sonar receives and processes these sound waves, a two-dimensional image of the underwater environment can be obtained. The sonar imaging process only retains the distance and azimuth angle, and loses the pitch angle for restoring the target height. Through derivation, the height of the target object can be expressed as:

[0073] ;

[0074] wherein, Z is the height of the target, it can be seen that the size of the pitch angle affects the height information of the target, and the pitch angle is solved by restoring the height of the target.

[0075] Preferably, the specific method of reconstructing the target area in Step 7 is to obtain the pitch angle of the target by using the geometric relationship the pitch angle of the trailing edge of the shadow area formed by the target occlusion The same, the point of the target surface needs to be obtained by calculation, and the pitch angle of the shadow area formed by the target occlusion can be used is solved, the pitch angle of the target surface point is solved by linear interpolation of the target front edge and the highest point pitch angle, after the target pitch angle is solved, the height of the target can be restored by using the target pitch angle, thereby completing the reconstruction of the underwater scene.

[0076] The present application has the following beneficial effects:

[0077] 1. The present application uses the imaging principle of multi-beam forward-looking sonar as the research object, analyzes the sending, reflection and receiving process of sound waves, realizes a kind of reverse reasoning of sound propagation, and obtains the three-dimensional reconstruction of the target in the sonar image;

[0078] 2. The present application adopts the method of guided filtering, effectively solves the problems of low resolution, beam-to-beam crosstalk and complex environmental interference of sonar imaging, and retains the effective details and edge features in the image;

[0079] 3. The present application uses SLIC algorithm to segment the superpixel of the denoised image, uses the method of connected domain to mark the superpixel, and better separates the unnecessary small target area;

[0080] 4. The present application combines traditional filtering method and modern image processing algorithm, enhances the spatial resolution of sonar imaging system, and reduces the influence of noise and artifacts on image quality;

[0081] 5. The present application proposes a filtering method based on connected domain for the common artifact problem in sonar image, which can effectively remove the artifact interference in the image, so as to improve the detection accuracy of the target and the image quality;

[0082] 6. The present application uses the imaging principle of sonar image, solves the pitch angle of the target surface of the defective target in the sonar image by linear interpolation of the pitch angle of the target front edge and the highest point, and the pitch angle diagram combines the gray scale information and the corresponding segmentation area, which can more accurately reconstruct the scene in three dimensions. BRIEF DESCRIPTION OF DRAWINGS

[0083] Figure 1 is the method step flow chart of the present application;

[0084] Figure 2 Schematic diagram of 4-neighborhood in the connected domain marking method of the application;

[0085] Figure 3 Schematic diagram of point taking in the spherical coordinate system and the spatial coordinate system of the application;

[0086] Figure 4 Schematic diagram of the center position of sound wave emission of the sonar of the application;

[0087] Figure 5 Schematic diagram of the pitch angle of the application;

[0088] Figure 6 Schematic diagram of the sonar data in the embodiment of the application;

[0089] Figure 7 Schematic diagram of the pipe bottom echo simulation in the embodiment of the application;

[0090] Figure 8 Schematic diagram of the sonar imaging in the embodiment of the application;

[0091] Figure 9 Schematic diagram of the window being smaller than the detection distance in the embodiment of the application;

[0092] Figure 10 Schematic diagram of the window being equal to the detection distance in the embodiment of the application;

[0093] Figure 11 Schematic diagram of the window being greater than the detection distance in the embodiment of the application;

[0094] Figure 12 Schematic diagram of the sonar incident path in the embodiment of the application;

[0095] Figure 13 Schematic diagram of the sonar imaging corresponding to the incident point position in the embodiment of the application. DETAILED DESCRIPTION

[0096] The application will be further described below in combination with the drawings and embodiments:

[0097] Referring to Figures 1 to 5 A sonar image three-dimensional reconstruction method based on target shadow and gray scale, which comprises the following steps:

[0098] Step 1, converting the original sonar image into a gray scale image;

[0099] Step 2, denoising the image;

[0100] Step 3, performing shadow edge calibration;

[0101] Step4, segmenting the target region;

[0102] Step5, removing small target regions;

[0103] Step6, solving the pitch angle of the target region;

[0104] Step7, reconstructing the target region. Blueprint provides the source code of the Oculus SDK software, through which the storage structure of the sonar data can be found, and through the data structure, the project header and the sonar transmission data structure of the stored data can be obtained.

[0105] Preferably, the specific method for converting the original sonar image to a grayscale image in Step 1 is to use the component method, that is, to use any one channel component of the three channels of the color image, and here the R channel is selected as the grayscale value of the grayscale image.

[0106] Preferably, the image denoising processing in Step 2 uses a guided filter with strong filtering capability. Guided filter is a local linear image filter, which has the advantages of smoothing image details and edge preservation, can keep more image details near the edge, and the single pixel k exists in the linear relationship within the window center. The linear function can be expressed as a complex function.

[0107] Preferably, the core of the guided filter is to obtain the linear model of each pixel, that is:

[0108] ;

[0109] In the above formula is the pixel value of the guided image, is the pixel value of the output image, is a square window with a radius of , , is the linear coefficient in the window;

[0110] In order to solve the linear coefficient , , a regularization parameter is introduced to prevent , that is:

[0111] ;

[0112] wherein , wherein is the noise value in the image, is the input noise-containing image, that is:

[0113] ;

[0114] ;

[0115] wherein, and represent the mean and variance of the windowed directed image, is the number of pixels in the window is the mean of the in the window ;

[0116] As an input in the filter constraint, the output of the process of , n (noise or redundant texture) is an unwanted component of the input , which can be removed, and its formula can be expressed as: .

[0117] Preferably, the specific steps of carrying out shadow edge calibration in Step 3 are as follows:

[0118] Step 3.1, the edge of the shadow area starts, each pixel point is the shadow trailing edge, and the pixel point is marked;

[0119] Step 3.2, the first pixel point of the target area in each column in front of each column where the marked shadow trailing edge is located is the target trailing edge of the region, and the target trailing edge is marked while the shadow trailing edge is retained;

[0120] Step 3.3, continue to mark forward, when passing through the last pixel of the target area, the boundary pixel of the background area is marked, which is the leading edge of the target region;

[0121] Step 3.4, three edges can be found, which are complete calibration, and they are retained.

[0122] Preferably, the SLIC method is used to carry out superpixel segmentation on the sonar image in Step 4, and the specific implementation steps of the SLIC method are as follows:

[0123] Step 4.1, cluster center initialization: after the number of superpixels is set, the cluster centers in the image are uniformly distributed, assuming that the number of pixel points in the image is N, the image is pre-segmented into K superpixels of the same size by pre-segmentation, at this time each superpixel contains N / K pixel points, and the distance between each adjacent cluster center can be approximated as ;

[0124] Step 4.2, in the range of​​ In the neighborhood, the cluster center can reselect the cluster center, first calculate the gradient value of all pixel points in the neighborhood, and place the cluster center in the place with the smallest gradient in the neighborhood, which effectively prevents the boundary of the contour with large gradient from placing the cluster center, and does not affect the subsequent clustering effect;

[0125] Step4.3, assign a class label to each pixel point in the neighborhood of each cluster center: set the size of the superpixel to ;

[0126] Step4.4, search each pixel point by distance measurement, including color distance and spatial distance, and calculate the distance from the cluster center;

[0127] Step4.5, iterative optimization is performed on the above calculation process until the error converges.

[0128] Compared with other superpixel segmentation, SLIC algorithm generates more compact and neat superpixels, and the pixel feature expression between neighborhoods is easier. This process can effectively solve the problems of multiple connectivity, too small superpixel size, and single pixel being cut into multiple discontinuous superpixels.

[0129] Preferably, the small target region in Step 5 is removed by using a connected domain marking method, which can divide the points meeting the conditions into the same connected domain, and separate the points not adjacent to other target points, thereby achieving the effect of removing small target points. The method of processing two-dimensional sonar image usually includes 4-neighbor connected marking and 8-channel connected marking algorithm, for the center pixel, the four pixels in the 4-neighborhood of the pixel , , , , , for the center pixel, the eight pixels in the 8-neighborhood of the pixel , , , , , , , , If the positions of two pixel points satisfy the above relationship and have the same gray value, it is said that the two pixel points belong to the same connected region, and if A and B are connected, and B and C are connected, then A and C are connected, and thus the pixels are connected one by one to form a relatively adjacent region with the same characteristic points, and the target points after the threshold screening are marked as 1, and the other points are marked as 0.

[0130] Preferably, the specific method for solving the normal vector of the planar point of the target region in Step 6 is to first perform three-dimensional reconstruction on the defect, and the defect needs to be represented in three-dimensional space and two-dimensional plane, and a space coordinate system and a plane coordinate system are established respectively.

[0131] Take a point P in the space coordinate system, and the corresponding three-dimensional space coordinates of the point P are The corresponding polar coordinates are The angle between the point P and the XY plane is the pitch angle of the sonar The conversion relationship between the three-dimensional space coordinates and the polar coordinates can be expressed as:

[0132] ;

[0133] The corresponding polar coordinate expression is:

[0134] ;

[0135] Wherein, R represents the straight-line distance from the point P to the center of the sonar;

[0136] From the above formula, the height expression of the point P in the space scene is obtained, wherein the values of the points R are known, and how to solve the pitch angle of the target surface point is a difficulty in three-dimensional reconstruction, assuming that the defect target is located in a flat pipeline, the positional relationship between the pipeline and the target needs to be determined, and the normal vector of the bottom plane of the pipeline is defined as Assuming that the three-dimensional feature position point ball coordinates P of the P on the flat pipeline bottom can be expressed as:

[0137] ;

[0138] Since the normal vector has the uncertainty of arbitrary scaling, The value of n is a negative constant, and let Substitute the coordinates of P and n into the above formula and arrange to obtain the formula, that is:

[0139] ;

[0140] The point O of the three-dimensional coordinate system is the position of the sonar sound wave emission center, and the height of the point O relative to the pipe bottom is h, the vertical foot is H, and the intersection of the negative half of the Z axis and the pipe bottom plane is K, The data measured by the sensor in the original sonar software can be analyzed to obtain Then the following formula can be obtained:

[0141] ;

[0142] The spatial coordinates P of the pipe bottom satisfy Therefore, the relationship between the K point and the n point can be obtained:

[0143] ;

[0144] The start of the window is set to the minimum inclination angle of the sonar, and the farthest is the maximum inclination angle of the sonar, assuming The number of points on the minimum detection distance on the sonar image is

[0145] ;

[0146] ;

[0147] Substitute the coordinates of each point into the formula to obtain

[0148] ;

[0149] which can be written in the form of where A and X can be represented as:

[0150] ;

[0151] ;

[0152] will be obtained and the normal vector n can be represented as:

[0153] ;

[0154] where is The linear regression method is used in the above solving process, and the least squares method is used to fit the coordinate point plane using the pixel points of the pipe bottom plane points, and the equation of the plane is the coordinate of the normal vector of the pixel point. The normal vector of the pipe bottom plane point in the region can be obtained by calculation.

[0155] Preferably, the pitch angle in Step 6 is solved by The specific method for restoring the target height Z is that there is a point P in the spatial coordinate system, and the corresponding three-dimensional spatial coordinates are The corresponding polar coordinates are The transformation relationship between three-dimensional spatial coordinates and polar coordinates can be expressed as:

[0156] ;

[0157] The corresponding conversion to polar coordinates is shown in the following formula:

[0158] ;

[0159] in, R This represents the straight-line distance from point P to the center of the sonar.

[0160] Point P is the point where its projection onto the XY plane of sonar imaging intersects the Y-axis. The included angle is The angle between point P and the XY plane is the elevation angle of the sonar. The projections of point P onto the XY plane intersect at a single point, the coordinates of which are... When the elevation angle of point P relative to the sonar is zero, the correspondence between the point in the sonar coordinate system and the projected point is as follows:

[0161] ;

[0162] Consider pitch angle When the coordinates are not zero, the transformation relationship between the coordinates of the projection point P in the XY projection plane coordinate system of the sonar and the coordinate system of the spatial point P is as shown in the following formula:

[0163] ;

[0164] Sonar relies on emitting and receiving sound waves in an aqueous environment. When the sound waves reach the target surface and are reflected by the background environment, the reflected sound waves have different intensities. After receiving and processing these sound waves, the sonar can obtain a two-dimensional image of the underwater environment. The sonar imaging process only retains the range and azimuth angles, losing the elevation angle needed to recover the target's altitude. Through derivation, the height of the target object can be expressed as:

[0165] ;

[0166] in, Z Given the height of the target object, it can be seen that the pitch angle affects the target's height information. By solving for the pitch angle... Restore target altitude.

[0167] Preferably, the specific method for reconstructing the target region in Step 7 is to derive the target's pitch angle using geometric relationships. Pitch angle along the trailing edge of the shadow area formed by target occlusion The same, the point of the target surface needs to be obtained by calculation, the elevation angle of the shadow area formed by the target occlusion can be used The elevation angle of the target surface point is obtained by linear interpolation of the target front and the highest point elevation angle. After the target elevation angle is solved, the height of the target can be recovered using the target elevation angle, thereby completing the reconstruction of the underwater scene.

[0168] Embodiment 1

[0169] Referring to Figure 6 As shown, the Blueprint company provides Oculus SDK software source code, through which the storage structure of sonar data can be found, and through which the project header and sonar transmission data structure of the stored data can be obtained. The project header structure includes header type, project header size, identifier of data type contained, data type version, project creation time, compression type, compression pre-load size, size of blank area after load, and other information.

[0170] In the 1ping data transmitted by the sonar, the image data size is 122 bytes, of which the transmission message occupies 53 bytes. The transmission message mainly stores the sending pulse information, mainly including the message header, transmission mode, transmission rate, network transmission speed, gamma correction parameter, flag bit, range, time-varying gain, sound speed, and salinity information. The structure in the message header includes sonar ID, source device ID, target device ID, message identifier, message version, message load size, and other information.

[0171] The forward-looking multi-beam sonar is different from the single-beam sonar. It can transmit and receive reflected sound waves in multiple directions. The beam width of each beam in the horizontal and vertical directions can be represented as and respectively. The sonar can receive all reflected echoes with a certain intensity in the underwater scene within its azimuth angle range. These reflected sound waves mainly come from the pipe bottom plane and the defect surface. The sonar image can be represented as , , which represents the angle between a point on the defect target surface and the center point of the sonar, R , which represents the distance between the two points. The sonar image can represent the small reflection intensity change on the defect surface, i.e. the distance resolution and the beam width The angle range of this part is , the distance range is , and the azimuth angle between the point on the sonar original image data and the sonar can be obtained from the coordinates of the point , the sound wave range is determined according to the point coordinates , in order to obtain the elevation angle of the target surface , according to the analysis of the influencing factors of the echo intensity of the point on the sonar image in the sonar imaging process, the intensity model may be expressed as:

[0172]

[0173] From the above equation, it can be seen that the intensity of the acoustic echo is mainly related to the change in the pitch angle of the acoustic wave after scanning the point. When the change in the pitch angle is small, the area in the sonar image is dark, and vice versa.

[0174] Referring to Figure 7 , the relatively flat bottom of the drainage pipe network may have some impact on sonar imaging. In addition to the echoes generated by the target surface, the acoustic wave may also be reflected at the bottom of the pipe, causing the sonar to mistakenly regard it as the echo of the target area. In order to simulate this situation, the mirror image of the incident source O and the bottom plane of the pipe is simulated to simulate the echo of the pipe bottom; the scattering of the target surface can cause part of the acoustic wave to be incident on the pipe bottom, forming the echo of the pipe bottom plane. However, due to the absorption and scattering of the pipe bottom to the acoustic wave, the energy of the acoustic wave incident on the target surface is relatively low, so the echo interference caused by it can be ignored. In addition, due to the rapid energy attenuation of the acoustic wave in the reflection process, the energy of multiple reflections is weak, and the acoustic wave component scattered to the pipe bottom can be ignored.

[0175] Forward-looking sonar imaging is a mapping from 3-D space to 2-D, and the pitch angle will be lost in this process. If 2-D images are to be reconstructed in three dimensions, the lost pitch angle needs to be solved. The space inside the pipe is relatively narrow, and the sonar needs to be installed in a small sweep angle. This not only makes the sonar detection distance far, but also makes the protruding objects inside the pipe or the obstacles suspended in the water produce strong reflections due to the change in the pitch angle, which appears as a high-light area in the image. The sonar small sweep angle will cause a large shadow area behind the target, which is closely adjacent to the boundary of the target high-light area and is more obvious in the image.

[0176] In the internal environment of the drainage pipe network, the bottom of the pipe is a relatively flat area with little obvious height change, while the target surface has a high amplitude change. Under this condition, relative to the flat area of the pipe bottom, the echo intensity of the target surface is higher, which appears as a high-light area in the sonar image. On the flat pipe bottom, due to the certain height of the target, it will have a certain shielding effect on the pipe bottom area behind the target in the direction of the sonar incident. The pipe bottom area behind the target has little acoustic wave irradiation, and the acoustic wave reflected back to the sonar receiver is also less. This area appears in the sonar image with lower brightness than the flat pipe bottom area. Considering the case where an object with a certain height is placed on the flat pipe bottom, without considering the case where the pipe bottom is concave, according to the above analysis, the target, target shadow and background can be distinguished from the image according to the different light and dark performances of the target and background in the sonar image.

[0177] Referring to Figure 8 As shown in the figure, the sound wave incident point is O, the sound wave is incident to the pipe bottom plane AB and DE through the lower part of the pipe, this part of the sound wave is directly incident, but there is no obvious height fluctuation change, the echo intensity is low, the gray in the image is low, the target area is in BC, the shadow area CD part is not reached by the sound wave blocked by the target, the sonar reflects different light and dark areas in the image due to the different intensity of the received sound wave. The sound wave is directly incident to the BH plane, the plane where BH is located has a certain height difference, as known from the above, the greater the height difference, the brighter the brightness displayed in the sonar image, this area is the BP area in the figure, the brightness of this area is the highest; B and P respectively represent the front edge of the target and the trailing edge of the target; Because the height of the target will block the back area of the target, i.e. the PD area, this part is not irradiated by the sound wave, and there is no sound wave returning from this area, the smaller the intensity of the sound wave, the lower the brightness displayed in the image. The shadow PD part of the target, the D point is the trailing edge of the sonar target shadow, the gray of this part is lower than that of AB and DE, it is relatively easy to distinguish in the image, each pixel point of the original sonar image contains the distance of the point from the sonar and the azimuth angle data from the center of the sonar, the boundary of the target and the shadow also contains the relevant information of the target itself, so it is necessary to extract the edge of the target area in the sonar image.

[0178] The sonar software provided by Blueprint sonar company can adjust the size of the sonar window and range, the size adjustment of these parameters also has a certain influence on the sought pitch angle, which can be adjusted according to the actual situation in use, and the caliber of urban underground drainage pipe is relatively uniform, the maximum caliber of the drainage pipe used in the city is 3000mm, so the range of the sonar can be fixed; The minimum detection distance of the sonar is set by the height of the sonar device relative to the bottom of the drainage pipe and the detection inclination angle of the sonar device fixed on the carrier and the window value of the detection distance . Moderate adjustment of these two parameters can obtain the maximum range of the target in the detection scene and obtain a relatively complete sonar image; Figure 9 The adjustment limit of the two parameters described in the description affects the imaging area of the two-dimensional forward-looking sonar, assuming that the coordinates of the nearest spatial point that the sonar can detect in the horizontal direction are , the coordinates of the farthest point in the horizontal direction are , the vertical pitch angle of the sonar is , the corresponding detection distance is , the vertical pitch angle of the sonar is , and the corresponding detection distance is .

[0179] Referring to Figure 9 as shown, and The two cases, i.e. the minimum distance of the detected target area is less than the minimum detection distance set by the sonar itself, and the maximum distance of the detected target area is greater than the maximum detection distance set by the sonar itself, will affect the imaging of the sonar, and the front end area and the rear end area of the target will not be scanned by the sonar during the sonar process, resulting in a part of information being lost in the sonar detection process, so that only the middle part of the target is displayed in the sonar image of the defect target in the pipeline.

[0180] Referring to Figure 10 , when the two cases of and occur, in this case, whether it is the minimum detection distance set by the sonar or the maximum detection distance, they are equal to the minimum and maximum distances actually required to be detected, and the elevation angle of the point on the front edge of the sonar image area is exactly , and the elevation angle of the point on the rear edge of the image area is .

[0181] Referring to Figure 11 , when the two cases of and occur, i.e. the minimum detection distance set by the sonar is less than the minimum distance of the target area required to be detected, and the maximum detection distance set by the sonar is greater than the maximum distance of the target area actually detected; it can be known that if the window range set by the sonar is greater than the actual imaging area, there are some dark areas outside the imaging area in the sonar image, and these dark areas will appear in front and rear of the imaging area; the three-dimensional reconstruction based on the two-dimensional single frame multi-beam sonar image is not suitable for collecting data in this situation; the maximum elevation angle value and the minimum elevation angle value on the sonar image have corresponding points, and these points can be obtained by adjusting the window parameters of the sonar, and in actual application, the nearest and farthest of the detected target should satisfy the following formula:

[0182]

[0183]

[0184] In the case of knowing the pipe plane normal vector, the pitch angle value of any point of the defect in the pipe in the sonar image can be solved by the formula, and then the pitch angle map of the sonar image is obtained, and the sonar image gray value can be divided into three regions: the region with the highest gray value corresponds to the target area with a certain height in the pipe; the gray medium region corresponds to the bottom plane of the pipe; and the gray lowest region corresponds to the shadow area behind the target which is not directly incident by the sound wave due to the shielding of the target.

[0185] Referring to Figure 12 and Figure 13As shown, the pitch angle of each point on the bottom of the pipeline in the sonar image can be directly calculated by the formula, but the pitch angle of the highest point on the target surface cannot be directly obtained; therefore, the pitch angle relationship between the target and the shadow is considered to indirectly calculate the path of the sound wave when the pipeline bottom is flat and there is a target with a certain height when the two-dimensional forward-looking sonar is incident.

[0186] The target existing in the area where the sonar sound wave reaches, the local schematic diagram formed by the target in the sonar image, the point H on the target surface is the highest point on the target surface where the sound wave can reach, which represents the highest point of the sound wave incident at the pitch angle value, and when the slightly high pitch angle value is incident on the surface of the target, the sound wave is reflected over the target and directly incident on the D point on the pipeline bottom plane, and the sound wave with the pitch angle value is the boundary, and the area reached by the sound wave greater than the angle is the pipeline bottom plane, the DE area in the figure; it can be known that the point is the boundary between the target shadow and the imaging of the pipeline bottom, and the point corresponds to the H point in Figure 12 , and the D point is the boundary point of the shadow area, corresponding to Figure 13 in , it is assumed that the sound waves passing through points P and H are in the same plane, which is reflected in the sonar image as points , point and point , the three points are located on the same straight line, thereby establishing a spherical coordinate system, and the coordinates of points H and D are respectively , , it can be known that the pitch angle values of the two points are equal, that is .

[0187] The edge of the shadow area starts, and each pixel point is the trailing edge of the shadow, which is marked, and the first pixel point of each column in the target area in front of each column marked with the trailing edge of the shadow is the trailing edge of the target in that area, which is marked as the trailing edge of the target while retaining the trailing edge of the shadow. Continue to mark forward, and when the last pixel of the target area is passed, the boundary pixel of the background area is forward, which is marked, and the area is the front edge of the target. Three edges can be found, which are complete and retained.

[0188] The target trailing edge point of the same column of pixel points in the sonar image has the same pitch angle as the trailing edge point of the shadow on the same straight line, the pitch angle values of each point except the highest point on the target surface are solved, and the pitch angle of the front and rear edges of the target is estimated. This process can be simplified to calculate the pitch angle values of the front and rear edges of the target by linear interpolation, the pitch angle of the back of the target cannot be irradiated by the sonar, and the pitch angle values of each point remain unchanged.

[0189] The above embodiments are only preferred technical solutions of the present application, and should not be regarded as a limitation of the present application. The protection scope of the present application should be the technical solutions recited in the claims, including equivalent replacement solutions of the technical features recited in the claims. That is, equivalent replacement improvements within this scope are also within the protection scope of the present application.

Claims

1. A sonar image 3D reconstruction method based on target shadow and gray scale, characterized in that: It comprises the following steps: Step 1, the original sonar image is converted into a gray image; Step 2, the image is denoised; Step 3, the shadow edge is calibrated; Step 4, the target region is segmented, and the SLIC method is used for superpixel segmentation of the sonar image, and the SLIC method specifically comprises the following steps: Step 4.1, cluster center initialization: after the number of superpixels is set, the cluster centers in the image are uniformly distributed. Assuming that the number of pixel points in the image is N, the image is pre-segmented into K superpixels of the same size. At this time, the number of pixels contained in each superpixel is N / K, and the middle distance between each adjacent cluster center can be approximated as ; Step 4.2, in the range of In the neighborhood, the cluster center can be reselected. First, the gradient value of all pixel points in the neighborhood is calculated, and the cluster center is placed at the place with the minimum gradient in the neighborhood. In this way, the cluster center is effectively prevented from being placed at the boundary of the outline with a large gradient, which will not affect the subsequent clustering effect. Step 4.3, assigning class labels to each pixel point within the neighborhood of each cluster center: set the size of superpixels to ; Step 4.4, through distance measurement, including color distance and spatial distance, each pixel point searched needs to calculate its distance to the cluster center; Step 4.5, the above calculation process is iteratively optimized until the error converges; Step 5, small target regions are removed; Step 6, the pitch angle of the target region is solved, and the specific method for solving the normal vector of the planar point of the target region is that the defect is first reconstructed in three-dimensional space and two-dimensional plane, and spatial and plane coordinate systems are established respectively; Take the point P in the spatial coordinate system, and its corresponding three-dimensional space coordinates are The corresponding polar coordinates are The angle between point P and the XY plane is the sonar pitch angle The conversion relationship between three-dimensional space coordinates and polar coordinates can be expressed as: ； The corresponding polar coordinate expression is: ； Wherein, R represents the straight line distance from point P to the sonar center; The height expression of the point P in the space scene can be obtained from the above formula, wherein the height of each point is R The value is known, how to solve the target surface point pitch angle is a difficult point of three-dimensional reconstruction, assuming that the defect target is located in a flat pipeline, the positional relationship between the pipeline and the target needs to be determined, and the normal vector of the pipeline bottom plane is defined , assuming that the spherical coordinates P of the P three-dimensional feature position point on the flat pipeline bottom can be expressed as: ； Because the normal vector has the uncertainty of arbitrary scaling, is a negative constant value, let Substitute the coordinates of P and n into the above equation and organize it to get the equation, that is: ； Point O of the three-dimensional coordinate system is the position of the sonar sound wave emission center, whose height relative to the pipe bottom is h , OH is perpendicular to the pipe bottom plane through O, the foot is H, and K is the intersection of the negative half-axis of the Z-axis with the pipe bottom plane, The data measured by the sensor in the original sonar software can be obtained, assuming The following formula can be obtained: ； The spatial coordinates P located at the bottom of the pipe satisfy Thus the relationship between K and n points: ； The start of the window is set to the minimum tilt of the sonar, and the farthest point is set to the maximum tilt of the sonar, assuming is the number of points on the sonar image at the minimum detection distance, and the coordinates of the points can be written as follows: ； ； The coordinates of each point are substituted into, and ； may be written in the form of wherein A and X can be represented as follows, respectively: ； ； The obtained The normal vector n can be expressed as ； wherein is In the above solving process, a linear regression method is used. A pixel point of a bottom plane point of the pipeline is used to fit a coordinate point plane by using a least square method. Equation coefficients of the plane are coordinates of a normal vector of the pixel point. A normal vector of a bottom plane point of the pipeline can be obtained by calculation. Step 7, the target region is reconstructed.

2. The method of claim 1, wherein: The specific method for converting the original sonar image into a gray image in Step 1 is to use the component method, that is, to use any one channel component of the three channels of the color image, and here the R channel is selected as the gray value of the gray image.

3. The method of claim 1, wherein: The guided filter with strong filtering ability is used for denoising the image in Step 2. Guided filter is a kind of local linear image filter, which has the advantages of smoothing image details and edge preservation, can keep more image details near the edge, and single pixel k exists in the linear relationship within the window center. The linear function can be expressed as a complex function.

4. The method of claim 3, wherein: The core of the guided filter is to obtain the linear model of each pixel, that is: ； In the above formula is the pixel value of the guided image, is the pixel value of the output image, is a square window of radius , , is the linear coefficient in the window; To solve for the linear coefficients , , one needs to introduce a regularization parameter that prevents oversmoothing, i.e.: ； wherein wherein is a noise value in the image, is the input noisy image, i.e.: ； ； wherein, and are the mean and variance of the window of the guided image , is the number of pixels in the window is the mean of the window of the guided image ; As an input in filter constraints, output on model in the process, n， noise or redundant texture, input unwanted components can be removed, its formula can be expressed as: .

5. The method of claim 1, wherein: The specific steps for calibrating the shadow edge in Step 3 are as follows: Step 3.1, the shadow region edge starts, and each pixel point is the shadow trailing edge, which is marked; Step 3.2, the first pixel point of the target region in each column in front of the column where the marked shadow trailing edge is located is the target trailing edge of the region, which is marked but the shadow trailing edge is retained; Step 3.3, continue to mark forward, when passing through the last pixel of the target region, the boundary pixel of the background region is marked, which is the leading edge of the target region; Step 3.4, three edges are found, which are complete calibration, and they are retained.

6. The method of claim 1, wherein: The connected domain marking method is used to remove small target regions in Step 5, which can divide the points meeting the conditions into the same connected domain, separate the points not adjacent to other target points, and realize the function of removing small target points.

7. The method of claim 1, wherein: Solving the pitch angle in Step 6 The specific method of restoring the target height Z is that there is a point P in the spatial coordinate system, and the corresponding three-dimensional spatial coordinates of the point P are The corresponding polar coordinates are The conversion relationship between the three-dimensional spatial coordinates and the polar coordinates can be expressed as: ； The corresponding conversion to polar coordinates is shown in the following formula: ； wherein, R represents the straight-line distance from point P to the sonar center; Point P is the intersection of the sonar imaging XY plane projection and the Y axis , the included angle is , the included angle between point P and the XY plane is the sonar pitch angle , the projection of point P on the XY plane intersects at a point, and the coordinates of the point are When the size of the pitch angle of point P relative to the sonar is zero, the corresponding relationship between the point and the projection point in the sonar coordinate system is as follows: ； Considering the pitch angle In the case that the above-mentioned condition is not zero, the projection point P' coordinate of the sonar's XY projection plane coordinate system and the coordinate system conversion relationship of the space point P is shown in the following formula, that is: ； Sonar relies on the emission and reception of sound waves in the water environment, when the sound waves reach the target surface and the background environment where the target is located, the reflected sound waves have different intensities, and when the sonar receives and processes these sound waves, a two-dimensional image of the underwater environment can be obtained. The sonar imaging process only retains the distance and azimuth angle, and loses the recovery of the target height pitch angle The height of the target object can be represented as: ； wherein, Z is the height of the object, it can be seen that the size of the pitch angle affects the height information of the target, and the pitch angle is solved to restore the height of the target.

8. The method of claim 1, wherein: The specific method of reconstructing the target region in Step 7 is to obtain the pitch angle of the target by using geometric relationship The pitch angle of the trailing edge of the shadow region formed by the target occlusion The pitch angle of the shadow region formed by the target occlusion is calculated, and the pitch angle of the point on the target surface is calculated The pitch angle of the point on the target surface is calculated by linear interpolation of the pitch angle of the target leading edge and the highest point. After the pitch angle of the target is calculated, the height of the target is recovered using the pitch angle of the target, thereby completing the reconstruction of the underwater scene.

Images