Sonar data processing and image fusion method

Abstract

The invention discloses a sonar data processing and image fusion method, which belongs to the technical field of image processing, and comprises the following steps of: performing geometric correction and registration processing on forward-looking sonar data by using respective advantages of side-scan sonar data and forward-looking sonar data and taking the side-scan sonar data as a benchmark reference source, so that the imaging mode of the forward-looking sonar data is matched with that of the side-scan sonar; multiple frames of images of the forward-looking sonar are integrated through an image splicing algorithm, and a water bottom image of a complete area is constructed; and finally, the blind area of the side-scan sonar is compensated by using a splicing result, so that complete reconstruction of underwater topography information is realized. The method is easy and convenient to operate and wide in application prospect, blind area compensation of the side-scan sonar image blind area can be achieved, and a complete underwater image is obtained.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and particularly relates to a sonar data processing and image fusion method. Background Art

[0002] With the continuous exploitation and utilization of global resources, the strategic value of the ocean has become increasingly prominent and has become an important spatial carrier for human survival and development. The development and utilization of ocean resources are not only an important engine for promoting economic growth and achieving sustainable development, but also a key area for enhancing comprehensive national strength and safeguarding and expanding national interests. Especially for major ocean countries, its importance is self-evident. Based on this background, the observation of the true situation of the seabed has become the focus of international attention, and the development and breakthrough of technology will have a profound impact on the future global ocean strategic pattern.

[0003] Side-scan sonar and forward-looking sonar are the core equipment for ocean mapping and play an important role in seabed terrain detection and target recognition. The side-scan sonar has a wide detection range and high accuracy, and can detect seabed terrain and targets such as fish schools, but there is a blind area in the center of its image; the forward-looking sonar detects objects in front by obliquely emitting sound waves, which helps ships avoid obstacles and is especially suitable for underwater equipment such as submarines, but its detection range is narrow, and the target image is severely distorted due to oblique detection. Summary of the Invention

[0004] In order to solve the technical problems in the background art, the present invention aims to provide a method for processing side-scan sonar and forward-looking sonar data and image fusion, which integrates the two sonar systems and fuses the data, and can give full play to their complementary advantages. By mounting a side-scan sonar and a forward-looking sonar system on the same detection platform, on the basis of ensuring that the forward-looking sonar completes its established functions, the detection data of the forward-looking sonar is systematically corrected. Subsequently, using image matching and fusion algorithms in computer graphics, autonomous stitching of the forward-looking sonar image is achieved, and the detection blind area of the side-scan sonar is effectively filled.

[0005] In order to solve the technical problems, the technical solution of the present invention is:

[0006] A data processing and image fusion method, the method comprising:

[0007] Input the original side-scan sonar data, perform water bottom line tracking, slant range correction, horizontal gray level equalization, spatial smoothing filtering, geocoding and gap filling processing to obtain a restored complete side-scan sonar image, that is, a side-scan sonar image to be matched;

[0008] Input the original forward-looking sonar data. Based on beam mean filtering + growth rate calculation, detect the point with the largest slope change to determine the water bottom line. Use the geometric projection relationship to correct the distortion error of the forward-looking sonar, accurately calculate the true coordinates of the target point, and combine the inertial navigation data of the carrier to map the forward-looking sonar image to the actual geographic coordinate system. Then, use edge detection to extract the target mask, and use the Criminisi algorithm to fill in the missing areas. Perform image enhancement and noise reduction using adaptive Gamma correction + adaptive median filtering. Finally, use the image registration algorithm combined with spatial position information for registration to obtain the self-fusion image of the forward-looking sonar.

[0009] Register the side-scan sonar image to be matched with the self-fusion image of the forward-looking sonar, perform fusion through an improved wavelet transform method, calculate the matching degree based on regional energy to enhance the fusion effect, and combine the data of both in a unified coordinate system to output a high-quality fused image.

[0010] Furthermore, process the original side-scan sonar data, including:

[0011] Input and parse the original side-scan sonar data, extract the bottom echo data and position information, perform water bottom line tracking, and detect the water bottom line based on the echo intensity threshold method. Perform slant range correction, calculate the horizontal distance using geometric relationships, and correct the position of the water bottom line. Perform horizontal gray level equalization, use sliding window mean filtering and normalization to enhance the contrast. Perform spatial smoothing filtering, use non-local mean filtering to reduce noise. Perform geocoding and gap filling, calculate the geographic coordinates of each pixel based on the heading angle and sampling rate, and fill in the data missing areas. Output the complete side-scan sonar image, that is, the side-scan sonar image to be matched.

[0012] Furthermore, process the original side-scan sonar data, specifically including:

[0013] Water bottom line tracking of side-scan sonar data:

[0014] After obtaining the side-scan sonar data, first parse the side-scan sonar data, and its data packet extracts the bottom echo data and position information; the quantization formula for the echo intensity is:

[0015]

[0016] where E is the echo intensity, max and min are respectively the maximum and minimum values of the echo in a single Ping data. In this way, the data is quantized to [0, 255], and then each Ping is arranged longitudinally to generate the side-scan sonar waterfall image. Through an appropriate threshold T, in the pixel point sequence, find the first echo with an intensity I greater than the threshold, and this point is the first bottom echo. The distance from this point to the fish support is the slant range distance from the fish support to the bottom. The set of these points is the water bottom line.

[0017] Slant Range Correction of Sonar Data:

[0018] On the premise that the sound speed is assumed to be constant and the seabed topography is flat, a strict right triangle relationship is formed among the horizontal distance, the slant range, and the height from the towed fish to the seabed. Based on this, the corresponding horizontal distance can be calculated using geometric relationships. Finally, by performing a horizontal adjustment operation on the pixel positions, slant range correction is achieved, and the slant range r is calculated according to the sound speed and time:

[0019]

[0020] After obtaining the slant range, the horizontal distance is obtained according to the Pythagorean theorem, and each Ping data is corrected to complete the slant range correction. The formula is as follows:

[0021]

[0022] In the formula, d i : The horizontal distance of the side-scan sonar water bottom line, that is, the value obtained after slant range correction; c: The sound speed in water (m / s), t i : The two-way propagation time (s) of the echo signal, that is, the time from transmission to reception, h: The installation height (m) of the sonar, which is the vertical distance from the sensor to the seabed;

[0023] Horizontal Gray Scale Equalization:

[0024] Select 1 Ping of data, set a sliding window, define the mean filter kernel according to the size of the window, perform mean filtering on the pixel values within the window, and smooth the image by continuously moving the window. After completing the mean filtering, normalize the image, calculate the standard deviation of each point, and use the original pixel value of the image minus the quotient of the average value and the standard deviation, as shown in the following formula:

[0025]

[0026] Side-Scan Sonar Spatial Smoothing Filtering:

[0027] Perform filtering on the side-scan sonar image, use non-local means filtering NL-means to denoise the image, find the pixel points with similar neighborhood structures to the pixel point to be processed in the entire image, and obtain the filtering effect of the pixel point to be processed according to the weighted average of the similar pixel points;

[0028] Side-Scan Sonar Data Geocoding and Gap Filling:

[0029] In a coordinate system, for single Ping data, the geographical coordinates corresponding to the projection point directly below the fish tow are (X0, Y0). From the left and right sides of the ship, the scanning range of the sidescan sonar on one side is R, the sampling rate of each channel of the sonar is N, and the course azimuth angle is α. Since each Ping data is perpendicular to its course direction, the direction angle of the left side is θ = α - π / 2, and the right side has a direction angle difference of π from the left side, so the right side direction angle is θ = α + π / 2. Pi is the i-th pixel point of a certain Ping data, and its geographical coordinates are:

[0030] X i = X0 + R×cos(α ± π / 2)×i / N

[0031] Y i = Y0 + R×sin(α ± π / 2)×i / N (5).

[0032] Further, the determination of the water bottom line includes:

[0033] By analyzing the intensity of a certain beam of the forward-looking sonar, first perform mean filtering on it:

[0034]

[0035] Process the beam after mean value, and calculate its growth rate in sequence order. The formula is as follows:

[0036]

[0037] Since the signal intensity change from the water area to the bottom area is relatively significant and changes from dark to bright, which is manifested as a change in the signal slope, the bottom position can be determined by detecting the maximum value of the signal slope; the sampling points corresponding to the bottom area should have the largest slope change, and the set of these points can be recognized as the water bottom line.

[0038] Further, the correction of the distortion error of the forward-looking sonar using the geometric projection relationship includes:

[0039] For the position error problem existing in the forward-looking sonar image, use the system correction method of the geometric projection relationship for correction; first, based on the oblique forward detection mechanism of the forward-looking sonar, establish a spatial geometric model of sound wave propagation, and determine the spatial position relationship between the target point, the sonar, and the echo path; secondly, calculate the propagation distance according to the sound wave propagation time, and construct a spatial circle equation with the sonar position as the center; finally, by solving the intersection points of the spatial circle and the water bottom plane equation, accurately calculate the true spatial coordinates of the target point, thereby realizing the correction of the position error; the result of the correction is:

[0040]

[0041] Further, the coordinate transformation by combining the carrier position information includes:

[0042] After the position distortion correction of the forward-looking sonar image, the geocoding of its image can be started. Considering that the sonar device itself does not directly access the position information and relies on the position and motion data provided by the carrier's inertial navigation system for geodetic coordinate reduction, by combining the position information of the carrier, each pixel point in the sonar image is mapped to the actual geodetic coordinate system, so as to achieve the precise correspondence between the sonar image and the geographical space. The forward-looking sonar geocoding formula is:

[0043] X i = X0 + R·cos[π / 2 - (α + Δθ·i)] / N

[0044] Y i = Y0 + R·sin[π / 2 - (α + Δθ·i)] / N (11).

[0045] Further, the extraction of the target mask by edge detection and the filling of the missing area by the Criminisi algorithm includes:

[0046] Detect the target in the image. Accurately detecting the target can clearly define the range that needs to be corrected for the shadow area, separating the target area from the shadow area to be corrected, so as to ensure that the shadow area correction process is not affected by the target pixel values. After detecting the target mask, start to extract the shadow area. First, enhance the contrast between the shadow area and other areas, then roughly extract the area through edge detection, and use morphological operations and connected region analysis to obtain the shadow mask, that is, the shadow mask, and use the Criminisi algorithm to obtain the repaired image.

[0047] Further, the forward-looking sonar image is processed by means of adaptive Gamma correction and adaptive median filtering to obtain the enhanced + noise-reduced forward-looking sonar image.

[0048] Further, the registration by using the image registration algorithm in combination with the spatial position information includes:

[0049] The SuperPoint + LightGlue combination method is used for feature detection and matching. For the area where the target features are not obvious, the previously geocoded position information is adopted to match the geographical position of the image. After the preliminary registration, the affine transformation is performed on the image to obtain the fully registered image.

[0050] Further, the registration of the side-scan sonar image to be matched and the forward-looking sonar self-fusion image includes:

[0051] After the images to be fused are registered, they are mapped into a unified coordinate system and directly fused; in view of the characteristics of sonar images, the wavelet transform fusion is improved. By selecting the regional energy of the images after wavelet decomposition and calculating the regional matching degree, the formula is as follows:

[0052] E = ∑ Δx∈J,Δy∈K [I(x + Δx, y + Δy)] 2 (12)

[0053] M ed (x, y) = M edian [I(x + Δx, y + Δy)] (13)

[0054]

[0055] The wavelet fusion rule is optimized through the matching degree, and the matching degree is judged to select an appropriate calculation method to realize the fusion of high-frequency components.

[0056] Compared with the prior art, the advantages of the present invention are as follows:

[0057] The present invention uses image processing methods and technologies to realize the fusion of side-scan sonar and forward-looking sonar images, achieving the complementary advantages between sonars.

[0058] The present invention processes side-scan sonar data, corrects and geocodes the data, and truly reflects the bottom characteristics.

[0059] The present invention processes forward-looking sonar data. In view of the imaging characteristics of the forward-looking sonar, it is corrected to be closer to the side-scan sonar imaging mode, making the quality of the fused image higher. Brief Description of the Drawings

[0060] Figure 1 - The single Ping data diagram of the amplitude threshold method of the present invention;

[0061] Figure 2 - The water bottom line tracking result diagram of the present invention;

[0062] Figure 3 - The pixel distribution diagram of the 150th beam of the forward-looking sonar of the present invention;

[0063] Figure 4 - The pixel distribution diagram of the 150th beam of the forward-looking sonar after equalization of the present invention;

[0064] Figure 5 - The slope distribution diagram of the 150th beam after difference of the present invention;

[0065] Figure 6 - The shadow area correction flow chart of the present invention;

[0066] Figure 7 1. The Gamma value of the present invention is 1.617 enhanced result graph;

[0067] Figure 8 2. The adaptive median filtering result graph of the present invention;

[0068] Figure 9 3. The differential registration flowchart of the present invention;

[0069] Figure 10 4. The wavelet transform image fusion flowchart of the present invention;

[0070] Figure 11 5. The overall technical roadmap of the present invention;

[0071] Figure 12 6. The Criminisi algorithm flowchart;

[0072] Figure 13 7. The SuperPoint algorithm flowchart;

[0073] Figure 14 8. The LightGlue algorithm flowchart;

[0074] Figure 15 9. The fusion result graph;

[0075] Figure 16 10. The side-scan sonar (SSS) data processing graph;

[0076] Figure 17 11. The forward-looking sonar (FLS) data processing graph. Detailed implementation manners

[0077] The following describes the specific implementation manners of the present invention in combination with embodiments:

[0078] It should be noted that the structures, ratios, sizes, etc. shown in this specification are only used to cooperate with the content disclosed in the specification for those skilled in this technology to understand and read, and are not used to limit the implementation conditions of the present invention. Any modification of the structure, change of the ratio relationship or adjustment of the size, without affecting the effects that the present invention can produce and the purposes that can be achieved, should still fall within the scope covered by the technical content disclosed in the present invention.

[0079] At the same time, the terms such as "upper", "lower", "left", "right", "middle" and "one" cited in this specification are only for the convenience of clear narration, and are not used to limit the scope of implementation of the present invention. The change or adjustment of their relative relationships, without substantial change in the technical content, should also be regarded as the scope of implementation of the present invention.

[0080] Embodiment 1:

[0081] AsFigure 11 As shown in the figure, the present invention provides a sonar data processing and image fusion method. This method utilizes the respective advantages of side-scan sonar and forward-looking sonar data. Taking the side-scan sonar data as the reference source, geometric correction and registration processing are performed on the forward-looking sonar data to make its imaging mode match that of the side-scan sonar. The multi-frame images of the forward-looking sonar are integrated through an image stitching algorithm to construct a complete underwater image of the area. Finally, the blind area of the side-scan sonar is filled using the stitching result to achieve the complete reconstruction of the underwater terrain information. As Figure 16 shown, first, aiming at the problems existing in the process of heterogeneous image fusion of side-scan sonar, water bottom line tracking, slant range correction, spatial filtering, and geocoding are performed on the side-scan sonar data. Secondly, as Figure 17 shown, based on the imaging mechanism and image features of the forward-looking sonar, first, the water bottom line is tracked using the difference method to correct the position distortion of the forward-looking sonar image. Secondly, to improve the consistency of the imaging features between the forward-looking sonar image and the side-scan sonar, a shadow area detection method combining Canny edge detection and morphological operations is proposed, and the detected shadow area is effectively filled using the Criminisi algorithm. Finally, after geocoding the forward-looking sonar image, image enhancement and denoising processing are performed. Finally, registration is performed using an image registration algorithm combined with spatial position information, and wavelet transform is selected for image fusion. The specific steps are as follows:

[0082] Step 1: Geocoding of side-scan sonar

[0083] Considering that the side-scan sonar data is displayed in the form of a waterfall diagram, which does not contain position information, geocoding is performed on the data and gap filling is carried out to restore the real seabed situation.

[0084] Step 2: Forward-looking sonar data processing

[0085] After obtaining the forward-looking sonar data, the data needs to be parsed according to the agreed format to obtain the parsed data of the forward-looking sonar. After obtaining the original detection data of the forward-looking sonar, first, the original data needs to be parsed according to the pre-defined data protocol. Subsequently, based on the physical detection principle of the sonar, the conversion of the spatial coordinate system of the data is performed. On this basis, the tracking of the water bottom line and the elimination of water body data are realized. Finally, to ensure the spatial accuracy of the data, position distortion correction is also required, and data geocoding is completed at the same time.

[0086] Step 3: Forward-looking sonar image processing

[0087] To meet the purpose of fusing with side-scan sonar images and filling in the blind areas, it is necessary to perform shadow area correction on the forward-looking sonar. The shadow area and target area are detected through Canny, and the detected shadow area is filled in the blind area by the Criminisi algorithm, so as to reduce the impact of the shadow area of the forward-looking sonar image on the fusion quality. Then, enhancement and filtering operations are performed on the forward-looking sonar image to obtain the forward-looking sonar image to be registered and fused.

[0088] Step 4: Image registration

[0089] Image registration is the basis for completing image fusion. The SuperPoint+LightGlue combination method is used for feature detection and matching. For areas with unclear target features, the previously geocoded position information is adopted to perform geographical location matching on the image. After preliminary registration, an affine transformation is performed on the image to complete the registration.

[0090] Step 5: Image fusion

[0091] After the images to be fused are registered, they will be mapped to a unified coordinate system. At this time, the geometric position differences of the images have been basically eliminated, and image fusion can be directly performed. Wavelet transform with regional energy selection and optimized fusion rules is used for image fusion.

[0092] In Step 1, before geocoding, the position reduction of the fish towing needs to be carried out. It is necessary to perform conversion according to the hull coordinate system VFS and the geographical coordinate system GRF, and then obtain the real fish towing coordinates.

[0093] In Step 1, the coordinates of each pixel point of each Ping data are calculated, and the side-scan sonar image after geocoding can be obtained through the formula. However, because the longitudinal and transverse sampling rates of the side-scan sonar are inconsistent, there may be gaps between each Ping data in the encoded image. This gap is more obvious on the outer side of the bend along the track line, which affects the quality of the bottom image and is also not conducive to subsequent image stitching work. Therefore, it is necessary to fill the gaps. The methods include but are not limited to the scan filling method.

[0094] In Step 2, according to the characteristics of the forward-looking sonar image, the water bottom line is tracked by the sequence difference method. After obtaining the seabed distance, its spatial position distortion is corrected to facilitate geocoding.

[0095] In Step 3, since the shadow areas of the forward-looking sonar and the side-scan sonar are different, without correction, it will affect the quality of the fused image. Therefore, it is necessary to detect the shadow area and the target by combining Canny with morphological filtering, and then repair the shadow area through the Criminisi algorithm.

[0096] In step 4, differential registration is performed according to the number of features in the image. First, the image is registered using SuperPoint + LightGlue. For images with less than 10 feature pairs, geographic location registration is performed. After the initial registration is completed, the RANSAC algorithm is used to match the feature pairs, and then an affine transformation is performed to obtain a fully registered image.

[0097] In step 5, the region energy selected for image fusion with the optimized fusion rule of wavelet transform is an improvement on the wavelet transform fusion method, making it more suitable for sonar image fusion.

[0098] Embodiment 2:

[0099] This Embodiment 2 is a specific expansion of Embodiment 1. This embodiment provides a sonar data processing and image fusion method, and the specific steps are as follows:

[0100] 1. Side-scan sonar data water bottom line tracking

[0101] After obtaining the side-scan sonar data, first, the side-scan sonar data needs to be parsed to extract the water bottom echo data and position information from its data packet. The quantization formula for the echo intensity is:

[0102]

[0103] where E is the echo intensity, max and min are respectively the maximum and minimum values of the echo in a single Ping data. In this way, the data is quantized to [0, 255], and then each Ping is arranged vertically to generate a side-scan sonar waterfall image. By using an appropriate threshold T, in the pixel point sequence, the first echo with an intensity I greater than the threshold is found, and this point is the first water bottom echo. As Figure 1 shown, the distance from this point to the fish sled is the slant distance from the fish sled to the water bottom. The set of these points is the water bottom line. The water bottom line tracking result is as Figure 2 shown.

[0104] 2. Sonar data slant distance correction

[0105] On the premise of assuming that the sound speed is constant and the water bottom terrain is flat, a strict right triangle relationship is formed among the horizontal distance, the slant distance, and the height from the fish sled to the water bottom. Based on this, using geometric relationships, the corresponding horizontal distance can be calculated. Finally, by performing a horizontal adjustment operation on the pixel point positions, slant distance correction is achieved. The slant distance r is calculated according to the sound speed and time:

[0106]

[0107] After obtaining the slant distance, the horizontal distance is obtained according to the Pythagorean theorem, and each Ping data is corrected to complete the slant distance correction. The formula is as follows:

[0108]

[0109] In the formula, it is the slant range of the water bottom line traced in Step 1 and is calculated according to the side-scan sonar parameters.

[0110] 3. Lateral gray level equalization

[0111] Select 1Ping data and set a sliding window with a suitable size. The size of the window should be selected according to the actual situation. If the window is too large, although the image can be smoothed, the equalization is not thorough and the gray level of the image is overly uneven. On the contrary, if the window is too small, it will affect the contour of the target and lose the details of the image. Define the mean filter kernel according to the size of the window, perform mean filtering on the pixel values within the window, and smooth the image by continuously moving the window. After completing the mean filtering, normalize the image, calculate the standard deviation of each point, and divide the difference between the original pixel value of the image and the average value by the standard deviation, so that the pixel values are more concentrated in a certain range while ensuring the relative size relationship, thereby enhancing the contrast, as shown in the following formula:

[0112]

[0113] 4. Side-scan sonar spatial smoothing filtering

[0114] Filtering the side-scan sonar image is crucial in image processing work to reduce the noise effect and improve the image quality. Nonlocal Means (NL-means) is a non-linear image denoising algorithm. This algorithm is based on the assumption that pixels in the image have similar structures and uses the global information of the image to denoise the image. It does not only consider the neighborhood information of pixel points like traditional local filtering algorithms, but searches for pixel points with similar neighborhood structures to the pixel point to be processed in the entire image, and obtains the filtering effect of the pixel point to be processed according to the weighted average of these similar pixel points.

[0115] 5. Side-scan sonar data geocoding and gap filling

[0116] In the coordinate system, for single Ping data, the geographical coordinates corresponding to the projection point directly below the fish tow are (X0, Y0). Looking from the port and starboard sides, the scan width of the side-scan sonar on one side is R, the sampling rate of each channel of the sonar is N, and the course azimuth angle is α. Since each Ping data is perpendicular to its course direction, the direction angle of the port side can be obtained as θ = α - π / 2, and the direction angle of the starboard side differs from that of the port side by π, so the direction angle of the starboard side is θ = α + π / 2. Pi is the i-th pixel point of a certain Ping data, and its geographical coordinates are:

[0117] X i = X0 + R × cos(α ± π / 2) × i / N

[0118] Y i = Y0 + R×sin(α ± π / 2)×i / N (19)

[0119] 6. Forward-looking sonar data water bottom line tracking

[0120] By analyzing the intensity of a certain beam of the forward-looking sonar, as Figure 3 shown.

[0121] First, perform mean filtering on it:

[0122]

[0123] The processed beam after mean value is as Figure 4 shown;

[0124] Calculate its growth rate according to the sequence order, and the formula is as follows

[0125]

[0126] The slope distribution of the 150th beam after differentiation, as Figure 5 shown.

[0127] Since the signal intensity changes significantly from the water area to the bottom area and changes from dark to bright, which is manifested as the change of the signal slope, the bottom position can be determined by detecting the maximum value of the signal slope. The sampling points corresponding to the bottom area should have the largest slope change, and the set of these points can be identified as the water bottom line.

[0128] 7. Forward-looking sonar data position correction

[0129] Aiming at the position error problem existing in the forward-looking sonar image, a system correction method using geometric projection relationship is used for correction. The main steps of this method are divided into three key steps: First, based on the oblique forward detection mechanism of the forward-looking sonar, establish a spatial geometric model of sound wave propagation to determine the spatial position relationship between the target point, the sonar and the echo path;

[0130]

[0131] Second, calculate the propagation distance according to the sound wave propagation time, and construct a spatial circle equation with the sonar position as the center;

[0132]

[0133] Finally, by solving the intersection points of the spatial circle and the bottom plane equation, accurately calculate the true spatial coordinates of the target point, so as to realize the correction of the position error. The result of the correction is:

[0134]

[0135] 8. Forward-looking sonar geocoding

[0136] After correcting the position distortion of the forward-looking sonar image, the geocoding of the image can be started. Since the sonar device itself does not directly access the position information, it is necessary to rely on the position and motion data provided by the inertial navigation system of the carrier for geodetic coordinate reduction. By combining the position information of the carrier, each pixel point in the sonar image can be mapped to the actual geodetic coordinate system, thus realizing the accurate correspondence between the sonar image and the geographical space. Similar to the side-scan sonar geocoding, the forward-looking sonar geocoding formula is:

[0137] X i = X0 + R·cos[π / 2 - (α + Δθ·i)] / N

[0138] Y i = Y0 + R·sin[π / 2 - (α + Δθ·i)] / N (26)

[0139] 9. Shadow area correction based on edge detection and Criminisi

[0140] The shadow area in the sonar image refers to the underwater area where the sound wave is blocked by the target and fails to reach. This area shows a state of information loss in the image, seriously affecting the integrity and accuracy of the image. Moreover, the forward-looking sonar shadow area will affect both the feature matching and fusion of the forward-looking sonar image and the side-scan sonar image. Correcting the shadow area of the forward-looking sonar image is a key link to improve the accuracy of feature matching and the quality of image fusion of the two. In the process of shadow area correction, an important factor is to avoid the interference of the pixel values of the target on the correction result. The target is usually a brighter area with a contour in the image, and its pixel values are different from the surrounding background. If the target is not effectively distinguished during the shadow area correction, the target pixel values may be wrongly included in the correction calculation, resulting in distortion of the corrected image and unable to truly reflect the actual situation of the underwater bottom.

[0141] Therefore, in order to achieve accurate shadow area correction, the primary task is to detect the target in the image. Accurately detecting the target can clearly define the range where the shadow area correction is needed, separating the target area from the shadow area to be corrected, thus ensuring that the shadow area correction process is not affected by the target pixel values.

[0142] After obtaining the target mask, start to extract the shadow area. First, enhance the contrast between the shadow area and other areas, then roughly extract the area through edge detection, and obtain the shadow mask by using morphological operations and connected region analysis to get the shadow mask. Use the Criminisi algorithm such as Figure 12 to obtain the repaired image, as shown in Figure 6 shown.

[0143] 10. Forward-looking Sonar Image Enhancement and Noise Reduction

[0144] The forward-looking sonar image is processed by means of adaptive Gamma correction and adaptive median filtering, significantly improving the clarity and realism of the image, with the overall effect being the best, as shown in Figure 7 and 8 .

[0145] 11. Differential Registration

[0146] During the image registration process, although algorithms such as SuperPoint+LightGlue in Figure 13 and 14 can effectively complete the registration task, in regions where the target features are sparse or the texture information is insufficient, such algorithms often encounter problems of matching failure. To ensure the registration accuracy, it is necessary to dynamically select an appropriate registration algorithm according to the image characteristics. When the image contains rich feature points, a registration method based on feature matching is preferred; while in regions with scarce features, it is necessary to switch to a registration algorithm based on geographical location, and its flowchart is as shown in Figure 9 .

[0147] 12. Image Fusion

[0148] After the images to be fused are registered, they will be mapped to a unified coordinate system. At this time, the geometric position differences of the images have been basically eliminated, and the images can be directly fused. In view of the characteristics of sonar images, the wavelet transform fusion is improved. By selecting the regional energy of the images after wavelet decomposition and calculating the regional matching degree, the flowchart is as shown in Figure 10 .

[0149] The formula is as follows:

[0150]

[0151] M ed (x,y) = M edian [I(x + Δx, y + Δy)] (28)

[0152]

[0153] The wavelet fusion rule is optimized through the matching degree, and the matching degree is judged to select an appropriate calculation method to achieve the fusion of high-frequency components. The final fusion result is as shown in Figure 15 .

[0154] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the present invention can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memory, CD-ROM, optical memory, etc.) that contain computer-usable program code.

[0155] The present invention is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to the embodiments of the present invention. It should be understood that each flow and / or block in the flowchart and / or block diagram, as well as the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices generate means for implementing the functions specified in one Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0156] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing devices to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured article including instruction means that implement the functions specified in one Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0157] These computer program instructions can also be loaded onto a computer or other programmable data processing devices, such that a series of operation steps are executed on the computer or other programmable devices to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable devices provide steps for implementing the functions specified in one Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0158] The preferred embodiments of the present invention have been described in detail above, but the present invention is not limited to the above embodiments. Within the knowledge scope of those of ordinary skill in the art, various changes can be made without departing from the purpose of the present invention.

[0159] Many other changes and modifications can be made without departing from the concept and scope of the present invention. It should be understood that the present invention is not limited to the specific embodiments, and the scope of the present invention is defined by the appended claims.

Claims

1. A sonar data processing and image fusion method, characterized in that, The method includes: Input the original side-scan sonar data, and perform water bottom line tracking, slant range correction, horizontal gray level equalization, spatial smoothing filtering, geocoding, and gap filling processing to obtain the repaired complete side-scan sonar image, that is, the side-scan sonar image to be matched; Input the original forward-looking sonar data. Based on beam mean filtering + growth rate calculation, detect the point with the largest slope change to determine the water bottom line; correct the distortion error of the forward-looking sonar using the geometric projection relationship, accurately calculate the true coordinates of the target point, and combine the inertial navigation data of the carrier to map the forward-looking sonar image to the actual geographic coordinate system; then use edge detection to extract the target mask, and use the Criminisi algorithm to fill the missing area; perform image enhancement and noise reduction using adaptive Gamma correction + adaptive median filtering, and finally use the image registration algorithm to perform registration in combination with the spatial position information to obtain the forward-looking sonar self-fusion image; Register the side-scan sonar image to be matched with the forward-looking sonar self-fusion image, perform fusion through an improved wavelet transform method, calculate the matching degree based on regional energy to enhance the fusion effect, combine the data of the two in a unified coordinate system, and output a high-quality fused image.

2. The sonar data processing and image fusion method according to claim 1, characterized in that Process the original side-scan sonar data, including: Input and parse the original side-scan sonar data, extract the water bottom echo data and position information, perform water bottom line tracking, and detect the water bottom line based on the echo intensity threshold method; perform slant range correction, calculate the horizontal distance using geometric relationships, and correct the position of the water bottom line; perform horizontal gray level equalization, use sliding window mean filtering and normalization to enhance the contrast; perform spatial smoothing filtering, use non-local mean filtering to reduce noise; perform geocoding and gap filling, calculate the geographic coordinates of each pixel based on the heading angle and sampling rate, and fill the data missing area; output the complete side-scan sonar image, that is, the side-scan sonar image to be matched.

3. A sonar data processing and image fusion method according to claim 2, characterized in that, Process the original side-scan sonar data, specifically including: Water bottom line tracking of side-scan sonar data: After obtaining the side-scan sonar data, first parse the side-scan sonar data, and its data packet extracts the water bottom echo data and position information; the quantization formula for the echo intensity is: where E is the echo intensity, max and min are respectively the maximum and minimum values of the echo in a single Ping data, so that the data is quantized to [0, 255], and then each Ping is arranged longitudinally to generate the side-scan sonar waterfall image; through an appropriate threshold T, in the pixel point sequence, find the first echo with an intensity I greater than the threshold, then this point is the first water bottom echo, and the distance from this point to the fish sled is the slant range distance from the fish sled to the water bottom, and the set of these points is the water bottom line; Slant range correction of sonar data: On the premise that the sound speed is assumed to be constant and the water bottom terrain is flat, a strict right triangle relationship is formed among the horizontal distance, the slant distance, and the height from the fish sled to the water bottom; based on this, using geometric relationships, the corresponding horizontal distance can be calculated; finally, through horizontal adjustment of the pixel point position, slant range correction is achieved, and the slant range r is calculated according to the sound speed and time: After obtaining the slant range, the horizontal range can be calculated according to the Pythagorean theorem, and each Ping of data is corrected to complete the slant range correction. The formula is as follows: where d i : the horizontal distance of the side-scan sonar water bottom line, i.e., the value obtained after slant range correction; c: the sound speed in water (m / s), t i : the two-way propagation time (s) of the echo signal, i.e., the time from transmission to reception, h: the installation height (m) of the sonar, which is the vertical distance from the sensor to the water bottom; Horizontal gray level equalization: Select 1 Ping of data, set a sliding window, define a mean filter kernel according to the size of the window, perform mean filtering on the pixel values within the window, and smooth the image by continuously moving the window; after completing the mean filtering, normalize the image, calculate the standard deviation of each point, and divide the difference between the original pixel value of the image and the average value by the standard deviation, as shown in the following formula: Side-scan sonar spatial smoothing filter: Perform filtering on the side-scan sonar image, use non-local means filtering (NL-means) to denoise the image, find the pixel points with similar neighborhood structures to the pixel point to be processed in the entire image, and obtain the filtering effect of the pixel point to be processed according to the weighted average of the similar pixel points; Side-scan sonar data geocoding and gap filling: In the coordinate system, for a single Ping of data, the corresponding geographical coordinates of the projection point directly below the fish are (X0, Y0). From the left and right sides, the scan width of the side-scan sonar on one side is R, the sampling rate of each channel of the sonar is N, and the course azimuth angle is α; because each Ping of data is perpendicular to its course direction, the direction angle of the left side is θ = α - π / 2, and the right side differs from the left side by π, so the right side direction angle is θ = α + π / 2. Pi is the i-th pixel point of a certain Ping of data, and its geographical coordinates are: X i = X0 + R × cos(α ± π / 2) × i / N Y i = Y0 + R × sin(α ± π / 2) × i / N (5).

4. A sonar data processing and image fusion method according to claim 1, characterized in that The determination of the water bottom line includes: By analyzing the intensity of a certain beam of the forward-looking sonar, first perform mean filtering on it: Process the beam after mean value, and calculate its growth rate in sequence order. The formula is as follows: Since the signal intensity changes significantly from the water body area to the water bottom area and changes from dark to bright, which is manifested as the change of the signal slope, the water bottom position can be determined by detecting the maximum value of the signal slope; the sampling points corresponding to the water bottom area should have the largest slope change, and the set of these points can be recognized as the water bottom line.

5. A sonar data processing and image fusion method according to claim 1, characterized in that The correction of the distortion error of the forward-looking sonar by using the geometric projection relationship includes: Aiming at the position error problem existing in the forward-looking sonar image, use the system correction method of the geometric projection relationship for correction; first, based on the oblique forward detection mechanism of the forward-looking sonar, establish a spatial geometric model of sound wave propagation, and determine the spatial position relationship between the target point, the sonar, and the echo path; second, calculate the propagation distance according to the sound wave propagation time, and construct a spatial circle equation with the sonar position as the center; finally, by solving the intersection points of the spatial circle and the water bottom plane equation, accurately calculate the true spatial coordinates of the target point, so as to realize the correction of the position error; the result of the correction is: z k = -d (3).

6. A sonar data processing and image fusion method according to claim 1, characterized in that The coordinate transformation in combination with the carrier position information includes: After correcting the position distortion of the forward-looking sonar image, the geocoding of the image can be started. Considering that the sonar device itself does not directly access the position information and relies on the position and motion data provided by the carrier's inertial navigation system for geodetic coordinate reduction, by combining the position information of the carrier, each pixel point in the sonar image is mapped into the actual geodetic coordinate system, so as to achieve the precise correspondence between the sonar image and the geographical space. The geocoding formula for the forward-looking sonar is as follows: X i = X0 + R·cos[π / 2 - (α + Δθ·i)] / N Y i = Y0 + R·sin[π / 2 - (α + Δθ·i)] / N (4).

7. A sonar data processing and image fusion method according to claim 1, characterized in that, The method of using edge detection to extract the target mask and using the Criminisi algorithm to fill the missing area includes: Detect the target in the image. Accurately detecting the target can clearly define the range that needs to be corrected for the shadow area, separating the target area from the shadow area to be corrected, so as to ensure that the shadow area correction process is not affected by the target pixel values. After detecting the target mask, start to extract the shadow area. First, enhance the contrast between the shadow area and other areas, then roughly extract the area through edge detection, and use morphological operations and connected component analysis to obtain the shadow mask, that is, the shadow mask. Use the Criminisi algorithm to obtain the repaired image.

8. A sonar data processing and image fusion method according to claim 1, characterized in that Process the forward-looking sonar image through adaptive Gamma correction and adaptive median filtering to obtain the enhanced + noise-reduced forward-looking sonar image.

9. A sonar data processing and image fusion method according to claim 1, characterized in that, The method of using the image registration algorithm to combine the spatial position information for registration includes: Adopt the combination of SuperPoint + LightGlue for feature detection and matching. For areas with unclear target features, take the position information of the previous geocoding, perform geographical location matching on the image, and perform affine transformation on the image after preliminary registration to obtain the fully registered image.

10. A sonar data processing and image fusion method according to claim 1, characterized in that Register the side-scan sonar image to be matched with the self-fused image of the forward-looking sonar, including: After the image to be fused is registered through image registration, it is mapped into a unified coordinate system and directly fused. Considering the characteristics of sonar images, improve the wavelet transform fusion. By selecting the regional energy of the image after wavelet decomposition and calculating the regional matching degree, the formula is as follows: E = ∑ Δx∈J,Δy∈K [I(x + Δx, y + Δy)] 2 (5) M ed (x,y) = M edian [I(x + Δx, y + Δy)] (6) Optimize the wavelet fusion rule through the matching degree, and judge the matching degree to select the appropriate calculation method to achieve the fusion of high-frequency components.

Images