Underwater target tracking method and system integrating sonar and camera

Abstract

The invention discloses an underwater target tracking method and system fusing sonar and a camera. According to the underwater target tracking method, underwater tracking is conducted on a tracking target through the underwater target tracking system composed of an underwater holder fixing platform, the underwater camera, the sonar, a switch, an image processor and an underwater steering engine. In the target tracking process, an optical image and a sonar image are collected respectively, and the processed optical image and sonar image are obtained after optical image sectorization preprocessing, acousto-optic registration and image enhancement are carried out in sequence; performing high and low frequency decomposition and adaptive weighted fusion on the processed optical image and sonar image to obtain a final fused image; and the mass center position of the tracked target is obtained based on the final fusion image, and a combined control strategy is provided to drive the holder to steer stably and quickly, so that the underwater camera and the sonar can be continuously aligned with the tracked target.

Description

Technical Field

[0001] This invention belongs to the field of target tracking technology, specifically relating to an underwater target tracking method and system that integrates sonar and camera. Background Technology

[0002] In recent years, the identification and tracking of underwater targets has received widespread attention. In underwater applications, the imaging capabilities of underwater cameras are significantly limited by light attenuation, suspended particles, and variations in turbidity. While optical cameras can acquire high-resolution images, their visibility range is typically short, and in high turbidity conditions, image information is easily lost, making it difficult to reflect the true shape and outline of the target object. In contrast, sonar offers advantages such as long detection range and clear object outlines, but its resolution is affected by beamwidth, resulting in limited precision in identifying small targets at close range. Therefore, a single sensing method often struggles to balance image clarity and detection range in complex underwater environments, posing significant challenges to underwater target tracking and identification. However, these two sensing technologies are complementary, and effectively fusing them is currently a research hotspot in the field of underwater visual perception. Current research on acoustic-optical fusion mainly focuses on image registration and information enhancement. Some studies utilize optical images for color mapping and brightness compensation of sonar images. However, in target identification and tracking research, achieving high target recognition capabilities through complementary image fusion is a crucial consideration. Blindly combining sonar and camera sensors may weaken the target's edge characteristics, reducing the accuracy of target identification and localization. In the target identification and localization process, changes in target angle and delayed response can easily lead to lag and overshoot. Therefore, it is necessary to propose novel underwater target tracking methods and systems that integrate sonar and camera sensors. Summary of the Invention

[0003] The purpose of this invention is to provide an underwater target tracking method and system that integrates sonar and camera.

[0004] In a first aspect, the present invention provides an underwater target tracking method that integrates sonar and camera, the method comprising:

[0005] The original optical image and the original sonar fan-shaped image of the target are acquired by an underwater camera and sonar respectively; the original optical image and the original sonar fan-shaped image are registered, and the registered original optical image is transformed by polar coordinates to obtain the original optical fan-shaped image.

[0006] Low-frequency decomposition is performed on the original optical fan-shaped image and the original sonar fan-shaped image to obtain the camera low-frequency image, camera high-frequency image, sonar low-frequency image and sonar high-frequency image; the quality of the low-frequency image and the high-frequency image are evaluated respectively to obtain the image quality of the low-frequency image and the high-frequency image.

[0007] Based on the image quality of the low-frequency image, a low-frequency fusion coefficient is constructed by introducing the distance from the tracking target to the imaging center. The low-frequency image from the camera and the low-frequency image from the sonar are weighted and summed according to the low-frequency fusion coefficient to obtain the low-frequency fused image. Based on the image quality of the high-frequency image, a high-frequency fusion coefficient is constructed. The high-frequency image from the camera and the high-frequency image from the sonar are weighted and summed according to the high-frequency fusion coefficient to obtain the high-frequency fused image. The low-frequency fused image and the high-frequency fused image are fused to obtain the final fused image.

[0008] The centroid position of the target is obtained based on the final fused image, and the alignment angles of the underwater camera and sonar are adjusted according to the centroid position to align them with the target.

[0009] Repeat the above process until the target is accurately aligned and tracked.

[0010] Preferably, before performing low-frequency decomposition, image enhancement is performed on the original optical sector image and the original sonar sector image respectively, and the brightness dynamic range of the enhanced optical sector grayscale image is adjusted to the same range as that of the enhanced sonar sector grayscale image.

[0011] Preferably, the method for enhancing the original optical sector image is as follows: sequentially performing grayscale processing, Gaussian kernel filtering, adaptive filtering, and Gamma nonlinear mapping on the original optical sector image to obtain the enhanced optical sector grayscale image. .

[0012] Preferably, the method for enhancing the original sonar fan-shaped image is as follows: the original sonar fan-shaped image is sequentially subjected to median filtering, grayscale stretching, and Gamma nonlinear mapping to obtain an enhanced sonar fan-shaped grayscale image. .

[0013] As a preferred method, the brightness dynamic range is as follows:

[0014] The mean gray level and standard deviation of the enhanced optical sector grayscale image and the enhanced sonar sector grayscale image are obtained respectively. Using the enhanced sonar sector grayscale image as a reference, brightness matching is performed on the enhanced optical sector grayscale image to obtain the matched optical sector grayscale image C', expressed as:

[0015]

[0016] in For the pixels in the matched optical sector grayscale image; These represent the average gray level and standard deviation of the optical sector grayscale image, respectively. These represent the average gray level and standard deviation of the sonar sector grayscale image, respectively. For pixels in the enhanced optical sector grayscale image; This is the brightness balance coefficient.

[0017] After grayscale clipping of the optical sector grayscale image C', the clipped optical sector grayscale image is obtained. Complete the dynamic range adjustment of brightness.

[0018] As a preferred method, the underwater camera and sonar alignment angle adjustment method is as follows: The difference between the polar axis angle and the line-of-sight angle of the target's centroid is obtained, and a first-order exponential smoothing filter is used to obtain the input angle deviation; the input angle deviation is processed by a proportional-integral controller to obtain the desired angular velocity; two thresholds are set to segment the input angle deviation, and the angular velocity in each segment is obtained as a reference angular velocity based on the desired angular velocity; an S-curve is planned for the reference angular velocity to obtain the smoothed angular velocity at each moment; the control output at each moment is obtained based on the difference between the smoothed angular velocity and the actual angular velocity.

[0019] Preferably, the method for obtaining the image quality of the low-frequency image is as follows: obtaining the low-frequency gray-level variance and the low-frequency average gray-level of the low-frequency image respectively; obtaining the image quality of the low-frequency image based on the quotient of the low-frequency gray-level variance and the low-frequency average gray-level.

[0020] The method for obtaining the image quality of the high-frequency image is as follows: the average value of the gradient amplitude of the high-frequency image is used as the image quality of the high-frequency image.

[0021] Preferably, the low-frequency decomposition method is as follows: extract the low-frequency image of the image using a Gaussian filter, and obtain the high-frequency image using residuals to complete the low-frequency decomposition of the image.

[0022] Preferably, the registration process between the original optical image and the original sonar fan-shaped image is as follows:

[0023] Multiple one-to-one camera feature points and sonar feature points are set on the optical image and sonar fan-shaped image measuring the same target, respectively; a covariance matrix is ​​constructed based on the camera feature points and sonar feature points; the covariance matrix is ​​decomposed by singular value decomposition, and a rigid compensation transformation matrix is ​​obtained based on the decomposition result; the original optical image is multiplied by the rigid compensation transformation matrix to complete the registration of the original optical image and the original sonar fan-shaped image.

[0024] Secondly, the present invention provides an underwater target tracking system that integrates sonar and camera, used to execute the aforementioned underwater target tracking method; the underwater target tracking system includes an underwater gimbal and an underwater camera, sonar, switch, image processor, and underwater servo mounted on a fixed platform of the underwater gimbal; the underwater camera and sonar are used to acquire optical images and sonar images of the tracked target, respectively, and transmit them to the image processor through the switch; the image processor is used to fuse the optical images and sonar images, and obtain the angular deviation of the tracked target relative to the underwater gimbal based on the fused image; the underwater servo is used to adjust the underwater gimbal according to the angular deviation.

[0025] The beneficial effects of this invention are:

[0026] 1. The present invention first decomposes and then fuses optical and sonar images at low and high frequencies, which effectively suppresses noise, highlights important features, enhances the edge features of the tracked target, and reduces the accuracy of target recognition. At the same time, the present invention constructs different fusion coefficients in low-frequency fusion and high-frequency fusion respectively. By adding a distance component in low-frequency fusion, the dominant role of the camera and sonar under different conditions is introduced into the fusion coefficient, thereby improving the accuracy of target localization.

[0027] 2. This invention adjusts the alignment angle of the underwater platform through a dual-loop control structure to ensure smooth turning of the gimbal. At the same time, this invention adopts a dual-threshold hysteresis control strategy to segment different deviation angles, further improving tracking stability and preventing frequent reverse jitter of the servo motor due to small angle errors and noise. In addition, this invention smooths the angular velocity to prevent servo motor shock and oscillation caused by sudden speed changes. Attached Figure Description

[0028] Figure 1 This is a block diagram of the underwater target tracking system in this invention.

[0029] Figure 2 This is a flowchart of the coordinate registration process for sonar image and optical image fusion in this invention.

[0030] Figure 3 This is a flowchart illustrating the fusion process of sonar images and optical images in this invention.

[0031] Figure 4 This is a flowchart illustrating the strategy for controlling the gimbal to align and track the target in this invention. Detailed Implementation

[0032] The present invention will be further described below with reference to the accompanying drawings.

[0033] like Figure 1As shown, an underwater target tracking method integrating sonar and camera is employed. The underwater target tracking system includes an underwater gimbal and an underwater camera, sonar, switch, image processor, and underwater servo motor mounted on a fixed platform of the underwater gimbal. The underwater camera and sonar are used to acquire optical and sonar images of the tracked target, respectively, which are transmitted to the image processor via the switch. The image processor fuses the optical and sonar images and obtains the deflection angle of the tracked target relative to the underwater gimbal based on the fused image. The underwater servo motor adjusts the alignment angle of the underwater gimbal according to the deflection angle of the tracked target relative to the underwater gimbal, so that the tracked target is positioned between the optical and sonar images.

[0034] The underwater target tracking and detection method includes the following steps:

[0035] Step 1: Fan-shaped preprocessing of optical images

[0036] Optical images and sonar fan plots of the target were acquired using underwater cameras and sonar, respectively. In the optical image, N points are designated as camera feature points. Through polar coordinate resampling, the camera feature points in the optical image are transformed from Cartesian matrix coordinates to coordinates with the sonar sector. Figure 1 The sector polar coordinates can be represented as:

[0037]

[0038] in Camera feature points in optical images sector polar coordinates; Camera feature points Cartesian coordinates of the matrix; To track the imaging center of the target; The effective radius of the sonar sector diagram; The central angle of the sonar sector diagram; , This represents the number of feature points.

[0039] Step 2: Initial rigidity geometric compensation

[0040] like Figure 2 As shown, points in the sonar sector image that correspond to camera feature points are used as sonar feature points, meaning there is a one-to-one correspondence between camera and sonar feature points. The camera and corresponding sonar feature points detect the same location of the tracked target. For ease of calculation, the feature points in the optical image and sonar sector image are converted from sector polar coordinates to Cartesian coordinates, which can be represented as:

[0041]

[0042] in Camera feature points Cartesian coordinates; sonar feature points Cartesian coordinates; sonar feature points The sector polar coordinates.

[0043] Let the sets of sonar feature points and camera feature points be denoted as follows: and The centroids are respectively denoted as and Point sets are obtained through decentralization. Hedianji Calculate the point set Hedianji covariance matrix The covariance matrix is ​​decomposed using the singular value decomposition method, and the rigid compensation transformation matrix is ​​obtained based on the decomposition results. While ensuring that the relative positions of the underwater camera and sonar remain unchanged, the rigid compensation transformation matrix This method can be used indefinitely. The above process can be expressed by the formula:

[0044]

[0045] in and These represent the left and right singular vector matrices, respectively; R represents the singular value matrix; R is the rotation matrix; t is the transformation bias.

[0046] Step 3: Enhancement of optical and sonar sector images

[0047] 3-1. Coordinate Registration

[0048] like Figure 3 As shown, raw optical images and raw sonar fan-shaped images of the target were acquired using an underwater camera and sonar, respectively. Using a rigid compensation transformation matrix The original optical image is processed using the following expression:

[0049]

[0050] in and These are the pixel coordinates in the optical images before and after processing with the rigid compensation transformation matrix, respectively.

[0051] After processing with the rigid compensation transformation matrix, the original optical image is transformed from matrix Cartesian coordinates to sector polar coordinates to obtain the original optical sector image. .

[0052] 3-2. Enhancement of the original optical fan-shaped image

[0053] To simplify subsequent filtering and brightness control, the computational load and accuracy of the fusion process are reduced by converting multi-channel color to single-channel, thus simplifying the original optical fan-shaped image. Convert to optical fan-shaped grayscale image To suppress random noise and underwater particle reflections in the image, optical sector grayscale images were processed. Perform a two-dimensional Gaussian kernel Filtering achieves smoothing and noise reduction, resulting in the Gaussian-filtered image C. G To preserve edges while enhancing local details, the Gaussian-filtered image C... G Adaptive filtering is performed, increasing the enhancement amplitude when local areas show significant changes, and suppressing over-enhancement in smooth areas, resulting in the adaptively filtered image C. a Finally, the adaptively filtered image C... a Gamma nonlinear mapping is performed to enhance contrast, resulting in an enhanced optical fan-shaped grayscale image C. z The above process can be expressed by the following formula:

[0054]

[0055] in For pixels in an optical sector grayscale image; These represent the brightness values ​​of the red, green, and blue channels in a color optical sector image, respectively. These are the pixels in the image after Gaussian filtering. The numbers represent the pixels in the image after adaptive filtering; α is the detail magnification factor. This represents the local mean of the image after Gaussian filtering. To enhance the pixels of the optical sector grayscale image; The comparison coefficients are adaptively adjusted based on lighting conditions; * represents convolution operations.

[0056] 3-3. Enhancement of the original sonar fan-shaped image

[0057] The original sonar fan-shaped image is prone to speckle noise, which severely affects the subsequent fusion effect. Median filtering is used to process the image, and after traversing the entire image, the median-filtered sonar fan-shaped image is obtained. Image contrast was enhanced by grayscale stretching and Gamma nonlinear enhancement to obtain an enhanced sonar fan-shaped grayscale image. The formula is as follows:

[0058]

[0059]

[0060] in This represents the center pixel value of a single sliding window in median filtering; These are the pixels of the original sonar fan-shaped image; This represents the median operation; A sliding window representing a preset area; These are the pixels of the sonar fan-shaped image after grayscale stretching. These are the pixels in the sonar sector image after median filtering; and These are the maximum and minimum grayscale values, respectively. β1 represents the enhanced sonar sector grayscale image pixels; β2 is the contrast coefficient, which is adaptively adjusted according to lighting conditions.

[0061] 3-4. Joint optimization processing

[0062] Due to the differences in sensor types between sonar and camera, to ensure that the enhanced optical sector grayscale image and the sonar sector grayscale image maintain as much consistency as possible in grayscale distribution and brightness range, the brightness dynamic range of the optical image is adjusted to the same range as the sonar image. First, the enhanced optical sector grayscale image is calculated. and enhanced sonar fan-shaped grayscale image The average gray level and standard deviation are calculated. Using the enhanced sonar sector grayscale image as a reference, brightness matching is performed on the enhanced optical sector grayscale image to obtain the matched optical sector grayscale image C', expressed as:

[0063]

[0064] in For the matched optical sector grayscale image pixels; Optical sector grayscale image The average gray level and standard deviation; These are sonar sector grayscale images. The average gray level and standard deviation; This is the brightness balance coefficient (with a value of 0.9-1.1).

[0065] The optical sector grayscale image C' may exceed the standard 8-bit grayscale range (0-255 pixel values), therefore further grayscale clipping is required to obtain the clipped optical sector grayscale image. The expression is:

[0066]

[0067] in These are the pixels in the optically limited sector grayscale image.

[0068] Step 4: Low- and high-frequency decomposition of the image

[0069] Two-dimensional Gaussian filter Extract optical sector grayscale images separately and sonar fan-shaped grayscale image The low-frequency image is used to obtain the high-frequency image using residuals, and the expression is:

[0070]

[0071] in For pixels in a low-frequency image; For pixels in a grayscale image; For pixels in a high-frequency image; and denoted as the standard deviations of distance and angle in a two-dimensional Gaussian filter, respectively.

[0072] Based on the above-mentioned low-frequency and high-frequency image extraction methods, the optical sector grayscale image is extracted respectively. and sonar fan-shaped grayscale image Decomposed into low-frequency and high-frequency images, the expression is:

[0073]

[0074] in and These are the pixels in the low-frequency image and the high-frequency image of the camera, respectively. and These are pixels in the low-frequency and high-frequency sonar images, respectively.

[0075] Step 5: Joint Weighted Image Fusion

[0076] 5-1. Q-evaluation of low-frequency and high-frequency image quality

[0077] Low-frequency images primarily represent the overall brightness and structural hierarchy of the tracked target. The smoothness of brightness and the stability of transitions between levels represent image quality. These two metrics are used to evaluate low-frequency images and obtain their image quality. , can be represented as:

[0078]

[0079] in Image quality for low-frequency images; Low-frequency image; Represents low-frequency grayscale variance; ε represents the average gray level at low frequencies; ε is a constant term.

[0080] High-frequency images primarily represent the edges, contours, and texture information of the tracked target. Gradient analysis is used to evaluate the quality of high-frequency images, thus obtaining the image quality of the high-frequency images. The expression is:

[0081]

[0082] Where N is the number of pixels in the high-frequency image; This represents the gradient magnitude of a high-frequency image.

[0083] The image quality of low-frequency and high-frequency images from the camera is obtained through evaluation methods using low-frequency and high-frequency images, respectively. and the image quality of low-frequency and high-frequency sonar images. This is used for subsequent weighted fusion.

[0084] 5-2. Obtain the final fused image

[0085] Set the maximum boundary R respectively max and minimum boundary R min If the distance r between the tracked target and the imaging center is greater than the maximum boundary R max If the distance r between the tracked target and the imaging center is less than the minimum boundary R, ​​then the underwater camera can be considered to be malfunctioning. min If the sonar resolution is too low, it is determined that the sonar has completely failed. At close range, the low-frequency information from the underwater camera is reliable, so the weighting of low-frequency information should be dominated by the camera; at greater distances, the high-frequency information from the sonar is reliable, so the weighting of high-frequency information should be dominated by the sonar. When abnormal frame rates occur, another non-dominant device takes over. Due to the influence of water, the signal attenuation with distance is non-linear; to ensure a smooth transition of weights, an exponential function is applied to the distance. Dividing the sector image into several regions and adaptively weighting each region yields more reliable results. The low-frequency fusion formula is shown below:

[0086]

[0087] in Indicates the low-frequency camera fusion coefficient; This represents the low-frequency sonar fusion coefficient; Represents pixels in a low-frequency fused image; This is the low-frequency attenuation coefficient.

[0088] High-frequency images primarily reflect edge features and are less sensitive to distance. Therefore, image sharpness is the main factor in fusion, and distance attenuation is not required. The high-frequency fusion formula is shown below:

[0089]

[0090] in Indicates the high-frequency camera fusion coefficient; Indicates the high-frequency sonar fusion coefficient; This represents the pixels in a high-frequency fused image.

[0091] Low-frequency fusion images High-frequency fusion image After overlaying, a fused image is obtained. The expression is:

[0092]

[0093] in This represents the pixels in the final merged image.

[0094] Step Six: Underwater Servo Control Strategy

[0095] like Figure 4 As shown, the images will be merged. After processing by the image processing module, the angle of the target is given as the input for the underwater servo control. The underwater servo is then used to control the underwater gimbal platform to rotate smoothly, quickly, and efficiently to align with the target. The specific process is as follows:

[0096] 6-1. Angular deviation of the tracking target

[0097] Set brightness threshold For the final fused image Perform threshold segmentation; when the final fused image The pixel value is greater than the brightness threshold. Determined as the target area The centroid coordinates are obtained by applying a brightness-weighted average to the angle and distance of the target area. The expression is:

[0098]

[0099] Let the current underwater gimbal's fixed platform's line-of-sight angle and distance be denoted as . The difference between the centroid coordinates of the tracked target and the target's center of mass is calculated. Due to underwater image noise and acoustic reflection, the target's position estimation fluctuates. A first-order exponential smoothing filter is applied to the difference angle to obtain the controller input angle deviation. The expression is:

[0100]

[0101] Where Δθ and Δr are the angular and distance deviations between the current gimbal and the centroid of the tracked target, respectively; Δθ is the filter coefficient. pThe controller input angle deviation is the value from the previous moment.

[0102] 6-2. Outer ring angle control

[0103] To ensure smooth steering of the underwater servo motor, a dual-loop control structure is adopted: the outer loop is responsible for angle control, defining the starting and ending angles; the inner loop controls the angular velocity to achieve smooth and rapid steering, and prevents overshoot.

[0104] The outer ring receives the underwater servo input angle deviation from the image processing module. The desired angular velocity is obtained through a proportional-integral controller. Integrals are mainly used to control deviation drift. To prevent excessive deviation from causing system oscillation, appropriate upper and lower limits are set. Angular velocity... The method to obtain it is as follows:

[0105]

[0106] in This is the proportionality coefficient; The integral coefficient; Indicates time The gimbal input angle deviation; I max and I max These are the upper and lower limits of integral saturation, respectively.

[0107] 6-3. Dual-threshold hysteresis control

[0108] To further improve the tracking stability of the underwater servo motor and prevent frequent reverse jitter caused by minute angle errors and noise, a dual-threshold hysteresis control strategy is adopted. Dual-threshold hysteresis control is implemented when the gimbal inputs an angle deviation... Three zones are established based on thresholds T1 and T2: a large deviation zone, a medium deviation zone, and a small deviation zone, each corresponding to a different control strategy. In the large deviation zone, a fast response requires the system to use the angular velocity ω obtained from the outer-loop strategy. p Full-speed tracking; in the medium-deviation region, linear transition requires the angular velocity to decrease proportionally; in the small-deviation region, steady-state maintenance requires the angular velocity to approach 0 to avoid frequent reversals; the reference angular velocity ω obtained under different conditions... r , can be represented as:

[0109]

[0110] 6-4. Adaptive S-curve Programming

[0111] To prevent sudden changes in angular velocity from the outer loop input to the inner loop, which could cause servo motor shock and oscillation, an adaptive S-curve programming approach is introduced for a smooth transition. The S-curve programming divides the angular velocity change process into seven stages (starting acceleration, uniform acceleration, deceleration, constant speed, starting deceleration, uniform deceleration, and the end of deceleration), using the angular acceleration j and angular acceleration... By restricting the angular velocity to ensure continuous change, a smooth angular velocity can be obtained. The method to obtain it is as follows:

[0112]

[0113] in The angular velocity at the current time t; The angular acceleration at the current time t; The angular acceleration at the next moment; Represents a symbolic function; The smoothed angular velocity at the current time t; This is the maximum angular acceleration; This is the maximum acceleration; This is the smoothed angular velocity at the next moment.

[0114] Based on the current angle deviation and angular velocity deviation To dynamically adjust the maximum angular acceleration and maximum angular acceleration The expression is:

[0115]

[0116] in and Basic parameters; and This is the adjustment coefficient; ; This represents the actual angular velocity.

[0117] 6-5. Inner Loop Speed ​​Control

[0118] The inner loop primarily controls the specific angular velocity. It uses PWM (Pulse Width Modulation) feedback control of the servo to adjust the angular velocity in real time, continuously correcting it to a smooth angular velocity close to the current moment. The inner loop operates at a high sampling frequency to ensure good transient response. The control output U is expressed as:

[0119]

[0120] in This indicates the speed loop gain. When the actual angular velocity is lower than the smooth angular velocity, the duty cycle is increased to improve the servo torque; conversely, the duty cycle is decreased to achieve feedback regulation of the speed.

[0121] 6-6 Dead Zone Compensation

[0122] Because of the static friction between the servo drive circuit and the mechanical mechanism, if the PWM duty cycle is too small, the gimbal may fail to start under slight angular deviations. Therefore, a piecewise compensation function is introduced to obtain the compensated control signal. , can be represented as:

[0123]

[0124] in For fixed compensation gain, This is the threshold for servo motor activation.

[0125] Image processing module outputs control signals This drives the underwater servo motor to adjust its direction.

[0126] Step 7: Repeat steps 3 to 6, using a 20Hz sampling frequency to provide real-time feedback and iterative correction to the tracked target, achieving rapid and stable rotation of the gimbal until underwater tracking of the target is completed.

Claims

1. An underwater target tracking method integrating sonar and camera, characterized in that: The method includes: The original optical image and the original sonar fan-shaped image of the target are acquired by an underwater camera and sonar respectively; the original optical image and the sonar fan-shaped image are registered, and the registered original optical image is transformed by polar coordinates to obtain the original optical fan-shaped image. Low-frequency decomposition is performed on the original optical fan-shaped image and the original sonar fan-shaped image to obtain the camera low-frequency image, camera high-frequency image, sonar low-frequency image and sonar high-frequency image; the quality of the low-frequency image and the high-frequency image are evaluated respectively to obtain the image quality of the low-frequency image and the high-frequency image. Based on the image quality of the low-frequency image, a low-frequency fusion coefficient is constructed by introducing the distance from the tracking target to the imaging center. The low-frequency image from the camera and the low-frequency image from the sonar are weighted and summed according to the low-frequency fusion coefficient to obtain the low-frequency fused image. Based on the image quality of the high-frequency image, a high-frequency fusion coefficient is constructed. The high-frequency image from the camera and the high-frequency image from the sonar are weighted and summed according to the high-frequency fusion coefficient to obtain the high-frequency fused image. The low-frequency fused image and the high-frequency fused image are fused to obtain the final fused image. The centroid position of the target is obtained based on the final fused image, and the underwater camera and sonar are rotated according to the centroid position to align with the target. Repeat the above process until underwater tracking of the target is complete.

2. The underwater target tracking method integrating sonar and camera according to claim 1, characterized in that: Before performing low-frequency decomposition, image enhancement was performed on the original optical fan-shaped image and the sonar fan-shaped image respectively, and the brightness dynamic range of the enhanced optical fan-shaped grayscale image was adjusted to the same pixel range as the enhanced sonar fan-shaped grayscale image.

3. The underwater target tracking method integrating sonar and camera according to claim 2, characterized in that: The method for enhancing the original optical sector image is as follows: the original optical sector image is sequentially subjected to grayscale processing, Gaussian kernel filtering, adaptive filtering, and Gamma nonlinear mapping to obtain the enhanced optical sector grayscale image. .

4. The underwater target tracking method integrating sonar and camera according to claim 2, characterized in that: The method for enhancing the original sonar fan-shaped image is as follows: the original sonar fan-shaped image is sequentially subjected to median filtering, grayscale stretching, and Gamma nonlinear mapping to obtain the enhanced sonar fan-shaped grayscale image. .

5. The underwater target tracking method integrating sonar and camera according to claim 2, characterized in that: The method for adjusting the brightness dynamic range is as follows: The mean gray level and standard deviation of the enhanced optical sector grayscale image and the enhanced sonar sector grayscale image were obtained respectively; Using the enhanced sonar sector grayscale image as a reference, brightness matching is performed on the enhanced optical sector grayscale image to obtain the matched optical sector grayscale image C', whose expression is: in For the pixels in the matched optical sector grayscale image; These represent the average gray level and standard deviation of the optical sector grayscale image, respectively. These represent the average gray level and standard deviation of the sonar sector grayscale image, respectively. For pixels in the enhanced optical sector grayscale image; This is the brightness balance coefficient; After grayscale clipping of the optical sector grayscale image C', the clipped optical sector grayscale image is obtained. Complete the dynamic range adjustment of brightness.

6. The underwater target tracking method integrating sonar and camera according to claim 1, characterized in that: The method for adjusting the alignment angle of the underwater camera and sonar is as follows: obtain the difference between the polar axis angle and the line-of-sight angle of the target's centroid, use a first-order exponential smoothing filter to obtain the input angle deviation; process the input angle deviation through a proportional-integral controller to obtain the desired angular velocity; Two thresholds are set to segment the input angle deviation, and the angular velocity in each segment is obtained as the reference angular velocity based on the desired angular velocity. S-curve planning is performed on the reference angular velocity to obtain the smoothed angular velocity at each moment; based on the difference between the smoothed angular velocity and the actual angular velocity, the control output at each moment is obtained.

7. The underwater target tracking method integrating sonar and camera according to claim 1, characterized in that: The method for obtaining the image quality of the low-frequency image is as follows: obtain the low-frequency grayscale variance and low-frequency average grayscale of the low-frequency image respectively. Image quality of low-frequency images is obtained by quotient of low-frequency gray-level variance and low-frequency average gray-level. The method for obtaining the image quality of the high-frequency image is as follows: the average value of the gradient amplitude of the high-frequency image is used as the image quality of the high-frequency image.

8. The underwater target tracking method integrating sonar and camera according to claim 1, characterized in that: The low-frequency decomposition method is as follows: a Gaussian filter is used to extract the low-frequency image of the image, and the residual is used to obtain the high-frequency image, thus completing the low-frequency decomposition of the image.

9. The underwater target tracking method integrating sonar and camera according to claim 1, characterized in that: The registration process between the original optical image and the original sonar sector image is as follows: Multiple one-to-one camera feature points and sonar feature points are set on optical images and sonar fan-shaped images measuring the same target, respectively; a covariance matrix is ​​constructed based on the camera feature points and sonar feature points; The covariance matrix is ​​decomposed using singular value decomposition, and a rigid compensation transformation matrix is ​​obtained based on the decomposition result. The original optical image is multiplied by the rigid compensation transformation matrix to complete the registration of the original optical image and the original sonar fan-shaped image.

10. An underwater target tracking system integrating sonar and camera, characterized in that: The underwater target tracking system is used to perform the underwater target tracking method that integrates sonar and camera as described in claim 1. The underwater target tracking system includes an underwater gimbal and an underwater camera, sonar, switch, image processor and underwater servo mounted on the underwater gimbal. The underwater camera and sonar are used to acquire optical images and sonar images of the tracked target, respectively, and are transmitted to the image processor through the switch. The image processor is used to fuse optical images and sonar images, and obtain the alignment angle deviation of the tracking target relative to the underwater gimbal based on the fused image; the underwater servo is used to adjust the alignment angle of the underwater gimbal based on the angle deviation.

Images