Sonar image continuous frame target detection method and system based on tracker fusion

Abstract

The application discloses a sonar image continuous frame target detection method and system based on tracker fusion, and belongs to the technical field of underwater image processing. The method comprises the following steps: acquiring a target detection result of a sonar image at a moment, wherein the target detection result is composed of at least one detection point; predicting a track point of each target at the moment based on a track trajectory stored by a tracker; and generating a target detection result at the moment based on an association cost matrix between all detection points and all track points. The application can effectively process dynamic and static targets and improve the practicability for real application scenarios.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically relating to a method and system for continuous frame target detection in sonar images based on tracker fusion. Background Technology

[0002] Forward-looking sonar (FLS) is a core sensor for underwater vehicles (such as unmanned underwater vehicles (UUVs / ROVs) and autonomous underwater vehicles (AUVs) to acquire information about the water environment ahead. It utilizes the efficient propagation characteristics of sound waves in water to provide real-time, two-dimensional range-azimuth images. FLS images have wide applications in scenarios such as submarine pipeline and cable inspection, underwater structure detection, port and near-shore security, and deep-sea scientific exploration. Before the rise of deep learning, forward-looking sonar image recognition mainly relied on traditional image processing and pattern recognition techniques, such as using Support Vector Machines (SVM), Adaboost, or manually designed rule sets for target classification based on geometric features (such as the target's acoustic shadow length, shape, and area) and acoustic features (such as echo intensity). However, due to the inherent low resolution, high noise, lack of texture information, and strong acoustic shadow dependence of forward-looking sonar images, the recognition accuracy and generalization ability of traditional methods are severely limited. Currently, cutting-edge research on forward-looking sonar image recognition and detection mainly focuses on utilizing deep learning techniques to overcome the inherent defects of sonar images. The mainstream approach aims to automatically extract high-dimensional semantic features from complex acoustic images using a data-driven method. A representative work is the use of CNN-based detectors, transferring classic target detection frameworks such as Faster R-CNN, YOLO, and SSD, originally applied to natural optical images, to forward-looking sonar images. However, because forward-looking sonar images inherently lack texture information and are highly dependent on sound and shadow, while also possessing more temporal information than natural optical images, their target differentiation relies heavily on comprehensive recognition across multiple frames. Therefore, directly applying single-image target detection frameworks to forward-looking sonar image recognition is inappropriate.

[0003] Deep learning-based object detection methods are mainly divided into two major technical schools, with significant differences in their working principles. 1) Two-stage detectors divide the detection process into two stages: first, candidate regions are generated, and then each candidate region is classified and its bounding box is regressed. Their characteristic is generally high accuracy, but relatively slow speed. 2) Single-stage detectors omit the region proposal stage, directly performing dense sampling on the feature map to predict the object's category and bounding box in one go. Their characteristic is very high speed, meeting the needs of real-time detection, and in recent years their accuracy has approached that of two-stage methods. Regardless of the detection framework used, each recognition is only performed on the current frame.

[0004] However, in real underwater operations, forward-looking sonar imaging requires a frame rate of 1 FPS or higher to quickly and continuously scan and display underwater targets and terrain. When a target approaches, it will appear continuously within the forward-looking sonar imaging range, and its movement trajectory and continuous imaging exhibit significant temporal characteristics. According to the aforementioned general image recognition process, each recognition only targets the current frame, ignoring the continuous temporal information between frames. Furthermore, it is inevitably affected by random environmental noise, resulting in a high false alarm rate problem.

[0005] Currently, some technical solutions exist to attempt to address the aforementioned problems. For example, Chinese patent application CN120318662A discloses a target detection fusion method suitable for forward-looking sonar, but its single Euclidean distance matching suffers from mismatches, and the accumulation of multiple errors can lead to target bias, potentially resulting in dynamic targets causing bias in the identification of static targets. Chinese patent application CN119831883A discloses a continuous frame noise suppression method for sonar images based on frame difference, which uses the grayscale values ​​of corresponding pixels in the nth and (n-1th)th frames of the forward-looking sonar image to perform differential operations and converts them to frequency domain noise reduction to obtain a noise-removed image. The invention has the goal, but its application to consecutive frames is limited to between two adjacent frames. It lacks corresponding means for modeling long-term, rapid motion information of objects. Chinese patent application CN115993602A discloses an underwater target detection and target localization method based on forward-looking sonar. It extracts target feature information based on the echo intensity imaging of forward-looking sonar, and finally maps the information to the latitude and longitude of the physical world through target detection and localization. However, this method uses all the information from the first frame to the nth frame. As the number of accumulated frames increases, the amount of computation increases exponentially, which is not conducive to long-term unattended operation. Summary of the Invention

[0006] This invention provides a method and system for continuous frame target detection in sonar images based on tracker fusion, which can effectively handle dynamic and static targets and improve its practicality for real-world application scenarios.

[0007] To achieve the above objectives, the technical solution of the present invention includes the following:

[0008] A method for continuous frame target detection in sonar images based on tracker fusion, the method comprising: Get The target detection result of the sonar image at any time, wherein the target detection result consists of at least one detection point; Based on tracker storage Track trajectories in real time and predict the location of each target. The trajectory points at any given moment; Based on the association cost matrix between all detection points and all trajectory points, generate The target detection result at any given time; wherein, the association cost matrix includes spatial distance cost, appearance feature cost, and physical size similarity cost.

[0009] Furthermore, to obtain Target detection results from real-time sonar images include: right The sonar images are preprocessed, including enhancement processing and filtering processing. Generate the target detection bounding box and its confidence level and category in the preprocessing result; Based on the confidence level and the pixel width and pixel height of the target detection box, the target detection box is filtered. Based on the position representation of the filtered target detection boxes in 3D space, we obtain Target detection results from real-time sonar images.

[0010] Furthermore, on The time-lapse sonar images are enhanced, including: Get The echo intensity of the sonar image at time t is calculated, and the echo intensity is logarithmically compressed to obtain the compressed sonar image. The compressed sonar image is divided into several non-overlapping local regions, and the histogram of each local region is subjected to contrast cropping. The number of cropped pixels is evenly distributed across all gray levels, and the histogram is equalized based on the distribution results to generate a gray-level mapping function for the local region. Bilinear interpolation is used to merge the grayscale mapping results of all local regions to obtain the enhanced result. Real-time sonar images.

[0011] Furthermore, on The sonar image at any given time undergoes filtering processing, including: Computational enhancement The structural tensor of the time-series sonar image; Calculate the first and second eigenvalues ​​of the structure tensor, and generate the first and second eigenvectors corresponding to the first and second eigenvalues ​​respectively; wherein the first eigenvalue represents the gradient change intensity of the edge direction, and the second eigenvalue represents the gradient change intensity of the vertical edge direction; Design a diffusion tensor based on the first and second eigenvectors; Based on the diffusion tensor and the enhanced processing The gradient of the image intensity function corresponding to the sonar image at time step 1 is used to obtain the partial differential equation describing the diffusion process. Based on this partial differential equation, the filtered result is obtained. Real-time sonar images.

[0012] Further, the preprocessed result generates the target detection bounding box and the confidence and category of the target detection bounding box, including: Construct an object detection model, which includes: a backbone network and a decoupled detection head; A feature pyramid network structure is introduced to obtain feature maps of different levels of the backbone network; For a feature map of a certain level output by the feature pyramid network structure, the target detection box in the corresponding level and the confidence and category of the target detection box are obtained based on the decoupled detection head; wherein, the decoupled detection head includes a classification branch and a regression branch, the classification branch is used to calculate the confidence and category of the target detection box, and the regression branch is used to predict the position of the target detection box; The loss used to train the object detection model includes: classification loss, regression loss, and object property loss; wherein the object property loss is used to distinguish between foreground and background.

[0013] Further, based on the confidence level and the pixel width and pixel height of the target detection box, the target detection box is filtered, including: Set a first confidence threshold and perform preliminary screening of target detection boxes based on the confidence of the target detection boxes; Based on the initial screening results, the physical size of the target is estimated according to the pixel width and pixel height of the target detection box, and the target detection box is further screened based on the target physical size to obtain a set of target detection boxes; Add the target detection box with the highest confidence score from the target detection box set to the final output list; Calculate the intersection-union ratio (IU) between the highest-confidence bounding box and the remaining bounding boxes; Calculate the orientation penalty factor based on the x-coordinate of the center pixel of the target detection box with the highest confidence and the remaining target detection boxes; Based on the intersection-over-union ratio and the orientation penalty factor, the remaining target detection boxes are... Update the confidence level; Set a second confidence threshold, and after filtering the remaining target detection boxes based on the confidence update results, update the target detection box set; Based on the updated set of target detection boxes, the process of adding the target detection box with the highest confidence in the set to the final output list is repeated until the set of target detection boxes is empty.

[0014] Further, the process of obtaining the position representation of the filtered target detection box in 3D space includes: Calculate the slant range and azimuth angle of the target detection box based on the center coordinates of the target detection box; Calculate the latitude and longitude corresponding to the center of the target detection box based on the slant distance and azimuth angle; Calculate the target depth based on binocular sonar, depth sounding sonar, or vertical sonar; By combining the latitude and longitude of the center of the target detection box with the target depth, the position representation of the target detection box in 3D space is obtained.

[0015] Furthermore, based on the association cost matrix between all detection points and all trajectory points, a... The target detection results at any given time include: Based on the association cost matrix, the Hungarian algorithm or greedy algorithm is applied to obtain the matching results between the detection points and the trajectory points; wherein, the matching results include: matching pairs, unmatched detections, and unmatched trajectories; Based on the matching results, generate The trajectory is tracked in real time; for matched pairs, a Kalman filter is used to connect the detection points with the target points. Real-time tracking trajectory fusion; for unmatched detections, if the confidence of the detection point is higher than the set threshold and appears continuously in N frames, it is initialized as a new tracking trajectory; for unmatched trajectories, if the trajectory point fails to match successfully in N consecutive frames, the corresponding tracking trajectory is terminated. By tracking the length and stability of the trajectory Continuously track and filter the trajectory; Calculate the average speed of each tracking trajectory, and determine whether the target is a static or dynamic target based on the average speed.

[0016] A target detection system for consecutive frames of sonar images based on tracker fusion, the system comprising: The detection point generation module is used to obtain... The target detection result of the sonar image at any time, wherein the target detection result consists of at least one detection point; The trajectory point generation module is used for tracking points stored by the tracker. Track trajectories in real time and predict the location of each target. The trajectory points at any given moment; The detection result generation module is used to generate a result based on the association cost matrix between all detection points and all trajectory points. The target detection result at any given time; wherein, the association cost matrix includes spatial distance cost, appearance feature cost, and physical size similarity cost.

[0017] A computer device includes: a processor and a memory storing computer program instructions; the processor, when executing the computer program instructions, implements the sonar image continuous frame target detection method based on tracker fusion as described above.

[0018] Compared with the prior art, the present invention has at least the following beneficial effects.

[0019] 1) This invention achieves the transformation from “underwater blurred image blocks” to “stable, geotagged, dynamic and static underwater situation” through the deep integration of acoustic imaging geometry, platform navigation information and multi-target tracking technology.

[0020] 2) This invention skips inputs with low image quality or unsatisfactory experimental conditions, thus improving detector efficiency. Attached Figure Description

[0021] Figure 1 A flowchart of a continuous frame target detection method for sonar images based on tracker fusion.

[0022] Figure 2 Block diagram of a continuous frame target detection system for sonar images based on tracker fusion.

[0023] Figure 3 A block diagram of computer equipment. Detailed Implementation

[0024] The present invention will be further described below with reference to possible accompanying drawings and specific embodiments, but this does not constitute any limitation on the present invention.

[0025] The core process of this invention is as follows: Inputting raw sonar images and carrier state information, the system undergoes image enhancement and denoising, deep learning-based target detection, and high-confidence post-processing. Finally, through precise mapping from 2D pixels to 3D geographic space, the target information is updated to the spatiotemporal tracker, achieving continuous state estimation and trajectory management of underwater targets. The innovation of this invention lies in its processing pipeline optimized for sonar image characteristics, and its lightweight, high-precision real-time solution model that integrates information from multiple sensor sources.

[0026] Specifically, such as Figure 1 As shown, the sonar image continuous frame target detection method based on tracker fusion of the present invention includes the following steps S1 to S3.

[0027] Step S1: Obtain The target detection result of the sonar image at any time, wherein the target detection result consists of at least one detection point.

[0028] Step S1.1: Enhancement and preprocessing of forward-looking sonar images.

[0029] Because forward-looking sonar images are characterized by a large dynamic range, low contrast, and severe speckle noise, directly applying general detection algorithms yields poor results. This step aims to improve image quality and provide optimized input for subsequent detection. First, the input image is contrast-enhanced by compressing the dynamic range of the sonar image echo intensity (e.g., logarithmic transformation) and normalizing it to improve contrast. Then, filtering algorithms targeting sonar speckle noise, such as median filtering and anisotropic diffusion filtering, are applied to suppress noise in the sonar image while preserving target edges as much as possible.

[0030] (1) Dynamic range compression and contrast enhancement Echo intensity of raw sonar images The (x,y) dynamic range is extremely wide (up to 60dB or more). Direct linear normalization can cause weak signals to be submerged. This invention employs a combined strategy of adaptive logarithmic transformation and local contrast-limited histogram equalization (CLAHE).

[0031] Logarithmic transformation: Logarithmically compresses pixel intensity, converting multiplicative noise into additive noise and enhancing details in dark areas.

[0032] in, This is the gain coefficient, used to adjust the compression intensity, typically based on the maximum intensity value of the image. Adaptive settings: =255 / .

[0033] Even after logarithmic transformation, the overall image contrast remains insufficient. CLAHE divides the image into several non-overlapping local regions, performs histogram equalization on each region, and uses contrast clipping to suppress noise amplification. Let the local region histogram be... The cropping limit is Calculate the total number of pixels to be cropped. .Will The image is redistributed evenly across all gray levels. A cumulative distribution function (CDF) is calculated on the cropped histogram, and this CDF is used for gray-level mapping. Finally, bilinear interpolation is used to smooth the boundaries of each region, resulting in the enhanced image. .

[0034] (2) Anisotropic diffusion filtering for speckle noise.

[0035] Speckle noise in sonar images exhibits multiplicative properties, and traditional median filtering severely damages edges during denoising. This invention employs coherently enhanced anisotropic diffusion filtering to smooth uniform regions while preserving and enhancing directional target edges. The diffusion process is described by partial differential equations: in, Let t be the image intensity, and t be the iteration time (equivalent to the number of iterations). Let be the gradient, div be the divergence operator, and D be the diffusion tensor.

[0036] The diffusion tensor D is the key factor, and its construction method is as follows.

[0037] (i). Calculate the structure tensor of the image (the covariance matrix of the gradient): in, It is a Gaussian kernel Smoothed image, Representing an image gradient, It is used for integral scale ( A Gaussian kernel with a value greater than 0, * indicates convolution. , , , For structure tensor Matrix elements.

[0038] (ii). Calculate the structure tensor eigenvalues , ( ) and the corresponding feature vector , . Indicates the direction of the local edge. Characterizes local coherence (edge ​​strength).

[0039] (iii). Designing the diffusion tensor , so that the diffusion tensor eigenvectors and structure tensors eigenvectors , Same, but eigenvalues , Reset based on coherence: in, and For control parameters, It is a small constant (such as 0.001). To prevent small amounts from being divided by zero, it is typically 1e-6. This design ensures that in the edge normal direction ( The diffusion is relatively weak (preserving the edges), in the tangential direction of the edges ( Strong diffusion (smoothing noise), resulting in a filtered final output image. .

[0040] Step S1.2: General neural network detection based on lightweight decoupling head.

[0041] To balance the limited computing resources and detection accuracy requirements of underwater platforms, this invention employs a strategy of "general backbone network pre-training + sonar data fine-tuning + decoupled detection head". Feature extraction is performed using a general lightweight deep neural network, which is pre-trained on a large number of natural optical images for single-frame detection and recognition, and then fine-tuned on actual acquired sonar data. The feature matrix is ​​then input into the decoupled detection head module to generate detection boxes, confidence scores, and categories.

[0042] (1) Enhance the backbone network and feature pyramid.

[0043] Models pre-trained on ImageNet using lightweight networks (such as MobileNetV3 or ShuffleNetV2) as the backbone already possess powerful general feature extraction capabilities. For multi-scale targets in sonar images, a Feature Pyramid Network (FPN) structure is introduced, fusing different layers of the backbone network. The feature maps are respectively responsible for detecting small, medium and large targets.

[0044] (2) Decoupling the detection head.

[0045] Unlike traditional coupled detection heads (which directly regress category and bounding box coordinates), decoupled detection heads separate the classification task from the bounding box regression task, and have been shown to significantly improve convergence speed and final accuracy. Let the feature map of a certain level output by the FPN be... Through a classification branch—a lightweight convolutional subnetwork Predict the probability that each anchor point or location belongs to one of K categories: Through the regression branch, which is another parallel convolutional subnetwork Predict the offset of the bounding box. For center coordinate format The prediction target is: in( () represents the actual annotation box, ( () represents the preset anchor frame. Regression branch output. .

[0046] (3) Loss function and fine-tuning.

[0047] Model fine-tuning employs weighted multi-task loss: ,in To address the class imbalance problem, we use classification loss. To better measure bounding box overlap for regression loss; This is for object-oriented loss (distinguishing between foreground and background). This represents the weight, used to adjust the ratio of different loss penalty terms during training.

[0048] Step S1.3: Post-processing based on high threshold confidence and geometric constraints.

[0049] Sonar images have relatively simple backgrounds and sparse targets, but contain a small number of high-intensity false alarms. This invention employs a rigorous post-processing workflow.

[0050] (1) High confidence threshold filtering.

[0051] For each candidate box output by the detector, set a high confidence threshold (e.g., 0.7) based on its category confidence to filter out obviously unreliable detections.

[0052] (2) Improved orientation-weighted nonmaximum suppression (NMS).

[0053] Since targets are typically sparse in sonar images, an improved azimuth-weighted Non-maximum Suppression (NMS) algorithm is used to eliminate duplicate detection boxes for the same target. Standard NMS, relying solely on IoU and confidence, may suppress true neighboring targets at the same distance with large IoU values ​​but different azimuth angles. This invention proposes an improved azimuth-weighted NMS as follows: (i) For the retained set of detection boxes Each box has its own confidence level. and the x-coordinate of the center pixel (Horizontal axis corresponds to azimuth angle); (ii) By confidence level Sort in descending order; (iii) Select the box with the highest confidence level Add it to the final output list; (iv) Calculation With all remaining boxes IoU, denoted as ; (v) Calculate the orientation penalty factor: ,in This is a parameter related to sonar azimuth resolution. When the azimuths of two frames are close... , When the variances differ greatly, ; (vi) Weighted suppression: for the remaining boxes Its confidence level is updated to .like If the value is below a certain threshold (e.g., 0.001), it is removed. This step, when suppressing overlapping boxes, penalizes boxes with large azimuth differences more lightly, thus protecting adjacent targets that are azimuthally separated; Repeat steps (iii)-(vi) until set B is empty.

[0054] Step S1.4: From 2D pixels (x,y) to 3D geographic coordinates ( A precise mapping of ).

[0055] Mapping 2D pixel bounding boxes to 3D geospatial coordinates is key to transforming image perception into knowledge of the physical world. This invention comprehensively considers sonar imaging geometry, vehicle attitude, and multi-source depth information to perform high-precision calculations.

[0056] Forward-looking sonar typically emits sound waves in a fan shape. The pixel coordinates (u, v) of the original multibeamform rectangular image are defined as follows: u (horizontal axis): azimuth dimension, range [0, W], corresponding to azimuth angle For the horizontal opening angle of the sonar; v (vertical axis): Slope distance dimension, range [0, H], corresponding to the slope distance. This represents the current measurement range.

[0057] For the center of the detection frame ( , ), and its corresponding slant range R and azimuth angle for: The Earth is actually an ellipsoid, and the center of the detection frame can be accurately calculated using the Vincenty forward calculation formula. , The corresponding latitude and longitude (t_lat, t_lon) can be obtained. Similarly, the target depth can be calculated from binocular sonar, depth sounding sonar, or vertical sonar. Finally, by substituting the transformation formula into the vehicle state information, the true latitude and longitude corresponding to the width, height, and center of each detection box can be obtained, thus obtaining the position representation of the target point in 3D space. ).

[0058] Step S1.5: Target physical size estimation.

[0059] The pixel width and height of the detection box can also be converted to the physical distance corresponding to each pixel based on the measurement range, mapping it to the physical world to obtain the physical size of the detected target. For detection boxes with unreasonable pixel sizes (too large or too small) based on prior knowledge (such as the minimum / maximum target physical size and the current estimated distance), filters are removed. Detections appearing in physically invalid areas of the sonar image (such as sonar near-field blind spots, areas with severe image edge distortion) are excluded. Results identified when the vehicle's state is unstable (such as excessive speed, excessive pitch angle, excessive roll angle, etc.) are also excluded.

[0060] Step S2: Based on tracker storage Track trajectories in real time and predict the location of each target. The trajectory points at any given moment.

[0061] The tracker of this invention is designed to correlate the detected trajectory at time t with the historical trajectory, update the target state, and filter out noise. For each existing tracking trajectory... Using its current state (position, velocity) and motion model (such as uniform velocity or uniform acceleration model), predict its position at time t. .

[0062] Step S3: Based on the association cost matrix between all detection points and all trajectory points, generate... The target detection result at any given time; wherein, the association cost matrix includes spatial distance cost, appearance feature cost, and physical size similarity cost.

[0063] First, this invention calculates all detection points. With all predicted trajectory points The cost matrix is ​​determined by considering three types of costs and combining them to obtain the final cost matrix: spatial distance cost (geospatial 3D distance), appearance feature cost (using the CNN features extracted in the previous steps to calculate the cosine distance between feature vectors), and physical size similarity cost (similarity of width and height). Optimal matching is performed using the Hungarian algorithm or a greedy algorithm to obtain matched pairs, unmatched detections (new targets), and unmatched trajectories (target loss). Each tracking trajectory... Record the target's velocity and the target distance at time t, which matches the same trajectory. Distance from the recognition result at time t-1 The distance offset is calculated by the difference between the two values. Then, the time offset is obtained by subtracting time t-1 from time t. The two values ​​are then divided to obtain the distance offset.

[0064] Next, the Kalman filter is updated. For successfully matched "trajectory-detection" pairs, the Kalman filter is used to update the detection points. With predicted trajectory points The system fuses the data to obtain a smoothed target state (including velocity) and updates the covariance. Trajectory initialization: For unmatched detections, if their confidence is high and they appear in N consecutive frames, they are initialized as a new trajectory. Trajectory management: Unmatched trajectories are subject to "loss counting." If a trajectory is lost for more than a certain number of consecutive frames, it is determined that the target has left or disappeared, and the trajectory is terminated.

[0065] Finally, calculate each trajectory The average speed is used to determine the target. If the speed remains below a certain threshold for an extended period, the target is classified as a static target (e.g., reefs, shipwrecks); otherwise, it is classified as a dynamic target. Static targets should maintain a highly stable geographical location across multiple frames. Targets with large positional fluctuations or that appear and disappear intermittently are likely noise and can be filtered out using trajectory length and stability.

[0066] In this scheme, all thresholds can be set to any floating-point number greater than or equal to 0, and there are no restrictions on the feature extraction network structure (any general detection and recognition network structure can be used).

[0067] In summary, this invention uses the forward-looking sonar image at time t (size: The detection and recognition results are calculated into latitude, longitude, and depth (if any) information in the actual physical world, denoted as a three-dimensional coordinate system. The system identifies a point in the track and records its state information (boundary, width, height, latitude, longitude, depth, velocity, etc.), updating this information in the tracker's state queue. Finally, it generates the target detection result using the association cost matrix between the detection point and all trajectory points.

[0068] Based on the same concept, this invention also discloses a continuous frame target detection system for sonar images based on tracker fusion, such as... Figure 2 As shown, the system includes: The detection point generation module is used to obtain... The target detection result of the sonar image at any time, wherein the target detection result consists of at least one detection point; The trajectory point generation module is used for tracking points stored by the tracker. Track trajectories in real time and predict the location of each target. The trajectory points at any given moment; The detection result generation module is used to generate a result based on the association cost matrix between all detection points and all trajectory points. The target detection result at any given time; wherein, the association cost matrix includes spatial distance cost, appearance feature cost, and physical size similarity cost.

[0069] Based on the same concept, this invention also discloses a computer device, which may be a terminal, a laptop computer, a desktop computer, a server, a computer cluster, or other types of computer devices. For example... Figure 3 As shown, the computer device may include at least one processor and memory. The processor can execute instructions stored in the memory. The processor is communicatively connected to the memory via a data bus. In addition to the memory, the processor can also be communicatively connected to input devices, output devices, and communication devices via the data bus.

[0070] The processor can be any conventional processor. Processors may include central processing units (CPUs), graphics processing units (GPUs), field-programmable gate arrays (FPGAs), systems on chips (SoCs), application-specific integrated circuits (ASICs), or combinations thereof.

[0071] Memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0072] In this embodiment of the invention, an executable instruction is stored in a memory. The processor can read the executable instruction from the memory and execute the instruction to implement all or part of the steps of the method of the invention.

[0073] Based on the same concept, the present invention also discloses a computer-readable storage medium including a computer program product or storing the computer program product. The computer product includes computer program instructions that can be executed by a processor to perform all or part of the steps described in the exemplary embodiments above.

[0074] Computer program products can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this application. These programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages, and scripting languages ​​(e.g., Python). The program code can be executed entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0075] Computer-readable storage media can take the form of any combination of one or more readable media. A readable medium can be a readable signal medium or a readable storage medium. A readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media include: static random access memory (SRAM) having one or more electrically connected wires; electrically erasable programmable read-only memory (EEPROM); erasable programmable read-only memory (EPROM); programmable read-only memory (PROM); read-only memory (ROM); magnetic storage; flash memory; magnetic disk or optical disk; or any suitable combination thereof.

[0076] The above embodiments are merely illustrative of the technical solutions of the present invention and are not intended to limit it. Those skilled in the art can modify or make equivalent substitutions to the above technical solutions based on the concept of the present invention, and such modifications or equivalent substitutions should all be covered within the protection scope of the present invention. The protection scope of the present invention is defined by the claims.

Claims

1. A method for continuous frame target detection in sonar images based on tracker fusion, characterized in that, The method includes: Get The target detection result of the sonar image at any time, wherein the target detection result consists of at least one detection point; Based on tracker storage Track trajectories in real time and predict the location of each target. The trajectory points at any given moment; Based on the association cost matrix between all detection points and all trajectory points, generate The target detection results at any given time; wherein, the association cost matrix includes spatial distance cost, appearance feature cost, and physical size similarity cost; Among them, obtaining Target detection results from real-time sonar images include: right The sonar images are preprocessed, including enhancement processing and filtering processing. Generate the target detection bounding box and its confidence level and category in the preprocessing result; Based on the confidence level and the pixel width and pixel height of the target detection box, the target detection box is filtered. Based on the position representation of the filtered target detection boxes in 3D space, we obtain Target detection results in real-time sonar images; The process of filtering target detection boxes based on the confidence level and the pixel width and pixel height of the target detection box includes: Set a first confidence threshold and perform preliminary screening of target detection boxes based on the confidence of the target detection boxes; Based on the initial screening results, the physical size of the target is estimated according to the pixel width and pixel height of the target detection box, and the target detection box is further screened based on the target physical size to obtain a set of target detection boxes; Add the target detection box with the highest confidence score from the target detection box set to the final output list; Calculate the intersection-union ratio (IU) between the highest-confidence bounding box and the remaining bounding boxes; Calculate the orientation penalty factor based on the x-coordinate of the center pixel of the highest confidence bounding box and the remaining bounding boxes. , These are parameters related to sonar azimuth resolution; The confidence of the remaining target detection boxes is updated based on the intersection-union ratio and the orientation penalty factor; A second confidence threshold is set, and the target detection box set is updated after filtering the remaining target detection boxes based on the confidence update results; wherein, the confidence update results... ; Based on the updated set of target detection boxes, the process of adding the target detection box with the highest confidence in the set to the final output list is repeated until the set of target detection boxes is empty.

2. The method according to claim 1, characterized in that, right The time-lapse sonar images are enhanced, including: Get The echo intensity of the sonar image at time t is calculated, and the echo intensity is logarithmically compressed to obtain the compressed sonar image. The compressed sonar image is divided into several non-overlapping local regions, and the histogram of each local region is subjected to contrast cropping. The number of cropped pixels is evenly distributed across all gray levels, and the histogram is equalized based on the distribution results to generate a gray-level mapping function for the local region. Bilinear interpolation is used to merge the grayscale mapping results of all local regions to obtain the enhanced result. Real-time sonar images.

3. The method according to claim 1, characterized in that, right The sonar image at any given time undergoes filtering processing, including: Computational enhancement The structural tensor of the time-series sonar image; Calculate the first and second eigenvalues ​​of the structure tensor, and generate the first and second eigenvectors corresponding to the first and second eigenvalues ​​respectively; wherein the first eigenvalue represents the gradient change intensity of the edge direction, and the second eigenvalue represents the gradient change intensity of the vertical edge direction; Design a diffusion tensor based on the first and second eigenvectors; Based on the diffusion tensor and the enhanced processing The gradient of the image intensity function corresponding to the sonar image at time step 1 is used to obtain the partial differential equation describing the diffusion process. Based on this partial differential equation, the filtered result is obtained. Real-time sonar images.

4. The method according to claim 1, characterized in that, Generate the target detection bounding box and its confidence level and category in the preprocessing result, including: Construct an object detection model, which includes: a backbone network and a decoupled detection head; A feature pyramid network structure is introduced to obtain feature maps of different levels of the backbone network; For a feature map of a certain level output by the feature pyramid network structure, the target detection box in the corresponding level and the confidence and category of the target detection box are obtained based on the decoupled detection head; wherein, the decoupled detection head includes a classification branch and a regression branch, the classification branch is used to calculate the confidence and category of the target detection box, and the regression branch is used to predict the position of the target detection box; The loss used to train the object detection model includes: classification loss, regression loss, and object property loss; wherein the object property loss is used to distinguish between foreground and background.

5. The method according to claim 1, characterized in that, The process of obtaining the position representation of the filtered target detection box in 3D space includes: Calculate the slant range and azimuth angle of the target detection box based on the center coordinates of the target detection box; Calculate the latitude and longitude corresponding to the center of the target detection box based on the slant distance and azimuth angle; Calculate the target depth based on binocular sonar, depth sounding sonar, or vertical sonar; By combining the latitude and longitude of the center of the target detection box with the target depth, the position representation of the target detection box in 3D space is obtained.

6. The method according to claim 1, characterized in that, Based on the association cost matrix between all detection points and all trajectory points, generate The target detection results at any given time include: Based on the association cost matrix, the Hungarian algorithm or greedy algorithm is applied to obtain the matching results between the detection points and the trajectory points; wherein, the matching results include: matching pairs, unmatched detections, and unmatched trajectories; Based on the matching results, generate The trajectory is tracked in real time; for matched pairs, a Kalman filter is used to connect the detection points with the target points. Real-time tracking trajectory fusion; for unmatched detections, if the confidence of the detection point is higher than the set threshold and appears continuously in N frames, it is initialized as a new tracking trajectory; for unmatched trajectories, if the trajectory point fails to match successfully in N consecutive frames, the corresponding tracking trajectory is terminated. By tracking the length and stability of the trajectory Continuously track and filter the trajectory; Calculate the average speed of each tracking trajectory, and determine whether the target is a static or dynamic target based on the average speed.

7. A continuous frame target detection system for sonar images based on tracker fusion, characterized in that, The system includes: The detection point generation module is used to obtain... The target detection result of the sonar image at any time, wherein the target detection result consists of at least one detection point; The trajectory point generation module is used for tracking points stored by the tracker. Track trajectories in real time and predict the location of each target. The trajectory points at any given moment; The detection result generation module is used to generate a result based on the association cost matrix between all detection points and all trajectory points. The target detection results at any given time; wherein, the association cost matrix includes spatial distance cost, appearance feature cost, and physical size similarity cost; Among them, obtaining Target detection results from real-time sonar images include: right The sonar images are preprocessed, including enhancement processing and filtering processing. Generate the target detection bounding box and its confidence level and category in the preprocessing result; Based on the confidence level and the pixel width and pixel height of the target detection box, the target detection box is filtered. Based on the position representation of the filtered target detection boxes in 3D space, we obtain Target detection results in real-time sonar images; The process of filtering target detection boxes based on the confidence level and the pixel width and pixel height of the target detection box includes: Set a first confidence threshold and perform preliminary screening of target detection boxes based on the confidence of the target detection boxes; Based on the initial screening results, the physical size of the target is estimated according to the pixel width and pixel height of the target detection box, and the target detection box is further screened based on the target physical size to obtain a set of target detection boxes; Add the target detection box with the highest confidence score from the target detection box set to the final output list; Calculate the intersection-union ratio (IU) between the highest-confidence bounding box and the remaining bounding boxes; Calculate the orientation penalty factor based on the x-coordinate of the center pixel of the highest confidence bounding box and the remaining bounding boxes. , These are parameters related to sonar azimuth resolution; The confidence of the remaining target detection boxes is updated based on the intersection-union ratio and the orientation penalty factor; A second confidence threshold is set, and the target detection box set is updated after filtering the remaining target detection boxes based on the confidence update results; wherein, the confidence update results... ; Based on the updated set of target detection boxes, the process of adding the target detection box with the highest confidence in the set to the final output list is repeated until the set of target detection boxes is empty.

8. A computer device, characterized in that, The computer device includes: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, it implements the sonar image continuous frame target detection method based on tracker fusion as described in any one of claims 1-6.

Images