Deep sea target detection and tracking method and system

Abstract

The present invention relates to a detection and tracking method and system, in particular, a deep-sea target detection and tracking method and system. According to the technical solution provided by the present invention, a deep-sea target detection and tracking method, the target detection and tracking method comprises: acquiring a sonar image of a deep-sea target; performing non-local mean filtering processing on the acquired sonar image in a GPU, wherein when the sonar image is processed by the non-local mean filtering method, the non-local mean filtering operation processing is performed in the form of an integral graph and accelerated by the three-dimensional thread of the GPU; the sonar image after the non-local mean filtering processing is subjected to the required target detection and tracking to generate a target detection and tracking ROI area. The present invention can effectively realize the detection and tracking of underwater targets, assist underwater vehicles in locking deep-sea targets, and improve the efficiency of target detection and tracking.

Description

Technical Field

[0001] The present invention relates to a detection and tracking method and system, and in particular to a deep-sea target detection and tracking method and system. Background Art

[0002] The ocean is the cradle of life, a treasure of resources, a channel for trade, and a barrier for national defense. The ocean is increasingly valued by governments and favored by scientists. Deep-sea scientific research, deep-sea resource exploration, and deep-sea engineering underwater operations are inseparable from high-tech marine detection equipment and require the support of high-tech marine technology.

[0003] With the deepening of marine scientific research, higher requirements are placed on marine survey operation technology. Ships can carry scientists, engineering and technical personnel and various detection equipment to the destination quickly and accurately for efficient exploration and scientific investigation, becoming an important technical means to explore the mysteries of the deep sea.

[0004] At present, underwater vehicles operate in the deep sea for a long time. Due to the lack of automatic detection mechanism for underwater targets, manual target determination is inefficient and accurate. In addition, humans are unable to effectively track targets and cannot collaborate with other automated equipment, which seriously affects the progress of unmanned automation systems for underwater vehicles.

[0005] Therefore, in deep-sea navigation operations, an automated method is needed to improve the efficiency and accuracy of target determination and promote the unmanned automation of underwater vehicles. However, there is currently a blank in the automatic detection and tracking methods for underwater targets. Summary of the invention

[0006] The purpose of the present invention is to overcome the deficiencies in the prior art and provide a deep-sea target detection and tracking method and system, which can effectively realize the detection and tracking of underwater targets, assist underwater vehicles in locking deep-sea targets, and improve the efficiency of target detection and tracking.

[0007] According to the technical solution provided by the present invention, a deep-sea target detection and tracking method comprises:

[0008] Acquire sonar images of deep-sea targets;

[0009] The acquired sonar image is processed by non-local mean filtering in the GPU, wherein when the sonar image is processed by the non-local mean filtering method, the non-local mean filtering operation is performed in the form of an integral graph and accelerated by using the three-dimensional thread of the GPU;

[0010] The sonar image processed by non-local mean filtering is subjected to the required target detection and tracking to generate a target detection and tracking ROI area.

[0011] The sonar images of deep-sea targets are collected by multi-beam sonar, where:

[0012] The sonar image collected by the multi-beam sonar is transmitted to the target detection tracker, so that the target detection tracker obtains the sonar image after receiving it;

[0013] After the target detection tracker acquires the sonar image, it sends the acquired sonar image to the video memory of the GPU so as to perform non-local mean filtering on the sonar image in the GPU.

[0014] When performing the operation of non-local mean filtering in the form of an integral graph in the GPU, the process of performing the operation includes:

[0015] For sonar images, a grid model of non-local mean filtering is constructed in the GPU, wherein any layer in the grid model calculates a mean square integral map of the non-local mean filtering;

[0016] When calculating the integral graph, first calculate the row integral graph;

[0017] Perform column integration on the above row integral graph to obtain the corresponding mean square value integral graph.

[0018] The process of calculating the row integral graph includes:

[0019] Each row of the matrix is ​​divided into several groups, and each thread calculates the sum of the data prefixes of each group;

[0020] Calculate the prefix sum of the last data in each group;

[0021] Starting from group 2, add the numbers in each column (except the last group) to the last number in the previous group. This will give you a complete row integral diagram.

[0022] When performing column integration based on a row integral graph, this includes:

[0023] Each column of the matrix is ​​divided into several groups, and each thread calculates the sum of the data prefixes of each group;

[0024] Calculate the prefix sum of the last data in each group;

[0025] Starting from group 2, add the numbers in each column (except the last group) to the last number in the previous group to get the mean square value integral graph.

[0026] When performing target detection and tracking, it includes manually selected target tracking and automatic target detection and tracking, among which,

[0027] When manually selecting target tracking, manually selecting a to-be-detected and tracked ROI region on the sonar image, so as to generate a target detection and tracking ROI region based on the manually selected to-be-detected and tracked ROI region;

[0028] When automatic target detection is adopted, n to-be-detected and tracked ROI regions are configured to generate n target detection and tracking ROI regions based on the configured n to-be-detected and tracked ROI regions.

[0029] When generating a target detection and tracking ROI area based on the manually selected target detection and tracking ROI area, it includes:

[0030] For the manually selected ROI area to be detected and tracked, the target area is matched by using the template matching method, so as to determine the position information of the target area to be determined after the target area is matched;

[0031] The target detection is determined for the position information of the determined target area. When the determined target area is compatible with the manually selected target detection and tracking ROI area, the target area is generated as a target detection and tracking ROI area by using a circumscribed rectangle.

[0032] When the template matching method is used to match the target area, the template matching method used is normalized square difference matching;

[0033] When generating the target detection and tracking ROI area, the Canny operator is used to perform edge detection on the target area, so that after the edge detection, the edge of the target area is used as a circumscribed rectangle to generate the target detection and tracking ROI area.

[0034] For automatic target detection, the target detection includes:

[0035] Identify the target on the sonar image, extract the edge contour of the identified target, and use the template matching method to match the target area after the edge contour is extracted, so as to determine the position information of the target area to be determined after the target area is matched;

[0036] The target detection is determined for the position information of the determined target area. When the determined target area is compatible with the manually selected target detection and tracking ROI area, the target area is generated as a target detection and tracking ROI area by using a circumscribed rectangle.

[0037] A deep-sea target detection and tracking system, comprising:

[0038] Multi-beam sonar, used to collect sonar images of deep-sea targets;

[0039] The target detection tracker obtains the sonar image collected by the multi-beam sonar and sends the obtained sonar image to the GPU memory to perform non-local mean filtering on the sonar image in the GPU, where:

[0040] When the sonar image is processed by the non-local mean filtering method, the non-local mean filtering operation is performed in the form of an integral image and accelerated by using the three-dimensional thread of the GPU;

[0041] The sonar image processed by non-local mean filtering is subjected to the required target detection and tracking to generate a target detection and tracking ROI area.

[0042] The advantages of the present invention are as follows: the sonar image is denoised by using non-local mean filtering, which utilizes the non-correlated characteristics of noise. In an image, there are many image blocks with the same pixels, and the noise therein is irrelevant. However, there are many image blocks with the same pixels in the sonar image, so the non-local mean filtering is very suitable for multi-beam sonar images. However, it has extremely high computational complexity. The present invention performs the processing of the sonar image by non-local mean filtering in the GPU in an integral graph manner, which can not only maintain a good filtering effect, but also ensure good real-time performance.

[0043] Compared with optical image target detection and tracking, sonar images have the characteristics of low resolution, unclear target features, and severe noise, which brings certain challenges to target detection and tracking. Compared with advanced algorithms that rely on target features for tracking, the present invention adopts a method assisted by template matching algorithm, target segmentation, GPU acceleration, etc., which achieves the effect of target tracking in low-resolution images while ensuring the stability and real-time performance of target detection and tracking.

[0044] The non-local mean filtering of sonar images is performed in the GPU, which opens up far more computing cores than the image processing unit has. As the number of image processing unit cores increases, the real-time performance of deep-sea target detection and tracking will be further improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 A schematic diagram of an embodiment of a deep-sea detection target tracking system of the present invention.

[0046] Figure 2 A flow chart of an embodiment of target detection and tracking according to the present invention.

[0047] Figure 3 A schematic diagram of the present invention performing non-local mean filtering in a GPU.

[0048] Figure 4This is a schematic diagram of an embodiment of the first step of integration according to the present invention.

[0049] Figure 5 This is a schematic diagram of an embodiment of the second step of the integration process of the present invention.

[0050] Figure 6 This is a schematic diagram of an embodiment of the third step of integration of the present invention.

[0051] Figure 7 Schematic diagram of the first step of the column integration embodiment of the present invention.

[0052] Figure 8 It is a schematic diagram of an embodiment of the second step of column integration of the present invention.

[0053] Fig. 9 It is a schematic diagram of an embodiment of the third step of column integration of the present invention.

[0054] Fig.10 It is a schematic diagram of an embodiment of non-uniform non-average value filtering of the present invention.

[0055] Explanation of reference numerals: 1 - target detection tracker and 2 - multi-beam sonar. DETAILED DESCRIPTION

[0056] The present invention will be further described below in conjunction with specific drawings and embodiments.

[0057] In order to effectively realize the detection and tracking of underwater targets and improve the efficiency of target detection and tracking, a deep-sea target detection and tracking method is provided. In one embodiment of the present invention, the target detection and tracking method includes:

[0058] Acquire sonar images of deep-sea targets;

[0059] The acquired sonar image is processed by non-local mean filtering in the GPU, wherein when the sonar image is processed by the non-local mean filtering method, the non-local mean filtering operation is performed in the form of an integral graph and accelerated by using the three-dimensional thread of the GPU;

[0060] The sonar image processed by non-local mean filtering is subjected to the required target detection and tracking to generate a target detection and tracking ROI area.

[0061] Figure 1 An embodiment of deep-sea target detection and tracking is shown, which includes at least one multi-beam sonar 2 and at least one target detection tracker 1 adapted to be connected to the multi-beam sonar. The multi-beam sonar 2 can adopt an existing commonly used form, and the specific form shall be based on whether it can meet the requirements of obtaining the required sonar image.

[0062] In one embodiment of the present invention, the sonar image of the deep-sea target is collected by a multi-beam sonar 2, wherein:

[0063] The sonar image collected by the multi-beam sonar 2 is transmitted to the target detection tracker 1, so that the target detection tracker 1 obtains the sonar image after receiving it;

[0064] After the target detection tracker 1 acquires the sonar image, the acquired sonar image is sent to the video memory of the GPU so as to perform non-local mean filtering on the sonar image in the GPU.

[0065] The target detection tracker 1 can use existing commonly used computer equipment. In order to improve the efficiency of target detection and tracking, a GPU (graphics processing unit) unit needs to be configured in the target detection tracker 1. The configured GPU unit can be an NVIDIA GEFORCE RTX3070 image processing unit. Of course, the configured GPU unit can also adopt other forms, which shall be based on whether it can meet the requirements of using GPU for non-local mean filtering processing.

[0066] After the multi-beam sonar 2 collects the sonar image, it transmits the collected sonar image to the target detection tracker 1, that is, the target detection tracker 1 obtains the sonar image. When using the GPU for non-local mean filtering, the target detection tracker 1 must first apply for the GPU's video memory space, such as applying for the memory space in the GPU through the API function in CUDA. The size of the space applied for is dynamically allocated according to the image size, and its memory first address is recorded in the pointer variable. After applying for the GPU's memory space, the sonar data is copied to the GPU's video memory, specifically sending the sonar image to the GPU's video memory, and controlling the GPU to perform non-local mean filtering on the sonar image. The method and process can be consistent with the existing ones.

[0067] In specific implementation, the non-local mean filtering algorithm can effectively process sonar images to remove noise from sonar images, but the real-time performance of the processing is poor. In one embodiment of the present invention, the operation processing of non-local mean filtering is performed in the form of an integral graph and accelerated by the three-dimensional thread of the GPU, thereby achieving the purpose of improving the efficiency of sonar image processing using non-local mean filtering. In addition, in the process of calculating the integral graph, the segmented calculation method is used to further accelerate. The original four-layer loop can be reduced to a single-layer loop, which greatly reduces the running time of the non-local mean filtering method, so that it can be used in the real-time processing of sonar target tracking.

[0068] In one embodiment of the present invention, when performing a non-local mean filtering operation in a GPU in an integral graph manner, the process of performing the operation includes:

[0069] For sonar images, a grid model of non-local mean filtering is constructed in the GPU, wherein any layer in the grid model calculates a mean square integral map of the non-local mean filtering;

[0070] When calculating the integral graph, first calculate the row integral graph;

[0071] Perform column integration on the above row integral graph to obtain the corresponding mean square value integral graph.

[0072] by Fig.10 Take as an example to explain the principle of non-local mean filtering algorithm. Fig.10 In the figure, A is the point to be filtered, then for the point to be filtered A, we have:

[0073]

[0074] Where P(A) represents the pixel value of point A, and w is the size of the search box of P(A) (the box formed by points B and C). P(i,j) is the weight of the pixel W(i,j) in the search box. W(i,j) can be determined by the following formula:

[0075]

[0076] in, Where m is the size of the neighborhood window of point A (the search box where point A is located), M A is the area of ​​point A, M x is the domain of point x, point x is the location of the search box where point A is located, and h is the Gaussian parameter.

[0077]

[0078] From the above description, we can see that for a point to be filtered, since the size of the search box is w*w, we need to calculate w 2 MSE (similarity). Assume the coordinates of the point A to be filtered are (i, j), M (i,j) is the neighborhood of point (i,j), then:

[0079]

[0080] For any filtering point, formula (5) is all the MSEs that need to be calculated. In the above calculation, the similarity (MSE) is calculated with each pixel point (to be filtered) as a unit. Each time a pixel point is traversed, the MSE is calculated w*w times. Specifically, after calculating a point to be filtered, the next filtering point is calculated. By taking different values ​​for i and j, the entire image can be traversed, but a large number of repeated calculation parts will appear in the calculation process.

[0081] In one embodiment of the present invention, the integral graph of the entire image matrix is ​​calculated, and the corresponding MSE can be obtained by table lookup. The specific method and process of obtaining the corresponding MSE by table lookup using the integral graph are consistent with the existing ones.

[0082] Assume that the image matrix of the sonar image is src. When calculating the similarity MSE first, all the squares of the differences that need to be calculated can be obtained by formula (6), specifically:

[0083]

[0084] Among them, diff (a,b) 2 (i, j) is the square of the difference; all the squares of the difference in formula (6) are the MSEs of all the sonar images to be calculated. Therefore, when calculating the integral map, the image matrix src is calculated to obtain the corresponding integral map. For the calculation of the square of the difference in formula (6), since each calculation formula is independent of each other, the GPU can be used for calculation, and then the mean square value calculated above can be used to obtain the integral map.

[0085] Figure 3 To construct a grid model for non-local mean filtering, each layer of the grid (composed of blocks) is used to calculate a diff 2 Integral graph (Formula 6), w integral graphs are calculated in one loop. After w loops, all required integral graphs can be calculated. One layer of the grid model calculates one formula in Formula 6 and integrates it. Because the grid model has only w layers, only w formulas can be calculated, so one layer of loop is required.

[0086] Figure 4 The process shown shows the process of calculating an integral graph: the general process is to accumulate the data in rows and then in columns to obtain the integral graph. In the specific implementation, the calculation of the row accumulation and column accumulation uses the group calculation method. Figure 4 The squares in the middle represent the diff of a certain layer. (a,b) 2 (i,j).

[0087] Figure 4 In the process of row integration, each row of data is divided into a group of T data. Figure 4 In the example, 8 data are grouped together and summed up separately. It can be expressed by the following formula, with each thread calculating one formula.

[0088]

[0089] In the above formula, a is the position of the image matrix src; b is the result after executing the first step; T represents the amount of data in each group; j represents the column index. For k, when there are Nz data in a row, k is Nz / T-1. The value of k in the following is the same.

[0090] Figure 5 The second step of grouped integral summation is shown. Since the row integral is calculated in groups, the part that was not accumulated in the first step is accumulated. In this step, the last data of each group (for example, when 8 data are divided into 1 group, the 8th data in each group) is integrated. At this point, the integration of the last data in each group is completed, which can be expressed by the following formula.

[0091]

[0092] In the above formula, c is the result after executing the second step, and j represents the column index.

[0093] Figure 6 The third step of grouping integral summation is shown in Figure 1. Now the missing data in each group (except the last data in each group) is the integral of the previous group. Therefore, the integral of the previous group, that is, the last data, is added to each group. This process can be expressed by the following formula:

[0094]

[0095] In the above formula, c is the result after executing the second step, j represents the column index, i represents the row index, and d is the result after executing the third step.

[0096] From the above description, it can be concluded that the process of calculating the row integral graph includes:

[0097] Each row of the image matrix is ​​divided into several groups, and each thread calculates the sum of the data prefixes of each group;

[0098] Calculate the prefix sum of the last data in each group;

[0099] Starting with group 2, add the numbers in each column (except the last group) to the last number in the previous group to get a complete row integral graph.

[0100] In the above, for the sum of data prefixes, a specific embodiment is given below. Specifically, for one-dimensional data: 1, 2, 3, 4, 5; the process of calculating the sum of prefixes for this data is as follows: 1

[0102] 1+2=3

[0103] 3+3=6

[0104] 6+4=10

[0105] 10+5=15

[0106] That is, integration, and finally we get the following set of data: 1, 3, 6, 10, 15.

[0107] During specific implementation, the process of summing the prefixes may refer to the above description and will not be repeated here.

[0108] In the specific implementation, during the column integration process, each column of data is divided into a group of T data. Figure 7 In the example, 8 data are grouped together and summed up respectively. It can be expressed by the following formula, with each thread calculating one formula:

[0109]

[0110] In the above formula, d is the result of the third step; e is the result of the fourth step. The formula here is the same as Figure 4 The calculation method is the same as in . The row integral is completed through the first three steps to obtain the d matrix. Here, column integral is performed on the basis of the row integral, that is, on the basis of the d matrix, and each column is grouped, also with T data in one group.

[0111] Figure 8 The second step of summing up the grouped column integral is shown in the figure. Since the column integral is calculated in groups, the part that was not accumulated in the first step needs to be accumulated. In this step, the last data (red part) of each group is integrated. In this way, the integration of the last data in each group is completed. It can be expressed by the following formula.

[0112]

[0113] superior Fig. 9 The third step of grouping integral summation is shown in . Now the missing data in each group (except the last data in each group) is the integral of the previous group. Therefore, the integral of the previous group is added to each group, that is, the last data. This process can be expressed by the following formula:

[0114]

[0115] In summary, column integration based on row integral graph includes:

[0116] Each column of the matrix is ​​divided into several groups, and each thread calculates the sum of the data prefixes of each group;

[0117] Calculate the prefix sum of the last data in each group;

[0118] Starting from group 2, add the numbers in each column (except the last group) to the last number in the previous group to get the mean square value integral graph.

[0119] From the above description, we can see that the entire algorithm needs to calculate w*w integral images. Any layer in the grid model calculates a mean square integral image of the non-local mean filter; opening w layers for parallel calculations can achieve three-dimensional thread acceleration.

[0120] When using sequential devices such as CPU to complete filtering, 4 layers of loops are required. To calculate the MSE of a point to be filtered within its search box, 2 layers of loops are required to control it. To filter the entire image, it is necessary to traverse the entire image, which is a 2-layer loop. In the embodiment of the present invention, calculations are performed on the entire image. The 2-layer loop is removed, but the entire algorithm requires w*w integral images for table lookup. Since w integral images can be calculated at one time, a loop is required to calculate w*w integral images to complete the filtering of the entire image, which is a layer of loop. The above calculation is a calculation step for one layer in the entire three-dimensional grid model, with a total of w layers, all performing the same operation.

[0121] In one embodiment of the present invention, target detection and tracking include manually selected target tracking and automatic target detection and tracking, wherein:

[0122] When manually selecting target tracking, manually selecting a to-be-detected and tracked ROI region on the sonar image, so as to generate a target detection and tracking ROI region based on the manually selected to-be-detected and tracked ROI region;

[0123] When automatic target detection is adopted, n to-be-detected and tracked ROI regions are configured to generate n target detection and tracking ROI regions based on the configured n to-be-detected and tracked ROI regions.

[0124] In specific implementation, according to different target detection and tracking scenarios, target detection and tracking can generally be performed manually selected target tracking or automatic target detection and tracking. The following describes the process of implementing target detection and tracking under manual selected target tracking and automatic target detection and tracking respectively.

[0125] Further, when generating the target detection and tracking ROI area based on the manually selected target detection and tracking ROI area, it includes:

[0126] For the manually selected ROI area to be detected and tracked, the target area is matched by using the template matching method, so as to determine the position information of the target area to be determined after the target area is matched;

[0127] The target detection is determined for the position information of the determined target area. When the determined target area is compatible with the manually selected target detection and tracking ROI area, the target area is generated as a target detection and tracking ROI area by using a circumscribed rectangle.

[0128] In specific implementation, after the sonar image is processed by non-local mean filtering in the GPU, the processed sonar image needs to be copied back to the video memory of the target detection tracker 1. When manually selecting the ROI area to be detected and tracked, it is generally necessary to determine whether the ROI area to be detected and tracked has been selected. Generally, when it has not been selected, it is necessary to wait or repeat the above process of acquiring the sonar image and performing non-local mean filtering on the sonar image until the ROI area to be detected and tracked is manually selected. Specifically, the commonly used technical means in the technical field can be used to determine whether the ROI area to be detected and tracked has been selected.

[0129] For the selected target detection and tracking ROI area, the target area is matched using the template matching method. Generally, the template used for template matching is a known small image. Therefore, template matching is to search for the target in a large image. It is known that the image contains the target to be found, and the target has the same size, direction and image elements as the template. In specific implementation, the selected target detection and tracking ROI area is used as the template for matching. Template matching principle: Let the template matrix be T(x',y'), the source image matrix be I(x,y), and the result matrix be R(x,y), then the method for measuring the similarity between the template and the source image matrix can be obtained.

[0130] In one embodiment of the present invention, the template matching uses the normalized square difference matching method, which can fully meet the matching requirements and ensure real-time performance. The normalized square difference matching method is used to find the area most similar to the target detection and tracking ROI area in all areas, and the coordinates of the center position of the similar area, i.e., the coordinates of the target, are solved.

[0131] After matching the sonar images using the template matching method, the Euclidean distance between the matching position and the previous matching position is calculated and recorded. Check whether the Euclidean distance exceeds the set threshold. Generally, the threshold is set based on the recorded Euclidean distance. The moving speed of the object can be calculated based on the recorded Euclidean distance. The moving speed of the object is multiplied by the movement time (the interval between two frames of images) to obtain the value of the object's movement range. 1.5 times this value is configured as the threshold. At this point, dynamic thresholds can be implemented for objects with different movement speeds to achieve the effect of tracking different objects.

[0132] When the Euclidean distance exceeds the threshold, it indicates that the template matching is in a matching error state; since there are many similar areas in the sonar image, when the matching position changes greatly, it proves that the target matching at this time is an incorrect result or the target has been lost. Template matching is to find the most similar area in the image. When the target is lost, a result will still be given. The existence of the threshold is to verify whether the result is correct.

[0133] In specific implementation, when the Euclidean distance does not exceed the threshold, the detection and tracking of the target is maintained. When the Euclidean distance exceeds the threshold, the Euclidean distance between the current position of all targets and the position detected by the last template matching is output, and the target closest to the last detected position is found, and the minimum enclosing rectangle is drawn for this target, and the ROI area is automatically generated using the coordinates of the enclosing rectangle. In addition, when generating the target detection and tracking ROI area, the Canny algorithm is used to perform edge detection on the target area, so that after edge detection, the edge of the target area is used as a enclosing rectangle to generate the target detection and tracking ROI area. The method and process of edge detection of the target area using the Canny algorithm are consistent with the existing ones.

[0134] Further, for automatic target detection, the target detection includes:

[0135] Identify the target on the sonar image, extract the edge contour of the identified target, and use the template matching method to match the target area after the edge contour is extracted, so as to determine the position information of the target area to be determined after the target area is matched;

[0136] The target detection is determined for the position information of the determined target area. When the determined target area is compatible with the manually selected target detection and tracking ROI area, the target area is generated as a target detection and tracking ROI area by using a circumscribed rectangle.

[0137] When performing automatic target detection, the target is segmented on the sonar image, and the edge contours of all segmented targets are extracted. False targets are excluded based on the target's area, length and other information. Among all real targets, the set n targets are found, and n templates are selected for template matching. The template matching method and process can be referred to the above description and will not be repeated here. In specific implementation, the n targets are the n targets automatically selected in the first image, that is, the selected n matching templates.

[0138] The Euclidean distance between the n target matching positions and the last matching positions is calculated to check whether the Euclidean distance exceeds the set threshold. Since n targets are being tracked, the moving positions of the n targets are stored here, and the moving ranges of the n targets are calculated to determine the threshold of the n targets.

[0139] When automatically detecting and tracking a target, the method and process of using thresholds to detect and track a target are similar. For details, please refer to the above description. For manual target tracking and automatic target detection and tracking, the specific implementation process of target detection and tracking can be referred to Figure 2 Flowchart of the process.

[0140] In summary, a deep-sea target detection and tracking system can be obtained. In one embodiment of the present invention, the system includes:

[0141] Multibeam sonar 2, used to collect sonar images of deep-sea targets;

[0142] The target detection tracker 1 obtains the sonar image collected by the multi-beam sonar 2, and sends the obtained sonar image to the GPU memory to perform non-local mean filtering on the sonar image in the GPU, wherein:

[0143] When the sonar image is processed by the non-local mean filtering method, the non-local mean filtering operation is performed in the form of an integral image and accelerated by using the three-dimensional thread of the GPU;

[0144] The sonar image processed by non-local mean filtering is subjected to the required target detection and tracking to generate a target detection and tracking ROI area.

[0145] During specific implementation, the multi-beam sonar 2 and the target detection tracker 1 can refer to the above description and will not be repeated here.

[0146] From the above description, it can be seen that when the present invention detects and tracks sonar targets, it must first have a good denoising effect to provide a basis for target segmentation and facilitate target tracking. Secondly, the actual application should have good real-time performance and room for performance improvement.

[0147] Therefore, compared with the previous target detection and tracking, the present invention adopts non-local mean filtering to denoise the sonar image, and the non-local mean filtering utilizes the non-correlated characteristics of noise. In an image, there are many image blocks with the same pixels, and the noise therein is uncorrelated. However, there are many image blocks with the same pixels in the sonar image, so the non-local mean filtering is very suitable for multi-beam sonar images. However, it has extremely high computational complexity. The present invention performs non-local mean filtering on the sonar image in an integral graph manner in the GPU, which can not only maintain a good filtering effect, but also ensure good real-time performance.

[0148] Compared with optical image target detection and tracking, sonar images have the characteristics of low resolution, unclear target features, and severe noise, which brings certain challenges to target detection and tracking. Compared with advanced algorithms that rely on target features for tracking, the present invention adopts a method assisted by template matching algorithm, target segmentation, GPU acceleration, etc., which achieves the effect of target tracking in low-resolution images while ensuring the stability and real-time performance of target detection and tracking.

[0149] The non-local mean filtering of sonar images is performed in the GPU, which opens up far more computing cores than the image processing unit has. As the number of image processing unit cores increases, the real-time performance of deep-sea target detection and tracking will be further improved.

Claims

1. A deep-sea target detection and tracking method, Its characteristics are: The target detection and tracking method comprises: Acquire sonar images of deep-sea targets; The acquired sonar image is processed by non-local mean filtering in the GPU, wherein when the sonar image is processed by the non-local mean filtering method, the non-local mean filtering operation is performed in the form of an integral graph and accelerated by using the three-dimensional thread of the GPU; Performing the required target detection and tracking on the sonar image processed by the non-local mean filtering to generate the target detection and tracking ROI area; In the GPU, when performing the operation of non-local mean filtering in the form of an integral graph, in the process of calculating the integral graph, a segmented calculation method is used to perform row accumulation and column accumulation in sequence to further accelerate the calculation, reducing the original four-layer loop to a single-layer loop. The process of performing the operation includes: For sonar images, a grid model of non-local mean filtering is constructed in the GPU. In this model, a mean square integral map of non-local mean filtering is calculated for each layer of the grid model, taking the entire sonar image as a unit, and w layers are enabled for parallel calculation to achieve three-dimensional thread acceleration. When calculating the integral graph, first calculate the row integral graph; Perform column integration on the above row integral graph to obtain the corresponding mean square value integral graph; The process of calculating the row integral graph includes: Each row of the matrix is ​​divided into several groups, and each thread calculates the sum of the data prefixes of each group; Calculate the prefix sum of the last data in each group; Starting from group 2, add the numbers in each column (except the last group) to the last number in the previous group to get a complete row integral graph; When performing column integration based on a row integral graph, this includes: Each column of the matrix is ​​divided into several groups, and each thread calculates the sum of the data prefixes of each group; Calculate the prefix sum of the last data in each group; Starting from group 2, add the numbers in each column (except the last group) to the last number in the previous group to get the mean square value integral graph.

2. The deep-sea target detection and tracking method according to claim 1, Its characteristics are: Sonar images of deep-sea targets are collected by multi-beam sonar, where: The sonar image collected by the multi-beam sonar is transmitted to the target detection tracker, so that the target detection tracker obtains the sonar image after receiving it; After the target detection tracker acquires the sonar image, it sends the acquired sonar image to the video memory of the GPU so as to perform non-local mean filtering on the sonar image in the GPU.

3. The deep-sea target detection and tracking method according to any one of claims 1 to 2, Its characteristics are: When performing target detection and tracking, it includes manually selected target tracking and automatic target detection and tracking, among which, When manually selecting target tracking, manually selecting a to-be-detected and tracked ROI region on the sonar image, so as to generate a target detection and tracking ROI region based on the manually selected to-be-detected and tracked ROI region; When automatic target detection is adopted, n to-be-detected and tracked ROI regions are configured to generate n target detection and tracking ROI regions based on the configured n to-be-detected and tracked ROI regions.

4. The deep-sea target detection and tracking method according to claim 3, Its characteristics are: When generating a target detection and tracking ROI area based on the manually selected target detection and tracking ROI area, it includes: For the manually selected ROI area to be detected and tracked, the target area is matched by using the template matching method, so as to determine the position information of the target area to be determined after the target area is matched; The target detection is determined for the position information of the determined target area. When the determined target area is compatible with the manually selected target detection and tracking ROI area, the target area is generated as a target detection and tracking ROI area by using a circumscribed rectangle.

5. The deep-sea target detection and tracking method according to claim 4, Its characteristics are: When the template matching method is used to match the target area, the template matching method used is normalized square difference matching; When generating the target detection and tracking ROI area, the Canny operator is used to perform edge detection on the target area, so that after the edge detection, the edge of the target area is used as a circumscribed rectangle to generate the target detection and tracking ROI area.

6. The deep-sea target detection and tracking method according to claim 3, Its characteristics are: For automatic target detection, the target detection includes: Identify the target on the sonar image, extract the edge contour of the identified target, and use the template matching method to match the target area after the edge contour is extracted, so as to determine the position information of the target area to be determined after the target area is matched; The target detection is determined for the position information of the determined target area. When the determined target area is compatible with the manually selected target detection and tracking ROI area, the target area is generated as a target detection and tracking ROI area by using a circumscribed rectangle.

7. A deep-sea target detection and tracking system, Its characteristics are: The method for detecting and tracking deep-sea targets according to claim 1 comprises: Multi-beam sonar, used to collect sonar images of deep-sea targets; The target detection tracker obtains the sonar image collected by the multi-beam sonar and sends the obtained sonar image to the GPU memory to perform non-local mean filtering on the sonar image in the GPU. When the sonar image is processed by the non-local mean filtering method, the non-local mean filtering operation is performed in the form of an integral image and accelerated by using the three-dimensional thread of the GPU; Performing the required target detection and tracking on the sonar image processed by the non-local mean filtering to generate the target detection and tracking ROI area; When performing the operation of non-local mean filtering in the form of an integral graph in the GPU, the process of performing the operation includes: For sonar images, a grid model of non-local mean filtering is constructed in the GPU, wherein each layer in the grid model calculates a mean square integral map of the non-local mean filtering; When calculating the integral graph, first calculate the row integral graph; Perform column integration on the above row integral graph to obtain the corresponding mean square value integral graph.

Images