Fusion method suitable for foresight sonar target detection

Abstract

According to the foresight sonar target detection fusion method provided by the invention, the target detection performance in a foresight sonar image is improved through fusion of a model framework and multi-frame information; the method comprises the following steps: collecting a forward-looking sonar image sequence; image preprocessing including coordinate conversion, denoising and the like; a target detection model is constructed based on deep learning, and single-frame target detection is completed; carrying out detection result fusion on the single-frame image, wherein the detection result fusion comprises coordinate transformation, image slicing, distance matrix construction, fusion threshold setting and the like; performing target tracking and multi-frame fusion on different frames of images; outputting the processed target state information; the detection robustness is improved through a model framework, and detection fusion is performed by using multi-frame information, so that the false alarm phenomenon in forward-looking sonar target detection is reduced; experimental results show that the method can effectively improve the forward-looking sonar target detection performance and reduce the false alarm rate, and has practical application value.

Description

Technical Field

[0001] The present invention relates to the field of target fusion, and particularly relates to a method for fusing forward-looking sonar target detection Background Art

[0002] With the development of underwater unmanned equipment, underwater target detection has become a key technology. Currently, there is a certain false alarm rate in the target detection method based on forward-looking sonar, which directly affects the performance of target tracking. To solve the false alarm problem in forward-looking sonar target detection, it is necessary to effectively fuse the target detection information in multiple time-series images to improve the accuracy and robustness of target detection. Summary of the Invention

[0003] In view of this, the present invention provides a method for fusing forward-looking sonar target detection, including the following steps:

[0004] Step S1: Collect forward-looking sonar beam data;

[0005] Step S2: Analyze the forward-looking sonar beam data to generate a forward-looking sonar image, and preprocess the image;

[0006] Step S3: Construct a training set and a test set for forward-looking sonar pictures, and construct a horizontal box target detection network based on deep learning; train and test the horizontal box target detection network based on deep learning;

[0007] Step S4: Detect the forward-looking sonar image based on the horizontal box target detection method based on deep learning to obtain target position information;

[0008] Step S5: Perform intra-frame fusion on each frame of forward-looking sonar image;

[0009] Step S6: Perform inter-frame fusion on the forward-looking sonar images of different frames.

[0010] Particularly, step S1 specifically includes deploying an underwater target, and collecting forward-looking sonar data under different working conditions by carrying a forward-looking sonar with an underwater unmanned vehicle; the different working conditions include different sonar ranges and the headings of the underwater unmanned vehicle.

[0011] Particularly, step S2 specifically includes: for the beam data of the forward-looking sonar beam data in the two-dimensional plane coordinates, generating a sequence of forward-looking sonar image frames in the polar coordinate system through coordinate mapping, and performing mean filtering smoothing processing on each frame of the picture to remove noise.

[0012] Specifically, the step S3 specifically includes: the network used is a horizontal box target detection network based on yolov5 deep learning; training hyperparameters are set, and the forward-looking sonar image dataset images are used as the input of the network, and the network is trained to obtain a horizontal box target detection model based on deep learning.

[0013] Specifically, the step S4 specifically includes: slicing the a-th frame of forward-looking sonar image according to the preset horizontal and vertical overlap rates (ξ1, ξ2) and equal sizes (w slice ,h slice ) to obtain a set of sliced images {I k ,k = 1, 2, 3…} Detecting the set of sliced images {I k ,k = 1, 2, 3…} using the model obtained in step S3 to obtain the target detection results {R k ,k = 1, 2, 3…}, where R k is the detection result of the k-th sliced image, {R k = (x i ,y i ,w i ,h i ),i = 1, 2, 3,...}, where x i 、y i 、w i 、h i are the abscissa, ordinate, width, and height of the center position of the target detection box respectively.

[0014] Specifically,

[0015] The step S5 specifically includes: performing in-frame fusion on the a-th frame of forward-looking sonar image {R k ,k = 1, 2, 3…}, and the specific steps are as follows:

[0016] Solve the distance matrix D of R1 and R2. Let the size of R1 be n and the size of R2 be m, then:

[0017]

[0018] Among them:

[0019]

[0020] R1(i)∩R2(i) is the number of pixels in the intersection area of the two detection boxes

[0021] R1(i)∪R2(i) is the number of pixels in the merged area of the two detection boxes

[0022] Set a threshold ε. If d ijIf it is ≥ ε, then R1(i) and R2(j) are regarded as the same target, and the two are averaged and weighted to obtain an updated target detection frame. The updated target detection frame and all target detection frames that satisfy d ij ≤ ε together constitute the fused result R';

[0023] For the remaining {R k , k = 3, 4,...}, update R' in the same way, and finally obtain the target detection result after fusion of the forward-looking sonar image of the a-th frame, denoted as T a .

[0024] In particular,

[0025] Step S6 includes: setting the set of target detection results after fusion of the first n frames as Φ n , and Φ1 is initialized as T1

[0026] According to the latitude and longitude, heading angle of the vehicle carrier, and the center position, length and width of the target detection frame, the latitude and longitude of each target can be calculated, and further the Euclidean distance between the targets can be calculated. Let the target detection result of the new forward-looking image frame be T n+1

[0027] Solve the distance matrix S between Φ n and T n+1 . Suppose the size of Φ n is h, and the size of T n+1 is l, then:

[0028]

[0029] where: s ij is the Euclidean distance between Φ n (i) and T n+1 (j);

[0030] Set the threshold η. If s ij ≥ η, then Φ n (i) and R n+1 (j) are regarded as the same target, and the number of times this target appears is incremented by 1. If the number of appearances exceeds the preset times, then confirm that this target exists and continue to retain it in Φ n+1 . If a certain target does not appear for τ consecutive frames, then remove it from Φ n+1 ;

[0031] After the above inter-frame fusion, the fusion result Φ n+1 of the first n + 1 target detection frames can be obtained.

[0032] Beneficial effects:

[0033] Through the multi-frame information fusion method of the present invention, the false alarm rate of forward-looking sonar target detection can be effectively reduced;

[0034] Through the intra-frame fusion processing of the present invention, the target detection accuracy of a single-frame forward-looking sonar image can be improved;

[0035] Through the inter-frame fusion processing of the present invention, the multi-frame target detection results can be made more coherent and robust;

[0036] The forward-looking sonar target detection model based on deep learning established by the present invention can be extended and applied to various forward-looking sonar target detection tasks;

[0037] Through the fusion threshold setting method of the present invention, the detection performance and false alarm rate can be balanced;

[0038] In summary, the forward-looking sonar target detection fusion method proposed by the present invention can effectively improve the accuracy and coherence of forward-looking sonar target detection, reduce the false alarm rate, expand the target detection application scenario, and has important practical value. Description of the Drawings

[0039] Figure 1 It is a schematic flowchart of a forward-looking sonar target detection fusion method applicable to the present invention. Detailed Embodiments

[0040] The following combines the drawings and gives embodiments to describe the present invention in detail.

[0041] The present invention provides a forward-looking sonar target detection fusion method, as Figure 1 shown, including the following steps:

[0042] Step S1: Collect forward-looking sonar beam data; deploy underwater targets; collect forward-looking sonar data through an underwater unmanned vehicle equipped with an Oculus850d forward-looking sonar under different working conditions. The different working conditions are different sonar ranges and the headings of the underwater unmanned vehicle. In this example, the sonar ranges are 50m and 100m, and the headings of the underwater unmanned vehicle are 45 degrees, 90 degrees, and 135 degrees;

[0043] Step S2: Parse the forward-looking sonar beam data to generate a forward-looking sonar image frame sequence, and preprocess the images; the forward-looking sonar beam data is beam data in a two-dimensional plane coordinate system, generate a forward-looking sonar image in a polar coordinate system through coordinate mapping, and perform mean filter smoothing processing on each frame of the picture to remove noise;

[0044] Step S3: Construct a training set and a test set for forward-looking sonar images, and construct a horizontal box object detection network based on deep learning; train and test the horizontal box object detection network based on deep learning; construct the training set and test set of forward-looking sonar images according to a preset ratio, which is set to 3:1 in this example; construct a horizontal box object detection network based on deep learning. The network specifically used in this example is a horizontal box object detection network based on yolov5 deep learning; set training hyperparameters, such as setting the learning rate to 0.00001, the number of iterations to 450, and the network depth to 12. Use the forward-looking sonar image dataset images as the input of this network, and train the network to obtain a horizontal box object detection model based on deep learning; test the trained network. If the index requirements are met, stop. Otherwise, modify the hyperparameters and continue training until the index requirements are met.

[0045] Step S4: Detect the forward-looking sonar image based on the horizontal box object detection method based on deep learning to obtain target position information;

[0046] The specific steps of Step S4 include: slice the a-th frame of forward-looking sonar image according to the preset horizontal and vertical overlap rates (ξ1, ξ2) and equal sizes (w slice , h slice ). For example, when (ξ1, ξ2) is (0.2, 0.2) and (w slice , h slice ) is (640, 640), obtain a set of sliced images {I k , k = 1, 2, 3...}. Use the model obtained in Step S3 to detect {I k , k = 1, 2, 3...} to obtain the target detection results of the sliced image set {R k , k = 1, 2, 3...}, where R k is the detection result of the k-th sliced image, {R k = (x i , y i , w i , h i ), i = 1, 2, 3,...}, where x i , y i , w i , h i are the abscissa, ordinate, width, and height of the center position of the target detection box respectively.

[0047] Step S5: Perform intra-frame fusion on each frame of forward-looking sonar image;

[0048] The specific steps of Step S5 include: perform intra-frame fusion on the a-th frame of forward-looking sonar image {R k , k = 1, 2, 3...}. The specific steps are as follows:

[0049] Solve the distance matrix D between R1 and R2. Assume the size of R1 is n and the size of R2 is m, then:

[0050]

[0051] Where:

[0052]

[0053] The number of pixels in the intersection area of the two detection frames is R1(i)∩R2(i)

[0054] The number of pixels in the merged area of the two detection frames is R1(i)∪R2(i)

[0055] Set the threshold ε. For example, if ε = 0.6, if d ij ≥ε, then R1(i) and R2(j) are regarded as the same target, and they are averaged and weighted to obtain an updated target detection frame. The updated target detection frame and all target detection frames that satisfy d ij ≤ε together form the fused result R';

[0056] For the remaining {R k , k = 3, 4,...}, update R' in the same way. Finally, obtain the fused target detection result of the forward-looking sonar image of the a-th frame, denoted as T a .

[0057] Step S6: Perform inter-frame fusion on forward-looking sonar images of different frames;

[0058] The said step S6 includes: Let the set of fused target detection results of the first n frames be Φ n , and Φ1 is initialized as T1

[0059] According to the longitude, latitude, heading angle of the vehicle carrier, and the center position, length, and width of the target detection frame, the longitude and latitude of each target can be calculated, and further the Euclidean distance between the targets can be calculated. Let the target detection result of the new forward-looking image frame be T n+1

[0060] Solve the distance matrix S between Φ n and T n+1 . Assume the size of Φ n is h and the size of T n+1 is l, then:

[0061]

[0062] Where: s ij is the Euclidean distance between Φ n (i) and T n+1 (j);

[0063] Set a threshold η, such as η = 0.9. If s ij ≥ η, then Φ n (i), R n+1 (j) are regarded as the same target, and the number of times this target appears is incremented by 1. If the number of appearances exceeds the preset times, such as setting to 7, then confirm the existence of this target and continue to retain it in Φ n+1 . If a certain target does not appear for consecutive τ frames, such as setting τ to 3, then remove it from Φ n+1 ;

[0064] After the above inter-frame fusion, the fusion results of the first n + 1 target detection frames Φ n+1 can be obtained.

[0065] In summary, the above is only a preferred embodiment of the present invention and is not used to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

[0066] For those skilled in the art, it is obvious that the embodiments of the present invention are not limited to the details of the above exemplary embodiments, and without departing from the spirit or basic characteristics of the embodiments of the present invention, the present invention can be implemented in other specific forms. Therefore, in any aspect, the embodiments should be regarded as exemplary and non-limiting. The scope of the embodiments of the present invention is defined by the appended claims rather than the above description. Therefore, it is intended to cover all changes falling within the meaning and scope of the equivalent elements of the claims in the embodiments of the present invention. Any reference signs in the claims should not be regarded as limiting the claimed rights. In addition, obviously the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. The multiple units, modules or devices stated in the system, apparatus or terminal claims can also be implemented by the same unit, module or device through software or hardware. First, second, etc. are used to represent names and do not represent any specific order.

[0067] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the embodiments of the present invention and not to limit them. Although the technical solutions of the embodiments of the present invention have been described in detail with reference to the above preferred embodiments, those of ordinary skill in the art should understand that modifications or equivalent replacements can be made to the technical solutions of the embodiments of the present invention without departing from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A forward-looking sonar target detection fusion method, characterized in that It includes the following steps: Step S1: Collect forward-looking sonar beam data; Step S2: Parse the forward-looking sonar beam data to generate a forward-looking sonar image, and preprocess the image; Step S3: Construct a training set and a test set for the forward-looking sonar pictures, and construct a horizontal box object detection network based on deep learning; Train and test the horizontal box object detection network based on deep learning; Step S4: Detect the forward-looking sonar image based on the horizontal box object detection method based on deep learning to obtain target position information; Step S5: Perform intra-frame fusion on each frame of the forward-looking sonar image; Step S6: Perform inter-frame fusion on the forward-looking sonar images of different frames.

2. The forward-looking sonar target detection fusion method according to claim 1, wherein Step S1 specifically includes deploying an underwater target, and collecting forward-looking sonar data under different working conditions by carrying a forward-looking sonar with an underwater unmanned vehicle; the different working conditions include different sonar ranges and the headings of the underwater unmanned vehicle.

3. The forward-looking sonar target detection fusion method according to claim 1, wherein Step S2 specifically includes: For the beam data of the forward-looking sonar beam data in the two-dimensional plane coordinate, generate a forward-looking sonar image in the polar coordinate system through coordinate mapping, and perform mean filtering smoothing processing on each frame of the image to denoise.

4. The forward-looking sonar target detection fusion method according to claim 1, wherein Step S3 specifically includes: The network used is a horizontal box object detection network based on yolov5 deep learning; set training hyperparameters, use the forward-looking sonar picture data set pictures as the input of the network, and train the network to obtain a horizontal box object detection model based on deep learning.

5. The forward-looking sonar target detection fusion method according to claim 1, wherein The specific steps of step S4 include: slicing the a-th frame of forward sonar image according to a preset horizontal and vertical overlap rate (ξ1, ξ2) and equal size (w slice , h slice ) to obtain a set of sliced images {I k , k = 1, 2, 3...} Detecting the set of sliced images {I k , k = 1, 2, 3...} with the model obtained in step S3 to obtain the target detection results of the set of sliced images {R k , k = 1, 2, 3...}, where R k is the detection result of the k-th sliced image, {R k = (x i , y i , w i , h i ), i = 1, 2, 3,...}, where x i , y i , w i , h i are the abscissa, ordinate, width, and height of the center position of the target detection box, respectively.

6. The forward-looking sonar target detection fusion method according to claim 1, wherein The specific steps of step S5 include: performing intra-frame fusion on the a-th frame of forward-looking sonar images {R k , k = 1, 2, 3...}, and the specific steps are as follows: Solve the distance matrix D of R1 and R2. Let the size of R1 be n and the size of R2 be m, then: Where: R1(i)∩R2(i) is the number of pixels in the intersection area of the two detection boxes Set a threshold ε for the number of pixels in the merged region of the two detection boxes R1(i) ∪ R2(i). If d ij ≥ ε, then R1(i) and R2(j) are regarded as the same target, and they are averaged and weighted to obtain an updated target detection box. The updated target detection box and all target detection boxes that satisfy d ij ≤ ε together form the fused result R'. For the remaining {R k , k = 3, 4,...}, update R' in the same way, and finally obtain the target detection result after fusing the forward sonar images of the a-th frame, denoted as T a .

7. The forward-looking sonar target detection fusion method according to claim 6, characterized in that The step S6 includes: setting the set of target detection results after fusing the first n frames as Φ n , initializing Φ1 as T1 Based on the latitude and longitude of the vehicle carrier, the heading angle, and the center position, length, and width of the target detection box, the latitude and longitude of each target can be calculated, and further the Euclidean distance between the targets can be calculated. Let the target detection result of the new forward-looking image frame be T n+1 , Solve for Φ n and T n+1 distance matrix S. Let Φ n have size h, and T n+1 have size l, then: Where: s ij is the Euclidean distance between n (i) and T n+1 (j). Set a threshold η. If s ij ≥ η, then Φ n (i), R n+1 (j) are regarded as the same target, and the number of occurrences of this target is incremented by 1. If the number of occurrences exceeds the preset times, then confirm the existence of this target and continue to retain it in Φ n+1 . If a certain target does not appear for τ consecutive frames, then remove it from Φ n+1 ; After the above frame fusion, the fusion result Φ of the first n+1 object detection boxes can be obtained n+1 .

Images