Underwater target detection method for generating virtual sonar based on physical model and optical image

Abstract

The invention discloses an underwater target detection method for generating a virtual sonar based on a physical model and an optical image. The underwater target detection method comprises the following steps: acquiring the optical image; based on a physical model of sonar imaging and an imaging principle, generating a virtual sonar image from the optical image; multi-modal feature extraction is carried out; carrying out multi-modal feature fusion; and performing target detection on the multi-modal fusion feature map by using a deep learning model, and outputting a target detection result. According to the invention, the technical problems of high system cost and difficult deployment of a traditional multi-modal detection method are effectively solved, the detection effect of optical and sonar multi-modal fusion can be achieved only by using a common optical camera, the hardware cost and the application threshold are greatly reduced, and the detection efficiency is improved. According to the method, the target detection accuracy and the robustness in a complex and changeable environment can be remarkably improved, the detection operation is simple, rapid and real-time detection can be realized, the generalization ability is relatively high, and the method has extremely high application value and economic benefits for underwater target detection.

Description

Technical Field

[0001] The present invention relates to the technical field of underwater target detection, and in particular to an underwater target detection method based on generating virtual sonar based on physical models and optical images. Background Art

[0002] Underwater target detection is a key technology in fields such as ocean development, environmental monitoring, underwater search and rescue, and national defense security. Currently, mainstream underwater perception technologies are categorized into two main categories: optical imaging and acoustic imaging. Optical imaging can provide rich details such as color and texture, but its image quality degrades dramatically in turbid water and low-light conditions, severely impacting the performance of target detection algorithms. Sonar (acoustic imaging) technology detects targets by emitting sound waves and receiving echoes. It is unaffected by water transparency and lighting conditions, operates at a long range, and is a reliable means of underwater detection. However, sonar images typically have low resolution, lack color information, and have difficulty discerning the fine structure of targets.

[0003] To combine the advantages of both, academia and industry have proposed multimodal detection solutions that fuse optical and sonar data. These solutions simultaneously deploy optical cameras and sonar equipment on underwater platforms to capture both modal data for the same scene. Feature fusion techniques are then used to improve detection accuracy and robustness. However, this approach presents significant practical challenges: 1) High cost: High-performance sonar equipment is expensive, significantly increasing system hardware costs; 2) Complex deployment and calibration: Integrating, synchronizing, and accurately calibrating two sensors based on different physical principles on the same platform is technically challenging and cumbersome; 3) Difficult data acquisition: The sonar data acquisition process is time-consuming, making it difficult to meet real-time detection requirements and to obtain well-registered paired optical-sonar datasets on a large scale, limiting its widespread application.

[0004] In addition, although there are some preprocessing methods in the existing technology aimed at improving the quality of single-modal (optical) images, such as improving the contrast and color of underwater images through image enhancement algorithms, these methods are essentially still information optimization in the optical domain and do not introduce new physical dimension information. When the optical information itself is severely missing due to environmental factors, its performance improvement is limited. They fail to solve the fundamental problem of how to introduce the unique physical information that sonar can provide when only optical sensors are available.

[0005] Therefore, this field urgently needs a new technology that can achieve low-cost and high-precision underwater target detection by integrating optical and sonar multimodal features. Summary of the Invention

[0006] In view of the above problems and needs existing in the prior art, the purpose of the present application is to provide an underwater target detection method based on a physical model and an optical image to generate a virtual sonar, which can realize underwater target detection by fusing optical and sonar multi-modal features at low cost and high precision.

[0007] To achieve the above-mentioned purpose, the present application adopts the following technical solutions:

[0008] An underwater target detection method based on a physical model and an optical image to generate a virtual sonar, comprising the following steps:

[0009] S1) Collecting an optical image: using an optical camera to collect an optical image of a target underwater area;

[0010] S2) Generating a virtual sonar image: based on the physical model and imaging principle of sonar imaging, generating a virtual sonar image from the optical image obtained in step S1);

[0011] S3) Multi-modal feature extraction: using a dual-path parallel feature extraction network, one path takes the optical image obtained in step S1) as input to extract an optical image feature map; the other path takes the virtual sonar image generated in step S2) as input to extract a sonar image feature map;

[0012] S4) Multi-modal feature fusion: through a feature fusion module, adaptively weighting and fusing the optical image feature map and the sonar image feature map extracted in step S3) to generate a multi-modal fusion feature map;

[0013] S5) Target detection and output: using a deep learning model to perform target detection on the multi-modal fusion feature map and output the target detection result.

[0014] An embodiment, the specific steps of step S2) for generating a virtual sonar image are as follows:

[0015] S21) Depth estimation is performed on the optical image obtained in step S1) to convert a disparity map into a depth map;

[0016] S22) Based on the camera intrinsic parameters and the depth map obtained in step S21), using an inverse camera model to perform three-dimensional point cloud reconstruction;

[0017] S23) Based on the physical model and imaging principle of sonar imaging, converting the three-dimensional point cloud obtained in step S22) into a virtual sonar image.

[0018] Further embodiments, step S21) specifically comprises the following steps:

[0019] S211) Using a pre-trained EfficientNet as an encoder, performing multi-scale feature extraction on the optical image using an image pyramid;

[0020] S212) Recovering the spatial resolution of each scale by decoder step-by-step upsampling, and outputting the disparity map of the corresponding scale;

[0021] S213) Converting the disparity map to the depth map according to the formula "depth = f x b / disparity", wherein: f represents the focal length of the camera, and b represents the baseline length of the camera.

[0022] Further implementation, step S22) specifically includes the following steps:

[0023] S221) Converting each pixel coordinate (u, v) in the depth map to normalized plane coordinates (x', y'), i.e.

[0024] , wherein K −1 is the inverse matrix of the camera intrinsic matrix K;

[0025] S222) According to the depth value Z and the normalized plane coordinates (x', y') of each pixel point (u, v), calculating the three-dimensional coordinates of each pixel point in the world coordinate system:

[0026] , , ;

[0027] S223) Repeating steps S221) ~ S222) for each pixel point in the depth map, respectively generating three-dimensional coordinate points (X, Y, Z), storing all generated three-dimensional coordinate points as point cloud data, i.e. obtaining the three-dimensional point cloud data representing the three-dimensional scene.

[0028] Further implementation, step S23) specifically includes the following steps:

[0029] S231) Coordinate system conversion: converting the three-dimensional point cloud from the world coordinate system to the sonar coordinate system, and the conversion formula is: , wherein: R is a rotation matrix, t is a translation vector, and R and t are obtained by pre-calibration;

[0030] S232) Beam simulation: simulating the geometric shape of the sonar beam, and judging whether the points in the three-dimensional point cloud are located within the beam range;

[0031] S233) Echo intensity calculation: for each point located within the beam range, calculating its echo intensity I according to the following formula:

[0032] , wherein: is the initial sound intensity of the sonar emission; r is the distance from the point to the simulated sonar; a scattering cross section of a target, the target referring to an object illuminated by a simulated sonar beam and generating echoes; a beam directionality attenuation function, an included angle between a point direction and a beam main axis, a beam width of a simulated sonar;

[0033] S234) Image generation: projecting the calculated echo intensity to a two-dimensional imaging plane according to a sonar imaging model, and generating a gray-scale image as a virtual sonar image.

[0034] In a preferred embodiment, the feature extraction network in step S3) is two parallel independent YOLOv11 backbone networks.

[0035] In an embodiment, the feature fusion module in step S4) is composed of an input layer, a channel adjustment layer, a weight generation module, a weighted fusion layer, and an output layer.

[0036] In a further embodiment, the specific steps of implementing multi-modal feature fusion using the feature fusion module are as follows:

[0037] S41) Input layer: inputting the received optical image feature map F optical and the sonar image feature map F sonar to the channel adjustment layer;

[0038] S42) Channel adjustment layer: adjusting the channel numbers of the input optical feature map and the sonar feature map to be consistent using 1x1 convolution;

[0039] S43) Weight generation module: splicing the adjusted optical feature map and the sonar feature map, and then generating two weight maps W optical and W sonar representing the contribution weights of the optical feature and the sonar feature, respectively, through two layers of convolutional neural networks;

[0040] S44) Weighted fusion layer: performing weighted summation of the optical feature map and the sonar feature map using the generated weight maps to obtain the fused feature map F fused =F optical ×W optical +F sonar ×W sona ;

[0041] S45) Output layer: performing 3x3 convolution on the fused feature map and outputting the multi-modal fused feature map F output .

[0042] In a preferred solution, the deep learning model in step S5) is based on a YOLOv11 architecture, 1 feature extraction network is changed to 2 parallel feature extraction networks, and a feature fusion module in step S4) is introduced, and the YOLOv11 architecture includes a backbone network, a feature enhancement network and a detection head.

[0043] In an embodiment, the target detection result in step S5) includes a target category label, position information and a detection confidence.

[0044] Compared with the prior art, the beneficial technical effects of the present application are:

[0045] ①The present application does not require expensive sonar equipment, but only uses a common optical camera to realize multi-modal detection, greatly reducing the hardware cost and application threshold;

[0046] ②Since the virtual sonar image adopted by the present application is generated by calculation in the coordinate system of the optical image, the two are naturally accurately aligned in space, fundamentally avoiding the complex and error-prone calibration process between real multi-sensor systems;

[0047] ③Unlike pure optical image enhancement technology, the present application generates a virtual sonar image based on a physical model and an optical image, providing sonar modal information that optical images do not have, achieving the detection effect of multi-modal determination of underwater targets, and can be used for target detection in complex underwater environments (such as turbid water and similar target and background colors);

[0048] ④The present application effectively improves the accuracy of target detection and the robustness in complex and variable environments by fusing real optical features and virtual sonar features;

[0049] ⑤The present application uses an efficient deep learning model, which not only realizes fast and real-time detection of targets, but also has strong generalization ability;

[0050] In summary, compared with existing underwater target detection technology, the present application not only has the above beneficial effects and significant progress, but also has significant application value. BRIEF DESCRIPTION OF DRAWINGS

[0051] Figure 1 is the overall flowchart of the underwater target detection method based on a physical model and an optical image to generate a virtual sonar provided by the present application;

[0052] Figure 2 is the flowchart of generating a virtual sonar image in an embodiment of the present application;

[0053] Figure 3 is the original optical image collected;

[0054] Figure 4 is a virtual sonar image generated by the method of the present application;

[0055] Figure 5 is a detection result finally output by the method of the present application. DETAILED DESCRIPTION

[0056] The present application will be further described in conjunction with the accompanying drawings and examples. It should be understood that the following examples are only used to illustrate the present application, but not to limit the protection scope of the present application.

[0057] Embodiment

[0058] Please see Figure 1 As shown in the figure, the underwater target detection method for generating virtual sonar based on physical model and optical image provided by the present embodiment comprises the following steps:

[0059] S1) Collecting optical image: using an optical camera to collect the optical image of the target underwater area;

[0060] S2) Generating virtual sonar image: based on the physical model and imaging principle of sonar imaging, generating a virtual sonar image from the optical image obtained in step S1);

[0061] S3) Multi-modal feature extraction: using a dual-path parallel feature extraction network (in the present embodiment, two parallel independent YOLOv11 backbone networks are used), one path takes the optical image obtained in step S1) as input to extract the optical image feature map; the other path takes the virtual sonar image generated in step S2) as input to extract the sonar image feature map;

[0062] S4) Multi-modal feature fusion: through a feature fusion module, the optical image feature map and the sonar image feature map extracted in step S3) are adaptively weighted and fused to generate a multi-modal fusion feature map; the feature fusion module is composed of an input layer, a channel adjustment layer, a weight generation module, a weighted fusion layer and an output layer, and the specific steps of realizing multi-modal feature fusion using the feature fusion module are as follows:

[0063] S41) Input layer: receiving the optical image feature map F optical and the sonar image feature map F sonar extracted in step S3) and inputting them to the channel adjustment layer;

[0064] S42) Channel adjustment layer: using 1x1 convolution to adjust the channel numbers of the input optical feature map and sonar feature map to be consistent;

[0065] S43) Weight generation module: the adjusted optical feature map and sonar feature map are spliced, and then two weight maps W representing the contribution weights of the optical features and sonar features, respectively, are generated through two layers of convolutional neural networks optical and W sonar ;

[0066] S44) Weighted fusion layer: the optical feature map and the sonar feature map are weighted and summed using the generated weight map to obtain a fused feature map F fused =F optical ×W optical +F sonar ×W sona ;

[0067] S45) Output layer: a 3x3 convolution is performed on the fused feature map to output a multi-modal fusion feature map F output ;

[0068] S5) Target detection and output: a deep learning model is used to perform target detection on the multi-modal fusion feature map to output a target detection result, the deep learning model being an improved YOLOv11 model in which 1 feature extraction network is replaced by 2 parallel feature extraction networks and the feature fusion module described in step S4) is introduced, the YOLOv11 architecture including a backbone network, a feature enhancement network and a detection head; the input multi-modal fusion feature map is first subjected to target prediction by the detection head of the improved YOLOv11 model, and then the target prediction result is post-processed, the post-processing including: non-maximum suppression, boundary box regression fine-tuning, confidence threshold filtering, restoration to the original image size, and then outputting the final target detection result, the target detection result including a target class label, position information and detection confidence.

[0069] Referring to Figure 2 , the embodiment implements generation of a virtual sonar image, and specifically includes the following steps:

[0070] S21) Depth estimation is performed on the optical image obtained in step S1) to convert the disparity map into a depth map, specifically including the following steps:

[0071] S211) A pre-trained EfficientNet is used as an encoder to perform multi-scale feature extraction on the optical image using an image pyramid; in this embodiment, a Laplacian pyramid is constructed to extract features at different levels and generate feature maps of different scales;

[0072] S212) Recovering spatial resolution of each scale by decoder step-by-step upsampling and outputting the disparity map of the corresponding scale; in the embodiment, the decoder is composed of multiple upsampling blocks, each block containing an upsampling layer, a skip connection and a convolution layer, wherein: the upsampling layer uses transpose convolution to recover the spatial resolution, the skip connection splices the features of the corresponding scale in the encoder (data features captured at different abstract levels) and the features after upsampling (data representation obtained after converting low-resolution feature maps into high-resolution feature maps) to enhance the detail recovery capability; the convolution layer uses multiple convolution layers and down-sampling layers to gradually refine the feature representation by the multi-level features of the input image extracted by the encoder; for each scale, the disparity map under the scale is generated by convolution combining the upsampling feature map and the skip connection input;

[0073] S213) Converting the disparity map into a depth map according to the formula "depth = f x b / disparity", wherein: f represents the focal length of the camera, and b represents the baseline length of the camera;

[0074] S22) Using an inverse camera model to perform three-dimensional point cloud reconstruction based on the camera intrinsic parameters and the depth map obtained in step S21), specifically including the following steps:

[0075] S221) Converting each pixel coordinate (u, v) in the depth map into normalized plane coordinates (x', y'), i.e.:

[0076] , wherein K −1 is the inverse matrix of the camera intrinsic matrix K;

[0077] S222) According to the depth value Z and the normalized plane coordinates (x', y') of each pixel point (u, v), calculating the three-dimensional coordinates of each pixel point in the world coordinate system:

[0078]

[0079] S223) Repeating steps S221) to S222) for each pixel point in the depth map to generate three-dimensional coordinate points (X, Y, Z) respectively, and storing all generated three-dimensional coordinate points as point cloud data, i.e. obtaining three-dimensional point cloud data representing a three-dimensional scene;

[0080] S23) Converting the three-dimensional point cloud obtained in step S22) into a virtual sonar image based on the physical model and imaging principle of sonar imaging, specifically including the following steps:

[0081] S231) Coordinate system conversion: converting the three-dimensional point cloud from the world coordinate system to the sonar coordinate system, and the conversion formula is: ​​​wherein: R is a rotation matrix, t is a translation vector, R and t are obtained through pre-calibration;

[0082] S232) Beam simulation: simulate the geometry of the sonar beam, and determine whether the points in the three-dimensional point cloud are within the beam range;

[0083] S233) Echo intensity calculation: for each point within the beam range, calculate its echo intensity I according to the following formula:

[0084] wherein: is the initial sound intensity of the sonar emission; r is the distance from the point to the simulated sonar; is the scattering cross-sectional area of the target, which refers to the object irradiated by the simulated sonar beam and producing echoes; is the beam direction attenuation function, is the angle between the point direction and the beam principal axis, is the beam width of the simulated sonar;

[0085] S234) Image generation: project the calculated echo intensity according to the sonar imaging model to the two-dimensional imaging plane, and the generated gray-scale image is the virtual sonar image. The specific implementation process of this embodiment is as follows:

[0086] ① Use the formulas x = r x cos θ and y = r x sin θ to convert the points within the simulated beam range from sonar coordinates (r, θ) to Cartesian coordinates (x, y), where r is the distance from the point to the simulated sonar, and θ is the azimuth angle of the point;

[0087] ② Create a blank two-dimensional image array with a size determined by the resolution requirement; then, for each point converted from the sonar data (calculated echo intensity data), update the gray value of the corresponding pixel according to its position on the Cartesian two-dimensional coordinate plane;

[0088] ③ Use an image processing library (such as PIL in Python) to render this two-dimensional array into a visual gray-scale image and display it, which is the virtual sonar image.

[0089] It should be noted that the target class labels mentioned in step S5) are pre-set, and the deep learning model learns to recognize and distinguish them through learning a large number of labeled samples. When training the model, an optical image dataset with annotated real underwater scenes is used, the input image size is set to 640x640, the initial learning rate is 0.01, the weight decay rate is 0.0005, the number of network training samples is 16 each time, and the Adam optimizer is used for gradient update.

[0090] In addition, in order to verify the effectiveness of the present application, the inventors conducted experiments on multiple underwater environment test data sets, such as: Figure 3 is the original optical image in the test data set, Figure 4 is the virtual sonar image generated by the method of the present application, Figure 5 is the detection result finally output by the method of the present application, including the position of the target in the image, the classification label "sea urchin" and the confidence "94%"; as shown by Figures 3 to 5 It can be seen that by using the method of the present application, high-precision detection of underwater targets can be achieved.

[0091] As can be seen from the above, by creatively using optical images and sonar physical models to generate virtual sonar images, the present application achieves the detection effect of optical and sonar multi-modal fusion using only a common optical camera, which not only greatly reduces the hardware cost and application threshold, but also significantly improves the accuracy of target detection and the robustness in complex and variable environments, and the detection operation is simple, fast and real-time, and has strong generalization ability. In short, the present application effectively solves the technical pain points of high system cost and difficult deployment of traditional multi-modal detection methods, significantly reduces the hardware threshold of underwater multi-modal perception, and has very high application value and economic benefit for underwater target detection.

[0092] Finally, it is necessary to point out that: the above is only a specific embodiment of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can easily think of changes or replacements within the technical scope disclosed by the present application. In the case of no conflict, the embodiments in the present application and the features in the embodiments can be combined with each other. And for the foregoing method embodiments, in order to simply describe, they are all expressed as a combination of a series of actions, but those skilled in the art should know that the present application is not limited by the order of the described actions, because according to the present application, certain steps can be performed in other order or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification all belong to preferred embodiments.

Claims

1. A method for underwater target detection based on physical models and optical images to generate virtual sonar, characterized in that: The steps include: S1) collecting optical images: using an optical camera to collect optical images of the target underwater area; S2) generating a virtual sonar image: generating a virtual sonar image from the optical image obtained in step S1) based on the physical model and imaging principle of sonar imaging; S3) Multimodal feature extraction: A two-way parallel feature extraction network is used, where one way takes the optical image obtained in step S1) as input and extracts an optical image feature map; the other way takes the virtual sonar image generated in step S2) as input and extracts a sonar image feature map; S4) Multimodal feature fusion: performing adaptive weighted fusion of the optical image feature map extracted in step S3) and the sonar image feature map through a feature fusion module to generate a multimodal fusion feature map; S5) Object detection and output: Use the deep learning model to perform object detection on the multimodal fusion feature map and output the object detection results.

2. The underwater target detection method according to claim 1, characterized in that: Step S2) generates the virtual sonar image in the following steps: S21) performing depth estimation on the optical image obtained in step S1) and converting the disparity map into a depth map; S22) reconstructing a three-dimensional point cloud using an inverse camera model based on the camera intrinsic parameters and the depth map obtained in step S21); S23) Based on the physical model and imaging principle of sonar imaging, the three-dimensional point cloud obtained in step S22) is converted into a virtual sonar image.

3. The underwater target detection method according to claim 2, characterized in that: Step S21) specifically includes the following steps: S211) Using the pre-trained EfficientNet as the encoder, perform multi-scale feature extraction on the optical image using an image pyramid; S212) Upsampling the image layer by layer through the decoder to restore the spatial resolution of each scale and output a disparity map of the corresponding scale; S213) Converting the disparity map into a depth map according to the formula "depth = f×b / disparity", where: f represents the focal length of the camera, and b represents the baseline length of the camera.

4. The underwater target detection method according to claim 2, characterized in that: Step S22) specifically includes the following steps: S221) Convert each pixel coordinate (u, v) in the depth map into a normalized plane coordinate (x', y'), that is: , where K −1 is the inverse matrix of the camera intrinsic parameter matrix K; S222) Calculate the three-dimensional coordinates of each pixel in the world coordinate system based on the depth value Z and normalized plane coordinates (x', y') of each pixel point (u, v): ， ， ； S223) Repeat steps S221) to S222) for each pixel in the depth map to generate three-dimensional coordinate points (X, Y, Z) respectively, and store all generated three-dimensional coordinate points as point cloud data, that is, obtain three-dimensional point cloud data representing the three-dimensional scene.

5. The underwater target detection method according to claim 2, characterized in that: Step S23) specifically includes the following steps: S231) Coordinate system conversion: Convert the 3D point cloud from the world coordinate system to the sonar coordinate system. The conversion formula is: , where: R is the rotation matrix, t is the translation vector, R and t are obtained by pre-calibration; S232) Beam simulation: simulating the geometry of the sonar beam and determining whether a point in the three-dimensional point cloud is within the beam range; S233) Echo intensity calculation: For each point within the beam range, calculate its echo intensity I according to the following formula: ,in: is the initial sound intensity emitted by the sonar; r is the distance from the point to the simulated sonar; is the scattering cross-section of the target, where the target is the object that is illuminated by the simulated sonar beam and generates an echo; , is the beam directivity attenuation function, is the angle between the point direction and the main axis of the beam, is the beam width of the simulated sonar; S234) Image generation: The calculated echo intensity is projected onto a two-dimensional imaging plane according to the sonar imaging model, and the generated grayscale image is the virtual sonar image.

6. The underwater target detection method according to claim 1, characterized in that: The feature extraction network described in step S3) is two parallel independent YOLOv11 backbone networks.

7. The underwater target detection method according to claim 1, characterized in that: The feature fusion module described in step S4) is composed of an input layer, a channel adjustment layer, a weight generation module, a weighted fusion layer and an output layer.

8. The underwater target detection method according to claim 7, characterized in that: The specific steps of implementing multimodal feature fusion using the feature fusion module are as follows: S41) Input layer: receives the optical image feature map F optical and sonar image feature map F sonar Input to the channel adjustment layer; S42) Channel adjustment layer: Use 1×1 convolution to adjust the number of channels of the input optical feature map and sonar feature map to be consistent; S43) Weight generation module: The adjusted optical feature map and sonar feature map are spliced ​​together, and then two weight maps W representing the contribution weights of the optical feature and sonar feature are generated through a two-layer convolutional neural network. optical and W sonar ; S44) Weighted fusion layer: Use the generated weight map to perform weighted summation on the optical feature map and the sonar feature map to obtain the fused feature map F fused =F optical ×W optical +F sonar ×W sona ; S45) Output layer: Perform 3×3 convolution on the fused feature map and output the multimodal fusion feature map F output .

9. The underwater target detection method according to claim 1, characterized in that: The deep learning model described in step S5) is based on the YOLOv11 architecture, which changes one feature extraction network into two parallel feature extraction networks and introduces the feature fusion module described in step S4). The YOLOv11 architecture includes a backbone network, a feature enhancement network and a detection head.

10. The underwater target detection method according to claim 1, characterized in that: The target detection result in step S5) includes target category label, location information and detection confidence.

Images