A method for detecting and locating targets above and below water based on acoustic and optical fusion

Abstract

The present invention relates to the technical field of above-water and underwater target recognition, and discloses an above-water and underwater target detection and positioning method based on acoustic-optical fusion, comprising: obtaining an optical image of the above-water part of the target and a sonar image of the underwater part of the target, and preprocessing the optical image and the sonar image to obtain acoustic-optical fusion pair data with time synchronization and spatial alignment; inputting the acoustic-optical fusion pair data into a pre-trained target detection model to obtain a target detection result; obtaining inertial guidance information, and determining the above-water position information and underwater position information of the target based on the inertial guidance information; determining the position information of the target according to the preset credibility of the above-water position information and the underwater position information. The present invention can effectively detect and locate targets having both above-water and underwater parts by fusing above-water optical image information with underwater sonar information, and combining inertial guidance information to detect and locate targets, thereby improving detection accuracy and positioning precision.

Description

Technical Field

[0001] The present invention relates to the technical field of above-water and underwater target recognition, and in particular to an above-water and underwater target detection and positioning method based on acoustic-optical fusion. Background Art

[0002] Optical and acoustic sensors are widely used in various marine vehicles and ships. In water scenes, optical sensors have the characteristics of color information and high details, but single cameras and other similar sensors are difficult to obtain accurate target position information, and cannot obtain underwater target information. In underwater scenes, sonar technology can provide long-distance object detection and ranging in underwater environments, but due to resolution limitations, it is impossible to obtain detailed object shape and texture information, and it is also impossible to obtain water target information. For targets that are partially above the water surface and partially below the water surface, the information obtained by a single acoustic or optical device is difficult to accurately detect the target.

[0003] Optical sensors mainly include optical cameras, laser radar and other methods, which can obtain rich target color and distance information on the water. Optical devices have the advantages of non-contact and high precision. Acoustic sensors mainly include sonar, which can obtain information and distance information of underwater targets. The existing single sensor detection method is difficult to effectively detect and locate targets both on the water and underwater at the same time. Summary of the invention

[0004] In view of this, the present invention provides a method for detecting and positioning targets above and below water based on acoustic and optical fusion, so as to solve the problem that the detection method of a single sensor is difficult to effectively detect and position targets both above and below water at the same time.

[0005] In a first aspect, the present invention provides a method for detecting and locating targets above and below water based on acoustic-optical fusion, the method comprising:

[0006] Acquire an optical image of the above-water part of the target and a sonar image of the underwater part of the target, and pre-process the optical image and the sonar image to obtain time-synchronized and space-aligned acoustic-optical fusion pair data;

[0007] Input the sound-light fusion data into the pre-trained target detection model to obtain the target detection result;

[0008] Acquire inertial guidance information, and determine the above-water position information and underwater position information of the target based on the inertial guidance information;

[0009] The target's location information is determined based on the preset credibility of the above-water location information and the underwater location information.

[0010] The method for detecting and positioning targets above and below the water based on acoustic and optical fusion provided by the present invention can effectively detect and position targets having both above-water parts and underwater parts by fusing above-water optical image information with underwater sonar information and combining inertial guidance information to detect and position targets, thereby improving detection accuracy and positioning precision.

[0011] In an optional implementation, the acoustic-optical fusion data includes: optical fusion data and sonar fusion data. The acoustic-optical fusion data is input into a pre-trained target detection model to obtain a target detection result, including:

[0012] Inputting the optical fusion data and the sonar fusion data into different feature extraction branches respectively to obtain optical image features and sonar image features;

[0013] The optical image features and the sonar image features are fused according to a preset fusion strategy to obtain fused features, wherein the preset fusion strategy includes dynamically adjusted optical image feature weights and sonar image feature weights;

[0014] The targets are classified and detected based on the fusion features to determine the target detection results.

[0015] The above-water and underwater target detection and positioning method based on acoustic-optical fusion provided by the present invention processes optical images and sonar images through a multi-modal input and dual-branch composite network structure to achieve target detection, ensure that two data of different data types can effectively enter the network, enhance multi-modal adaptability, fuse the multi-modal features of the target for detection, and improve the accuracy of target detection.

[0016] In an optional implementation, the training process of the pre-trained object detection model includes:

[0017] Acquire multiple optical images of the above-water part and multiple sonar images of the underwater part of the same target;

[0018] Matching the obtained optical image and sonar image according to the acquisition time to obtain multiple matching data pairs;

[0019] Obtain target image information and target position information corresponding to each matching data pair, and form a sample data set with multiple matching data pairs, corresponding target image information and target position information;

[0020] The sample data set is used to train the neural network model to obtain a pre-trained target detection model.

[0021] The above-water and underwater target detection and positioning method based on acoustic-optical fusion provided by the present invention adopts a composite network structure as the structural basis of a pre-trained target detection model, and combines optical data and sonar data. It can provide more accurate target information in complex environments, improve detection accuracy, make up for the defects of single-modality detection, and improve the robustness and adaptability of target detection.

[0022] In an optional implementation, acquiring inertial guidance information, and determining the above-water position information of the target based on the inertial guidance information, includes:

[0023] Obtaining the optical image and radar information of the above-water part of the target, and superimposing the optical image and radar information to obtain the above-water position information of the target in the optical camera coordinate system. The optical image is acquired by using the optical camera;

[0024] Acquire inertial guidance information, and perform coordinate conversion between the inertial guidance information and the optical camera to obtain the target's position information on the water in three-dimensional space.

[0025] In an optional implementation, acquiring inertial guidance information, and determining underwater position information of the target based on the inertial guidance information, includes:

[0026] Use sonar equipment to obtain the underwater position information of the target in the sonar coordinate system;

[0027] Acquire inertial guidance information, and perform coordinate conversion between the inertial guidance information and the sonar coordinate system of the sonar equipment to obtain the underwater position information of the target in three-dimensional space.

[0028] The above-water and underwater target detection and positioning method based on acoustic-optical fusion provided by the present invention determines the three-dimensional spatial position information of the optical camera and the sonar device in the world coordinate system through an inertial guidance device, and further determines the above-water position information and underwater position information of the target in the three-dimensional space by combining the optical image and the sonar image of the target, thereby overcoming the defects of a single sensor in above-water or underwater detection and positioning, and realizing unified detection and positioning of targets in a mixed above-water and underwater environment.

[0029] In an optional implementation, determining the location information of the target according to the preset credibility of the above-water location information and the underwater location information includes:

[0030] If the difference between the preset credibility of the above-water position information and the preset credibility of the underwater position information is greater than a preset threshold, comparing the preset credibility of the above-water position information with the preset credibility of the underwater position information;

[0031] If the preset credibility of the above-water position information is greater than the preset credibility of the underwater position information, the above-water position information is used as the position information of the target;

[0032] If the preset credibility of the above-water position information is less than the preset credibility of the underwater position information, the underwater position information is used as the position information of the target.

[0033] In an optional embodiment, if the difference between the preset credibility of the above-water position information and the preset credibility of the underwater position information is not greater than a preset threshold, the weighted position information of the target is calculated based on the weights of the above-water position information and the underwater position information, and the weighted position information is used as the position information.

[0034] The above-water and underwater target detection and positioning method based on acoustic-optical fusion provided by the present invention improves the accuracy of target positioning by selecting the credibility based on multi-sensor information, especially in complex environments, and can significantly reduce the false alarm rate and missed alarm rate, thereby improving the accuracy of target positioning.

[0035] In a second aspect, the present invention provides an above-water and underwater target detection and positioning device based on acoustic-optical fusion, the device comprising:

[0036] An image data acquisition module is used to acquire an optical image of the above-water part of the target and a sonar image of the underwater part of the target, and pre-process the optical image and the sonar image to obtain time-synchronized and space-aligned acoustic-optical fusion pair data;

[0037] The target detection module is used to input the sound and light fusion data into the pre-trained target detection model to obtain the target detection result;

[0038] An inertial guidance position determination module is used to obtain inertial guidance information and determine the above-water position information and underwater position information of the target based on the inertial guidance information;

[0039] The target positioning module is used to determine the position information of the target according to the preset credibility of the above-water position information and the underwater position information.

[0040] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0041] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to cause a computer to execute the method of the first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0043] Figure 1 is a flow chart of a method for detecting and locating targets above and below water based on acoustic and optical fusion according to an embodiment of the present invention;

[0044] Figure 2 is a schematic diagram of the installation of various devices in the above-water and underwater target detection and positioning method based on sound and light fusion according to an embodiment of the present invention;

[0045] Figure 3 It is a flowchart of the detection process and the positioning process in the above-water and underwater target detection and positioning method based on acoustic-optical fusion according to an embodiment of the present invention;

[0046] Figure 4 is a flow chart of another method for detecting and locating targets above and below water based on acoustic and optical fusion according to an embodiment of the present invention;

[0047] Figure 5 is a schematic diagram of the fusion of optical images and acoustic images in a method for detecting and positioning targets above and below water based on acoustic and optical fusion according to an embodiment of the present invention;

[0048] Figure 6 is a structural schematic diagram of target detection model training in a method for detecting and positioning targets above and below water based on acoustic and optical fusion according to an embodiment of the present invention;

[0049] Figure 7 is a structural block diagram of an above-water and underwater target detection and positioning device based on acoustic-optical fusion according to an embodiment of the present invention;

[0050] Figure 8 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0052] The embodiment of the present invention provides a method for detecting and locating targets above and below water based on acoustic-optical fusion, which achieves the effect of effectively detecting and locating targets above and below water simultaneously by fusing optical images and sonar images.

[0053] According to an embodiment of the present invention, an embodiment of a method for detecting and positioning targets above and below water based on acoustic and optical fusion is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in an order different from that shown here.

[0054] In this embodiment, a method for detecting and locating targets above and below water based on acoustic and optical fusion is provided, which can be used in the above-mentioned computer system. Figure 1 is a flow chart of a method for detecting and locating targets above and below water based on acoustic and optical fusion according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:

[0055] Step S101, acquiring an optical image of the above-water part of the target and a sonar image of the underwater part of the target, and preprocessing the optical image and the sonar image to obtain time-synchronized and space-aligned acoustic-optical fusion pair data.

[0056] Specifically, this embodiment is suitable for the case where the detection target has both an above-water part and an underwater part, such as Figure 2 As shown, an optical camera, a laser radar, a sonar and an inertial guidance device are installed on a movable surface platform, wherein the optical camera and the laser radar are located above the water surface, the optical camera is used to detect the shape and size of the target on the water, and the laser radar is used to detect the position of the target on the water; the sonar is located below the water surface and is used to detect the shape, size and position of the underwater target.

[0057] Before target detection and positioning, the initial position of the optical camera, laser radar, sonar and inertial guidance equipment is calibrated to establish the coordinate conversion relationship between the equipment. When the surface platform is stationary as the initial state, the position of each device in the initial state is measured and calibrated, including the installation position of the optical camera, laser radar, sonar and inertial guidance equipment and the conversion of the relative coordinate system.

[0058] For target detection, an optical camera is used to obtain an optical image of the above-water part, and a sonar is used to obtain a sonar image of the underwater part. The sonar image is converted into a grayscale image, and the optical image and sonar image are preprocessed, including image enhancement, denoising, and time synchronization and spatial position alignment of image acquisition. The specific process includes: matching the optical image and sonar image by timestamp to ensure that they correspond to the same target. In terms of hardware, the optical camera and sonar device are connected through a trigger signal line so that both start collecting at the same time to reduce time deviation; in terms of software, the timestamp is used to interpolate and align the data. The corresponding image matching algorithm is used to pair the above-water and underwater images. The optical camera collects image data, while the sonar usually collects sound wave reflection intensity maps (grayscale images). There are large differences in image features. Therefore, the resolution of the sonar image is adjusted to make it as close as possible to the resolution of the optical image for matching. Edge detection is performed on the optical image and sonar data respectively. The edge features of the two data can be matched during the fusion process to form acoustic-optical fusion pair data. The optical image of the above-water part and the sonar image of the underwater part of the same target are used as acoustic-optical fusion pair data.

[0059] Step S102, inputting the sound-light fusion data into a pre-trained target detection model to obtain a target detection result.

[0060] Specifically, the pre-trained target detection model is a neural network model trained in advance, and can detect targets using sound and light fusion data. By inputting the sound and light fusion data into the pre-trained target detection model, the target detection result is obtained. The target detection result may include information such as the size, appearance, and shape of the target.

[0061] Step S103, acquiring inertial guidance information, and determining the above-water position information and underwater position information of the target based on the inertial guidance information.

[0062] Specifically, the inertial guidance device is installed underwater to obtain the speed and position of the running object at different times. In this embodiment, the inertial guidance device is used to obtain the speed and position information of the optical camera and the sonar device as the inertial guidance information, and the surface position information of the target is determined based on the inertial guidance information, the coordinate conversion relationship between the inertial guidance device and the optical camera, and the optical image; the underwater position information of the target is determined based on the inertial guidance information, the coordinate conversion relationship between the inertial guidance device and the sonar device, and the sonar image. Figure 3As shown in the figure, it is a schematic diagram of the overall process of target detection and positioning. In target detection, the target detection results of the optical image and the acoustic image are fused to more accurately identify the target. Target positioning combines the navigation position information of the inertial guidance device, the distance information of the laser radar, and the distance information of the sonar device to jointly determine the target position. Finally, the target detection result and the position information are combined to achieve the purpose of target detection and positioning.

[0063] Step S104, determining the location information of the target according to the preset credibility of the above-water location information and the underwater location information.

[0064] Specifically, the credibility of the optical camera and the sonar device is set according to actual conditions as the preset credibility of the above-water position information and the underwater position information, respectively. The position information of the target in the three-dimensional space is calculated based on the preset credibility, and the obtained position information is more accurate.

[0065] The method for detecting and positioning targets above and below water based on acoustic and optical fusion provided in this embodiment can effectively detect and position targets having both above-water and underwater parts by integrating the above-water optical image information with the underwater sonar information and combining the inertial guidance information to detect and position the targets, thereby improving the detection accuracy and positioning precision.

[0066] In this embodiment, a method for detecting and locating targets above and below water based on acoustic and optical fusion is provided, which can be used in the above-mentioned computer system. Figure 4 is a flow chart of a method for detecting and locating targets above and below water based on acoustic and optical fusion according to an embodiment of the present invention. Figure 4 As shown, the process includes the following steps:

[0067] Step S201, obtain an optical image of the above-water part of the target and a sonar image of the underwater part of the target, and pre-process the optical image and the sonar image to obtain time-synchronized and space-aligned acoustic-optical fusion pair data. Figure 1 Step S101 of the illustrated embodiment will not be described in detail here.

[0068] Step S202, input the sound and light fusion data into a pre-trained target detection model to obtain a target detection result.

[0069] Specifically, the acoustic-optical fusion data includes: optical fusion data and sonar fusion data. The above step S202 includes:

[0070] Step S2021, inputting the optical fusion data and the sonar fusion data into different feature extraction branches respectively to obtain optical image features and sonar image features.

[0071] Specifically, the data acquisition process uses a composite network structure that supports multimodal input, and simultaneously receives optical images from optical cameras and sonar images from sonar devices. The optical camera provides visual information (such as edges, textures, etc.), and the sonar device provides intensity images (grayscale images) of sound wave reflections. The multimodal input layer ensures that two different data types can effectively enter the composite network. In the composite network, the feature extraction process is implemented through a dual-branch feature extraction module. For optical images, the optical branch uses convolutional neural networks (CNN) to extract image features, such as edges, colors, textures and other details. The optical branch contains multiple convolutional layers and pooling layers to extract high-level features layer by layer. For sonar images, the sonar branch contains more noise reduction layers and feature extraction layers that adapt to low resolution. The depth information and reflection intensity information of the sonar image are extracted through the convolutional layer. The sonar branch also contains multiple convolutional layers and pooling layers to specifically extract the target shape and reflection features unique to sonar.

[0072] The optical fusion data is the optical data obtained by preprocessing the optical image, and the sonar fusion data is the sonar data obtained by preprocessing the sonar image. The optical branch extracts features from the optical fusion data to obtain optical image features, and the sonar branch extracts features from the sonar fusion data to obtain sonar image features.

[0073] Step S2022: Fusing the optical image features and the sonar image features according to a preset fusion strategy to obtain fused features. The preset fusion strategy includes dynamically adjusted optical image feature weights and sonar image feature weights.

[0074] Specifically, the optical image features from the optical branch and the sonar image features from the sonar branch are fused at a specific level. The fusion methods include but are not limited to: feature concatenation, element addition, or weighted fusion based on the attention mechanism. The fusion strategy can balance the influence of the optical image features and the sonar image features according to the weight of the feature type to enhance the multimodal adaptability of the composite network. The optical image and the acoustic image enter the convolution module at the same time for feature extraction, and the sonar features and the optical features are fused, such as Figure 5 As shown, this is a structural schematic diagram of the sound and light fusion process.

[0075] Step S2023, classify and detect the target according to the fusion features to determine the target detection result.

[0076] Specifically, the composite network is the original network structure of the pre-trained target detection model. In the target detection process, the composite network represents the pre-trained target detection model and is the network structure of the trained target detection model. The composite network ultimately classifies the feature categories based on the fused features and predicts the position and size of the target at the same time, which is only an example, but not limited to this.

[0077] The above-water and underwater target detection and positioning method based on acoustic-optical fusion provided in this embodiment processes optical images and sonar images through a multi-modal input and dual-branch composite network structure to achieve target detection, ensure that two data of different data types can effectively enter the network, enhance multi-modal adaptability, fuse the multi-modal features of the target for detection, and improve the accuracy of target detection.

[0078] In some optional implementations, the training process of the pre-trained object detection model includes:

[0079] Acquire multiple optical images of the above-water part and multiple sonar images of the underwater part of the same target.

[0080] The obtained optical image and sonar image are matched according to the acquisition time to obtain multiple matching data pairs.

[0081] Specifically, the matching data composed of optical images and sonar images must meet the following conditions: the same target, the same time, the same location, and both the above-water part and the underwater part to ensure the validity of the data. Single above-water detection relies on optical images, which may be affected by water surface reflections and weather conditions (such as fog and rain); while single underwater detection is based on sonar images, which have low resolution and are prone to losing details. Figure 6 As shown in Figure 1, it is a schematic diagram of the training process of the target detection model. After the model is trained, the model is evaluated by testing target images, statistical target recognition accuracy, and above-water and underwater detection experiments.

[0082] The target image information and the target position information corresponding to each matching data pair are obtained, and a plurality of matching data pairs, the corresponding target image information and the target position information are formed into a sample data set.

[0083] The sample data set is used to train the neural network model to obtain a pre-trained target detection model.

[0084] Specifically, the target image information and target position information corresponding to each matching data pair are used as standard data, and multiple matching data pairs are input into the neural network model (composite network structure) respectively, and the predicted target image information and predicted target position information are output, and the predicted target image information is compared with the standard target image information, and the predicted target position information is compared with the standard target position information to obtain the image deviation and the position deviation. When the image deviation and the position deviation converge or meet the corresponding preset conditions, the model training process ends, and the neural network model used for the last time is used as the pre-trained target detection model. The detailed training process is a mature existing technology and will not be repeated here. Optical images are suitable for detecting details and color information, but are limited in turbid waters; sonar images can penetrate water and adapt to different water conditions. The composite network combines two modal information and can maintain a high recognition effect when the water is unclear, the light is insufficient, or the water is turbid.

[0085] The above-water and underwater target detection and positioning method based on acoustic-optical fusion provided in this embodiment adopts a composite network structure as the structural basis of a pre-trained target detection model, and combines optical data and sonar data. It can provide more accurate target information in complex environments, improve detection accuracy, make up for the defects of single-modality detection, and improve the robustness and adaptability of target detection.

[0086] Step S203, acquiring inertial guidance information, and determining the above-water position information and underwater position information of the target based on the inertial guidance information.

[0087] In some optional implementations, the above step S203 includes:

[0088] Step 2031, obtaining an optical image and radar information of the above-water part of the target, and superimposing the optical image and the radar information to obtain the above-water position information of the target in the optical camera coordinate system. The optical image is collected by using an optical camera.

[0089] Specifically, for the above-water part of the target, an optical camera is used to obtain an optical image, and a lidar is used to obtain radar information (point cloud information). The obtained optical image is used for target detection, and then the point cloud data obtained by the lidar and the optical image are converted into coordinates, and the position information of the point cloud is superimposed on the optical image to obtain the distance information of the target. The fusion of the optical image and the point cloud information can determine the above-water position information of the target in the optical camera coordinate system. The specific process is the same as the above-water target detection and positioning, including:

[0090] (1) In the optical image, the target is marked and trained so that the target can be detected directly in the optical image.

[0091] (2) Superimpose the distance and position information of the laser radar onto the optical camera. The specific conversion is as follows;

[0092]

[0093] Among them, R is the rotation matrix and T is the translation vector, which describes the relative position between the camera and the lidar. Represents the three-dimensional coordinates of the lidar point cloud in the lidar coordinate system, Represents the three-dimensional coordinates of the lidar point cloud in the optical camera coordinate system.

[0094] (3) The purpose of intrinsic calibration is to describe the optical characteristics of the camera, including focal length, optical center position and distortion parameters. The commonly used intrinsic parameter matrix is ​​as follows:

[0095]

[0096] Among them, fx and fy are the focal lengths of the optical camera, and cx and cy are the pixel coordinates of the optical center.

[0097] (4) Project the LiDAR point cloud to a two-dimensional image. After the projection is completed, the depth information of the LiDAR is: , the depth information is superimposed on the two-dimensional image of the optical camera in the form of grayscale or pseudo-color to form an RGB-D image or a superimposed image with depth information. The projection formula is:

[0098]

[0099] Where (u, v) is the pixel coordinate of the lidar point in the image plane, is the depth information, Represents the coordinate information of the two-dimensional image of the optical camera.

[0100] Step S2032, acquiring inertial guidance information, and performing coordinate conversion between the inertial guidance information and the optical camera to obtain the above-water position information of the target in three-dimensional space.

[0101] Specifically, the inertial guidance information is converted into coordinates by the optical camera to obtain the water position information of the optical camera in the inertial guidance device coordinate system (usually the world coordinate system). The laser radar and optical image overlay information are combined to determine the water position information of the target in three-dimensional space.

[0102] Step S2033, using the sonar device to obtain the underwater position information of the target in the sonar coordinate system.

[0103] Specifically, the sonar image is obtained by using the sonar device and image preprocessing is performed, the target recognition algorithm is detected by using the sonar image, the position information of the target is calculated in the sonar image, and the azimuth and position of the target are obtained. This process is the same as the target detection and positioning of only the underwater part, including:

[0104] (1) The sonar device transmits sound wave signals and receives echo signals to generate corresponding sonar images. Each pixel in the sonar image contains the intensity information of the echo, and each pixel corresponds to the arrival time of a certain sound wave.

[0105] (2) According to the echo time t of each pixel point, the distance d of the pixel corresponding to the target object is calculated using the sound wave propagation formula. The calculation formula is as follows:

[0106]

[0107] Where v represents the propagation speed of the wave in the medium, and t is the time from the emission of the sound wave to the receipt of the echo.

[0108] (3) Sonar equipment usually has a known scanning angle range. For each pixel, the corresponding azimuth is calculated using the following formula: :

[0109]

[0110] in and Represents the minimum and maximum angles of sonar scanning, is the horizontal or vertical index of the current pixel. is the total number of pixels in this direction.

[0111] (4) Based on the distance d of the target object and the azimuth angle of each pixel , convert the distance and angle in the polar coordinate system into two-dimensional or three-dimensional coordinates in the Cartesian coordinate system, and then generate different colors or brightness for pixels at different distances to visualize the distance and spatial position of the object. The specific two-dimensional coordinate conversion is as follows:

[0112]

[0113] The three-dimensional coordinate transformation is as follows:

[0114]

[0115] in, Indicates azimuth information, is the pitch angle.

[0116] Step S2034, acquiring inertial guidance information, and performing coordinate conversion between the inertial guidance information and the sonar coordinate system of the sonar device to obtain underwater position information of the target in three-dimensional space.

[0117] Specifically, the inertial guidance information is converted into coordinates by the sonar device to obtain the position information of the sonar device in the inertial guidance device coordinate system (usually the world coordinate system), and combined with the sonar image information to obtain the underwater position information of the target in three-dimensional space.

[0118] The method for detecting and positioning targets above and below water based on acoustic-optical fusion provided in this embodiment determines the three-dimensional spatial position information of the optical camera and the sonar device in the world coordinate system through an inertial guidance device, and further determines the above-water position information and underwater position information of the target in the three-dimensional space by combining the optical image and sonar image of the target, thereby overcoming the defects of a single sensor in above-water or underwater detection and positioning, and realizing unified detection and positioning of targets in a mixed environment above and underwater.

[0119] Step S204, determining the location information of the target according to the preset credibility of the above-water location information and the underwater location information.

[0120] Specifically, the above step S204 includes:

[0121] Step S2041, if the difference between the preset credibility of the above-water position information and the preset credibility of the underwater position information is greater than a preset threshold, compare the preset credibility of the above-water position information with the preset credibility of the underwater position information.

[0122] Step S2042: If the preset credibility of the above-water position information is greater than the preset credibility of the underwater position information, the above-water position information is used as the position information of the target.

[0123] Step S2043: If the preset credibility of the above-water position information is less than the preset credibility of the underwater position information, the underwater position information is used as the target position information.

[0124] Specifically, according to the actual conditions of the optical camera, laser radar, sonar equipment and environment, the preset credibility of the above-water position information and the preset credibility of the underwater position information are determined. The sum of the preset credibility of the above-water position information and the preset credibility of the underwater position information is usually "1". If the difference between the preset credibility of the above-water position information and the preset credibility of the underwater position information is greater than the preset threshold, the one with the greater credibility is used as the target position information. For example, if the preset threshold is 0.5, the preset credibility of the above-water position information is 0.8, and the preset credibility of the underwater position information is 0.2, the above-water position information is used as the target position information; if the preset credibility of the above-water position information is 0.15 and the preset credibility of the underwater position information is 0.85, the underwater position information is used as the target position information, which is only an example, but not limited to this.

[0125] In some optional embodiments, if the difference between the preset credibility of the above-water position information and the preset credibility of the underwater position information is not greater than a preset threshold, the weighted position information of the target is calculated based on the weights of the above-water position information and the underwater position information, and the weighted position information is used as the position information.

[0126] Specifically, if the difference between the preset credibility of the above-water position information and the preset credibility of the underwater position information is not greater than the preset threshold, the weights of the above-water position information and the underwater position information are determined according to the credibility of the two, and the weighted position information of the target is calculated according to the weights, and the weighted position information is used as the position information of the target. For example, if the preset threshold is 0.4, the preset credibility of the above-water position information is 0.4, and the preset credibility of the underwater position information is 0.6, then the weight of the above-water position information can be set to 0.4, and the weight of the underwater position information can be set to 0.6, and the position information of the target is: 0.4 above-water position information + 0.6 underwater position information, which is only an example, but not limited to this.

[0127] The above-water and underwater target detection and positioning method based on acoustic-optical fusion provided in this embodiment improves the accuracy of target positioning by selecting the credibility based on multi-sensor information, especially in complex environments, and can significantly reduce the false alarm rate and missed alarm rate, thereby improving the accuracy of target positioning.

[0128] In the present embodiment, a device for detecting and locating targets above and below the water based on the fusion of sound and light is also provided, and the device is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0129] This embodiment provides a device for detecting and locating targets above and below water based on acoustic and optical fusion. Figure 7 As shown, including:

[0130] The image data acquisition module 701 is used to acquire an optical image of the above-water part of the target and a sonar image of the underwater part of the target, and pre-process the optical image and the sonar image to obtain time-synchronized and space-aligned acoustic-optical fusion pair data.

[0131] The target detection module 702 is used to input the sound and light fusion data into a pre-trained target detection model to obtain a target detection result.

[0132] The inertial guidance position determination module 703 is used to obtain inertial guidance information and determine the above-water position information and underwater position information of the target based on the inertial guidance information.

[0133] The target positioning module 704 is used to determine the position information of the target according to the preset credibility of the above-water position information and the underwater position information.

[0134] In some optional implementations, the target detection module 702 includes:

[0135] The feature extraction unit is used to input the optical fusion data and the sonar fusion data into different feature extraction branches respectively to obtain optical image features and sonar image features.

[0136] The feature fusion unit is used to fuse the optical image features and the sonar image features according to a preset fusion strategy to obtain a fusion feature. The preset fusion strategy includes dynamically adjusted optical image feature weights and sonar image feature weights.

[0137] The classification and detection unit is used to classify and detect the target according to the fusion features and determine the target detection result.

[0138] In some optional implementations, the inertial guidance position determination module 703 includes:

[0139] The above-water position determination unit is used to obtain the optical image and radar information of the above-water part of the target, and superimpose the optical image and radar information to obtain the above-water position information of the target in the optical camera coordinate system. The optical image is collected by the optical camera.

[0140] The water position conversion unit is used to obtain inertial guidance information and perform coordinate conversion between the inertial guidance information and the optical camera to obtain the water position information of the target in three-dimensional space.

[0141] The underwater position determination unit is used to obtain the underwater position information of the target in the sonar coordinate system using the sonar equipment.

[0142] The underwater position conversion unit is used to obtain inertial guidance information and perform coordinate conversion between the inertial guidance information and the sonar coordinate system of the sonar device to obtain the underwater position information of the target in three-dimensional space.

[0143] In some optional implementations, the target positioning module 704 includes:

[0144] The credibility comparison unit is used to compare the preset credibility of the above-water position information with the preset credibility of the underwater position information if the difference between the preset credibility of the above-water position information and the preset credibility of the underwater position information is greater than a preset threshold.

[0145] The first position determination unit is used to use the above-water position information as the position information of the target if the preset credibility of the above-water position information is greater than the preset credibility of the underwater position information.

[0146] The second position determination unit is used to use the underwater position information as the position information of the target if the preset credibility of the above-water position information is less than the preset credibility of the underwater position information.

[0147] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0148] The above-water and underwater target detection and positioning device based on sound and light fusion in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above-mentioned functions.

[0149] The embodiment of the present invention also provides a computer device having the above Figure 7 The above-water and underwater target detection and positioning device based on acoustic-optical fusion is shown.

[0150] See also Figure 8 , Figure 88 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention. As shown in FIG8 , the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the computer device, including instructions stored in or on the memory to display graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 8 A processor 10 is taken as an example.

[0151] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0152] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0153] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0154] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0155] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0156] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0157] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A method for detecting and locating targets above and below water based on acoustic and optical fusion, characterized in that: The method comprises: Acquire an optical image of the above-water part of the target and a sonar image of the underwater part of the target, and pre-process the optical image and the sonar image to obtain time-synchronized and space-aligned acoustic-optical fusion pair data; Inputting the acoustic-optical fusion data into a pre-trained target detection model to obtain a target detection result; Acquire inertial guidance information, and determine the above-water position information and underwater position information of the target based on the inertial guidance information, including: acquire an optical image and radar information of the above-water part of the target, and superimpose the optical image and the radar information to obtain the above-water position information of the target in the optical camera coordinate system, wherein the optical image is acquired by using an optical camera; acquire inertial guidance information, and perform coordinate conversion between the inertial guidance information and the optical camera to obtain the above-water position information of the target in three-dimensional space, and use a sonar device to acquire the underwater position information of the target in the sonar coordinate system; acquire inertial guidance information, and perform coordinate conversion between the inertial guidance information and the sonar coordinate system of the sonar device to obtain the above-water position information of the target in three-dimensional space; Determining the location information of the target according to the preset credibility of the above-water location information and the underwater location information includes: If the difference between the preset credibility of the above-water position information and the preset credibility of the underwater position information is greater than a preset threshold, comparing the preset credibility of the above-water position information with the preset credibility of the underwater position information; If the preset credibility of the above-water position information is greater than the preset credibility of the underwater position information, taking the above-water position information as the position information of the target; If the preset credibility of the above-water position information is less than the preset credibility of the underwater position information, the underwater position information is used as the position information of the target.

2. The method according to claim 1, characterized in that The acoustic-optical fusion data includes: optical fusion data and sonar fusion data. The acoustic-optical fusion data is input into a pre-trained target detection model to obtain a target detection result, including: Inputting the optical fusion data and the sonar fusion data into different feature extraction branches respectively to obtain optical image features and sonar image features; The optical image feature and the sonar image feature are subjected to feature fusion according to a preset fusion strategy to obtain a fusion feature, wherein the preset fusion strategy includes dynamically adjusted optical image feature weights and sonar image feature weights; The target is classified and detected according to the fusion features to determine the target detection result.

3. The method according to claim 1 or 2, characterized in that: The training process of the pre-trained target detection model includes: Acquire multiple optical images of the above-water part and multiple sonar images of the underwater part of the same target; Matching the obtained optical image and the sonar image according to the acquisition time to obtain a plurality of matching data pairs; Obtain target image information and target position information corresponding to each matching data pair, and form a sample data set with multiple matching data pairs, corresponding target image information and target position information; The sample data set is used to train a neural network model to obtain a pre-trained target detection model.

4. The method according to claim 1, characterized in that If the difference between the preset credibility of the above-water position information and the preset credibility of the underwater position information is not greater than a preset threshold, the weighted position information of the target is calculated according to the weights of the above-water position information and the underwater position information, and the weighted position information is used as the position information.

5. A device for detecting and locating targets above and below water based on acoustic and optical fusion, characterized in that: The device comprises: An image data acquisition module is used to acquire an optical image of the above-water part of the target and a sonar image of the underwater part of the target, and pre-process the optical image and the sonar image to obtain time-synchronized and space-aligned acoustic-optical fusion pair data; A target detection module, used for inputting the acoustic-optical fusion data into a pre-trained target detection model to obtain a target detection result; The inertial guidance position determination module is used to obtain inertial guidance information, and determine the above-water position information and underwater position information of the target based on the inertial guidance information, including: obtaining an optical image and radar information of the above-water part of the target, and superimposing the optical image and the radar information to obtain the above-water position information of the target in the optical camera coordinate system, wherein the optical image is acquired by using an optical camera; obtaining inertial guidance information, and performing coordinate conversion between the inertial guidance information and the optical camera to obtain the above-water position information of the target in three-dimensional space, and using a sonar device to obtain the underwater position information of the target in the sonar coordinate system; obtaining inertial guidance information, and performing coordinate conversion between the inertial guidance information and the sonar coordinate system of the sonar device to obtain the above-water position information of the target in three-dimensional space; The target positioning module is used to determine the position information of the target according to the preset credibility of the above-water position information and the underwater position information, including: If the difference between the preset credibility of the above-water position information and the preset credibility of the underwater position information is greater than a preset threshold, comparing the preset credibility of the above-water position information with the preset credibility of the underwater position information; If the preset credibility of the above-water position information is greater than the preset credibility of the underwater position information, taking the above-water position information as the position information of the target; If the preset credibility of the above-water position information is less than the preset credibility of the underwater position information, the underwater position information is used as the position information of the target.

6. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the method according to any one of claims 1 to 4 by executing the computer instructions.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the method according to any one of claims 1 to 4.

Images