A multi-sensing fusion underwater frogman identification and positioning method and system

Abstract

The present invention discloses a multi-sensory fusion underwater frogman identification and positioning method and system, which obtains the geographic coordinates of a mother ship through the Beidou system to establish a reference coordinate system; the mother ship establishes hydroacoustic communication with an underwater vehicle; the phase difference of array signals, the speed of acoustic signals in water, signal frequency, and the time difference between transmission and reception are used to calculate the distance, azimuth, and pitch angle positioning information between the underwater vehicle and the origin of the mother ship coordinate system; the plane coordinates and depth of the underwater vehicle in the mother ship coordinate system are calculated according to the positioning information; the frogman is detected by using a visual camera and sonar on the underwater vehicle to obtain the perception fusion information and position information of the underwater frogman; the frogman coordinates are converted to the reference coordinate system through two conversions; the master station equipment fuses the perception fusion information of the frogman and superimposes the positioning information to realize the visual display of the perception fusion and positioning information. The present invention uses multi-sensor information fusion to identify and locate the underwater frogman with higher accuracy.

Description

Technical Field

[0001] The present invention belongs to the field of underwater sensing and positioning technology, and specifically relates to a multi-sensing fusion underwater frogman identification and positioning method and system. Background Art

[0002] my country has about 3 million square kilometers of sea area and is one of the few major maritime powers in the world. The vast sea area has led to an increasing demand for underwater sensing and positioning technology in my country's underwater emergency rescue, detection and maritime national defense security.

[0003] Given that the underwater environment has the characteristics of low visibility, strong concealment, weak sense of direction, and that underwater frogmen are small targets with poor detectability, the identification and positioning technology of underwater frogmen has become a topic that urgently needs research and development.

[0004] At present, most underwater frogman positioning technologies require the frogman to carry a transponder. In a non-friendly environment, when the frogman himself does not carry a transponder, the existing underwater frogman positioning technology that only uses long / short arrays cannot achieve underwater frogman detection. In addition, due to problems such as low contrast in the underwater environment and severe signal attenuation, it is difficult to ensure the perception and positioning accuracy of underwater targets using a single perception method. Therefore, in the case of no response from the underwater frogman, an underwater frogman detection technology that can perform high-precision perception and positioning of underwater frogmen in a natural environment is needed. Summary of the invention

[0005] In view of this, the present invention provides a multi-sensory fusion underwater frogman identification and positioning method and system, which can accurately locate the underwater frogman when the underwater frogman has no transponder.

[0006] The technical solution for implementing the present invention is as follows:

[0007] A multi-sensory fusion underwater frogman identification and positioning method comprises the following steps:

[0008] Step 1: Obtain the geographic coordinates of the mother ship through the Beidou system and perform two-dimensional coordinate conversion to establish a reference coordinate system;

[0009] Step 2: The mother ship establishes hydroacoustic communication with the underwater vehicle;

[0010] Step 3: Establish the mother ship coordinate system, and use the phase difference of the array signal, the speed of the acoustic signal in water, the signal frequency, and the time difference between transmission and reception to calculate the distance, azimuth, and pitch angle positioning information between the underwater vehicle and the origin of the mother ship coordinate system;

[0011] Step 4: Calculate the plane coordinates and depth of the underwater vehicle in the mother ship coordinate system according to the positioning information of the distance, azimuth and pitch angle;

[0012] Step 5: Use the visual camera and sonar on the underwater vehicle to detect the frogman and obtain the perception fusion information and position information of the underwater frogman;

[0013] Step 6: According to the rotation and translation theory, through two conversions, without the need to deploy a fixed acoustic array on the seabed and without the need for active response of the target to be detected, the frogman coordinates are converted to the reference coordinate system and converted into geographic coordinates;

[0014] Step 7: The master station equipment fuses the frogman's perception fusion information and superimposes the positioning information to achieve visual display of perception fusion and positioning information.

[0015] Furthermore, in step 4, the rotation and translation theory is used to obtain the conversion relationship between the mother ship coordinate system and the basic coordinate system, and the coordinate position of the underwater vehicle in the basic coordinate system is obtained.

[0016] Furthermore, the specific process of step five is as follows:

[0017] S51: Offline calibration of the visual camera and forward-looking sonar installed on the underwater vehicle, setting the underwater vehicle coordinate system so that the multi-sensor data are in the same coordinate system;

[0018] S52: When the underwater vehicle is operating under the sea, it uses a visual camera and a forward-looking sonar to detect the underwater frogman, and obtains visual images, acoustic images and positioning information respectively;

[0019] S53: Use the SSD algorithm to identify frogmen in visual images and the Cascade RCNN algorithm to identify frogmen in acoustic images;

[0020] S54: Calculate the intersection-over-union ratio (IoU) of the recognition areas of different sensor images. When IoU≥0.95, take the intersection area as the recognition area; when 0.95>IoU≥0.67, take the merged area as the recognition area; when IoU<0.67, it is judged that the frogman is not recognized; the average of the visual image recognition probability and the acoustic image recognition probability is taken as the recognition probability after fusion;

[0021] S55: The fused recognition area and recognition probability are used as the multi-sensor perception fusion result, and the position information is superimposed to obtain the final perception fusion information of the frogman and the position information in the underwater vehicle coordinate system.

[0022] Furthermore, in step seven, the visual images, acoustic images and positioning information collected by the underwater vehicle are transmitted back to the main station device, and positioning and perception fusion processing is performed through the recognition algorithm and multi-sensor data fusion model preset by the main station device, and displayed on the main screen, thereby realizing the visualization of underwater frogman recognition and positioning information.

[0023] A multi-sensory fusion underwater frogman identification and positioning system, including a main station device installed on a mother ship, a Beidou receiver, a motion sensor, a transmitting / receiving array (ultra-short baseline acoustic array), and an information processing unit, an ultra-short baseline transponder, a forward-looking sonar, a visual camera, and a motion sensor installed on an underwater vehicle; wherein:

[0024] The master station equipment is used to process the positioning information of the transmitting / receiving array, identify the image information sent back by the underwater vehicle and superimpose the positioning information, and visually display the positioning information for underwater frogmen to identify;

[0025] The motion sensor includes a gyroscope and a pressure sensor to obtain the roll angle α (around the X axis), pitch angle β (around the Y axis), heading angle γ (around the Z axis) and underwater depth of the carrier;

[0026] The Beidou receiver is used to locate multiple longitude and latitude positions of the transmitting / receiving array in real time;

[0027] The transmitting / receiving array and the ultra-short baseline transponder are used for hydroacoustic communication between the mother ship and the underwater vehicle, so as to determine the position of the underwater vehicle;

[0028] The visual camera and the forward-looking sonar detect the underwater frogman and obtain visual images, acoustic images and positioning information respectively;

[0029] The information processing unit is used to process the image information and position information collected by the visual camera and the forward-looking sonar.

[0030] Beneficial effects:

[0031] 1. The underwater frogman sensing and positioning technology of the present invention does not require the frogman to carry a transponder, and can be used in a wider range of conditions.

[0032] 2. Compared with underwater frogman positioning methods that rely solely on long baselines, short baselines or combined baselines, the present invention uses multi-sensor information fusion to identify and locate underwater frogmen, with higher accuracy.

[0033] 3. The present invention visually superimposes the positioning result and the visual recognition result, which is more intuitive.

[0034] 4. On the premise of ensuring accuracy, the present invention also has the advantages of simple operation process and integrated data processing and display. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 A schematic diagram of the working principle of the multi-sensory fusion underwater frogman identification and positioning system provided by an embodiment of the present invention;

[0036] Figure 2An intuitive schematic diagram of a multi-sensory fusion underwater frogman identification and positioning system provided by an embodiment of the present invention;

[0037] Figure 3 A flow chart of underwater vehicle coordinate calculation provided by an embodiment of the present invention;

[0038] Figure 4 A schematic diagram of the underwater frogman sensing information and obtaining positioning information provided by an embodiment of the present invention;

[0039] Figure 5 A flow chart of calculating the geographical coordinates of a frogman provided by an embodiment of the present invention;

[0040] Figure 6 Schematic diagram of coordinate system conversion provided by an embodiment of the present invention; (a) is a schematic diagram of coordinate conversion for an underwater vehicle, and (b) is a schematic diagram of coordinate conversion for a frogman;

[0041] Figure 7 A schematic diagram of the master station device information reception, processing and visualization process provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0042] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0043] The present invention provides a multi-sensory fusion underwater frogman identification and positioning method, such as Figure 1 As shown, the following steps S1-S7 are included:

[0044] S1. Obtain the geographic coordinates of the mother ship through the Beidou system and perform two-dimensional coordinate conversion to establish a reference coordinate system;

[0045] S2. The mother ship establishes hydroacoustic communication with the underwater vehicle;

[0046] S3, establish a mother ship coordinate system, and use the phase difference of the array signal, the speed of the acoustic signal in water, the signal frequency, and the time difference between transmission and reception to calculate the distance, azimuth, and pitch angle positioning information between the underwater vehicle and the origin of the mother ship coordinate system;

[0047] S4. Calculate the plane coordinates and depth of the underwater vehicle in the mother ship coordinate system according to the positioning information of the distance, azimuth, and pitch angle;

[0048] S5. Use the visual camera and sonar on the underwater vehicle to detect the frogman and obtain the perception fusion information and position information of the underwater frogman;

[0049] S6. According to the rotation-translation theory, through two conversions, without the need to deploy a fixed acoustic array on the seabed and without the need for active response of the target to be detected, the frogman coordinates are converted to the reference coordinate system and converted into geographic coordinates;

[0050] S7. The master station equipment fuses the frogman's perception fusion information and superimposes the positioning information to achieve visual display of perception fusion and positioning information.

[0051] Figure 2 The sensing positioning system for implementing the above process is shown, which includes a main station device, a Beidou receiver, a motion sensor, a transmitting / receiving array (ultra-short baseline acoustic array) installed on a mother ship, an information processing unit (not shown), an ultra-short baseline transponder, a forward-looking sonar, a visual camera, and a motion sensor installed on an underwater vehicle;

[0052] In this embodiment, the two-dimensional coordinate conversion of the geographic coordinates of the mother ship can be calculated according to the following formula:

[0053] X0=R*cos(lat)*cos(lng) (1)

[0054] Y0=R*cos(lat)*sin(lng) (2)

[0055] Z0=0 (3)

[0056] Among them, X0, Y0, Z0 are the coordinates of the mapping point of the mother ship on the basic coordinate system, R is the radius of the earth at the location, lat, lng are the longitude and latitude coordinates of the mother ship, and it is assumed that the longitude and latitude coordinates of the mother ship and the ultra-short baseline acoustic array are the same.

[0057] In this embodiment, the mother ship and the underwater vehicle establish underwater acoustic communication through the ultra-short array system, and establish a mother ship coordinate system with the ultra-short array as the origin, and obtain the positioning information of the underwater vehicle through the ultra-short array.

[0058] In this embodiment, S4 calculates the plane coordinates and depth of the underwater vehicle in the mother ship coordinate system, such as Figure 3 As shown, the specific steps are as follows S141-S144:

[0059] S141: Obtain the distance S and azimuth θ of the underwater vehicle through the ultra-short array positioning system x ,θ y and the pitch angle θ z Other positioning information;

[0060] S142: The coordinates (x1, y1, z1) of the underwater vehicle in the mother ship coordinate system can be calculated according to the following formula:

[0061] x1=S*cos(θ x ) (4)

[0062] y1=S*cos(θ y ) (5)

[0063] z1=S*cos(θ z ) (6)

[0064] S143: Figure 2 As shown, for the convenience of calculation, the base coordinate system and the mother ship coordinate system set the direction parallel to the mother ship as the x / X axis, the direction perpendicular to the mother ship as the y / Y axis, and the direction perpendicular to the sea surface as the z / Z axis. In summary, the rotation and translation matrix (R0, T0) can be calculated. Among them, the rotation matrix is:

[0065]

[0066] The translation matrix is:

[0067]

[0068] Among them, x n ,y n , z n is the coordinate of any point in the mother ship coordinate system.

[0069] S144: According to the rotation and translation matrix, the conversion relationship between the mother ship coordinate system and the base coordinate system can be obtained as follows:

[0070]

[0071] Substituting formulas (4), (5), (6), (7), and (8) into formula (9), the coordinates [X1, Y1, Z1] of the underwater vehicle in the basic coordinate system can be calculated as follows:

[0072]

[0073] In this embodiment, S5 uses multiple sensors to obtain the frogman's perception information and position information, such as Figure 4 As shown, the specific steps are as follows S151-S156:

[0074] S151: Offline calibration of the visual camera and forward-looking sonar installed on the underwater vehicle, setting the underwater vehicle coordinate system so that the multi-sensor data are in the same coordinate system;

[0075] S152: When the underwater vehicle is operating under the sea, it uses a visual camera and a forward-looking sonar to detect underwater frogmen and obtain visual images, acoustic images and positioning information respectively;

[0076] S153: Use the faster SSD algorithm to identify frogmen in visual images. At the same time, use the more accurate Cascade RCNN algorithm to identify frogmen in acoustic images.

[0077] S154: Calculate the intersection over union (IoU) of the recognition areas of different sensor images. When IoU>0.95, take the intersection area as the recognition area. When 0.95>IoU>0.67, take the merged area as the recognition area. When IoU<0.67, it is determined that the frogman is not recognized. In addition, the mean of the visual recognition probability and the acoustic image recognition probability is taken as the fused recognition result;

[0078] S155: The fused recognition area and recognition probability are used as the multi-sensor perception fusion result, and the position information is superimposed to obtain the final perception fusion information of the frogman and the position information in the underwater vehicle coordinate system.

[0079] In this embodiment, the coordinate system is converted in S6 to calculate the geographical coordinates of the frogman, such as Figure 5 As shown, the specific steps are as follows S161-S164:

[0080] S161: Obtain the position information of the frogman in the underwater vehicle coordinate system through the forward-looking sonar sensor;

[0081] S162: Figure 2 As shown in the figure, according to the altitude angle α and azimuth angle β measured by the forward-looking sonar, the coordinates (u1, v1, w1) of the frogman in the underwater vehicle coordinate system can be calculated as:

[0082]

[0083] S163: In order to obtain the coordinates of the frogman in the mother ship coordinate system, the rotation and translation matrix (R1, T1) needs to be calculated. Figure 6 As shown in (a), the x-axis deflection angle φ of the vehicle can be obtained from the motion sensor installed on the underwater vehicle x , y-axis deflection angle φ y 、z-axis deflection angle φ z , and then calculate the rotation matrix R1:

[0084]

[0085] According to the translation and rotation principle, the translation matrix T1 is the projection distance Δx, Δy, Δz of the underwater vehicle on the yoz plane, xoz plane, and xoy plane, as shown in Figure 6 As shown in (b), the sum of the projection distances Δx′1, Δy′1, and Δz′1 of the underwater frogman on the x′ axis, y′ axis, and z′ axis, that is:

[0086]

[0087] In the formula, the values ​​of Δx, Δy, and Δz are equal to the coordinates of the underwater vehicle in the mother ship coordinate system, and Δx1′, Δy1′, and Δz1′ can be calculated by the following formula:

[0088] Δx1′=cosφ x u1 (14)

[0089] Δy1′=cosφ y v1 (15)

[0090] Δz1′=cosφ z w1 (16)

[0091] Substituting equations (14), (15), and (16) into equation (13), we can obtain the translation matrix T1:

[0092]

[0093] S164: According to the rotation and translation matrix, the conversion relationship between the underwater vehicle coordinate system and the mother ship coordinate system can be obtained as follows:

[0094]

[0095] Substituting formula (18) into formula (9) yields the coordinates of the underwater frogman in the basic coordinate system:

[0096] [X2,Y2,Z2]:

[0097]

[0098] Then calculate the geographical coordinates of the frogman (lat w ,lng w ,h w ):

[0099] lng w =arctan(Y2 / X2) (21)

[0100] lat w =arccos(X2 / Rcos(lng w )) (twenty two)

[0101] h w = -Z2 (23)

[0102] In summary, the geographic coordinates of the underwater frogman can be calculated using equations (21), (22), and (23).

[0103] In this embodiment, the master station device in S7 performs visualization processing of frogman identification and positioning information, such as Figure 7As shown in the figure, the information collected by the motion sensor, visual camera, forward-looking sonar, Beidou receiver, ultra-short array system and other equipment is transmitted back to the main station equipment for processing or calling. For example, the visual image, acoustic image, and positioning information collected by the underwater vehicle are all transmitted back to the main station equipment, and the positioning and perception fusion processing is performed through the recognition algorithm preset by the main station equipment and the multi-sensor data fusion model, and displayed on the main screen, thereby realizing the visualization of underwater frogman recognition and positioning information.

[0104] In summary, the above are only preferred embodiments of the present invention and are not intended to limit the protection scope of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A multi-sensory fusion underwater frogman identification and positioning method, characterized in that: The following steps are involved: Step 1: Obtain the geographic coordinates of the mother ship through the Beidou system and perform two-dimensional coordinate conversion to establish a reference coordinate system; Step 2: The mother ship establishes hydroacoustic communication with the underwater vehicle; Step 3: Establish the mother ship coordinate system, and use the phase difference of the array signal, the speed of the acoustic signal in water, the signal frequency, and the time difference between transmission and reception to calculate the distance, azimuth, and pitch angle positioning information between the underwater vehicle and the origin of the mother ship coordinate system; Step 4: Calculate the plane coordinates and depth of the underwater vehicle in the mother ship coordinate system according to the positioning information of the distance, azimuth and pitch angle; Step 5: Use the visual camera and sonar on the underwater vehicle to detect the frogman and obtain the perception fusion information and position information of the underwater frogman; Step 6: According to the rotation and translation theory, through two conversions, without the need to deploy a fixed acoustic array on the seabed and without the need for active response of the target to be detected, the frogman coordinates are converted to the reference coordinate system and converted into geographic coordinates; Step 7: The master station equipment fuses the frogman's perception fusion information and superimposes the positioning information to achieve visual display of perception fusion and positioning information.

2. The multi-sensory fusion underwater frogman identification and positioning method according to claim 1, characterized in that: In step 4, the rotation and translation theory is used to obtain the transformation relationship between the mother ship coordinate system and the basic coordinate system, and the coordinate position of the underwater vehicle in the basic coordinate system is obtained.

3. The multi-sensory fusion underwater frogman identification and positioning method according to claim 1 or 2, characterized in that: The specific process of step five is as follows: S51: Offline calibration of the visual camera and forward-looking sonar installed on the underwater vehicle, setting the underwater vehicle coordinate system so that the multi-sensor data are in the same coordinate system; S52: When the underwater vehicle is operating under the sea, it uses a visual camera and a forward-looking sonar to detect the underwater frogman, and obtains visual images, acoustic images and positioning information respectively; S53: Use the SSD algorithm to identify frogmen in visual images and the Cascade RCNN algorithm to identify frogmen in acoustic images; S54: Calculate the intersection-over-union ratio (IoU) of the recognition areas of different sensor images. When IoU≥0.95, take the intersection area as the recognition area; when 0.95>IoU≥0.67, take the merged area as the recognition area; when IoU<0.67, it is judged that the frogman is not recognized; the average of the visual image recognition probability and the acoustic image recognition probability is taken as the recognition probability after fusion; S55: The fused recognition area and recognition probability are used as the multi-sensor perception fusion result, and the position information is superimposed to obtain the final perception fusion information of the frogman and the position information in the underwater vehicle coordinate system.

4. The multi-sensory fusion underwater frogman identification and positioning method according to claim 3, characterized in that: In step seven, the visual images, acoustic images and positioning information collected by the underwater vehicle are transmitted back to the main station device, and positioning and perception fusion processing is performed through the recognition algorithm and multi-sensor data fusion model preset by the main station device, and displayed on the main screen, thereby realizing the visualization of underwater frogman recognition and positioning information.

5. A multi-sensory fusion underwater frogman identification and positioning system using the positioning method described in claim 1, characterized in that: It includes the main station equipment, Beidou receiver, motion sensor, transmitting / receiving array installed on the mother ship, and the information processing unit, ultra-short baseline transponder, forward-looking sonar, visual camera, and motion sensor installed on the underwater vehicle; among which: The master station equipment is used to process the positioning information of the transmitting / receiving array, identify the image information sent back by the underwater vehicle and superimpose the positioning information, and visually display the positioning information for underwater frogmen to identify; The motion sensor includes a gyroscope and a pressure sensor to obtain the roll angle α, pitch angle β, heading angle γ and underwater depth of the carrier; The Beidou receiver is used to locate multiple longitude and latitude positions of the transmitting / receiving array in real time; The transmitting / receiving array and the ultra-short baseline transponder are used for hydroacoustic communication between the mother ship and the underwater vehicle, so as to determine the position of the underwater vehicle; The visual camera and the forward-looking sonar detect the underwater frogman and obtain visual images, acoustic images and positioning information respectively; The information processing unit is used to process the image information and position information collected by the visual camera and the forward-looking sonar.

Images