A method for rapid underwater sound and light matching based on forward-looking sonar and camera

Abstract

The present invention provides a method for underwater rapid acoustic-optical matching based on forward-looking sonar and camera, firstly, obtaining an acoustic image and an optical image at the same time during underwater target detection; selecting the target position on the acoustic image, obtaining distance, angle, and pixel coordinate information; establishing an epipolar constraint model between the acoustic image and the optical image based on the posture relationship between the forward-looking sonar and the camera, and realizing the correspondence between a pixel point on the acoustic image and a vertical line on the optical image; limiting the left and right boundaries of the target on the optical image, and limiting the upper and lower boundaries of the target on the optical image, further realizing the correspondence between the target frames on the acoustic image and the optical image, and completing the acoustic-optical matching. The present invention makes full use of the advantages of sound and light in underwater target detection, quickly constructs matching corresponding information, and realizes the spatial data interconnection and position information expression of real-time detection of sound and light in the underwater common domain in a lightweight computing manner.

Description

Technical Field

[0001] The invention belongs to the technical field of underwater acoustic-optical combined imaging, and in particular relates to a method for underwater rapid acoustic-optical matching based on a forward-looking sonar and a camera. Background Art

[0002] The particularity and complexity of the marine environment make it very difficult to effectively detect underwater targets. Sonar imaging and optical imaging are the two most commonly used technical means for underwater target detection. Sonar imaging determines the position, shape and motion state of the target by emitting sound waves and receiving the sound wave signals reflected by the target, analyzing the signal intensity, direction and time delay and other parameters. However, the wavelength of the sound wave is long and the image resolution is low, resulting in low reliability and accuracy of target recognition. Optical imaging technology can observe the surface features, color, texture and other detailed information of the target object, thereby achieving accurate identification and classification of the target object. However, light is seriously affected by the absorption and scattering of the water body during underwater transmission. Absorption causes the energy of light to decay rapidly with the increase of detection distance, which limits the detection distance. Backscattering will form a strong background, resulting in a decrease in contrast and a reduction in the image signal-to-noise ratio; forward scattering causes the points on the object surface to become spots on the image surface, making the image blurred and the resolution reduced. This feature is particularly prominent when detecting in muddy water areas.

[0003] At present, most of the acoustic-optical fusion technologies involve the fusion between the acoustic and optical image levels, and the two sensor images are spatially registered. The registration methods are mainly divided into regional registration and feature-level registration. Some acoustic-optical image matching technologies can be accurate to the pixel level, which usually requires that the acoustic image and the optical image show the same plane, and the image needs to be clear enough and have high resolution. However, in actual operation, it is difficult to obtain available acoustic-optical images at the same time, and it takes a long time to match, which is not suitable for observing moving targets. Feature-level registration requires finding the same feature information on the acoustic-optical images and fusing them. The underwater optical image itself has high enough detail information, and it is easier to extract features using conventional target detection algorithms. Acoustic images have more noise and poor image resolution. Generally, contour edges or regions are detected for feature extraction. By establishing an epipolar constraint model between the acoustic-optical imaging planes, data is fused at the feature level, external calibration is performed based on sonar and camera, and the volume model constructed by the two sensors is used for complementation, which can be further used for three-dimensional reconstruction. However, the extraction of some feature points still relies on manual matching. Summary of the invention

[0004] In response to the above problems, the present invention establishes an epipolar constraint model between the acoustic and optical sensors and uses the field of view of the forward-looking sonar to achieve rapid matching between images, thereby strengthening the demand for real-time underwater detection applications of acoustic and optical sensors.

[0005] The present invention proposes a method for rapid underwater sound and light matching based on forward-looking sonar and camera, comprising the following steps:

[0006] Step 1, obtaining an acoustic image and an optical image at the same time during underwater target detection;

[0007] Step 2: pre-process the acoustic image to obtain the distance and angle of the target object relative to the forward-looking sonar, select the target object on the image, and obtain the pixel coordinate information of the anchor frame;

[0008] Step 3: Based on the position relationship between the forward-looking sonar and the camera, and combined with the difference in the imaging planes of the two sensors, an epipolar constraint model between the acoustic image and the optical image is established to achieve the correspondence between a pixel point on the acoustic image and a vertical line on the optical image.

[0009] Step 4: Based on the pixel coordinates of the anchor box on the acoustic image and the established acoustic-optical polar line constraint model, the two corner points below the anchor box are mapped to two vertical lines on the optical image, thereby defining the left and right boundaries of the target object on the optical image;

[0010] Step 5, constrain the narrower forward-looking sonar field of view in the vertical direction to the optical image, limit the upper and lower boundaries of the target object on the optical image, further achieve the correspondence between the target frame and the frame on the acoustic image and the optical image, and complete the acoustic-optical matching.

[0011] Preferably, in step 2, the pixel information on the image is obtained using an opencv tool, and the origin position is transformed from the upper left corner to the forward-looking sonar probe position. The coordinates, distance resolution, and angle resolution of the forward-looking sonar probe are obtained through relevant functions in the forward-looking sonar SDK, and the actual distance and angle of the target position relative to the forward-looking sonar are calculated using the resolution information and pixel distance.

[0012] Preferably, the target object is framed out in step 2 to obtain pixel coordinate information of the anchor frame, which is achieved by training a YOLOv5 network model, deploying it in an acoustic image target detection system, automatically framed out the target object during the detection process, and obtaining pixel coordinate information of the anchor frame.

[0013] Preferably, the process of establishing the epipolar constraint model between the acoustic image and the optical image in step 3 is:

[0014] Detecting a 3D point in space The position of the rectangular coordinate system with the same origin And spherical coordinate system To indicate that the superscript represents transpose; in the rectangular coordinate system, Respectively represent "horizontal distance", "longitudinal distance" and "vertical distance"; in the spherical coordinate system and are the azimuth and elevation angles, Yes Distance to the origin; point The conversion relationship between the rectangular coordinate system and the spherical coordinate system is:

[0015]

[0016] The inverse of this transformation is as follows:

[0017]

[0018]

[0019] Forward-looking sonar detects 3D points at different locations To generate a two-dimensional beam image ; By converting the two measurable polar coordinates of the three-dimensional point, namely the distance and azimuth angle values ), which is converted to , generate a two-dimensional rectangular coordinate image , When used as a subscript, it means in the sonar coordinate system;

[0020] For a given 3D point A point on the acoustic image , find matching points in the optical image The trajectory of the forward-looking sonar and the camera is used to find the position correspondence of the point in the two sensor coordinate systems, that is, the rotation correspondence and displacement distance of the spatial position. The three-dimensional point in the two reference systems The optical position coordinates Harmony position coordinates The relationship between them is:

[0021]

[0022] in is a The rotation matrix of It is the displacement vector between two coordinate systems; it is determined by substituting the image measurement value of the selected known target into the following formula and Values:

[0023]

[0024]

[0025] Represents the rotation matrix OK, Represents the displacement vector component, and the acousto-optic constraint equation for the target position is established as follows:

[0026] )

[0027] The imaging planes of forward-looking sonar and camera are different. For the same target, the acoustic image is a top view, the optical image is a front view, and the elevation information Unknown in forward-looking sonar, using three-dimensional position relationship Substituting into the constraint equation, the equation is transformed into:

[0028]

[0029] The position relationship between the camera and the forward-looking sonar is defined as overlapping placement, that is,

[0030] Rotation Matrix

[0031] Displacement Vector

[0032] Using conversion relationships Get three-dimensional point of The corresponding relationship between the three components in the acoustic and optical coordinate systems is:

[0033]

[0034] The constraint equation is transformed into the following form:

[0035]

[0036] Thus, the epipolar constraint relationship model of the target position on the acoustic image and the optical image is obtained.

[0037] Preferably, in step 5, limiting the upper and lower boundaries of the target object on the optical image is specifically as follows:

[0038] In the vertical direction, the detection range of the forward-looking sonar is narrow, and the field of view of the camera is sufficient to cover the entire detection range of the forward-looking sonar. The overlapping detection area of ​​the forward-looking sonar and the camera in the vertical plane is the detection range of the forward-looking sonar. The position of this range mapped to the optical image is calculated based on the distance information of the target object obtained from the forward-looking sonar and the size of the field of view in the vertical direction, and this range is used to constrain the upper and lower boundaries of the target object in the optical image.

[0039] Preferably, after the sound-light matching is established, the LED light source is guided toward the target position for light compensation, which is used for the sound-light matching effect, specifically:

[0040] Establish a coordinate system with the sonar as the origin, find the position coordinates of the LED light source, and get the distance of the target object on the acoustic image and angle , converted to the rectangular coordinate system into the following form:

[0041]

[0042] Based on the position coordinates of the target , and the position coordinates of the LED light source Under the control of the PTZ, the LED light source can be guided horizontally toward the target object to deliver light field energy. .

[0043] Compared with the prior art, the present invention proposes a method for exploring the maximum detection distance under an underwater active light source illumination system, and produces the following beneficial effects:

[0044] The present invention is based on the forward-looking sonar and camera carried on the ROV at the same time, and uses the posture relationship and imaging characteristics between the two sensors to establish an epipolar constraint relationship between sound and light, and innovatively maps the vertical field of view range of the forward-looking sonar to the optical image, and finds a direct correspondence between the two different imaging planes of the acoustic image and the optical image. The present invention establishes an epipolar constraint model between the two sensors, the acoustic and optical sensors, and uses the field of view angle of the forward-looking sonar to achieve rapid matching between images, thereby strengthening the demand for sound and light in real-time underwater detection applications.

[0045] Applying the YOLOv5 network to acoustic image target detection, the ROV can quickly search for the target underwater and mark it in the image. Based on the pixel coordinate information of the marking frame and the established acoustic-optical constraint relationship, the location information of the target will be quickly matched on the optical image, even if it is beyond the visibility range. Light compensation is performed through the LED light source on the ROV, or the ROV is controlled to observe at a close distance until the shape, texture and other details of the target can be clearly seen on the optical image. Ensure that a relatively clear image of the target can be seen in both the acoustic image and the optical image.

[0046] Through the method provided by the present invention, the advantages of sound and light in underwater target detection are fully utilized, matching corresponding information is quickly constructed, and spatial data interconnection and position information expression of real-time detection of sound and light in the underwater common domain are realized in a lightweight computing manner, providing a fast, comprehensive and accurate model method for research and application in the field of underwater acoustic and optical imaging. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is the ROV used in the present invention for acoustic-optical matching.

[0048] Figure 2 The flowchart of the method for rapid underwater sound and light matching based on forward-looking sonar and camera.

[0049] Figure 3 It is the projection of a point in space on the zero elevation plane in the spherical coordinate system.

[0050] Figure 4 The invention discloses the difference in projection planes between the forward-looking sonar and the camera and the application of the field of view cone of the forward-looking sonar in the optical imaging plane.

[0051] Figure 5 Schematic diagram of the field of view relationship between the forward-looking sonar and the camera in the present invention.

[0052] Figure 6 It is a schematic diagram of constraining the upper and lower boundaries of the target object in the optical image by using the forward-looking sonar field of view angle in the vertical direction in the present invention.

[0053] Figure 7 This is the sound-light matching effect diagram in Example 1, and the experimental site is an indoor pool.

[0054] Figure 8 This is the sound-light matching effect diagram in Example 2, and the experimental site is a reservoir.

[0055] Fig. 9 This is the sound-light matching effect diagram in Example 3, and the experimental site is the dock.

[0056] Fig.10 This is a schematic diagram of the LED light source in Example 4 realizing precise lighting towards the detected target, where the target is located in the middle position.

[0057] Fig.11 This is a schematic diagram of the LED light source in Example 4 realizing precise lighting toward the detected target, where the target is located on the right.

[0058] Fig.12 This is a schematic diagram of the LED light source in Example 4 realizing precise lighting towards the detected target, where the target is located on the left. DETAILED DESCRIPTION

[0059] The invention will be further described below in conjunction with specific embodiments.

[0060] The present invention proposes a method for underwater rapid acoustic and optical matching based on a forward-looking sonar and a camera. The method takes underwater acoustic and optical imaging technology as the theoretical basis, establishes an epipolar constraint relationship based on the posture relationship between the forward-looking sonar and the camera and the different characteristics of the imaging plane, and uses the field of view of the forward-looking sonar to constrain the position of the target object in the optical image, thereby achieving a matching relationship between the acoustic and optical images of the same target object. By applying the YOLOv5 target detection network to the acoustic image, the target object can be quickly marked and the position information can be obtained, thereby automatically establishing a matching relationship in the optical image. Finally, the LED light source on the ROV can be guided toward the target position for illumination compensation to obtain a clear optical image. The ROV selected for the experiment is as follows: Figure 1 As shown, the overall process is as Figure 2 shown.

[0061] Embodiment 1:

[0062] 1. First, obtain the acoustic image and optical image at the same time during the underwater target detection process;

[0063] In this embodiment, an acoustic-optical image target detection interface is jointly developed based on Visual Studio and Qt, and the software development kit (SDK) and opencv tools of the forward-looking sonar are used to display underwater acoustic images and optical images on the interface.

[0064] Second, the acoustic image is preprocessed to obtain the distance and angle of the target position relative to the forward-looking sonar, and the target object is framed on the image to obtain the pixel coordinate information of the anchor frame.

[0065] Getting the anchor box:

[0066] This embodiment uses the YOLOv5 target detection network to select the target and obtain the pixel coordinate information of the anchor box. First, a large number of acoustic images are obtained, and the labelimg annotation tool is used to mark the targets on the images to make a forward-looking sonar dataset. Then, the dataset is used for YOLOv5 network model training. Finally, the trained YOLOv5 network is used to detect the targets on the acoustic image, quickly obtain the anchor box, and obtain the coordinate information of the anchor box.

[0067] Distance and angle acquisition:

[0068] In this embodiment, the pixel information on the acoustic image is obtained using the opencv tool, and the origin position of the image is transformed from the upper left corner to the position of the forward-looking sonar probe. The coordinates, distance resolution, and angle resolution of the probe are obtained through the relevant functions in the forward-looking sonar product SDK, and the actual distance and angle of the target position relative to the forward-looking sonar are calculated using the resolution information and pixel distance.

[0069] 3. Based on the position relationship between the forward-looking sonar and the camera, and combined with the difference in the imaging planes of the two sensors, an epipolar constraint model between the acoustic image and the optical image is established to achieve the correspondence between a pixel point on the acoustic image and a vertical line on the optical image.

[0070] The specific model building process is as follows:

[0071] S1 Conversion and correspondence between acoustic and optical coordinate systems

[0072] S1.1 Space coordinate system conversion

[0073] Detecting a 3D point in space The position of the rectangular coordinate system with the same origin And spherical coordinate system To indicate that the superscript represents transpose; in the rectangular coordinate system, Respectively represent "horizontal distance", "longitudinal distance" and "vertical distance"; in the spherical coordinate system and are the azimuth and elevation angles, Yes Distance to the origin; point The conversion relationship between the rectangular coordinate system and the spherical coordinate system is:

[0074]

[0075] The inverse of this transformation is as follows:

[0076]

[0077]

[0078] S1.2 Acoustic Image Coordinate Representation

[0079] Forward-looking sonar detects 3D points at different locations To generate a two-dimensional beam image ; By combining the two measurable polar coordinates of a 3D point (i.e., distance and azimuth values) ) is converted to , which can generate a two-dimensional image , defines the point on the zero elevation plane, The subscripts indicate that the coordinates are in the sonar coordinate system.

[0080] S2 Sound to Light Matching

[0081] For a given 3D point A point on the acoustic image , find matching points in the optical image The trajectory of the forward-looking sonar and the camera is used to find the position correspondence of the point in the two sensor coordinate systems, that is, the rotation correspondence and displacement distance of the spatial position. The three-dimensional point in the two reference systems The optical position coordinates Harmony position coordinates The relationship between them is:

[0082]

[0083] in is a The rotation matrix of It is the displacement vector between two coordinate systems; it is determined by substituting the image measurement value of the selected known target into the following formula and Values:

[0084]

[0085]

[0086] Represents the rotation matrix OK, Represents the displacement vector component, and the acousto-optic constraint equation for the target position is established as follows:

[0087] )

[0088] The imaging planes of forward-looking sonar and camera are different. For the same target, the acoustic image is a top view, the optical image is a front view, and the elevation information Unknown in forward-looking sonar, using three-dimensional position relationship Substituting into the constraint equation, the equation is transformed into:

[0089]

[0090] The position relationship between the camera and the forward-looking sonar is defined as overlapping placement, that is,

[0091] Rotation Matrix

[0092] Displacement Vector

[0093] Using conversion relationships Get three-dimensional point of The corresponding relationship between the three components in the acoustic and optical coordinate systems is:

[0094]

[0095] The constraint equation is transformed into the following form:

[0096]

[0097] Thus, the epipolar constraint relationship of the target position on the acoustic image and the optical image can be obtained.

[0098] Fourth, based on the horizontal coordinates of the left and right boundary pixels of the anchor box on the acoustic image and the established acoustic-optical polar line constraint model, the left and right boundaries of the target object on the optical image are limited;

[0099] The narrower forward-looking sonar field of view in the vertical direction is constrained to the optical image, and the upper and lower boundaries of the target object in the optical image are limited, further achieving the correspondence between the target object frames in the acoustic image and the optical image, completing the acoustic and optical matching.

[0100] The specific constraints are:

[0101] In the present invention, the fields of view of the forward-looking sonar and the camera are set to overlap, and the upper and lower boundaries of the target object are defined on the optical image. In the vertical direction, the field of view of the camera is sufficient to cover the entire range of the forward-looking sonar detection, and the overlapping detection area of ​​the forward-looking sonar and the camera on the vertical plane is the detection range of the forward-looking sonar.

[0102] Figure 5 The difference in the field of view between forward-looking sonar and camera is shown. The vertical detection angle of forward-looking sonar is small, while the vertical field of view of camera is enough to cover the entire vertical range of forward-looking sonar detection.

[0103] Based on the distance information of the target object obtained from the forward-looking sonar and the size of the field of view in the vertical direction, the position of the range mapped to the optical image is calculated, and the range is used to constrain the upper and lower boundaries of the target object in the optical image, such as Figure 6 As shown. The orange range represents the field of view of the camera, and the blue range represents the field of view of the forward-looking sonar. The distance of the target detected by the forward-looking sonar during optical imaging. is the vertical detection angle of the forward-looking sonar.

[0104] Application of acoustic-optical imaging plane difference and sonar field of view cone:

[0105] Combining the relationship between the acoustic-optical polar line constraint model and the field of view overlap, it is possible to find the corresponding object in the optical image based on the detection frame of the object identified in the acoustic image and form a field of view cone to mark the object, such as Figure 4 As shown, the mapping of objects on the acoustic image to the optical image is achieved.

[0106] The sound and light matching effect obtained from the indoor pool experiment is shown in the figure below. Figure 7As shown, the targets include an alloy rack, a small ROV, and a toy car. Since the pool is located indoors and is not easily affected by external impurities, the water is very clear and the visibility is high, which is very suitable for sound-light matching experiments.

[0107] 5. After the calibration frame of the same target object appears on both the acoustic image and the optical image, the LED light source can be guided toward the target object for illumination compensation based on the world coordinate system that previously regarded the sonar as the origin, so as to obtain a clearer image on the optical image and verify the acoustic-optical matching effect.

[0108] During the experiment of underwater target detection and acoustic-optical matching, the ROV is always in motion. As long as the target can be detected on the acoustic image, the method can always mark it on the optical image, even if it is beyond the visibility range. In addition, the running speed of this method mainly depends on the detection speed of the YOLOv5 network on the acoustic image, which can achieve real-time matching of the same target on the acoustic-optical image. The experimental results show that this method can construct matching information and mark the target detected by the acoustic-optical image in real time in turbid or dark environments, which facilitates the precise operation and further detection of the ROV.

[0109] Embodiment 2:

[0110] The difference between this embodiment and embodiment 1 is that the experimental location and experimental environment are different, and the underwater target selected is different, which is only a small ROV.

[0111] The experimental site is a reservoir with a visibility of 2.3 meters and a water depth of about 70 meters. Compared with an indoor pool, the open water area can reduce the impact of environmental noise on sonar observations, and the deeper water environment can create an experimental condition without natural light illumination. Figure 8 This is the acoustic-optical matching effect of the reservoir experiment. Compared with the indoor pool, there is no influence of the pool wall on the acoustic image, the noise is relatively reduced, and it is easier to detect the required target on the acoustic image. On the optical image, you can very intuitively see the details of the target ROV color, shell, components, etc. In terms of acoustic-optical matching, the marking boxes on the acoustic image and the corresponding optical image are also relatively accurate.

[0112] Embodiment 3:

[0113] The difference between this embodiment and embodiment 1 is that the experimental location and experimental environment are different, and the underwater target selected is different, which is only a small ROV.

[0114] The experimental site was chosen as the dock. The visibility of the nearby seawater is 1.5 meters, the water depth is 6 to 8 meters, the water quality is relatively poor and silt is easily stirred up at the bottom of the water. Fig. 9This is a set of acoustic-optical matching images obtained from the dock experiment. It can be seen that the acoustic image is not affected by the water body and can still successfully detect the target object. However, it can be clearly seen from the optical image that the image contrast is low and it is difficult to observe the details of the target object. However, based on this acoustic-optical matching method, even if the position of the target object on the optical image cannot be seen in turbid water, it can still be marked in the image relatively accurately.

[0115] Embodiment 4:

[0116] The difference between this embodiment and embodiment 1 is that the experimental emphasis is different and the part related to LED illumination compensation is supplemented.

[0117] The experimental site was an indoor pool. After the target was detected and marked on the acoustic image, it was quickly matched in the optical image. Although the target was found, the lack of necessary lighting made it difficult to effectively see the details of the target and the accuracy of the acoustic-optical matching could not be verified. At this point, in the sonar coordinate system, the LED light source can be guided to deflect toward the target according to the position of the target, and a certain amount of lighting compensation can be performed to see a clearer image of the target. The distance of the target can be obtained on the acoustic image. and angle , converted to the rectangular coordinate system into the following form:

[0118]

[0119] Know the location coordinates of the target , and the position coordinates of the LED light source Under the control of the PTZ, the LED light source can be guided horizontally toward the target object to deflect and deliver light field energy. . Figure 10~Figure 12 The comparison diagram shows the precise lighting effect of the alloy racks detected by the LED light source direction. The positions of the alloy racks are respectively in the middle, on the right and on the left.

[0120] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

[0121] Although the above describes the specific implementation methods of the present invention, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. A method for rapid underwater sound and light matching based on forward-looking sonar and camera, characterized in that: The following steps are involved: Step 1, obtaining an acoustic image and an optical image at the same time during underwater target detection; Step 2: pre-process the acoustic image to obtain the distance and angle of the target object relative to the forward-looking sonar, select the target object on the image, and obtain the pixel coordinate information of the anchor frame; The pixel information on the image is obtained using the opencv tool, and the origin position is transformed from the upper left corner to the forward-looking sonar probe position. The coordinates, distance resolution, and angle resolution of the forward-looking sonar probe are obtained through the relevant functions in the forward-looking sonar SDK. The actual distance and angle of the target position relative to the forward-looking sonar are calculated using the resolution information and pixel distance. The target object is framed out and the pixel coordinate information of the anchor frame is obtained by training a YOLOv5 network model and deploying it in an acoustic image target detection system. The target object is automatically framed out during the detection process to obtain the pixel coordinate information of the anchor frame. Step 3: Based on the position relationship between the forward-looking sonar and the camera, and combined with the difference in the imaging planes of the two sensors, an epipolar constraint model between the acoustic image and the optical image is established to achieve the correspondence between a pixel point on the acoustic image and a vertical line on the optical image. The process of establishing the epipolar constraint model between the acoustic image and the optical image is as follows: Detect a 3D point in space The position of the rectangular coordinate system with the same origin And spherical coordinate system To indicate that the superscript represents transpose; in the rectangular coordinate system, Respectively represent "horizontal distance", "longitudinal distance" and "vertical distance"; in the spherical coordinate system and are the azimuth and elevation angles, Yes Distance to the origin; point The conversion relationship between the rectangular coordinate system and the spherical coordinate system is: The inverse of this transformation is as follows: Forward-looking sonar detects 3D points at different locations To generate a two-dimensional beam image ; By converting the two measurable polar coordinates of the three-dimensional point, namely the distance and azimuth angle values ), which is converted to , generate a two-dimensional rectangular coordinate image , When used as a subscript, it means in the sonar coordinate system; For a given 3D point A point on the acoustic image , find matching points in the optical image The trajectory of the forward-looking sonar and the camera is used to find the position correspondence of the point in the two sensor coordinate systems, that is, the rotation correspondence and displacement distance of the spatial position. The three-dimensional point in the two reference systems The optical position coordinates Harmony position coordinates The relationship between them is: in is a The rotation matrix of It is the displacement vector between two coordinate systems; it is determined by substituting the image measurement value of the selected known target into the following formula and Values: Represents the rotation matrix OK, Represents the displacement vector component, and the acousto-optic constraint equation for the target position is established as follows: ) The imaging planes of forward-looking sonar and camera are different. For the same target, the acoustic image is a top view, the optical image is a front view, and the elevation information Unknown in forward-looking sonar, using three-dimensional position relationship Substituting into the constraint equation, the equation is transformed into: The position relationship between the camera and the forward-looking sonar is defined as overlapping placement, that is: Rotation Matrix Displacement Vector Using conversion relationships Get three-dimensional point of The corresponding relationship between the three components in the acoustic and optical coordinate systems is: The constraint equation is transformed into the following form: Thus, the epipolar constraint relationship model of the target position on the acoustic image and the optical image is obtained; Step 4: Based on the pixel coordinates of the anchor box on the acoustic image and the established acoustic-optical polar line constraint model, the two corner points below the anchor box are mapped to two vertical lines on the optical image, thereby defining the left and right boundaries of the target object on the optical image; Step 5, constrain the narrower forward-looking sonar field of view in the vertical direction to the optical image, limit the upper and lower boundaries of the target object on the optical image, further achieve the correspondence between the target frame and the frame on the acoustic image and the optical image, and complete the acoustic-optical matching.

2. The method for rapid underwater sound and light matching based on forward-looking sonar and camera as claimed in claim 1, characterized in that: In step 5, the upper and lower boundaries of the target object on the optical image are specifically limited as follows: In the vertical direction, the detection range of the forward-looking sonar is narrow, and the field of view of the camera is sufficient to cover the entire detection range of the forward-looking sonar. The overlapping detection area of ​​the forward-looking sonar and the camera in the vertical plane is the detection range of the forward-looking sonar. The position of this range mapped to the optical image is calculated based on the distance information of the target object obtained from the forward-looking sonar and the size of the field of view in the vertical direction, and this range is used to constrain the upper and lower boundaries of the target object in the optical image.

3. The method for rapid underwater sound and light matching based on forward-looking sonar and camera as claimed in claim 1, characterized in that: After the sound and light matching is established, the LED light source is guided toward the target position for light compensation, which is used for the sound and light matching effect. Specifically: Establish a coordinate system with the sonar as the origin, find the position coordinates of the LED light source, and get the distance of the target object on the acoustic image and angle , converted to the rectangular coordinate system into the following form: Based on the position coordinates of the target , and the position coordinates of the LED light source Under the control of the PTZ, the LED light source can be guided horizontally toward the target object to deliver light field energy. .

Images