Method for determining relative optical camera extrinsic parameters of an imaging sonar, recording medium and system

Abstract

This invention belongs to the field of multi-sensor fusion technology and discloses a method for obtaining the extrinsic parameters of an imaging sonar relative to an optical camera. The method includes: fixing and maintaining the relative pose of the imaging sonar and the optical camera; establishing a marker array and coordinate system within the field of view, and simultaneously acquiring acoustic and optical images of each array element; calculating the coordinates of the marker array in the sonar and camera coordinate systems respectively; and obtaining the rotation matrix and translation vector between the two coordinate systems through the coordinate values, which serve as the extrinsic parameters of the imaging sonar relative to the optical camera. This method physically ensures the spatial alignment of the two types of markers, reducing error accumulation; by standardizing the marker array to constrain the marker coordinates, it effectively compresses the solution space dimension, improving the accuracy, stability, and robustness of obtaining the extrinsic parameters, and is suitable for multimodal underwater sensing systems. This invention also provides a non-transient readable recording medium storing the program of this method and a system containing the medium, which can call the program through processing circuitry to execute the above method.

Description

Technical Field

[0001] This invention belongs to the field of multi-sensor fusion technology and relates to the registration of two types of sensors that use sound waves and light waves for underwater imaging. It discloses a method, recording medium and system for obtaining the extrinsic parameters of imaging sonar relative to an optical camera. Background Technology

[0002] Underwater environmental sensing technology is widely used in fields such as autonomous underwater robots, marine resource exploration, and environmental monitoring. To enhance the environmental adaptability and sensing capabilities of sensing systems, sonar and optical sensors are often used in combination due to their complementary sensing characteristics. Among them, two-dimensional multibeam sonar and optical cameras are the most commonly used sensor combinations. To achieve collaborative operation between the two, it is necessary to accurately obtain their spatial pose relationship, that is, to complete the extrinsic parameter estimation of the acousto-optic heterogeneous sensors. This is a key prerequisite technology for achieving multi-source information fusion and accurate sensing.

[0003] In the estimation of extrinsic parameters for underwater two-dimensional multibeam sonar and optical cameras, due to the significant differences in the imaging mechanisms of sonar and optical cameras, and the lack of complete three-dimensional information in the observations of both types of sensors, direct registration between the two different types of two-dimensional data often leads to multiple solutions and high error risk, making it difficult to guarantee the reliability of the extrinsic parameter estimation results.

[0004] To address this problem, existing technologies often employ a two-stage decoupling strategy: first, the extrinsic parameters of the acoustic and optical sensors relative to the world coordinate system are solved separately; then, their relative spatial poses are derived through matrix operations. This strategy involves multiple computational conversion processes, which easily introduce matrix operation errors and accumulated errors, limiting the accuracy and stability of the final extrinsic parameter estimation. Furthermore, the inconsistent acousto-optic feature positions on the calibration plates used in existing methods are also one of the reasons for the widespread application of this strategy.

[0005] Furthermore, the insufficient accuracy of extrinsic parameter estimation for two-dimensional acoustic sensors remains a major bottleneck limiting overall performance. Existing studies largely rely on general extrinsic parameter estimation algorithms, failing to fully utilize prior information about the geometric structure of calibration points in conjunction with sonar observation characteristics. This leads to significant errors under conditions of noise interference and incomplete marker point observations, making it difficult to meet the requirements for high accuracy and robustness.

[0006] A search revealed that existing patent literature proposes a calibration method for underwater acoustic-optical information integration. This method uses OpenCV functions to solve for the rotation matrix and translation vector between the sonar and the camera. However, for two-dimensional multibeam sonar observations, this method uses two-dimensional polar coordinate observations combined with manual measurement of the pitch height component using a measuring tape to obtain three-dimensional coordinates. This method is complex to operate and has limited accuracy, directly affecting the accuracy and stability of the final extrinsic parameter estimation. Summary of the Invention

[0007] To address the above problems, this invention provides a method for obtaining the extrinsic parameters of an imaging sonar relative to an optical camera, comprising the following steps: T1. Fix and maintain the relative pose of the imaging sonar and the optical camera unchanged; T2. Within the field of view of the imaging sonar and the optical camera, establish a marker array, set a world coordinate system that is stationary relative to the marker array, and mark the coordinates of each array element; set a camera coordinate system that is stationary relative to the optical camera and a sonar coordinate system that is stationary relative to the imaging sonar. T3. Imaging sonar and optical camera simultaneously acquire acoustic and optical images of the marked dot matrix; T4. Calculate the coordinates of the marker array in the camera coordinate system based on the optical images of each array element acquired synchronously; calculate the coordinates of the marker array in the sonar coordinate system based on the acoustic images of each array element acquired synchronously. T5. Calculate the rotation matrix and translation vector between the two coordinate systems based on the coordinate values ​​of the marker array in the camera coordinate system and the sonar coordinate system. Optimize the translation component by minimizing the two-dimensional reprojection error to obtain the extrinsic parameters of the imaging sonar relative to the optical camera.

[0008] Preferably, each element of the marker dot matrix is ​​disk-shaped, with an acoustically sensitive reflection point at the center of the disk, and a photosensitive reflection ring with the acoustically sensitive emission point as the center is drawn on the disk, with each acoustic and optical marker point being the center of the disk.

[0009] Preferably, step T3 further includes distortion correction, noise suppression, and contrast enhancement processing of the acquired acoustic and optical images. In the optical image, a marker detection algorithm based on circular coding is used to extract the positions and coding information of all marker points. In the acoustic image, a local adaptive thresholding method is used for binarization processing to extract the positions of all marker points, and matching and filtering are performed in combination with a predefined marker point array layout to remove noise points.

[0010] Preferably, in step T4, the coordinates of the marker array in the camera coordinate system based on the synchronously acquired optical image of each array element are calculated using any one of the following three methods: The perspective N-point method based on a monocular camera: By utilizing the known correspondence between the three-dimensional world coordinates of each marker point and its two-dimensional coordinates on the image plane, the rotation matrix and translation vector of the camera coordinate system relative to the world coordinate system are solved, thereby obtaining the coordinates of the marker point array in the camera coordinate system. Alternatively, a stereo vision-based 3D reconstruction method can be used to calculate the coordinates of the marker array in the camera coordinate system using synchronized images from two or more cameras and feature point matching and disparity calculation. Alternatively, based on monocular depth estimation or structured light methods, the coordinates of the marker dot matrix in the camera coordinate system can be calculated by combining monocular images with depth network or projection coding information.

[0011] Preferably, step T4, which calculates the coordinates of the marker array in the sonar coordinate system based on the synchronously acquired acoustic images of each array element, includes the following steps: A spatial position model of the marker array is established, and the coordinates of the marker array in the sonar coordinate system are calculated based on the basis vectors of the imaging sonar.

[0012] Furthermore, a spatial position model of the marker array is established, and the coordinates of the marker array in the sonar coordinate system are calculated based on the basis vectors of the imaging sonar using the spatial position model. This includes the following steps: All marker points are located in the same plane and arranged in a grid to form a marker point matrix. Let P be the three-dimensional coordinate of the i-th marker point in the sonar coordinate system. s,i Its grid row and column indices are (m i , n i The three-dimensional coordinates of the j-th marker point are P. s,j Its grid row and column indices are (m j , n j Introducing two spatial basis vectors: row basis vector g and column basis vector f, and a uniform grid spacing d; the theoretical geometric relationship between any two marker points is modeled as follows: P s,i - P s,j = d (m i - m j ) g + d (n i - n j f The spatial location model of the marker matrix introduces two unit orthogonal vectors, row basis vector g and column basis vector f, based on the known layout of the marker points. By combining the row and column basis vectors with the array step size, the three-dimensional coordinates of the marker matrix without elevation angle observations are constrained to a regular grid array.

[0013] Another aspect of the present invention is to provide a non-transient readable recording medium for storing one or more programs containing multiple instructions, which, when executed, cause a processing circuit to perform the above-described method for obtaining the extrinsic parameters of an imaging sonar relative to an optical camera.

[0014] Another aspect of the present invention provides a system for obtaining the extrinsic parameters of an imaging sonar relative to an optical camera, comprising a processing circuit and a memory electrically coupled thereto, the memory being configured to store at least one program, the program containing a plurality of instructions, the processing circuit running the program being able to execute the aforementioned method for obtaining the extrinsic parameters of an imaging sonar relative to an optical camera.

[0015] Compared with the prior art, the method, recording medium and system for obtaining the extrinsic parameters of imaging sonar relative to optical camera provided by the present invention have the following beneficial effects: (1) The two types of marker points in the present invention can be completely aligned in space. Therefore, the extrinsic parameters between the two sensors can be calculated directly using optical three-dimensional coordinates and acoustic two-dimensional observations. There is no need to solve the extrinsic parameters of each sensor relative to the world coordinates separately, or to perform additional three-dimensional coordinate position conversion. This simplifies the process and avoids the error accumulation caused by the traditional two-stage decoupling strategy, thereby improving the calibration accuracy and stability.

[0016] (2) This invention utilizes the prior geometric structure information of two types of markers to establish a gridded model of the calibration point matrix, and introduces two unit orthogonal basis vectors to estimate the extrinsic parameters based on the gridded model constraints, thereby reducing the dimension of the solution space and significantly improving the accuracy and stability of the extrinsic parameter estimation, which can meet the application requirements of high-precision, multimodal underwater sensing systems. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating the method for calculating extrinsic parameters between the imaging sonar and the optical camera in an embodiment of the present invention. Figure 2 This is a schematic diagram illustrating the coordinate system definition corresponding to the acousto-optic imaging model in this embodiment of the invention; Figure 3 This is a schematic diagram of a calibration board used for estimating the external parameters of an underwater acoustic-optic sensor in an embodiment of the present invention; Figure 4 This is a schematic diagram illustrating the specific calculation steps of the extrinsic parameters based on the gridded model in an embodiment of the present invention; Figure 5 This is a schematic diagram of the fixing of the acoustic and optical sensors in an embodiment of the present invention; Figure 6 This is a schematic diagram of an experimental scenario according to an embodiment of the present invention; Figure 7 This is a two-dimensional multibeam sonar image from an embodiment of the present invention; Figure 8 The optical image in this embodiment of the invention is displayed as a black and white image; Figure 9 This is a black and white image showing a comparison of the visual error between the acoustic observation of the marker point projected onto the optical image using the method of the present invention and the optical observation of the marker point. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be described below with reference to the accompanying drawings. The described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without innovative effort are within the scope of protection of the present invention.

[0019] This invention provides a method for obtaining the extrinsic parameters of an imaging sonar relative to an optical camera, comprising the following steps: Figure 1 As shown, the method includes the following steps: S1. Establish the imaging model and coordinate system of the acoustic and optical sensors; in this example, the acoustic sensor is the imaging sonar, and the optical sensor is the optical camera.

[0020] S2. In water, the acoustic and optical marker co-center calibration plate is placed within the field of view of the two-dimensional multibeam sonar (a specific type of imaging sonar) and the optical camera to simultaneously acquire acoustic and optical image pairs; S3. Preprocess the acquired images, detect the markers in the sound and light image pairs, and obtain their coordinates; S4. Calculate the three-dimensional coordinates of the marker point in the camera coordinate system; S5. Calculate the rotation matrix and translation vector between the sonar coordinate system and the camera coordinate system using a sonar feature point model based on a grid layout, i.e., the extrinsic parameter estimation results of the acousto-optic sensor.

[0021] Furthermore, the coordinate system corresponding to the imaging model established in step S1 is defined as follows: Figure 2 As shown.

[0022] The coordinate system includes the sonar coordinate system o. s - X s Y s Z s Camera coordinate system o c - X c Y c Z c Pixel coordinate system o-x c y c And the world coordinate system o with the upper left corner of the outer contour of the calibration plate as the origin. w - X w Y w Z w All signs are located in Z. w = 0 plane. All coordinate systems are right-handed.

[0023] Furthermore, the calibration board used in step S2 has markings for both acoustic and optical modes.

[0024] The method proposed in this invention is a supervised calibration using a calibration board, which estimates the extrinsic parameters between sensors by acquiring joint acoustic-optical observation results. The calibration board is as follows: Figure 3 As shown.

[0025] The calibration plate base is preferably made of carbon fiber composite material, which possesses high strength, dimensional stability, and water resistance, maintaining accuracy and reliability during long-term underwater use. Simultaneously, its low acoustic reflectivity and some absorption of visible light enhance acoustic transmission and optical image contrast. The optical markers are preferably white concentric ring codes printed on the base plate surface, offering high contrast and rotational invariance for easy identification and positioning. The acoustic markers are preferably stainless steel or copper cylinders (approximately 5mm in diameter and 8mm in height, with a hemispherical top), ensuring echo stability at different angles. The optical and acoustic markers are installed concentrically, preferably with a through-hole (approximately 4mm) in the center of the optical marker. The acoustic markers are fixed from the back with screws for easy maintenance. All markers are arranged in a regular array, preferably a 4×5 rectangular structure with a spacing of 60–100mm, to avoid feature overlap, reduce size, and improve portability.

[0026] Further, step S3 includes the following steps: First, the acquired acoustic and optical images are preprocessed. The preprocessing includes: distortion correction of the optical images to restore the true geometric structure and spatial proportions in the images; noise suppression to improve image quality; and contrast enhancement to highlight the landmarks, thereby providing clear and reliable input data for subsequent landmark localization and extrinsic parameter estimation.

[0027] Then, in the optical image, a marker detection algorithm based on circular coding is used to extract the optical pixel coordinates and coding information of the marker; in the acoustic image, a local adaptive thresholding method is used for binarization to extract the bright marker points, and a predefined row and column layout template is used for matching and filtering to remove noise and achieve marker point localization.

[0028] Further, step S4 includes the following steps: In this embodiment, a monocular camera is used, and the EPnP method is employed to measure the two-dimensional pixel coordinates p of the marker points observed optically. c And the corresponding three-dimensional world coordinates P of the marker point. w As input, combined with the camera intrinsic parameter matrix, the rotation matrix R between the camera coordinate system and the world coordinate system is calculated. wc With translation vector t wc Then, the estimated three-dimensional coordinates of the marker point in the camera coordinate system are calculated:

[0029] Furthermore, the process of step S5 is as follows: Figure 4 As shown, the process involves taking the three-dimensional coordinates of the marker point in the camera coordinate system and its two-dimensional observation in the sonar coordinate system as input, and includes the following steps: S51. Establish a gridded spatial location model of marker points.

[0030] S52. Estimate the basis vectors and elevation angles of the marker points based on the gridded model, and estimate the three-dimensional coordinates of the marker points in the sonar coordinate system.

[0031] S53. Using the two sets of three-dimensional coordinates of the common-center acoustic-optical marker in the camera and sonar coordinate systems, calculate the rotation matrix and translation vector between the two-dimensional multibeam sonar and the optical camera.

[0032] S54. Minimize the reprojection error in the sonar's two-dimensional imaging plane, and further optimize Y. s Directional translation amount.

[0033] Furthermore, the spatial location model of the gridded marker points established in step S51 is specifically represented as follows: Based on the prior knowledge of the spatial position of the coplanar regular array of marker points, the three-dimensional coordinates of the marker points are established as a meshed model. Let P be the three-dimensional coordinates of the i-th marker point in the sonar coordinate system. s,i Its grid row and column indices are (m i , n i The three-dimensional coordinates of the j-th marker point are P. s,j Its grid row and column indices are (m j , n j Introducing two spatial basis vectors: row basis vector g and column basis vector f, and a uniform grid spacing d; the theoretical geometric relationship between any two marker points is modeled as follows: P s,i - P s,j = d (m i - m j ) g + d (n i - n j f The gridded marker spatial location model utilizes a known layout of calibration points and introduces two unit orthogonal vectors, row basis vector g and column basis vector f. By combining the row and column basis vectors with the array step size, the three-dimensional position of the marker point set lacking elevation angle observation is constrained on a regular coplanar grid array.

[0034] Furthermore, step S52, which estimates the basis vectors and the elevation angle of the marker points based on the gridded model, is as follows: Considering the measurement errors of acoustic sensors and the positioning errors of marker points, the marker points often deviate from their true positions. This deviation is defined as the structural residual:

[0035] in, , These are the approximate three-dimensional coordinates of the i-th and j-th markers, calculated using sonar 2D observations and elevation angle estimation, respectively. The sonar 2D image observations are in the following form:

[0036] Where, r i For distance, φ i This is the azimuth angle. Approximate three-dimensional coordinates. The calculation method is as follows:

[0037] Where, θ i This is the estimated pitch angle.

[0038] A weight term based on grid distance is introduced to emphasize the impact of residuals on structural consistency for distant point pairs. The formula for calculating the weight term is as follows:

[0039] Furthermore, the weighted structural loss function is constructed as follows:

[0040] Where, {θ i} represents the set of pitch angle estimates for all marker points.

[0041] The unknowns in the loss function described above must satisfy certain constraints. Unit norm and orthogonality constraints are applied to the row basis vector g and the column basis vector f:

[0042] The constraint condition for the pitch angle of the marker point is: θ i Î [θ min , θ max ] Where, θ min With θ max These represent the maximum and minimum pitch angles, respectively, which are determined by the sensor parameters.

[0043] Specifically, the aforementioned constrained nonlinear minimization problem is solved using a sequential quadratic programming algorithm. The initial pitch angle is randomly initialized within a feasible range. Then, the 3D coordinates of the marker points at that pitch angle are calculated. Any two feature points located in the same row or column are selected and substituted into the gridded model to calculate the initial values ​​of the basis vectors. To improve robustness, if marker point observations are incomplete and no suitable pair of points in the same row or column is found, all initial coordinate estimates are used to perform a least-squares fit on the basis vectors, thus providing reliable initial values.

[0044] The aforementioned minimization problem aims to estimate the basis vectors g and f, as well as the elevation angle of the marker point, under the constraints of prior geometry and the effective field of view of the sensor. By introducing two basis vectors to construct a gridded model, the spatial dimension of the solution is effectively compressed, and the influence of ambiguity and noise is reduced, thereby achieving accurate reconstruction of the three-dimensional spatial position of the marker point in the sonar coordinate system.

[0045] Since sonar observation data cannot directly distinguish the elevation angle of feature points, it may lead to a situation where the elevation angle relative to the Y-axis is not directly identifiable. s = 0 plane has two symmetric positional solutions about this plane. Therefore, after obtaining the row basis vector g and column basis vector f, their cross product n = g × f is calculated according to the right-hand rule to obtain the normal vector, which is used to determine the orientation of the calibration plate. According to the definition of the meshed model, when the normal vector n is away from the sensor origin, it is determined to be the correct orientation; if the normal vector points to the sensor origin, it indicates that the calibration plate is in a reverse orientation, and the estimation result needs to be corrected by mirroring.

[0046] After obtaining the estimated elevation angle of the marker, calculate the estimated three-dimensional coordinates of the marker in the sonar coordinate system. ;

[0047] Further, step S53 is as follows: The estimated three-dimensional coordinates of the marker point in the camera coordinate system calculated in step S4. And the estimated three-dimensional coordinates of the marker point in the sonar coordinate system calculated in step S52. As input, the two sets of points are translated to their respective centroids to obtain decentralized coordinates. and, .

[0048] Based on two sets of decentralized 3D coordinate estimates and Construct the covariance matrix H:

[0049] Singular value decomposition of the covariance matrix H yields: H = UΣV T Where, U∈R3×3 It is an orthogonal matrix whose column vectors are left singular vectors; V∈R 3×3 It is an orthogonal matrix whose column vectors are right singular vectors; Σ∈R 3×3 It is a diagonal matrix whose diagonal elements are the singular values ​​of matrix H.

[0050] Furthermore, based on the decomposition results of the covariance matrix, the rotation matrix of the sonar coordinate system relative to the camera coordinate system is: R cs = VU T Translation vector of the sonar coordinate system relative to the camera coordinate system:

[0051] Furthermore, step S54 further optimizes Y. s The steps for directional translation are as follows: Due to the lack of sonar observation data, Y s The direction information is important, but the estimation accuracy of the translation component in this direction is lower than that in other directions. To improve accuracy, while keeping other extrinsic parameters constant, the Y-axis is optimized by minimizing the two-dimensional reprojection error function. s Directional translation t y The two-dimensional reprojection error is defined as:

[0052] Where pro(...) is the sonar projection function. The calculation is based on world coordinates, rotation matrix, and translation vector. This optimization is also subject to nonlinear constraints on pitch angle.

[0053] Finally, the optimized translation amount t y Substitute t cs At this time R cs and t cs This is the result of the external parameter estimation of the acoustic-optic sensor. For the true coordinates P of the marker point in the sonar coordinate system... s The true coordinates P of the marker point in the camera coordinate system c The estimation results approximately satisfy: P s ≈ R cs P c + t cs In this embodiment, the sensor used is as follows: Figure 5 As shown in the diagram. Data acquisition was conducted in a 2m × 0.6m × 0.75m glass water tank. Sound-absorbing rubber was laid on three sides and the bottom of the tank to reduce reflection from the tank walls. The experimental setup is illustrated in the diagram. Figure 6As shown, the distance between the sensor and the calibration board was approximately 1 meter during data acquisition. The image is only used to demonstrate the experimental environment for data acquisition and does not represent the actual positions of the sensor and calibration board during data acquisition.

[0054] Acquire sonar and optical image pairs, acoustic images such as Figure 7 As shown, the optical image is as follows Figure 8 As shown. The sensor extrinsic parameters between the two-dimensional multibeam sonar and the optical camera were estimated using the method described in this invention, and the results are as follows:

[0055] The calculation results closely matched the theoretical installation parameters in the CAD design drawings of the sensor bracket. The sonar device was located about 15 centimeters directly below the camera, which verified the effectiveness of the algorithm.

[0056] The sonar feature point cloud is mapped to the camera image plane through coordinate transformation, such as... Figure 9 As shown, the reprojection error between the original data and the actual optical observation point is calculated. The results show that the root mean square error (RMSE) of the reprojection is 2.2978 pixels. This error level indicates that the proposed algorithm exhibits excellent geometric consistency in the joint calibration process, effectively fusing sonar and optical data to ensure the spatial registration accuracy of the multi-sensor system.

[0057] Assembling the above methods and steps into a program and storing it on a hard disk or other non-transitory storage medium constitutes an embodiment of the present invention, "a non-transitory readable recording medium"; while connecting the storage medium electrically to a computer processor and processing data to obtain the extrinsic parameters of the imaging sonar relative to the optical camera constitutes an embodiment of the present invention, "a system for obtaining the extrinsic parameters of the imaging sonar relative to the optical camera".

[0058] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computers or available storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0059] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0060] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0061] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0062] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for determining the extrinsic parameters of an imaging sonar relative to an optical camera, characterized in that, Includes the following steps: T1. Fix and maintain the relative pose of the imaging sonar and the optical camera unchanged; T2. Within the field of view of the imaging sonar and the optical camera, establish a marker array, set a world coordinate system that is stationary relative to the marker array, and mark the coordinates of each array element; set a camera coordinate system that is stationary relative to the optical camera and a sonar coordinate system that is stationary relative to the imaging sonar. T3. Imaging sonar and optical camera simultaneously acquire acoustic and optical images of the marked dot matrix; T4. Calculate the coordinates of the marker array in the camera coordinate system based on the synchronously acquired optical images of each array element; The coordinates of the marker array in the sonar coordinate system are calculated based on the acoustic images of each array element acquired synchronously. T5. Calculate the rotation matrix and translation vector between the two coordinate systems based on the coordinate values ​​of the marker array in the camera coordinate system and the sonar coordinate system. Optimize the translation component by minimizing the two-dimensional reprojection error to obtain the extrinsic parameters of the imaging sonar relative to the optical camera.

2. The method for obtaining the extrinsic parameters of an imaging sonar relative to an optical camera according to claim 1, characterized in that, Each element of the marker dot matrix is ​​disk-shaped, with an acoustically sensitive reflection point at the center of the disk. A photosensitive reflection ring centered on the acoustically sensitive emission point is drawn on the disk, and each acoustic and optical marker point is at the center of the disk.

3. The method for obtaining the extrinsic parameters of an imaging sonar relative to an optical camera according to claim 2, characterized in that, Step T3 also includes distortion correction, noise suppression, and contrast enhancement processing of the acquired acoustic and optical images. In the optical image, a marker detection algorithm based on circular coding is used to extract the positions and coding information of all marker points. In the acoustic image, a local adaptive thresholding method is used for binarization processing to extract the positions of all marker points, and matching and filtering are performed in combination with a predefined marker point array layout to remove noise points.

4. The method for obtaining the extrinsic parameters of an imaging sonar relative to an optical camera according to claim 3, characterized in that, In step T4, the coordinates of the marker array in the camera coordinate system are calculated based on the synchronously acquired optical image of each array element using any of the following three methods: The perspective N-point method based on a monocular camera: By utilizing the known correspondence between the three-dimensional world coordinates of each marker point and its two-dimensional coordinates on the image plane, the rotation matrix and translation vector of the camera coordinate system relative to the world coordinate system are solved, thereby obtaining the coordinates of the marker point array in the camera coordinate system. Alternatively, a stereo vision-based 3D reconstruction method can be used to calculate the coordinates of the marker array in the camera coordinate system using synchronized images from two or more cameras and feature point matching and disparity calculation. Alternatively, based on monocular depth estimation or structured light methods, the coordinates of the marker dot matrix in the camera coordinate system can be calculated by combining monocular images with depth network or projection coding information.

5. The method for obtaining the extrinsic parameters of an imaging sonar relative to an optical camera according to claim 4, characterized in that, Step T4 calculates the coordinates of the marker array in the sonar coordinate system based on the synchronously acquired acoustic images of each array element, including the following steps: A spatial position model of the marker array is established, and the coordinates of the marker array in the sonar coordinate system are calculated based on the basis vectors of the imaging sonar.

6. The method for obtaining the extrinsic parameters of an imaging sonar relative to an optical camera according to claim 5, characterized in that, Establishing a spatial position model of the marker array, and calculating the coordinates of the marker array in the sonar coordinate system based on the spatial position model and the basis vectors of the imaging sonar, includes the following steps: All marker points are located in the same plane and arranged in a grid to form a marker point matrix. Let P be the three-dimensional coordinate of the i-th marker point in the sonar coordinate system. s,i Its grid row and column indices are (m i , n i The three-dimensional coordinates of the j-th marker point are P. s,j Its grid row and column indices are (m j , n j Introducing two spatial basis vectors: row basis vector g and column basis vector f, and a uniform grid spacing d; the theoretical geometric relationship between any two marker points is modeled as follows: P s,i - P s,j = d (m i - m j ) g + d (n i - n j ) f The spatial location model of the marker matrix introduces two unit orthogonal vectors, row basis vector g and column basis vector f, based on the known layout of the marker points. By combining the row and column basis vectors with the array step size, the three-dimensional coordinates of the marker matrix without elevation angle observations are constrained to a regular grid array.

7. A non-transitory readable recording medium for storing one or more programs containing multiple instructions, characterized in that, When the instruction is executed, the processing circuit will perform a method for obtaining the extrinsic parameters of an imaging sonar relative to an optical camera, as described in any one of claims 1-6.

8. A system for obtaining extrinsic parameters of an imaging sonar relative to an optical camera, comprising a processing circuit and a memory electrically coupled thereto, characterized in that, The memory is configured to store at least one program, the program containing multiple instructions, and the processing circuit runs the program to perform a method for obtaining the extrinsic parameters of an imaging sonar relative to an optical camera, as described in any one of claims 1-6.

Images