Calibration methods, systems, and calibration boards for underwater acoustic-optical spatial registration

Abstract

This application relates to the field of image processing and information fusion, and discloses a calibration method, system, and calibration plate for underwater acoustic-optic image spatial registration. The method includes: S1, acquiring sonar images and checkerboard images (details omitted); S2, calculating the transformation matrix between the sonar coordinate system and the centroid coordinate system of the calibration plate; S3, calculating the transformation matrix between the calibration plate coordinate system and the underwater optical camera coordinate system; S4, calculating the transformation matrix between the sonar coordinate system and the underwater optical camera coordinate system, completing the calibration. This application utilizes the stable bright spot feature generated by metallic protrusions in the sonar image to actively and accurately establish the coordinates of the sonar center and the camera optical center in the calibration plate coordinate system; it guides the adjustment process through the morphological features of the sonar image, rather than relying on preset external geometric conditions, avoiding the problems of requiring high-precision calibration objects and cumbersome operations in calibration.

Description

Technical Field

[0001] This application belongs to the field of image processing and information fusion, and specifically relates to a calibration method, system and calibration board for underwater acoustic-optical information spatial registration. Background Technology

[0002] Underwater robots rely mainly on optical cameras and sonar for environmental perception. Fusion of data from both can significantly improve perception capabilities. Acousto-optical fusion calibration, which unifies the coordinate systems of the camera and sonar, is a prerequisite for achieving fusion, and its accuracy directly determines the reliability of the entire system.

[0003] Currently, calibration methods are mainly divided into two categories: specific layout installation method: calibration is performed by precisely fixing the relative position of the sensor. This method requires extremely high installation accuracy and is difficult to meet high-precision requirements; feature point extraction method: coordinate transformation is calculated by manually matching feature points in acoustic and optical images. However, this method is limited by the characteristics of low signal-to-noise ratio and blurred edges of sonar images, making feature matching difficult and thus limiting calibration accuracy.

[0004] In feature point extraction methods, the individual calibration of the acoustic system is the primary step and the current bottleneck. Conventional acoustic calibration requires the calibration plate to be precisely placed in the sonar scanning plane and perpendicular to the beam axis. This process has fundamental limitations: it is highly dependent on human experience, lacks quantitative feedback, is inefficient, and is prone to introducing errors; underwater environmental disturbances make it difficult to maintain an ideal alignment.

[0005] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this application, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0006] To address or at least alleviate one or more of the above problems, a calibration method, system, and calibration board for underwater acoustic-optical information spatial registration are provided. A checkerboard calibration board with metal protrusions marked at specific locations serves as a medium to establish a connection between the sonar and the underwater optical camera. Robust vertical movement replaces complex and unstable three-dimensional spatial alignment. By driving the calibration board to move vertically and utilizing the stable bright spot features generated by the metal protrusions in the sonar image, the coordinates of the sonar center and the underwater optical camera's optical center in the calibration board's coordinate system are actively and accurately established, thereby achieving the goal of spatial registration of the underwater optical camera's optical center and the sonar center in the underwater acoustic-optical image.

[0007] To achieve the above objectives, according to a first aspect of this application, a calibration method for underwater acoustic-optical image spatial registration is provided, comprising: S1. Acquire sonar and checkerboard images: A checkerboard calibration plate with multiple metal protrusions along one diagonal surface is placed within the sonar detection range, and the surface of the calibration plate is made perpendicular to the sonar imaging plane. The calibration plate is then translated along the direction perpendicular to the sonar imaging plane, and a series of sonar images and checkerboard images of the calibration plate taken by an underwater optical camera are acquired simultaneously. S2. Calculate the transformation matrix between the sonar coordinate system and the calibration plate centroid coordinate system: By collecting a series of sonar images, the positional features of bright spots formed by metallic protrusions in the sonar images are extracted frame by frame. When multiple bright spots appear symmetrical about the center line of the sonar image and have the largest spacing in the horizontal direction, it is determined that the sonar scanning plane coincides with the center plane of the calibration plate. Based on the sonar image data at the time of coincidence and the known dimensions of the calibration plate, the transformation matrix between the sonar coordinate system and the centroid coordinate system of the calibration plate is calculated. S3. Calculate the transformation matrix between the calibration plate coordinate system and the underwater optical camera coordinate system: Using Zhang Zhengyou's calibration method, the internal matrix and distortion matrix of the underwater optical camera are solved by taking a checkerboard image of the calibration plate with an underwater optical camera. Using the known size of the calibration plate and the size of each checkerboard square, the transformation matrix between the calibration plate coordinate system and the underwater optical camera coordinate system is calculated by solving the PnP problem. S4. Calculate the transformation matrix between the sonar coordinate system and the underwater optical camera coordinate system to complete the calibration: By combining the transformation matrix between the sonar coordinate system and the calibration plate centroid coordinate system, and the transformation matrix between the calibration plate coordinate system and the underwater optical camera coordinate system, the transformation matrix between the sonar coordinate system and the underwater optical camera coordinate system is obtained, thus completing the calibration.

[0008] To achieve the above objectives, according to a second aspect of this application, a system for underwater acoustic-optical image spatial registration is provided, comprising a sonar, an underwater optical camera, a checkerboard calibration board, and a computing module; The sonar, the underwater optical camera, and the checkerboard calibration plate are used to acquire sonar images and checkerboard images: The checkerboard calibration plate, which has multiple metal protrusions set along a diagonal surface, is placed within the sonar detection range, and the surface of the calibration plate is made perpendicular to the imaging plane of the sonar. The calibration plate is vertically translated, and a series of sonar images and checkerboard images of the calibration plate captured by the underwater optical camera are acquired simultaneously. The calculation module is used to calculate the transformation matrix between the sonar coordinate system and the calibration plate centroid coordinate system: by extracting the position features of bright spots formed by metal protrusions in the sonar images frame by frame through a series of acquired sonar images, when multiple bright spots appear symmetrical about the center line of the sonar image and have the largest spacing in the horizontal direction, it is determined that the sonar scanning plane coincides with the center plane of the calibration plate; based on the sonar image data at the time of coincidence and the known dimensions of the calibration plate, the transformation matrix between the sonar coordinate system and the calibration plate centroid coordinate system is calculated. The calculation module is used to calculate the transformation matrix between the calibration plate coordinate system and the underwater optical camera coordinate system: using Zhang Zhengyou's calibration method, the internal matrix and distortion matrix of the underwater optical camera are solved by taking a checkerboard image of the calibration plate taken by the underwater optical camera; using the known size of the checkerboard and the size of each cell, the PnP problem is solved to calculate the transformation matrix between the calibration plate coordinate system and the underwater optical camera coordinate system. The calculation module is used to calculate the transformation matrix between the sonar coordinate system and the underwater optical camera coordinate system to complete the calibration: the transformation matrix between the sonar coordinate system and the centroid coordinate system of the calibration plate, and the transformation matrix between the calibration plate coordinate system and the underwater optical camera coordinate system are combined to obtain the transformation matrix between the sonar coordinate system and the underwater optical camera coordinate system, thus completing the calibration.

[0009] To achieve the above objectives, according to a third aspect of this application, a checkerboard calibration plate for underwater acoustic-optical image spatial registration is provided. The checkerboard calibration plate includes a calibration panel and calibration units: the surface of the calibration panel is provided with a plurality of checkerboard squares for calibration, the checkerboard squares including black squares and white squares; a plurality of calibration units are provided, equally spaced on a diagonal of the calibration panel, the calibration units being attached to and protruding from the surface of the calibration panel.

[0010] By adopting the above technical solution, this application has the following beneficial effects compared with the prior art: This application fully utilizes the advantages of a checkerboard calibration plate with metal protrusions marked on the diagonal. The metal protrusions generate stable bright spot features in the sonar image, actively and accurately establishing the coordinates of the sonar center and the camera optical center in the calibration plate coordinate system. The adjustment process is guided by the morphological features of the sonar image, rather than relying on preset external geometric conditions. This avoids the problems of requiring high-precision calibration materials and cumbersome operations, achieving higher accuracy than other calibration methods while avoiding high costs. The method is simple to manufacture, considers the projection distortion of the sonar image during calibration, increasing accuracy, and also exhibits good stability.

[0011] The specific embodiments of this application will be described in further detail below with reference to the accompanying drawings. Attached Figure Description

[0012] The accompanying drawings, which form part of this application, are used to provide a further understanding of the application. The illustrative embodiments and descriptions of the application are used to explain the application, but do not constitute an undue limitation of the application. Obviously, the drawings described below are merely some embodiments, and those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0013] In the attached diagram: Figure 1 This is a flowchart illustrating the calibration method for underwater acoustic-optical information spatial registration in this specific embodiment; Figure 2 This is a schematic diagram of the calibration plate used for spatial registration of underwater acoustic and optical information in this specific embodiment. Figure 3 This is a schematic diagram of the calibration method for underwater acoustic-optical information spatial registration in this specific embodiment; Figure 4 This is a schematic diagram of the calibration plate and the sonar imaging plane in this specific embodiment; Figure 5 This is a schematic diagram showing the positional relationship between the calibration plate and the sonar centroid in this specific embodiment; Figure 6 This is a schematic diagram illustrating the geometric relationship between the calibration plate and the sonar centroid in this specific embodiment; Figure 7 This is an example diagram of keyframes and feature points of sonar images in this specific embodiment; Figure 8 This is an example image of an underwater optical camera corresponding to a keyframe of a sonar image in this specific embodiment. Figure 9 This is a diagram showing the result of back-projecting key frame feature points of a sonar image onto an underwater optical camera image in this specific embodiment. Detailed Implementation

[0014] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions in the embodiments will be clearly and completely described below with reference to the accompanying drawings. The following embodiments are used to illustrate this application, but are not intended to limit the scope of this application.

[0015] Please refer to Figure 1 This application provides a calibration method for underwater acoustic-optical image spatial registration, comprising: S1. Acquire sonar and checkerboard images: A checkerboard calibration plate with multiple metal protrusions along one diagonal surface is placed within the sonar detection range, and the surface of the calibration plate is made perpendicular to the sonar imaging plane. The calibration plate is then translated along the direction perpendicular to the sonar imaging plane, and a series of sonar images and checkerboard images of the calibration plate taken by an underwater optical camera are acquired simultaneously. S2. Calculate the transformation matrix between the sonar coordinate system and the calibration plate centroid coordinate system: By collecting a series of sonar images, the positional features of bright spots formed by metallic protrusions in the sonar images are extracted frame by frame. When multiple bright spots appear symmetrical about the center line of the sonar image and have the largest spacing in the horizontal direction, it is determined that the sonar scanning plane coincides with the center plane of the calibration plate. Based on the sonar image data at the time of coincidence and the known dimensions of the calibration plate, the transformation matrix between the sonar coordinate system and the centroid coordinate system of the calibration plate is calculated. S3. Calculate the transformation matrix between the calibration plate coordinate system and the underwater optical camera coordinate system: Using Zhang Zhengyou's calibration method, the internal matrix and distortion matrix of the underwater optical camera are solved by taking a checkerboard image of the calibration plate with an underwater optical camera. Using the known size of the calibration plate and the size of each checkerboard square, the transformation matrix between the calibration plate coordinate system and the underwater optical camera coordinate system is calculated by solving the PnP problem. S4. Calculate the transformation matrix between the sonar coordinate system and the underwater optical camera coordinate system to complete the calibration: By combining the transformation matrix between the sonar coordinate system and the calibration plate centroid coordinate system, and the transformation matrix between the calibration plate coordinate system and the underwater optical camera coordinate system, the transformation matrix between the sonar coordinate system and the underwater optical camera coordinate system is obtained, thus completing the calibration.

[0016] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0017] In some embodiments, S1, acquiring sonar images and checkerboard images: a calibration plate with multiple metal protrusions set on the diagonal of the checkerboard is placed within the sonar detection range, and the surface of the calibration plate is made perpendicular to the sonar imaging plane. The calibration plate is vertically translated, and a series of sonar images and checkerboard images of the calibration plate taken by an underwater optical camera are acquired simultaneously.

[0018] Please see Figure 2 , Figure 2This is a schematic diagram of the calibration plate. The calibration plate is made of acrylic sheet with 12x12 corner points. Each checkerboard square has a side length of 50mm. The calibration plate has metal protrusions marked on the diagonals. Typically, metals that are resistant to rust in water, strongly reflect acoustic signals, and have clear color contrast in optical images, such as stainless steel, copper, or steel, are chosen. Visual data was acquired using a Stereolabs ZED2 binocular camera at 720p resolution. The underwater optical camera was placed in a waterproof chamber to isolate it from water. Sonar data was acquired using an Oculus M1200d multibeam sonar, operating at 2.1MHz with a horizontal opening angle of 80° and a vertical opening angle of 20°.

[0019] Please see Figure 4 , Figure 7 and Figure 8 , Figure 4 This is a schematic diagram showing the placement of the sonar imaging plane and the calibration plate. Figure 7 Keyframes for sonar images obtained by extracting image sequences from sonar. Figure 8 This is the optical image corresponding to the sonar image. The calibration plate is placed vertically so that its center is positioned within the sonar imaging plane. For example, the calibration plate is placed directly in front of the sonar's detection range. By observing the sonar image feedback in real time, the pitch and roll angles of the calibration plate are finely adjusted. When the calibration plate appears as a clear, thin bright line in the sonar image, it indicates that the plate surface is perpendicular to the sonar imaging plane. Maintaining the calibration plate's vertical position, the calibration plate is uniformly translated along a direction perpendicular to the sonar imaging plane. During this vertical movement, the sonar and underwater optical camera are simultaneously triggered, acquiring a sequence of 15 optical checkerboard calibration plate and sonar images.

[0020] In some embodiments, S2, the transformation matrix between the sonar coordinate system and the calibration plate centroid coordinate system is calculated: by extracting the position features of bright spots formed by metal protrusions in a series of acquired sonar images frame by frame, when multiple bright spots appear symmetrical about the center line of the sonar image and have the largest spacing in the horizontal direction, it is determined that the sonar scanning plane coincides with the center plane of the calibration plate; based on the sonar image data at the time of coincidence and the known size of the calibration plate, the transformation matrix between the sonar coordinate system and the calibration plate centroid coordinate system is calculated.

[0021] For the acquired sonar image sequence, Oculus ViewPoint software was used to perform frame-by-frame filtering, noise reduction, and feature enhancement. The location features of bright spots in the sonar images were marked frame by frame using Oculus ViewPoint software. When the bright spots in a certain frame were symmetrical and the spacing was maximized, that frame was selected as a keyframe from the sonar image sequence. The pose of this keyframe was the key pose where the sonar scanning plane coincided with the center plane of the calibration board. The three-dimensional coordinates of the origin of the sonar coordinate system in the calibration board coordinate system were solved using the keyframe.

[0022] Please refer to Figure 3 , Figure 3 A schematic diagram showing the various coordinate systems. Establishing the sonar coordinate system. and calibration plate centroid coordinate system For example, a sonar coordinate system is established with the sonar center as the origin. Establish a sonar coordinate system. shaft and The plane formed by the axis is parallel to the ground. Axis perpendicular to shaft and The plane formed by the axes establishes the coordinate system of the calibration plate. , and The axis satisfies the right-hand screw rule. Establish a coordinate system based on the centroid of the calibration plate, with the centroid of the calibration plate as the origin. Establish the centroid coordinate system of the calibration plate. shaft and The axes are the X and Y axes of the calibration board chessboard grid, respectively. Axis perpendicular to shaft and The plane formed by the axes establishes the centroid coordinate system of the calibration plate. , and The axis satisfies the right-hand screw rule.

[0023] Please refer to Figure 5 -(a), Figure 5 -(a) shows the positional relationship between the calibration plate and the sonar centroid. (Two endpoints of the calibration plate) , The distances from the origin of the sonar coordinate system are respectively , , angle is , The distance r between the origin of the sonar coordinate system and the origin of the centroid of the calibration plate can be obtained by averaging over the two endpoints. , angle is for .

[0024] Please refer to Figure 5-(b), Figure 5 -(b) represents the geometric relationship between the origin of the sonar coordinate system and the origin of the calibration plate's centroid coordinate system. Let the coordinate system of the unrotated calibration plate be... The sonar coordinate system is The distance r between the origin of the sonar coordinate system and the origin of the centroid coordinate system of the calibration plate, and the angle is... Therefore, the relationship between the two is as follows: the origin of the sonar coordinate system has the following coordinates in the unrotated calibration plate coordinate system: ( ,0, The coordinates of point p in the sonar coordinate system are (0, 0, ...). ), whose coordinates in the unrotated calibration plate centroid coordinate system are ( ,0,0): ; Please refer to Figure 6 , Figure 6 This relates to the rotational relationship of the calibration plate's centroid coordinate system. (Calibration plate centroid coordinate system) Axis and Sonar Coordinate System The axis forms an angle of Based on geometric relationships, the X-axis transformation formula and Z-axis transformation formula for the translational and rotated coordinates in the centroid coordinate system of the calibration plate can be derived as follows: ; ; in, , , To calibrate the translation coordinates in the centroid coordinate system of the plate, , , To calibrate the translational and rotational coordinates in the centroid coordinate system of the calibration plate; Rearrange the X-axis transformation formula and Z-axis transformation formula into matrix form and add the Y dimension to obtain the matrix: ; By simultaneously solving the translation and rotation matrices, we can obtain the transformation matrices between the sonar coordinate system and the calibration plate centroid coordinate system: .

[0025] For example, images are acquired using two-dimensional sonar. Figure 7 This is an image obtained from sonar. Open the sonar image using Oculus ViewPoint software, and use the software's scale to select the bright spot at the center of the fine bright line on the calibration board. , ) and the angle of the fine bright line on the calibration plate relative to the sonar coordinate axis Combining the translation and rotation matrices, we obtain the transformation matrices between the sonar coordinate system and the calibration plate centroid coordinate system. The transformation relationship matrix between the calibration plate centroid coordinate system and the sonar coordinate system is as follows: .

[0026] In some embodiments, S3, the transformation matrix between the calibration plate coordinate system and the underwater optical camera coordinate system is calculated: using the Zhang Zhengyou calibration method, the internal matrix and distortion matrix of the underwater optical camera are solved by taking a checkerboard image of the calibration plate taken by the underwater optical camera; using the known size of the calibration plate and the size of each checkerboard square, the transformation matrix between the calibration plate coordinate system and the underwater optical camera coordinate system is calculated by solving the PnP problem.

[0027] Solve for the transformation matrix between the calibration plate coordinate system and the underwater optical camera coordinate system. For example, based on the camera calibration principle, use the intrinsic parameter matrix of the calibrated underwater optical camera to solve for the transformation matrix between the calibration plate coordinate system and the underwater optical camera coordinate system.

[0028] For example, please see Figure 3 , Figure 3 This is a schematic diagram of the various coordinate systems. Establish the image pixel coordinate system. Underwater optical camera coordinate system Calibration plate coordinate system Set the image pixel coordinate system. Image pixel coordinate system Taking the top left corner of the image as the origin, axis, The axes are coordinate axes, and the unit of the coordinate axes is pixels. The x-coordinate of a pixel is... and the vertical axis These represent the row and column numbers in the image array, respectively. Define the coordinate system of the underwater optical camera. With camera optical center As the origin of the coordinate system, a straight line passing through the optical center and perpendicular to the imaging plane is drawn... axis, shaft and The coordinate system of the underwater optical camera is established by making the plane formed by the axes parallel to the imaging plane. A calibration plate coordinate system is also established. The calibration plate coordinate system takes the upper left corner of the calibration plate as its origin. , shaft and The axes are the X and Y axes of the calibration board chessboard grid, respectively. Axis perpendicular to shaft and axis. , and The axis satisfies the right-hand screw rule.

[0029] Obtain the intrinsic parameter matrix of the underwater optical camera: For example, the intrinsic parameter matrix of an underwater optical camera is a matrix describing the relationship between the camera's internal optical characteristics and imaging geometry. It is used to project three-dimensional points in the camera coordinate system onto two-dimensional pixel coordinates in the image plane. The software used for camera intrinsic parameter calibration is Python, using the `calibrateCamera` function built into the OpenCV library. Using checkerboard images captured by the underwater optical camera, the intrinsic parameter matrix of the underwater optical camera is estimated and calculated by reading the pixel coordinates of the corner points through the calibration program.

[0030] Input the physical dimensions of the checkerboard, including the length and width of each square, and the number of squares along the length and width directions on the calibration plate. Solve to calculate the intrinsic parameters of the underwater optical camera, and output the underwater optical camera intrinsic parameter matrix K, which includes the following: focal length. and Principal point coordinates and .focal length and Controlling image scaling is used to transform pixel coordinates to physical coordinates, with units of millimeters per pixel; the pixel coordinate system has its origin at the top left corner of the image, therefore principal point coordinates are needed ( , Translate to the optical center, in millimeters: ; in, , , The coordinates are in the underwater optical camera coordinate system, and u and v are in the pixel coordinate system.

[0031] The calibration results are obtained by running the calibration program. In this embodiment, the intrinsic parameter matrix K of the underwater optical camera is: .

[0032] Obtain the extrinsic parameter matrix of the underwater optical camera:

[0033] It should be noted that the extrinsic parameter matrix of the underwater optical camera is used to describe the transformation relationship between the calibration plate coordinate system with the upper left corner of the calibration plate as the origin and the underwater optical camera coordinate system with the camera optical center as the origin. The transformation relationship between the two is essentially a rigid body transformation, so the rotation matrix R and the translation matrix t are used to describe the transformation relationship.

[0034] Please refer to Figure 8 , Figure 8These are optical images corresponding to sonar. The optical images corresponding to the sonar images are acquired using an underwater optical camera. It should be noted that the software used for underwater optical camera calibration is Python, employing the `solvePnP` function built into the OpenCV library. The calibration program reads the pixel coordinates of the checkerboard corner points and the coordinates of the calibration board, and calculates the extrinsic parameter matrix of the underwater optical camera based on maximum likelihood estimation.

[0035] The extrinsic parameters of the underwater optical camera include a rotation matrix R and a translation matrix t. The rotation matrix R is a 3x3 orthogonal matrix used to describe the rotation from the calibration plate coordinate system to the camera coordinate system. Each column represents the direction of the X, Y, and Z axes of the camera coordinate system in the calibration plate coordinate system, and is dimensionless. The translation matrix t is a 1x3 matrix used to describe the translation from the calibration plate coordinate system to the camera coordinate system, with units in millimeters. ; in, , , The coordinates are in the coordinate system of the underwater optical camera. , , The coordinates are in the calibration plate coordinate system.

[0036] Run the calibration program to obtain the calibration results, and then concatenate the rotation matrix R and translation matrix t of the underwater optical camera. In this embodiment, the extrinsic parameter matrix of the underwater optical camera is: .

[0037] In some embodiments, S4, calculate the transformation matrix of the sonar coordinate system and the underwater optical camera coordinate system to complete the calibration: combine the transformation matrix of the sonar coordinate system and the centroid coordinate system of the calibration plate with the transformation matrix of the calibration plate coordinate system and the underwater optical camera coordinate system to obtain the transformation matrix of the sonar coordinate system and the underwater optical camera coordinate system, and complete the calibration.

[0038] For example, the origin of the calibration plate coordinate system is set to the upper left corner of the calibration plate, and the origin of the calibration plate centroid coordinate system is set to the centroid of the calibration plate. At the same time, the directions of the calibration plate coordinate system and the centroid coordinate system are set to be the same.

[0039] Since the coordinate systems of the calibration plate and the calibration centroid are aligned, the rotation matrix R is obtained as the identity matrix. The distance between the upper left corner of the calibration plate and its centroid is measured to be (300, 300, 0) in millimeters, yielding the translation matrix t. Combining the rotation matrix R and the translation matrix t yields the transformation matrix between the calibration plate's centroid coordinate system and the calibration plate's coordinate system: ; The transformation matrix between the calibration plate's centroid coordinate system and the calibration plate coordinate system describes the transformation relationship between the two coordinate systems. The transformation matrix between the calibration plate's centroid coordinate system and the sonar coordinate system in S2 realizes the transformation relationship between them. The extrinsic parameter matrix of the underwater optical camera realizes the transformation relationship from the calibration plate coordinate system to the underwater optical camera coordinate system. By simultaneously solving the transformation matrix between the calibration plate's centroid coordinate system and the calibration plate coordinate system, along with the transformation matrix between the sonar coordinate system and the calibration plate's centroid coordinate system in S2, and the extrinsic parameter matrix of the underwater optical camera, the transformation matrix between the sonar and underwater optical camera coordinate systems is obtained, thus realizing the joint calibration process of the sonar and underwater optical camera. The transformation matrix between the sonar and the underwater optical camera is: .

[0040] In this embodiment, a sonar image and a camera image are acquired, and the feature points of the sonar image are projected onto the camera image using the transformation matrix of the sonar and underwater optical camera obtained through joint calibration. Figure 9 As shown, the position of the projection point in the image is roughly aligned with the center of the calibration plate, verifying the accuracy of the transformation matrix between the sonar and the underwater optical camera.

[0041] Based on the same inventive concept, this application also provides a system for underwater acoustic-optical image spatial registration, including sonar, underwater optical camera, checkerboard calibration board and computing module; The sonar, the underwater optical camera, and the checkerboard calibration plate are used to acquire sonar images and checkerboard images: The checkerboard calibration plate, which has multiple metal protrusions set along a diagonal surface, is placed within the sonar detection range, and the surface of the calibration plate is made perpendicular to the imaging plane of the sonar. The calibration plate is vertically translated, and a series of sonar images and checkerboard images of the calibration plate captured by the underwater optical camera are acquired simultaneously. The calculation module is used to calculate the transformation matrix between the sonar coordinate system and the calibration plate centroid coordinate system: by extracting the position features of bright spots formed by metal protrusions in the sonar images frame by frame through a series of acquired sonar images, when multiple bright spots appear symmetrical about the center line of the sonar image and have the largest spacing in the horizontal direction, it is determined that the sonar scanning plane coincides with the center plane of the calibration plate; based on the sonar image data at the time of coincidence and the known dimensions of the calibration plate, the transformation matrix between the sonar coordinate system and the calibration plate centroid coordinate system is calculated. The calculation module is used to calculate the transformation matrix between the calibration plate coordinate system and the underwater optical camera coordinate system: using Zhang Zhengyou's calibration method, the internal matrix and distortion matrix of the underwater optical camera are solved by taking a checkerboard image of the calibration plate taken by the underwater optical camera; using the known size of the checkerboard and the size of each cell, the PnP problem is solved to calculate the transformation matrix between the calibration plate coordinate system and the underwater optical camera coordinate system. The calculation module is used to calculate the transformation matrix between the sonar coordinate system and the underwater optical camera coordinate system to complete the calibration: the transformation matrix between the sonar coordinate system and the centroid coordinate system of the calibration plate, and the transformation matrix between the calibration plate coordinate system and the underwater optical camera coordinate system are combined to obtain the transformation matrix between the sonar coordinate system and the underwater optical camera coordinate system, thus completing the calibration.

[0042] Based on the same inventive concept, please refer to Figure 2 This application also provides a checkerboard calibration board for underwater acoustic-optical image spatial registration, the checkerboard calibration board comprising: The calibration panel 100 has a surface covered with a number of checkerboard squares for calibration, the checkerboard squares including black squares and white squares; a number of black squares are arranged on the diagonal of the calibration panel 100, and one diagonal of the black squares coincides with one diagonal of the calibration panel 100. A plurality of calibration units 200 are provided. The calibration units 200 are attached to and protrude from the surface of the calibration panel 100 and are equally spaced on the diagonal of the calibration panel 100. Each black square has a calibration unit 200 at both ends of its diagonal. The calibration unit 200 is a metal protrusion.

[0043] The above are merely preferred embodiments of this application and are not intended to limit this application in any way. Although this application has disclosed preferred embodiments as described above, it is not intended to limit this application. Any person skilled in the art can make some modifications or alterations to the above-mentioned technical content to create equivalent embodiments without departing from the scope of the technical solution of this application. The implementation schemes in the above embodiments can also be further combined or replaced. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of this application without departing from the content of the technical solution of this application shall still fall within the scope of this application.

Claims

1. A calibration method for underwater acoustic-optical image spatial registration, characterized in that, include: S1. Acquire sonar and checkerboard images: A checkerboard calibration plate with multiple metal protrusions along one diagonal surface is placed within the sonar detection range, and the surface of the calibration plate is made perpendicular to the sonar imaging plane. The calibration plate is then translated along the direction perpendicular to the sonar imaging plane, and a series of sonar images and checkerboard images of the calibration plate taken by an underwater optical camera are acquired simultaneously. S2. Calculate the transformation matrix between the sonar coordinate system and the calibration plate centroid coordinate system: By collecting a series of sonar images, the positional features of bright spots formed by metallic protrusions in the sonar images are extracted frame by frame. When multiple bright spots appear symmetrical about the center line of the sonar image and have the largest spacing in the horizontal direction, it is determined that the sonar imaging plane coincides with the center plane of the calibration plate. Based on the sonar image data at the time of coincidence and the known dimensions of the calibration plate, the transformation matrix between the sonar coordinate system and the centroid coordinate system of the calibration plate is calculated. S3. Calculate the transformation matrix between the calibration plate coordinate system and the underwater optical camera coordinate system: Using Zhang Zhengyou's calibration method, the internal matrix and distortion matrix of the underwater optical camera are solved by taking a checkerboard image of the calibration plate with an underwater optical camera. Using the known size of the calibration plate and the size of each checkerboard square, the transformation matrix between the calibration plate coordinate system and the underwater optical camera coordinate system is calculated by solving the PnP problem. S4. Calculate the transformation matrix between the sonar coordinate system and the underwater optical camera coordinate system to complete the calibration: By combining the transformation matrix between the sonar coordinate system and the calibration plate centroid coordinate system, and the transformation matrix between the calibration plate coordinate system and the underwater optical camera coordinate system, the transformation matrix between the sonar coordinate system and the underwater optical camera coordinate system is obtained, thus completing the calibration.

2. The method according to claim 1, characterized in that, A checkerboard calibration board is placed within the sonar detection range, with its surface perpendicular to the sonar imaging plane. The calibration board is then vertically translated, simultaneously acquiring a series of sonar images and checkerboard images of the calibration board captured by an underwater optical camera; including: The calibration plate is placed in front of the sonar within its detection range. By observing the sonar image feedback in real time, the pitch and roll angles of the calibration plate are finely adjusted. When the calibration plate appears as a clear, thin bright line in the sonar image, it indicates that the surface of the calibration plate is perpendicular to the sonar imaging plane. Keeping the calibration plate perpendicular to the sonar imaging plane, the calibration plate is moved at a constant speed along a direction perpendicular to the sonar imaging plane. During the vertical movement of the calibration plate, the sonar and the underwater optical camera are triggered to work simultaneously, and the sonar image sequence and the corresponding underwater optical image sequence are acquired at a fixed sampling frequency.

3. The method according to any one of claims 1-2, characterized in that, Based on the sonar image data at the point of overlap and the known dimensions of the calibration plate, the transformation matrix between the sonar coordinate system and the centroid coordinate system of the calibration plate is calculated, including: The position of the central bright spot on the calibration board in the sonar imaging plane was determined by the sonar marking procedure as ( ). , At this point, the translation matrix between the calibration plate's centroid coordinate system and the sonar coordinate system can be obtained as follows: ； in, , , To calibrate the translation coordinates in the centroid coordinate system of the plate, , , These are coordinates in the sonar coordinate system. Through both ends of the calibration plate , relative to the origin of the sonar coordinate system The angle between the calibration plate surface and the sonar coordinate system is obtained using the geometric relationship of triangles. At this point, the rotation matrix between the calibration plate's centroid coordinate system and the sonar coordinate system is: ； in, , , To calibrate the translational and rotational coordinates in the centroid coordinate system of the calibration plate; By simultaneously establishing the displacement and rotation matrices between the calibration plate's centroid coordinate system and the sonar coordinate system, the transformation matrix between the sonar coordinate system and the calibration plate's centroid coordinate system can be obtained: 。 4. The method according to claim 3, characterized in that, Using Zhang Zhengyou's calibration method, the intrinsic parameter matrix of the underwater optical camera is solved from the checkerboard image of the calibration plate captured by the underwater optical camera. Using the known dimensions of the calibration plate and the size of each checkerboard square, a PnP problem is solved to calculate the transformation matrix between the calibration plate coordinate system and the underwater optical camera coordinate system, including: Establish pixel coordinate system Underwater optical camera coordinate system Calibration plate coordinate system Among them, the pixel coordinate system takes the upper left corner of the pixel as the origin, the underwater optical camera coordinate system takes the optical center of the camera as the origin, and the calibration board coordinate system takes the upper left corner of the calibration board as the origin. The internal parameters of an underwater optical camera are calibrated using OpenCV. The physical dimensions of a checkerboard pattern are input, including the length and width of each square, and the number of squares along the length and width directions on the calibration board. The internal parameters of the underwater optical camera are then calculated, including: focal length. and Principal point coordinates and Inclination coefficient, radial distortion coefficient, tangential distortion coefficient; Based on the scale transformation principle of underwater optical cameras, the intrinsic parameter matrix of the underwater optical camera is obtained as follows: ； in, , , The coordinates are in the underwater optical camera coordinate system, and u and v are in the pixel coordinate system. By combining the intrinsic parameter matrix of the underwater optical camera with the checkerboard image selected in S1, and considering the physical dimensions of the checkerboard, the extrinsic parameter matrix of the underwater optical camera is solved using OpenCV's solvePnP algorithm. The extrinsic parameter matrix of the underwater optical camera includes the rotation matrix R and the translation matrix t. The transformation matrix between the underwater optical camera coordinate system and the calibration board coordinate system is obtained as follows: ； in, , , The coordinates are in the calibration plate coordinate system.

5. The method according to claim 4, characterized in that, The transformation matrix between the sonar coordinate system and the calibration plate centroid coordinate system is combined with the transformation matrix between the calibration plate coordinate system and the underwater optical camera coordinate system to obtain the transformation matrix between the sonar coordinate system and the underwater optical camera coordinate system, thus completing the calibration. This includes: The origin of the calibration plate coordinate system is the upper left corner of the calibration plate, and the origin of the calibration plate centroid coordinate system is the centroid of the calibration plate; the distance between the upper left corner of the calibration plate and the centroid of the calibration plate is ( , At this point, the transformation matrix between the calibration plate centroid coordinate system and the calibration plate coordinate system is: ； The transformation matrix between the calibration plate's centroid coordinate system and the calibration plate coordinate system is multiplied together with the transformation matrix between the sonar coordinate system and the calibration plate's centroid coordinate system in S2, as well as the extrinsic parameter matrix of the underwater optical camera, to obtain the transformation matrix between the sonar coordinate system and the underwater optical camera coordinate system, thus realizing the joint calibration process of the sonar and the underwater optical camera. 。 6. A system for underwater acoustic-optical image spatial registration, characterized in that, Includes sonar, underwater optical camera, checkerboard calibration board, and computing module; The sonar, the underwater optical camera, and the checkerboard calibration plate are used to acquire sonar images and checkerboard images: The checkerboard calibration plate, which has multiple metal protrusions set along a diagonal surface, is placed within the sonar detection range, and the surface of the calibration plate is made perpendicular to the imaging plane of the sonar. The calibration plate is vertically translated, and a series of sonar images and checkerboard images of the calibration plate captured by the underwater optical camera are acquired simultaneously. The calculation module is used to calculate the transformation matrix between the sonar coordinate system and the calibration plate centroid coordinate system: by extracting the position features of bright spots formed by metal protrusions in the sonar images frame by frame through a series of acquired sonar images, when multiple bright spots appear symmetrical about the center line of the sonar image and have the largest spacing in the horizontal direction, it is determined that the sonar scanning plane coincides with the center plane of the calibration plate; based on the sonar image data at the time of coincidence and the known dimensions of the calibration plate, the transformation matrix between the sonar coordinate system and the calibration plate centroid coordinate system is calculated. The calculation module is used to calculate the transformation matrix between the calibration plate coordinate system and the underwater optical camera coordinate system: using Zhang Zhengyou's calibration method, the internal matrix and distortion matrix of the underwater optical camera are solved by taking a checkerboard image of the calibration plate taken by the underwater optical camera; using the known size of the checkerboard and the size of each cell, the PnP problem is solved to calculate the transformation matrix between the calibration plate coordinate system and the underwater optical camera coordinate system. The calculation module is used to calculate the transformation matrix between the sonar coordinate system and the underwater optical camera coordinate system to complete the calibration: the transformation matrix between the sonar coordinate system and the centroid coordinate system of the calibration plate, and the transformation matrix between the calibration plate coordinate system and the underwater optical camera coordinate system are combined to obtain the transformation matrix between the sonar coordinate system and the underwater optical camera coordinate system, thus completing the calibration.

7. A checkerboard calibration plate for underwater acoustic-optical image spatial registration, characterized in that, The checkerboard calibration plate includes a calibration panel and calibration units: the surface of the calibration panel is provided with a plurality of checkerboard squares for calibration, the checkerboard squares including black squares and white squares; a plurality of calibration units are provided, equally spaced on one diagonal of the calibration panel, the calibration units are attached to and protrude from the surface of the calibration panel, and the calibration units are metal protrusions.

Images