Methods, devices, electronic equipment, storage media, and computer program products for three-dimensional reconstruction of underwater objects

Abstract

This disclosure relates to a method, apparatus, electronic device, storage medium, and computer program product for 3D reconstruction of underwater objects. The method includes: calculating underwater structured light plane equation parameters at the current angle based on air structured light plane equation parameters and pre-acquired refractive interface parameters; obtaining pixel coordinates p0(u, v) corresponding to target underwater feature points on the underwater object at the current angle; calculating ray tracing vectors corresponding to the target underwater feature points based on pixel coordinates p0 and a preset pinhole imaging model; and calculating the 3D coordinates of the target underwater feature points based on pixel coordinates p0, the ray tracing vectors, and underwater structured light plane equation parameters at the current angle. Thus, considering the advantages of event cameras in high-speed and low-light scenarios, event cameras can be introduced into underwater structured light vision systems, thereby enabling fast, accurate, and robust 3D reconstruction of underwater moving objects in low-light conditions with weak textures.

Description

Technical Field

[0001] This disclosure relates to the field of three-dimensional reconstruction technology, and more specifically, to methods, apparatus, electronic devices, storage media, and computer program products for three-dimensional reconstruction of underwater objects. Background Technology

[0002] Underwater 3D reconstruction technology is widely used in underwater structure inspection, underwater topographic mapping, in-situ observation of marine life, and underwater archaeology. Furthermore, by acquiring 3D information about the underwater environment, underwater robots can better perform navigation and operational tasks. Currently, underwater 3D reconstruction technologies mainly include acoustic-based methods and optical-based methods. Acoustic-based methods have advantages such as long measurement distance and insensitivity to water turbidity. However, they also suffer from low measurement accuracy and low resolution, making them unsuitable for high-precision, dense underwater 3D reconstruction scenarios. Optical-based methods offer advantages such as high measurement accuracy, large information capacity, and low cost, making them the primary technology for short-range, high-precision underwater 3D reconstruction. Moreover, optical-based underwater 3D reconstruction methods mainly include passive optical vision methods and active optical vision methods.

[0003] Underwater passive light vision methods use only ambient light or underwater illumination to light the scene. The camera can then acquire images from different poses. Next, 3D reconstruction can be performed based on the image feature matching results. However, image feature matching computation heavily relies on the target's texture features and is computationally intensive. Worse still, in low-light and murky underwater environments, image feature matching is extremely difficult.

[0004] Underwater active light vision methods actively project specific patterns onto the target object to complete 3D measurements. As a typical example of active light vision methods, underwater structured light vision can actively project specific patterns onto the object being measured. Then, a camera can acquire the pattern information modulated by the object. Next, underwater 3D reconstruction can be completed using the principles of triangulation. Underwater structured light vision methods have advantages such as high measurement accuracy and robustness, and have attracted widespread attention from researchers in recent years. Furthermore, underwater structured light vision methods typically use blue-green lasers as the active light source, which have the advantages of high collimation and long propagation distance in water.

[0005] In related technologies, the cameras used in underwater structured light vision systems are all traditional fixed frame rate, global exposure cameras. However, in reality, the proportion of structured light information in the sensor pixels is very small, resulting in most invalid pixels still occupying a lot of bandwidth and processor resources. This leads to low measurement efficiency of underwater structured light vision systems, making them unsuitable for 3D reconstruction of underwater moving targets. Summary of the Invention

[0006] This disclosure provides a method, apparatus, electronic device, storage medium, and computer program product for three-dimensional reconstruction of underwater objects, in order to at least solve the problem that underwater structured light vision systems have low measurement efficiency and are not suitable for three-dimensional reconstruction of underwater moving targets in the aforementioned related technologies.

[0007] According to a first aspect of the present disclosure, a method for three-dimensional reconstruction of an underwater object is provided, applied to a structured light vision system. The structured light vision system includes a galvanometer, a laser, an event camera, and glass. The three-dimensional reconstruction method includes: acquiring structured light plane equation parameters in air at a current angle, wherein the structured light vision system is placed underwater; calculating structured light plane equation parameters in water at the current angle based on the structured light plane equation parameters in air at the current angle and pre-acquired refractive interface parameters; acquiring pixel coordinates p0(u, v) corresponding to target underwater feature points on the underwater object at the current angle; and calculating the structured light plane equation parameters in water based on the pixel coordinates p0 and pre-acquired refractive interface parameters. Using a pinhole imaging model, the ray tracing vector corresponding to the underwater feature point of the target is calculated. This ray tracing vector includes vectors Ocp1, p1p2, and p2p3. Vector Ocp1 is refracted through the inner surface of the glass to obtain vector p1p2, and vector p1p2 is refracted through the outer surface of the glass to obtain vector p2p3. p1 is the intersection of ray Ocp0 and the inner surface of the glass, and Oc is the optical center of the event camera. Based on the pixel coordinate point p0, the ray tracing vector, and the underwater structured light plane equation parameters at the current angle, the three-dimensional coordinates of the underwater feature point of the target are calculated.

[0008] Optionally, obtaining the air structured light plane equation parameters of the galvanometer at the current angle includes: obtaining the rotation angle of the galvanometer from the previous angle to the current angle; determining the rotation matrix based on the rotation angle and the pre-calibrated scanning axis linear equation parameters; and calculating the air structured light plane equation parameters at the current angle based on the air structured light plane equation parameters of the previous angle and the rotation matrix.

[0009] Optionally, before obtaining the rotation angle of the galvanometer from the previous angle to the current angle, the three-dimensional reconstruction method further includes: obtaining the air structured light plane equation parameters of the galvanometer at each of the multiple angles; and calculating the scanning axis linear equation parameters based on the multiple air structured light plane equation parameters that correspond one-to-one with the multiple angles.

[0010] Optionally, obtaining the structured light plane equation parameters in air at each of the multiple angles of the galvanometer includes: for each angle, obtaining laser stripe images at different poses between the preset checkerboard and the event camera, wherein the laser stripe image is an image obtained by the event camera capturing the preset checkerboard in image mode after the laser emitted by the laser is irradiated onto the galvanometer and the galvanometer reflects the received laser back to the preset checkerboard; and calculating the structured light plane equation parameters in air at that angle based on the laser stripe images at different poses between the preset checkerboard and the event camera.

[0011] Optionally, the refractive interface parameters include the normal vector n of the inner surface of the glass and the distance d0 between the optical center of the event camera and the inner surface of the glass; the 3D reconstruction method further includes: acquiring a first image and a second image when the relative pose between the preset checkerboard and the event camera is the target pose, wherein the first image is an image obtained by the event camera in image mode when both the structured light vision system and the preset checkerboard are placed in air, and the second image is an image obtained by the event camera in image mode when both the structured light vision system and the preset checkerboard are placed in water; calculating the coordinates of the corner points of the preset checkerboard based on the first image and the second image; calculating the normal vector n based on the coordinates of the corner points of the preset checkerboard and the camera parameters of the event camera obtained through pre-calibration; and calculating the distance d0 between the optical center of the event camera and the inner surface of the glass based on the normal vector n.

[0012] Optionally, before acquiring the first and second images when the relative pose between the preset checkerboard and the event camera is the target pose, the 3D reconstruction method further includes: acquiring a third image under different poses between the preset checkerboard and the event camera, wherein the third image is an image obtained by the event camera in the image mode when both the structured light vision system and the preset checkerboard are placed in the air; and calibrating the camera parameters of the event camera based on the third images under different poses between the preset checkerboard and the event camera.

[0013] Optionally, the pixel coordinate point p0 is the two-dimensional coordinate point of the underwater feature point of the target under the imaging plane coordinate system of the event camera, obtained by the event camera in event mode after the laser emitted by the laser shines on the galvanometer at the current angle and the galvanometer reflects the received laser to the target underwater feature point.

[0014] According to a second aspect of the present disclosure, a three-dimensional reconstruction device for underwater objects is provided, applied to a structured light vision system. The structured light vision system includes a galvanometer, a laser, an event camera, and glass. The three-dimensional reconstruction device includes: a parameter acquisition module configured to acquire air structured light plane equation parameters at a current angle, wherein the structured light vision system is placed underwater; an underwater equation parameter calculation module configured to calculate underwater structured light plane equation parameters at the current angle based on the air structured light plane equation parameters at the current angle and pre-acquired refractive interface parameters; a pixel coordinate point acquisition module configured to acquire pixel coordinate points p0(u, v) corresponding to target underwater feature points on the underwater object at the current angle; and a ray tracing vector counter. The calculation module is configured to calculate the ray tracing vector corresponding to the target underwater feature point based on the pixel coordinate point p0 and a preset pinhole imaging model. The ray tracing vector includes vectors Ocp1, p1p2, and p2p3. Vector Ocp1 is refracted through the inner surface of the glass to obtain vector p1p2, and vector p1p2 is refracted through the outer surface of the glass to obtain vector p2p3. p1 is the intersection of ray Ocp0 and the inner surface of the glass, and Oc is the optical center of the event camera. The three-dimensional coordinate calculation module is configured to calculate the three-dimensional coordinates of the target underwater feature point based on the pixel coordinate point p0, the ray tracing vector, and the underwater structured light plane equation parameters at the current angle.

[0015] Optionally, the parameter acquisition module is configured to: acquire the rotation angle of the galvanometer from the previous angle to the current angle; determine the rotation matrix based on the rotation angle and the pre-calibrated scanning axis linear equation parameters; and calculate the air structured light plane equation parameters at the current angle based on the air structured light plane equation parameters at the previous angle and the rotation matrix.

[0016] Optionally, the three-dimensional reconstruction device further includes: an air equation parameter acquisition module, configured to acquire air structured light plane equation parameters at each of the multiple angles in which the galvanometer is located; and a scanning axis linear equation parameter calculation module, configured to calculate the scanning axis linear equation parameters based on the multiple air structured light plane equation parameters that correspond one-to-one with the multiple angles.

[0017] Optionally, the air equation parameter acquisition module is configured to: for each angle, acquire laser stripe images of different poses between the preset checkerboard and the event camera, wherein the laser stripe image is an image obtained by the event camera capturing the preset checkerboard in image mode after the laser emitted by the laser is irradiated onto the galvanometer and the galvanometer reflects the received laser back to the preset checkerboard; and calculate the air structured light plane equation parameters at that angle based on the laser stripe images of different poses between the preset checkerboard and the event camera.

[0018] Optionally, the refractive interface parameters include the normal vector n of the inner surface of the glass and the distance d0 between the optical center of the event camera and the inner surface of the glass; the three-dimensional reconstruction device further includes: a first image and a second image acquisition module, configured to acquire a first image and a second image when the relative pose between the preset checkerboard and the event camera is the target pose, wherein the first image is an image obtained by the event camera in image mode when both the structured light vision system and the preset checkerboard are placed in air, and the second image is an image obtained by the event camera in image mode when both the structured light vision system and the preset checkerboard are placed in water; a corner coordinate calculation module, configured to calculate the coordinates of the corner points of the preset checkerboard based on the first image and the second image; a normal vector calculation module, configured to calculate the normal vector n based on the coordinates of the corner points of the preset checkerboard and the camera parameters of the event camera obtained through pre-calibration; and a distance calculation module, configured to calculate the distance d0 between the optical center of the event camera and the inner surface of the glass based on the normal vector n.

[0019] Optionally, the 3D reconstruction device further includes: a third image acquisition module, configured to acquire third images of the preset checkerboard and the event camera under different poses, wherein the third image is an image captured by the event camera in the image mode when both the structured light vision system and the preset checkerboard are placed in the air; and a camera parameter calibration module, configured to calibrate the camera parameters of the event camera based on the third images of the preset checkerboard and the event camera under different poses.

[0020] Optionally, the pixel coordinate point p0 is the two-dimensional coordinate point of the underwater feature point of the target under the imaging plane coordinate system of the event camera, obtained by the event camera in event mode after the laser emitted by the laser shines on the galvanometer at the current angle and the galvanometer reflects the received laser to the target underwater feature point.

[0021] According to a third aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement a three-dimensional reconstruction method for an underwater object according to the present disclosure.

[0022] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided that, when instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to perform a three-dimensional reconstruction method for an underwater object according to the present disclosure.

[0023] The technical solutions provided by the embodiments of this disclosure have at least the following beneficial effects:

[0024] In this disclosure, considering the advantages of event cameras in high-speed and low-light scenarios, event cameras can be introduced into underwater structured light vision systems, thereby enabling fast, accurate, and robust 3D reconstruction of underwater moving objects in low-light conditions with weak textures. Furthermore, the structured light vision system in this disclosure is simple to construct, consisting of only a laser, an event camera, and a galvanometer, without any additional hardware, resulting in low cost.

[0025] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0026] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0027] Figure 1 This is a schematic diagram illustrating the structure of a structured light vision system according to an exemplary embodiment of the present disclosure;

[0028] Figure 2 This is a schematic diagram illustrating the implementation process of underwater 3D reconstruction based on event structured light according to an exemplary embodiment of the present disclosure;

[0029] Figure 3 This is a schematic diagram illustrating the calibration process of event structured light parameters according to an exemplary embodiment of the present disclosure;

[0030] Figure 4 This is a flowchart illustrating a method for three-dimensional reconstruction of an underwater object according to an exemplary embodiment of the present disclosure;

[0031] Figure 5 This is a schematic diagram illustrating an event-structured light underwater three-dimensional measurement process according to an exemplary embodiment of the present disclosure;

[0032] Figure 6 This is a schematic diagram illustrating the rotational optical path transformation of a galvanometer according to an exemplary embodiment of the present disclosure;

[0033] Figure 7 This is a schematic diagram illustrating the calibration process of refractive interface parameters according to an exemplary embodiment of the present disclosure;

[0034] Figure 8 This is a block diagram illustrating a three-dimensional reconstruction apparatus for an underwater object according to an exemplary embodiment of the present disclosure;

[0035] Figure 9 This is a block diagram illustrating an electronic device according to exemplary embodiments of the present disclosure. Detailed Implementation

[0036] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0037] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following examples do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0038] It should be noted that the phrase "at least one of several items" in this disclosure refers to three parallel cases: "any one of the several items", "a combination of any number of the several items", and "all of the several items". For example, "including at least one of A and B" includes the following three parallel cases: (1) including A; (2) including B; (3) including A and B. As another example, "performing at least one of step one and step two" indicates the following three parallel cases: (1) performing step one; (2) performing step two; (3) performing both step one and step two.

[0039] Figure 1 This is a schematic diagram illustrating the structure of a structured light vision system according to an exemplary embodiment of the present disclosure.

[0040] Reference Figure 1 The structured light vision system provided in this disclosure may include a structured light projector, a scanning galvanometer, glass, an event camera, and a filter.

[0041] Structured light projectors, also known as lasers, are primarily used to project laser stripes. The laser can be, but is not limited to, blue-green light, which experiences minimal attenuation in water. For example, the laser wavelength can be 450 nm, and the power can be 400 mW.

[0042] Galvanometer: It can rotate at high speed, thus enabling rapid scanning of laser stripes. The scanning angle can be controlled by the input level.

[0043] Event camera: This is a bio-inspired sensor primarily used to obtain event stream information during laser stripe scanning. Specifically, it outputs asynchronous signals (including symbols for event, position, and brightness changes) by measuring the brightness changes of each pixel, suppressing redundant information. Compared to traditional cameras, event cameras feature high temporal resolution, low power consumption, and high dynamic range, offering broad application prospects in high-speed, high-dynamic-range scenarios. Additionally, the event camera may include an optical lens for imaging, in front of which a narrowband filter with a center wavelength matching the wavelength of the projected structured light can be mounted. For example, in this disclosure, the event frame pixel count can be 346 × 260, and the center wavelength of the narrowband filter in front of the lens can be 450 nm.

[0044] It should be noted that the above-mentioned structured light vision system can also be divided into: structured light parameter calibration module, event information processing module, and 3D reconstruction module. By executing the above modules in sequence, the 3D reconstruction of underwater moving objects with low light and weak texture can be realized.

[0045] Figure 2 This is a schematic diagram illustrating the implementation process of underwater 3D reconstruction based on event structured light according to an exemplary embodiment of the present disclosure.

[0046] Reference Figure 2 The structured light parameter calibration module can include camera parameter calibration, structured light plane equation parameter calibration in air, refraction interface parameter calibration, and scanning axis linear equation parameter calibration; the event information processing module can include acquiring event streams, preprocessing event streams, grouping event streams, and extracting features from grouped event streams; the 3D reconstruction module can include a self-scanning structured light visual measurement model, a structured light plane model considering refraction, a camera imaging model considering refraction, and 3D coordinate calculation of underwater feature points.

[0047] Figure 3 This is a schematic diagram illustrating the calibration process of event structured light parameters according to an exemplary embodiment of the present disclosure.

[0048] Reference Figure 3The structured light parameter calibration module can include camera parameter calibration, structured light plane equation parameter calibration in air, refractive interface parameter calibration, and scanning axis linear equation parameter calibration. Specifically, camera parameter calibration can include focal length, principal point image coordinates, and pixel size; refractive interface parameter calibration can include the normal vector of the inner surface of the glass and the distance from the camera optical center to the inner surface of the glass.

[0049] The event information processing module mainly includes acquiring event streams, preprocessing event streams, grouping event streams, and extracting features from the grouped event streams. Specifically, during rapid laser line scanning, the event camera can generate event stream information, which may include (x i ,y i ,t i ) information, where x i and y i This indicates that an event has occurred at this location, specifically, the change in light exceeds a threshold, t. i This indicates the time the event occurred. Event stream preprocessing involves removing noise data from the event stream. Typically, event points generated by laser line scanning are adjacent and continuous, while isolated event points are considered noise and therefore need to be removed. Event stream grouping involves grouping events according to their timestamps, with events within a time interval forming a single event packet. Finally, feature extraction can be performed on the grouped event packets to obtain the image coordinates of the laser stripe center point.

[0050] Figure 4 This is a flowchart illustrating a method for three-dimensional reconstruction of an underwater object according to an exemplary embodiment of the present disclosure, applied to a structured light vision system. As previously described, the structured light vision system may include a galvanometer, a laser, an event camera, and glass.

[0051] Reference Figure 4 In step 401, the parameters of the structured light plane equation in air at the current angle of the galvanometer can be obtained, wherein the structured light vision system is placed underwater. The parameters of the structured light plane equation in air can refer to the four coefficients A, B, C, and D contained in the structured light plane equation in air: Ax + By + Cz + D = 0.

[0052] According to an exemplary embodiment of this disclosure, the rotation angle of the galvanometer from the previous angle to the current angle can be obtained. Then, a rotation matrix can be determined based on this rotation angle and pre-calibrated scanning axis linear equation parameters. Next, the air structured light plane equation parameters at the current angle can be calculated based on the air structured light plane equation parameters at the previous angle and the rotation matrix. It should be noted that the air structured light plane equation parameters at the initial galvanometer angle can be obtained using the galvanometer angle obtained during the pre-calibration of the scanning axis linear equation parameters; this acquisition process will be described in detail later.

[0053] Figure 2 The self-scanning structured light vision measurement model can be obtained in the following ways, specifically:

[0054] Figure 5 This is a schematic diagram illustrating an event-structured light underwater three-dimensional measurement process according to an exemplary embodiment of the present disclosure. (Refer to...) Figure 5 The coordinate system of the event camera can be o c x c y c z c , where o c As the optical center of the event camera, z c The axis is the optical axis of the event camera, x c axis and y c The axes are parallel to the u-axis and v-axis of the event camera's image plane. We can define p0 as the point on the event camera's image plane corresponding to a feature point on an underwater object. Furthermore, the ray o... c The intersection point of p0 and the inner surface of the glass is p1,o. c The refracted ray p0 through the inner surface of the glass is p1p2, and the refracted ray p1p2 through the outer surface of the glass is p2p3. Additionally, the galvanometer coordinate system can be defined as o. w x w y w z w , where x w The axis is the rotation axis of the galvanometer.

[0055] Assume the image coordinates of point p0 are (u, v), and the coordinates of this point in the camera coordinate system are (x, v). c ,y c ,z c According to the pinhole imaging model, we can conclude that:

[0056]

[0057] Among them, (f x ,f y) are two magnification factors, (u0, v0) are the image coordinates of the camera's principal point, and M is the image coordinates of the principal point. c It is the camera intrinsic parameter matrix.

[0058] Assuming the initial mirror angle is 0, the equation of the structured light plane in air can be: A0x + B0y + C0z + D0 = 0, where A0, B0, C0, and D0 can be the parameters of the structured light plane equation in air.

[0059] Figure 6 This is a schematic diagram illustrating the rotational optical path transformation of a galvanometer according to an exemplary embodiment of the present disclosure. (Refer to...) Figure 6 When the galvanometer rotates by an angle of θ / 2, the structured light plane rotates by an angle of θ. Therefore, the equation parameters of the structured light plane in the air after rotation can be:

[0060]

[0061] Where A1, B1, and C1 are the parameters of the rotated structured light plane equation in air: A1x + B1y + C1z + D1 = 0, (s x ,s y ,s z ) is the direction vector of the mirror rotation axis.

[0062] Since a point (x0, y0, z0) on the rotation axis lies on the rotated structured light plane, we have A1x0 + B1y0 + C1z0 + D1 = 0. Therefore, we can calculate D1 = -(A1x0 + B1y0 + C1z0). Then, combining the event camera imaging model and the equation of the rotated structured light plane in air, we can obtain the structured light visual measurement model in air after the galvanometer rotation:

[0063]

[0064] According to an exemplary embodiment of this disclosure, before obtaining the rotation angle of the galvanometer from the previous angle to the current angle, the air-based structured light plane equation parameters for each of the multiple angles at which the galvanometer is located can also be obtained. Then, based on the multiple air-based structured light plane equation parameters corresponding one-to-one with the multiple angles, the scanning axis linear equation parameters can be calculated. Since the point on the scanning axis lies on the structured light plane, the structured light plane can be fitted using the least squares method to obtain the coordinates of a point on the scanning axis, thereby determining the scanning axis linear equation parameters. Specifically, the calibration process for the scanning axis linear equation parameters can be as follows:

[0065] Assume the equation of the linear axis of rotation of the galvanometer is:

[0066]

[0067] Among them, (s x ,s y ,s z (x0, y0, z0) is the direction vector of the rotation axis, (x0, y0, z0) is a point on the rotation axis, and t represents the straight line.

[0068] The parameters of the plane equation for structured light in air can be obtained at each galvanometer scanning angle using the calibration method for structured light plane equations. The specific calibration process will be described in detail later. Then, the galvanometer can be rotated at multiple angles; here, four angles are used as an example. At this point, a set of structured light plane equation parameters can be determined at each angle, and the scanning axis linear vector can be determined by least-squares fitting.

[0069]

[0070] Among them, (A) 0i B 0i C 0i Let X be the normal vector of the structured light plane equation at the i-th angle. By performing singular value decomposition on matrix X, the scanning axis line vector can be obtained.

[0071] Furthermore, since a point (x0, y0, z0) on the rotation axis lies on the structured light plane, we can conclude that:

[0072]

[0073] (x0, y0, z0) can be obtained by least squares fitting.

[0074] According to an exemplary embodiment of this disclosure, for each angle, laser stripe images under different poses between a preset checkerboard and an event camera can be acquired. These laser stripe images can be obtained by the event camera capturing the preset checkerboard in image mode after the laser emitted by the laser is incident on a galvanometer and reflected back to the preset checkerboard by the galvanometer. Then, based on the laser stripe images under different poses between the preset checkerboard and the event camera, the parameters of the structured light plane equation in the air at that angle can be calculated. It should be noted that by changing the pose of the preset checkerboard relative to the event camera, the two laser stripes can be made non-collinear in space, thus allowing a structured light plane to be uniquely determined by two non-collinear laser stripe lines.

[0075] The following section elaborates on the process of controlling the movement of a preset chessboard to change its pose relative to the event camera. Specifically:

[0076] Assuming the initial position, the laser stripe line can be denoted as L1. After changing the relative pose of the preset checkerboard and the event camera, the laser stripe line can be denoted as L2. Based on the positional relationship of the lines in space, the positional relationship between L1 and L2 can have three cases: L1 and L2 are collinear, in which case the structured light plane cannot be uniquely determined; L1 and L2 are parallel, in which case the structured light plane can be uniquely determined; L1 and L2 intersect, in which case the structured light plane can be uniquely determined. Assuming the structured light vision sensor is fixed, when the preset checkerboard rotates and translates within its own plane, L1 and L2 become collinear. Therefore, to prevent L1 and L2 from becoming collinear, the preset checkerboard should move along the z-axis or rotate around the x and y axes.

[0077] It should be noted that the extrinsic parameters of the event camera can be obtained during the calibration process. It can be used to represent the transformation relationship between the event camera coordinate system and the preset checkerboard coordinate system:

[0078]

[0079] Among them, [a x a y a z ] represents the normal vector of the predefined chessboard plane, [p x p y p z ] represents the origin of the coordinate system, [n x n y n z [o] represents the direction vector of the x-axis of the camera coordinate system in the preset checkerboard coordinate system. x o y o z ] represents the direction vector of the y-axis of the camera coordinate system in the preset checkerboard coordinate system, [x t y t , z t [] indicates the coordinate point in the preset chessboard coordinate system.

[0080] Assume the coordinates of any point within the predefined chessboard plane can be represented as [x c y c z c Then the following formula holds:

[0081] a x (x c -p x )+a y (y c -p y )+a z (z c -p z ) = 0

[0082] By combining the event camera imaging model, the coordinates of the laser stripe feature points in the event camera coordinate system can be obtained:

[0083]

[0084] x c =z c (u-u0) / k x

[0085] y c =z c (v-v0) / k y

[0086] Where, k x k y This is the intrinsic parameter of the event camera, specifically the focal length of the event camera.

[0087] Next, the structured light plane equation can be solved by performing least-squares fitting on three or more non-collinear laser stripe points. Furthermore, to improve calibration accuracy, multiple sets of relative poses can be set for the preset checkerboard pattern (calibration board) and event camera to obtain multiple non-collinear laser stripes. Then, the parameters of the structured light plane equation can be solved by performing least-squares fitting on all points on the laser stripes.

[0088] According to an exemplary embodiment of this disclosure, the refractive interface parameters may include the normal vector n of the inner surface of the glass and the distance d0 between the optical center of the event camera and the inner surface of the glass.

[0089] In this case, the three-dimensional reconstruction method for an underwater object according to an exemplary embodiment of the present disclosure may further include the following steps. First, a first image and a second image can be acquired when the relative pose between the preset checkerboard and the event camera is the target pose. The first image may be an image obtained by the event camera in image mode when both the structured light vision system and the preset checkerboard are placed in air; the second image may be an image obtained by the event camera in image mode when both the structured light vision system and the preset checkerboard are placed in water. Then, the coordinates of the corner points of the preset checkerboard can be calculated based on the first and second images. Next, the normal vector n can be calculated based on the coordinates of the corner points of the preset checkerboard and the event camera parameters of the pre-calibrated event camera. Then, the distance d0 between the optical center of the event camera and the inner surface of the glass can be calculated based on the normal vector n. Further, the distance d0 between the optical center of the event camera and the inner surface of the glass can be calculated based on the normal vector n combined with the event camera refraction model, specifically:

[0090] Figure 7 This is a schematic diagram illustrating the calibration process of refractive interface parameters according to an exemplary embodiment of the present disclosure. (Refer to...) Figure 7 Due to refraction in the medium, op3 and op1 should be on the same plane. Therefore, the refraction constraint of the normal vector n can be expressed as:

[0091] (op3×op1)·n=0

[0092] As mentioned earlier, the relative positions of the event camera and the preset checkerboard can be kept constant, and images of the preset checkerboard can be acquired in air and water, respectively. Then, image processing can be used to obtain the image coordinates of the corner points of the preset checkerboard. Next, combining the calibrated intrinsic parameters of the event camera, the refraction constraint equation for the normal vector n can be obtained based on the image coordinates of each preset checkerboard corner point. Thus, using a series of preset checkerboard corner points, a linear system of equations An = 0 can be constructed. Furthermore, by performing singular value decomposition on matrix A, the normal vector n of the inner surface of the glass can be obtained.

[0093] The calculation process of the distance d0 from the optical center of the event camera to the inner surface of the glass can mainly include four steps:

[0094] First, based on a pre-defined checkerboard image captured in the air, along with the camera's intrinsic and extrinsic parameters, the 3D coordinates X3 of any corner point p3 on the pre-defined checkerboard in the camera coordinate system can be calculated. Then, based on the event camera's intrinsic parameters and the image coordinates of any corner point p3, the direction vector of op1 can be calculated. Furthermore, the coordinates of points p1 and p2 can be calculated using ray tracing, which contains only one unknown, d0. Next, the direction vectors of p2 and p3 can be calculated using ray tracing. Finally, based on the 3D coordinates of point p2, the 3D coordinates of point p3, and the direction vectors of p2 and p3, d0 can be calculated.

[0095]

[0096] Where X2 is the three-dimensional coordinate of point p2, X3 is the three-dimensional coordinate of point p3, and d1 is the thickness of the glass.

[0097] According to an exemplary embodiment of this disclosure, before acquiring the first and second images when the relative pose between the preset checkerboard and the event camera is the target pose, a third image can also be acquired under different poses between the preset checkerboard and the event camera. This third image can be an image captured by the event camera in image mode when both the structured light vision system and the preset checkerboard are placed in air. Then, based on the third images under different poses between the preset checkerboard and the event camera, the event camera parameters can be calibrated. Specifically:

[0098] Event camera parameter calibration and structured light plane equation parameter calibration in air can be performed simultaneously. First, the structured light vision system can be mounted on the robot's end effector. Then, the event camera can be set to image mode, allowing the acquisition of third images of a preset checkerboard pattern in different poses. Next, the acquired third images of the preset checkerboard pattern in different poses can be used to calibrate the event camera parameters. It should be noted that the laser is turned off during the camera parameter calibration process, and the camera's intrinsic and extrinsic parameters can be obtained using Zhang Zhengyou's calibration method.

[0099] In step 402, the parameters of the structured light plane equation in the air at the current angle can be calculated based on the parameters of the structured light plane equation in the air at the current angle and the pre-acquired refraction interface parameters. Specifically, the plane equation of the inner glass surface can be calculated first based on the refraction interface parameters, namely the glass surface normal vector n and the distance d0 from the optical center of the event camera to the inner glass surface. Then, based on the plane equation of the inner glass surface and the parameters of the structured light plane equation in the air at the current angle, the plane equation of the structured light after refraction through the inner glass surface can be calculated according to Snell's law. Similarly, the parameters of the plane equation of the structured light in the water after refraction through the outer glass surface can also be calculated. Specifically, the structured light plane model considering refraction can be calculated in the following way:

[0100] Assuming the unit normal vector n of the inner surface of the glass is (a, b, c), and the distance from the optical center of the event camera to the inner surface of the glass is d0, then the plane equation of the inner surface of the glass can be: ax + by + cz - d0 = 0. Assuming the equation of the structured light plane obtained after refraction through the inner surface of the glass is: A²x + B²y + C²z + D² = 0, then according to Snell's law, we have:

[0101]

[0102] Where, μ air μ represents the refractive index of air. glass Let θ1 represent the refractive index of the glass, θ2 represent the angle of incidence of light from air through the inner surface of the glass into the glass, and θ3 represent the angle of incidence of light from the outer surface of the glass into the water. The direction vector of the line of intersection between the structured light plane I in air and the inner surface of the glass can be expressed as: (B1c-bC1, aC1-cA1, bA1-aB1). The direction vector of the line of intersection between the structured light plane II and the inner surface of the glass after refraction through the inner surface of the glass can be expressed as: (B2c-bC2, aC2-cA2, bA2-aB2).

[0103] Furthermore, based on the fact that the two straight line direction vectors are the same, we can conclude that:

[0104]

[0105] Furthermore, considering that the normal vector (A2, B2, C2) of the structured light plane II is a unit vector, we can have:

[0106]

[0107] The equation for the plane beam passing through the intersection of the inner surface of the glass and the structured light plane I in the air can be:

[0108] (A1x+B1y+C1z+D1)+k1(ax+by+cz-d)=0

[0109] Therefore, we can have:

[0110] Let l1, m1, n1, L1, k1 be intermediate variables, which can be calculated using the following formula:

[0111]

[0112] Furthermore, λ1, γ1, α1, β1 can be set as intermediate variables, which can be calculated using the following formula:

[0113]

[0114] At this point, the parameters of the structured light plane II after refraction through the inner surface of the glass can be calculated as follows:

[0115]

[0116] Similarly, assuming the equation of the structured light plane entering the water after refraction through the outer surface of the glass is: A3x + B3y + C3z + D3 = 0, then the parameters of the structured light plane III in the water after refraction through the outer surface of the glass can be obtained as follows:

[0117]

[0118] Where d1 represents the thickness of the glass, and λ2, γ2, α2, β2 are intermediate variables. Their calculation process is similar to that of the aforementioned intermediate variables λ1, γ1, α1, β1, and will not be repeated here.

[0119] In step 403, the pixel coordinates p0(u, v) corresponding to the target underwater feature points on the underwater object at the current angle can be obtained. It should be noted that in this disclosure, the pixel coordinates p0(u, v) can be obtained based on the x-th element that appears most frequently in the event packet. i Determine the region of interest. Then, for each y within the region of interest... i It can calculate several x. iThe average value is used as the center of the region of interest.

[0120] According to an exemplary embodiment of this disclosure, pixel coordinate point p0 can be the two-dimensional coordinate point of the underwater feature point of the target underwater feature point obtained by the event camera in event mode after the laser emitted by the laser illuminates the galvanometer at the current angle and the galvanometer reflects the received laser to the target underwater feature point.

[0121] It should be noted that the event camera in this disclosure can have both image mode and event mode. In image mode, the camera captures a color image. Furthermore, in image mode, the information transmitted by the event camera is synchronized. "Synchronization" means that at a certain time t, the camera takes an exposure, fills a matrix with all the pixels at that moment, and transmits it back. In this way, a photograph is created, and all the pixels in that photograph correspond to the same moment.

[0122] In event mode, events are returned asynchronously. That is, unlike image mode which returns all pixel values ​​simultaneously, in event mode, different events are returned at different times. Specifically, within the entire camera's field of view, as long as a single pixel value changes, an event is returned, and all these events occur asynchronously.

[0123] In step 404, the ray tracing vector corresponding to the underwater feature point of the target can be calculated based on the pixel coordinate point p0 and the preset pinhole imaging model. (Refer to...) Figure 5 A ray tracing vector can contain vectors Ocp1, p1p2, and p2p3. Vector Ocp1 is refracted through the inner surface of the glass to obtain vector p1p2, and vector p1p2 is refracted through the outer surface of the glass to obtain vector p2p3. p1 is the intersection point of ray Ocp0 and the inner surface of the glass, and Oc is the optical center of the event camera. θ1 represents the angle of incidence of the light ray from air through the inner surface of the glass into the interior of the glass, θ2 represents the angle of incidence of the light ray from inside the glass through the outer surface of the glass into the water, and θ3 represents the angle of refraction of the light ray from inside the glass through the outer surface of the glass into the water.

[0124] In this disclosure, based on the image coordinates (u,v) of pixel point p0 and a preset pinhole imaging model, the direction vector Ocp0 in the air and the intersection point p1 of the direction vector Ocp0 in the air with the inner surface of the glass can be calculated. Then, based on Snell's law, the direction vector p1p2 obtained after refraction through the inner surface of the glass and the intersection point p2 of the direction vector p1p2 with the outer surface of the glass can be calculated. Next, also based on Snell's law, the direction vector p2p3 in the water obtained after refraction through the outer surface of the glass can be calculated. Specifically:

[0125] As mentioned earlier, the image point of an underwater feature point in the image plane of the event camera can be denoted as p0, and the ray... The intersection point with the inner surface of the glass is p1, and the ray... The rays formed after refraction through the inner surface of the glass are ray The rays formed after refraction through the outer surface of the glass are

[0126] If the image coordinates of point P0 are (u, v), then according to the preset pinhole imaging model, the following can be calculated: Direction vector For: ((u-u0) / f x ,(v-v0) / f y ,1), where (u0,v0) represents the optical center of the event camera.

[0127] The coordinates of point p1 in the camera coordinate system are:

[0128]

[0129] in, Let d be the normal vector of the inner surface of the glass, and d0 represent the distance from the optical center of the event camera to the inner surface of the glass.

[0130] Furthermore, due to all direction vectors of the refraction plane and normal vector Since they are all in the same plane, we can have:

[0131]

[0132] Under the aforementioned refraction constraint, the ray passing through the refracting surface is a linear combination of the direction vector of the previous ray and the normal vector of the refracting surface. Therefore, the ray... vector It can be represented as:

[0133]

[0134] Where u1 represents the refractive index in air and u2 represents the refractive index in glass.

[0135] Then, the three-dimensional coordinates of point p2 in the camera coordinate system can be calculated using the following formula:

[0136]

[0137] in, d1 is the normal vector to the inner surface of the glass, and d1 is the thickness of the glass.

[0138] Similarly, rays vector It can be represented as:

[0139]

[0140] Where u2 represents the refractive index of glass and u3 represents the refractive index of water.

[0141] In step 405, the three-dimensional coordinates of the target underwater feature point can be calculated based on the pixel coordinate point p0, the ray tracing vector, and the underwater structured light plane equation parameters at the current angle. Specifically:

[0142] Combining the coordinates (x2, y2, z2) of point p2 in the camera coordinate system and vector Given (l3, m3, n3), the equation of the straight line for the refracted ray p2p3 in water can be expressed as:

[0143]

[0144] Furthermore, since feature point p3 on the underwater object is located on structured light plane III in the water, the three-dimensional coordinates of underwater feature point p3 can be obtained by calculating the intersection of the refracted light rays p2p3 in the water and structured light plane III in the water. The specific calculation formula is as follows:

[0145]

[0146] The equation of the structured light plane that enters the water after refraction through the outer surface of the glass is: A3x + B3y + C3z + D3 = 0. Thus, by utilizing the structured light plane model that considers refraction and the event camera imaging model that considers refraction, the three-dimensional coordinates of each feature point on the underwater target can be calculated, thereby enabling three-dimensional reconstruction of the underwater target.

[0147] In this disclosure, considering the advantages of event cameras in high-speed and low-light scenarios, event cameras can be introduced into underwater structured light vision systems, thereby enabling fast, accurate, and robust 3D reconstruction of underwater moving objects in low-light conditions with weak textures. Furthermore, this disclosure proposes an underwater self-scanning structured light vision measurement model that simultaneously considers camera refraction and structured light plane refraction, and presents a simple and efficient model parameter calibration method, eliminating the need for complex underwater calibration processes, thus simplifying the model parameter calibration process and improving calibration efficiency.

[0148] Figure 8This is a block diagram illustrating a three-dimensional reconstruction apparatus 800 for an underwater object according to an exemplary embodiment of the present disclosure. The three-dimensional reconstruction apparatus 800 for the underwater object can be applied to a structured light vision system, which may include a galvanometer, a laser, an event camera, and glass.

[0149] Reference Figure 8 The three-dimensional reconstruction device 800 for the underwater object may include a parameter acquisition module 801, an underwater equation parameter calculation module 802, a pixel coordinate point acquisition module 803, a ray tracing vector calculation module 804, and a three-dimensional coordinate calculation module 805.

[0150] The parameter acquisition module 801 can acquire the parameters of the structured light plane equation in air at the current angle of the galvanometer, wherein the structured light vision system is placed underwater. The parameters of the structured light plane equation in air can refer to the four coefficients A, B, C, and D contained in the structured light plane equation in air: Ax + By + Cz + D = 0.

[0151] According to an exemplary embodiment of this disclosure, the parameter acquisition module 801 can acquire the rotation angle of the galvanometer from the previous angle to the current angle. Then, the parameter acquisition module 801 can determine the rotation matrix based on the rotation angle and the pre-calibrated scanning axis linear equation parameters. Next, the parameter acquisition module 801 can calculate the air structured light plane equation parameters at the current angle based on the air structured light plane equation parameters at the previous angle and the rotation matrix. It should be noted that the air structured light plane equation parameters at the initial galvanometer angle can use the galvanometer angle obtained during the pre-calibration of the scanning axis linear equation parameters as the air structured light plane equation parameters at the initial angle; this acquisition process will be described in detail later.

[0152] According to an exemplary embodiment of this disclosure, the three-dimensional reconstruction apparatus 800 for underwater objects may further include an air equation parameter acquisition module and a scan axis linear equation parameter calculation module.

[0153] The air equation parameter acquisition module can acquire the air structured light plane equation parameters for each angle at which the galvanometer is positioned. Then, the scan axis linear equation parameter calculation module can calculate the scan axis linear equation parameters based on the multiple air structured light plane equation parameters that correspond one-to-one with the multiple angles.

[0154] According to an exemplary embodiment of this disclosure, for each angle, the air equation parameter acquisition module can acquire laser stripe images at different poses between a preset checkerboard and an event camera. These laser stripe images can be obtained by the event camera capturing the preset checkerboard in image mode after the laser emitted by the laser is reflected back to the preset checkerboard by a galvanometer. Then, the air equation parameter acquisition module can calculate the air structured light plane equation parameters at that angle based on the laser stripe images at different poses between the preset checkerboard and the event camera. It should be noted that by changing the pose of the preset checkerboard relative to the event camera, the two laser stripes can be made non-collinear in space, thus allowing a structured light plane to be uniquely determined by two non-collinear laser stripe lines.

[0155] According to an exemplary embodiment of this disclosure, the refractive interface parameters may include the normal vector n of the inner surface of the glass and the distance d0 between the optical center of the event camera and the inner surface of the glass. The aforementioned three-dimensional reconstruction apparatus 800 for underwater objects may further include a first image and a second image acquisition module, a corner coordinate calculation module, a normal vector calculation module, and a distance calculation module.

[0156] The first and second image acquisition modules can acquire a first image and a second image when the relative pose between the preset checkerboard and the event camera is the target pose. The first image can be an image captured by the event camera in image mode when both the structured light vision system and the preset checkerboard are in air; the second image can be an image captured by the event camera in image mode when both the structured light vision system and the preset checkerboard are in water. Then, the corner coordinate calculation module can calculate the coordinates of the corner points of the preset checkerboard based on the first and second images. Next, the normal vector calculation module can calculate the normal vector n based on the coordinates of the corner points of the preset checkerboard and the pre-calibrated event camera parameters. Then, the distance calculation module can calculate the distance d0 between the optical center of the event camera and the inner surface of the glass based on the normal vector n. Furthermore, the distance d0 between the optical center of the event camera and the inner surface of the glass can be calculated based on the normal vector n combined with the event camera refraction model.

[0157] According to an exemplary embodiment of this disclosure, the above-described three-dimensional reconstruction apparatus 800 for underwater objects may further include a third image acquisition module and a camera parameter calibration module.

[0158] The third image acquisition module can acquire third images of the preset checkerboard and the event camera in different poses. These third images can be images captured by the event camera in image mode, with both the structured light vision system and the preset checkerboard placed in the air. Then, the camera parameter calibration module can calibrate the event camera parameters based on these third images of the preset checkerboard and the event camera in different poses.

[0159] The underwater equation parameter calculation module 802 can calculate the underwater structured light plane equation parameters at the current angle based on the structured light plane equation parameters in the air at the current angle and the pre-acquired refraction interface parameters.

[0160] The pixel coordinate point acquisition module 803 can acquire the pixel coordinate point p0(u, v) corresponding to the target underwater feature point on the underwater object at the current angle.

[0161] According to an exemplary embodiment of this disclosure, pixel coordinate point p0 can be the two-dimensional coordinate point of the underwater feature point of the target underwater feature point obtained by the event camera in event mode after the laser emitted by the laser illuminates the galvanometer at the current angle and the galvanometer reflects the received laser to the target underwater feature point.

[0162] The ray tracing vector calculation module 804 can calculate the ray tracing vector corresponding to the underwater feature point of the target based on the pixel coordinate point p0 and the preset pinhole imaging model.

[0163] The 3D coordinate calculation module 805 can calculate the 3D coordinates of underwater feature points of a target based on pixel coordinate point p0, ray tracing vector, and underwater structured light plane equation parameters at the current angle.

[0164] Figure 9 This is a block diagram illustrating an electronic device 900 according to an exemplary embodiment of the present disclosure.

[0165] Reference Figure 9 The electronic device 900 includes at least one memory 901 and at least one processor 902. The at least one memory 901 stores instructions that, when executed by the at least one processor 902, perform a three-dimensional reconstruction method for an underwater object according to an exemplary embodiment of the present disclosure.

[0166] As an example, electronic device 900 can be a PC, tablet, personal digital assistant, smartphone, or other device capable of executing the aforementioned instructions. Here, electronic device 900 is not necessarily a single electronic device, but can be any collection of devices or circuits capable of executing the aforementioned instructions (or instruction sets) individually or in combination. Electronic device 900 can also be part of an integrated control system or system manager, or can be configured to interconnect with a portable electronic device locally or remotely (e.g., via wireless transmission) through an interface.

[0167] In electronic device 900, processor 902 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller, or a microprocessor. By way of example and not limitation, processor may also include analog processors, digital processors, microprocessors, multi-core processors, processor arrays, network processors, etc.

[0168] The processor 902 can execute instructions or code stored in the memory 901, which can also store data. Instructions and data can also be sent and received via a network through a network interface device, which can employ any known transmission protocol.

[0169] The memory 901 may be integrated with the processor 902, for example, by placing RAM or flash memory within an integrated circuit microprocessor. Alternatively, the memory 901 may include a separate device, such as an external disk drive, a storage array, or other storage device usable by any database system. The memory 901 and the processor 902 may be operatively coupled, or may communicate with each other, for example, via I / O ports, network connections, etc., enabling the processor 902 to read files stored in the memory.

[0170] In addition, the electronic device 900 may also include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, mouse, touch input device, etc.). All components of the electronic device 900 can be interconnected via a bus and / or network.

[0171] According to exemplary embodiments of this disclosure, a computer-readable storage medium may also be provided, which, when executed by a processor of an electronic device, enables the electronic device to perform the aforementioned three-dimensional reconstruction method for underwater objects. Examples of computer-readable storage media include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R, BD-R The computer program can be stored in a computer-readable storage medium such as a BD-RE, Blu-ray or optical disc storage device, hard disk drive (HDD), solid-state drive (SSD), card storage (such as a multimedia card, secure digital (SD) card, or ultra-fast digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid-state drive, or any other device configured to store a computer program and any associated data, data files, and data structures in a non-transitory manner and to provide the computer program and any associated data, data files, and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the aforementioned computer-readable storage medium can run in an environment deployed in computer devices such as clients, hosts, agent devices, servers, etc. Furthermore, in one example, the computer program and any associated data, data files, and data structures are distributed across a networked computer system, such that the computer program and any associated data, data files, and data structures are stored, accessed, and executed in a distributed manner through one or more processors or computers.

[0172] According to exemplary embodiments of the present disclosure, a computer program product may also be provided, including a computer program that, when executed by a processor, implements a method for three-dimensional reconstruction of underwater objects according to the present disclosure.

[0173] Based on the 3D reconstruction method, apparatus, electronic device, storage medium, and computer program product for underwater objects disclosed herein, and considering the advantages of event cameras in high-speed and low-light scenarios, event cameras can be introduced into underwater structured light vision systems. This enables rapid, accurate, and robust 3D reconstruction of underwater moving objects in low-light conditions with weak textures. Furthermore, the structured light vision system of this disclosure has a simple construction, comprising only a laser, an event camera, and a galvanometer, without any additional hardware, resulting in low cost.

[0174] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0175] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A method for three-dimensional reconstruction of underwater objects, applied to a structured light vision system, characterized in that, The structured light vision system includes a galvanometer, a laser, an event camera, and glass; the 3D reconstruction method includes: Obtain the structured light plane equation parameters in air at the current angle of the galvanometer, wherein the structured light vision system is placed underwater; Based on the structured light plane equation parameters in air at the current angle and the pre-acquired refraction interface parameters, calculate the structured light plane equation parameters in water at the current angle. Obtain the pixel coordinates p0(u, v) corresponding to the target underwater feature point on the underwater object at the current angle; Based on the pixel coordinate point p0 and the preset pinhole imaging model, the ray tracing vector corresponding to the underwater feature point of the target is calculated. The ray tracing vector includes vector Ocp1, vector p1p2 and vector p2p3. Vector Ocp1 is refracted through the inner surface of the glass to obtain vector p1p2, and vector p1p2 is refracted through the outer surface of the glass to obtain vector p2p3. p1 is the intersection of ray Ocp0 and the inner surface of the glass, and Oc is the optical center of the event camera. Based on the pixel coordinate point p0, the ray tracing vector, and the underwater structured light plane equation parameters at the current angle, calculate the three-dimensional coordinates of the target underwater feature point; The step of obtaining the structured light plane equation parameters in air at the current angle of the galvanometer includes: Obtain the rotation angle of the galvanometer from the previous angle to the current angle; Based on the rotation angle and the pre-calibrated linear equation parameters of the scanning axis, the rotation matrix is ​​determined; Based on the air structured light plane equation parameters at the previous angle and the rotation matrix, calculate the air structured light plane equation parameters at the current angle; Wherein, the pixel coordinate point p0 is the two-dimensional coordinate point of the underwater feature point of the target under the imaging plane coordinate system of the event camera, obtained by the event camera in event mode after the laser emitted by the laser shines on the galvanometer at the current angle and the galvanometer reflects the received laser to the target underwater feature point.

2. The three-dimensional reconstruction method as described in claim 1, characterized in that, Before obtaining the rotation angle of the galvanometer from the previous angle to the current angle, the three-dimensional reconstruction method further includes: Obtain the structured light plane equation parameters in air at each of the multiple angles in which the galvanometer is positioned; Based on the multiple air structured light plane equation parameters that correspond one-to-one with the multiple angles, the scanning axis linear equation parameters are calculated.

3. The three-dimensional reconstruction method as described in claim 2, characterized in that, The process of obtaining the structured light plane equation parameters in air at each of the multiple angles of the galvanometer includes: For each angle, laser stripe images are acquired under different poses between the preset checkerboard and the event camera. The laser stripe images are obtained by the event camera capturing the preset checkerboard in image mode after the laser emitted by the laser shines on the galvanometer and the galvanometer reflects the received laser back to the preset checkerboard. Based on the laser stripe images at different poses between the preset checkerboard and the event camera, the parameters of the structured light plane equation in the air at that angle are calculated.

4. The three-dimensional reconstruction method as described in claim 1, characterized in that, The refractive interface parameters include the normal vector n of the inner surface of the glass and the distance d0 between the optical center of the event camera and the inner surface of the glass. The three-dimensional reconstruction method also includes: Acquire a first image and a second image when the relative pose between the preset chessboard and the event camera is the target pose. The first image is an image captured by the event camera in image mode when both the structured light vision system and the preset chessboard are in the air. The second image is an image captured by the event camera in image mode when both the structured light vision system and the preset chessboard are in the water. Based on the first image and the second image, calculate the coordinates of the corner points of the preset chessboard grid; Based on the coordinates of the corner points of the preset chessboard and the camera parameters of the event camera obtained through pre-calibration, the normal vector n is calculated; Based on the normal vector n, calculate the distance d0 between the optical center of the event camera and the inner surface of the glass.

5. The three-dimensional reconstruction method as described in claim 4, characterized in that, Before acquiring the first and second images when the relative pose between the preset chessboard grid and the event camera is the target pose, the 3D reconstruction method further includes: Acquire third images of the preset chessboard and the event camera under different poses, wherein the third image is an image captured by the event camera in the image mode when both the structured light vision system and the preset chessboard are placed in the air; The camera parameters of the event camera are calibrated based on the third images of the preset checkerboard and the event camera under different poses.

6. A three-dimensional reconstruction device for underwater objects, applied to a structured light vision system, characterized in that, The structured light vision system includes a galvanometer, a laser, an event camera, and glass; the 3D reconstruction device includes: The parameter acquisition module is configured to acquire the structured light plane equation parameters in the air at the current angle of the galvanometer, wherein the structured light vision system is placed underwater; The underwater equation parameter calculation module is configured to calculate the underwater structured light plane equation parameters at the current angle based on the air structured light plane equation parameters at the current angle and the pre-acquired refractive interface parameters. The pixel coordinate point acquisition module is configured to acquire the pixel coordinate point p0(u, v) corresponding to the target underwater feature point on the underwater object at the current angle. The ray tracing vector calculation module is configured to calculate the ray tracing vector corresponding to the underwater feature point of the target based on the pixel coordinate point p0 and the preset pinhole imaging model. The ray tracing vector includes vector Ocp1, vector p1p2 and vector p2p3. Vector Ocp1 is refracted through the inner surface of the glass to obtain vector p1p2, and vector p1p2 is refracted through the outer surface of the glass to obtain vector p2p3. p1 is the intersection of ray Ocp0 and the inner surface of the glass, and Oc is the optical center of the event camera. The three-dimensional coordinate calculation module is configured to calculate the three-dimensional coordinates of the target underwater feature point based on the pixel coordinate point p0, the ray tracing vector, and the underwater structured light plane equation parameters at the current angle. The parameter acquisition module is configured as follows: Obtain the rotation angle of the galvanometer from the previous angle to the current angle; Based on the rotation angle and the pre-calibrated linear equation parameters of the scanning axis, the rotation matrix is ​​determined; Based on the air structured light plane equation parameters at the previous angle and the rotation matrix, calculate the air structured light plane equation parameters at the current angle; Wherein, the pixel coordinate point p0 is the two-dimensional coordinate point of the underwater feature point of the target under the imaging plane coordinate system of the event camera, obtained by the event camera in event mode after the laser emitted by the laser shines on the galvanometer at the current angle and the galvanometer reflects the received laser to the target underwater feature point.

7. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the three-dimensional reconstruction method for underwater objects as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is able to perform the three-dimensional reconstruction method of an underwater object as described in any one of claims 1 to 5.

Images