Three-dimensional reconstruction method and device for underwater object, electronic equipment, storage medium and computer program product

Abstract

The invention relates to a three-dimensional reconstruction method and device for an underwater object, electronic equipment, a storage medium and a computer program product, and the method comprises the steps: calculating plane equation parameters of underwater structured light at a current angle based on plane equation parameters of the structured light in the air at the current angle and pre-obtained refraction interface parameters; acquiring a pixel coordinate point p0 (u, v) corresponding to the target underwater feature point on the underwater object at the current angle; calculating a ray tracing vector corresponding to the target underwater feature point based on the pixel coordinate point p0 and a preset pinhole imaging model; and calculating a three-dimensional coordinate of the target underwater feature point based on the pixel coordinate point p0, the ray tracing vector and the underwater structured light plane equation parameter at the current angle. Therefore, in consideration of the advantages of the event camera in a high-speed and low-illumination scene, the event camera can be introduced into an underwater structured light vision system, so that rapid, accurate and robust three-dimensional reconstruction of low-illumination, weak-texture and underwater moving objects can be realized.

Description

Technical Field

[0001] The present disclosure relates to the field of three-dimensional reconstruction technology, and more particularly, to a three-dimensional reconstruction method, device, electronic device, storage medium, and computer program product for underwater objects. Background Art

[0002] Underwater 3D reconstruction technology is widely used in underwater structure inspection, underwater topographic mapping, in-situ observation of marine life, underwater cultural relics and archaeology, and other fields. In addition, by obtaining three-dimensional information of the underwater environment, underwater robots can also better complete navigation and operation tasks. At present, underwater 3D reconstruction technology mainly includes acoustic-based methods and optical-based methods. Acoustic-based methods have the advantages of long measurement distance and are not affected by water turbidity. However, acoustic-based methods have disadvantages such as low measurement accuracy and low resolution, and are not suitable for high-precision dense underwater 3D reconstruction scenes. Optical-based methods have the advantages of high measurement accuracy, large amount of information, and low cost. They are currently the main technical means for high-precision 3D reconstruction at close range underwater. In addition, optical-based underwater 3D reconstruction methods can mainly include passive light vision methods and active light vision methods.

[0003] Underwater passive light vision methods use only ambient light or underwater lighting to illuminate the scene. The camera then acquires images from various poses. Three-dimensional reconstruction is then performed based on image feature matching. However, this computational process relies heavily on the target's texture features and is computationally intensive. Furthermore, image feature matching is particularly challenging in low-light and turbid underwater environments.

[0004] Underwater active light vision methods actively project specific patterns onto the target to achieve three-dimensional measurement. As a typical example of active light vision methods, underwater structured light vision methods can actively project specific patterns onto the object being measured. A camera can then capture information about the pattern modulated by the object. Next, underwater three-dimensional reconstruction can be achieved by combining it with the principle of triangulation. Underwater structured light vision methods offer advantages such as high measurement accuracy and robustness, and have garnered widespread attention from researchers in recent years. Furthermore, underwater structured light vision methods typically use blue-green lasers as active light sources, which offer the advantages of high collimation and long propagation distance in water.

[0005] In related technologies, the cameras used in underwater structured light vision systems are all traditional fixed-frame-rate, global-exposure cameras. However, in reality, the proportion of structured light information in the sensor pixels is very small, resulting in most invalid pixels still occupying a large amount of bandwidth resources and processor resources. This will lead to low measurement efficiency of the underwater structured light vision system and is not suitable for three-dimensional reconstruction of underwater moving targets. Summary of the Invention

[0006] The present disclosure provides a method, device, electronic device, storage medium and computer program product for three-dimensional reconstruction of underwater objects, so as to at least solve the problem in the above-mentioned related technologies that the measurement efficiency of underwater structured light vision systems is low and they are not suitable for three-dimensional reconstruction of underwater moving targets.

[0007] According to a first aspect of an embodiment of the present disclosure, a three-dimensional reconstruction method for underwater objects is provided, which is applied to a structured light vision system, wherein the structured light vision system includes a galvanometer, a laser, an event camera, and glass. The three-dimensional reconstruction method includes: obtaining the parameters of the structured light plane equation in the air at a current angle of the galvanometer, wherein the structured light vision system is placed underwater; calculating the parameters of the structured light plane equation in the water at the current angle based on the parameters of the structured light plane equation in the air at the current angle and the pre-acquired refraction interface parameters; obtaining the pixel coordinate point p0 (u, v) corresponding to the target underwater feature point on the underwater object at the current angle; and calculating the parameters of the structured light plane equation in the water at the current angle based on the pixel coordinate point p0 and the pre-acquired refraction interface parameters. A pinhole imaging model is assumed to calculate the ray tracing vector corresponding to the target underwater feature point, wherein the ray tracing vector includes an Ocp1 vector, a p1p2 vector and a p2p3 vector, the Ocp1 vector is refracted by the inner surface of the glass to obtain the p1p2 vector, the p1p2 vector is refracted by the outer surface of the glass to obtain the p2p3 vector, p1 is the intersection of the ray Ocp0 and the inner surface of the glass, and Oc is the optical center of the event camera; based on the pixel coordinate point p0, the ray tracing vector and the parameters of the underwater structured light plane equation at the current angle, the three-dimensional coordinates of the target underwater feature point are calculated.

[0008] Optionally, obtaining the parameters of the structured light plane equation in the air when the galvanometer is at the current angle includes: obtaining the rotation angle of the galvanometer from the previous angle to the current angle; determining the rotation matrix based on the rotation angle and pre-calibrated scanning axis straight line equation parameters; and calculating the parameters of the structured light plane equation in the air at the current angle based on the parameters of the structured light plane equation in the air at the previous angle and the rotation matrix.

[0009] Optionally, before obtaining the rotation angle of the galvanometer from the previous angle to the current angle, the three-dimensional reconstruction method also includes: obtaining the parameters of the structured light plane equation in the air at each of the multiple angles of the galvanometer; and calculating the parameters of the scanning axis straight line equation based on multiple parameters of the structured light plane equation in the air corresponding one-to-one to the multiple angles.

[0010] Optionally, obtaining the parameters of the structured light plane equation in the air at each of the multiple angles of the galvanometer includes: for each angle, obtaining a laser stripe image at different postures between a preset checkerboard and the event camera, wherein the laser stripe image is an image obtained by the event camera photographing the preset checkerboard in image mode after the laser emitted by the laser is irradiated onto the galvanometer and the galvanometer reflects the received laser to the preset checkerboard; and calculating the parameters of the structured light plane equation in the air at the angle based on the laser stripe image at different postures between the preset checkerboard and the event camera.

[0011] Optionally, the refractive interface parameters include the normal vector n of the inner surface of the glass and the distance d0 between the optical center of the event camera and the inner surface of the glass; the three-dimensional reconstruction method also includes: obtaining a first image and a second image when the relative posture between the preset checkerboard and the event camera is the target posture, wherein the first image is an image obtained by the event camera shooting the preset checkerboard in the image mode when the structured light vision system and the preset checkerboard are both placed in the air, and the second image is an image obtained by the event camera shooting the preset checkerboard in the image mode when the structured light vision system and the preset checkerboard are both placed in water; based on the first image and the second image, calculating the coordinates of the corner points of the preset checkerboard; based on the coordinates of the corner points of the preset checkerboard and the camera parameters of the event camera obtained in advance, calculating the normal vector n; based on the normal vector n, calculating the distance d0 between the optical center of the event camera and the inner surface of the glass.

[0012] Optionally, before obtaining the first image and the second image when the relative posture between the preset checkerboard and the event camera is the target posture, the three-dimensional reconstruction method further includes: obtaining a third image at a different posture between the preset checkerboard and the event camera, wherein the third image is an image obtained by the event camera shooting the preset checkerboard in the image mode when the structured light vision system and the preset checkerboard are both placed in the air; and calibrating the camera parameters of the event camera based on the third image at different postures between the preset checkerboard and the event camera.

[0013] Optionally, the pixel coordinate point p0 is the two-dimensional coordinate point of the target underwater feature point in the imaging plane coordinate system of the event camera obtained by photographing the target underwater feature point in the event mode after the laser emitted by the laser is irradiated on the galvanometer at the current angle and the galvanometer reflects the received laser to the target underwater feature point.

[0014] According to a second aspect of an embodiment of the present disclosure, a three-dimensional reconstruction device for underwater objects is provided, which is applied to a structured light vision system, wherein the structured light vision system includes a galvanometer, a laser, an event camera and glass, and the three-dimensional reconstruction device includes: a parameter acquisition module, configured to obtain the parameters of the structured light plane equation in the air at the current angle of the galvanometer, wherein the structured light vision system is placed underwater; an underwater equation parameter calculation module, configured to calculate the parameters of the structured light plane equation in the water at the current angle based on the parameters of the structured light plane equation in the air at the current angle and the pre-acquired refraction interface parameters; a pixel coordinate point acquisition module, configured to obtain the pixel coordinate point p0 (u, v) corresponding to the target underwater feature point on the underwater object at the current angle; a ray tracing vector meter A calculation module is configured to calculate the ray tracing vector corresponding to the target underwater feature point based on the pixel coordinate point p0 and a preset pinhole imaging model, wherein the ray tracing vector includes an Ocp1 vector, a p1p2 vector and a p2p3 vector, the Ocp1 vector is refracted by the inner surface of the glass to obtain the p1p2 vector, the p1p2 vector is refracted by the outer surface of the glass to obtain the p2p3 vector, p1 is the intersection of the ray Ocp0 and the inner surface of the glass, and Oc is the optical center of the event camera; a three-dimensional coordinate calculation module is configured to calculate the three-dimensional coordinates of the target underwater feature point based on the pixel coordinate point p0, the ray tracing vector and the parameters of the underwater structured light plane equation at the current angle.

[0015] Optionally, the parameter acquisition module is configured to: obtain the rotation angle of the galvanometer from the previous angle to the current angle; determine the rotation matrix based on the rotation angle and the pre-calibrated scanning axis straight line equation parameters; and calculate the parameters of the structured light plane equation in the air at the current angle based on the parameters of the structured light plane equation in the air at the previous angle and the rotation matrix.

[0016] Optionally, the three-dimensional reconstruction device also includes: an air equation parameter acquisition module, configured to obtain the air structured light plane equation parameters at each of a plurality of angles of the galvanometer; a scanning axis straight line equation parameter calculation module, configured to calculate the scanning axis straight line equation parameters based on a plurality of air structured light plane equation parameters corresponding one-to-one to the plurality of angles.

[0017] Optionally, the air equation parameter acquisition module is configured to: for each angle, obtain laser stripe images in different postures between a preset checkerboard and the event camera, wherein the laser stripe image is an image obtained by the event camera photographing the preset checkerboard in image mode after the laser emitted by the laser is irradiated onto the galvanometer and the galvanometer reflects the received laser to the preset checkerboard; based on the laser stripe images in different postures between the preset checkerboard and the event camera, calculate the parameters of the structured light plane equation in the air at the angle.

[0018] Optionally, the refractive interface parameters include a normal vector n of the inner surface of the glass and a distance d0 between the optical center of the event camera and the inner surface of the glass; the three-dimensional reconstruction device also includes: a first image and a second image acquisition module, configured to acquire a first image and a second image when the relative posture between the preset checkerboard and the event camera is a target posture, wherein the first image is an image obtained by the event camera shooting the preset checkerboard in an image mode when the structured light vision system and the preset checkerboard are both placed in the air, and the second image is an image obtained by the event camera shooting the preset checkerboard in the image mode when the structured light vision system and the preset checkerboard are both placed in water; a corner point coordinate calculation module, configured to calculate the coordinates of the corner points of the preset checkerboard based on the first image and the second image; a normal vector calculation module, configured to calculate the normal vector n based on the coordinates of the corner points of the preset checkerboard and the camera parameters of the event camera obtained in advance; a distance calculation module, configured to calculate the distance d0 between the optical center of the event camera and the inner surface of the glass based on the normal vector n.

[0019] Optionally, the three-dimensional reconstruction device also includes: a third image acquisition module, configured to acquire a third image in different postures between the preset checkerboard and the event camera, wherein the third image is an image obtained by the event camera shooting the preset checkerboard in the image mode when the structured light vision system and the preset checkerboard are both placed in the air; a camera parameter calibration module, configured to calibrate the camera parameters of the event camera based on the third image in different postures between the preset checkerboard and the event camera.

[0020] Optionally, the pixel coordinate point p0 is the two-dimensional coordinate point of the target underwater feature point in the imaging plane coordinate system of the event camera obtained by photographing the target underwater feature point in the event mode after the laser emitted by the laser is irradiated on the galvanometer at the current angle and the galvanometer reflects the received laser to the target underwater feature point.

[0021] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement the three-dimensional reconstruction method of an underwater object according to the present disclosure.

[0022] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the three-dimensional reconstruction method of underwater objects according to the present disclosure.

[0023] The technical solutions provided by the embodiments of the present disclosure bring at least the following beneficial effects:

[0024] In this disclosure, considering the advantages of event cameras in high-speed and low-light scenarios, they can be incorporated into underwater structured light vision systems. This allows for fast, accurate, and robust 3D reconstruction of low-light, weak-texture, and underwater moving objects. Furthermore, the structured light vision system disclosed in this disclosure is simple in construction, consisting of only a laser, an event camera, and a galvanometer, without any additional hardware, resulting in low cost.

[0025] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The accompanying drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute an improper limitation of the present disclosure.

[0027] Figure 1 is a schematic structural diagram illustrating a structured light vision system according to an exemplary embodiment of the present disclosure;

[0028] Figure 2 is a schematic diagram illustrating an implementation flow of underwater 3D reconstruction based on event structured light according to an exemplary embodiment of the present disclosure;

[0029] Figure 3 is a schematic diagram illustrating a calibration process of event structured light parameters according to an exemplary embodiment of the present disclosure;

[0030] Figure 4 is a flowchart illustrating a method for three-dimensional reconstruction of an underwater object according to an exemplary embodiment of the present disclosure;

[0031] Figure 5 is a schematic diagram illustrating an underwater three-dimensional measurement process using event structured light according to an exemplary embodiment of the present disclosure;

[0032] Figure 6 is a schematic diagram illustrating a rotational optical path transformation of a galvanometer mirror according to an exemplary embodiment of the present disclosure;

[0033] Figure 7 is a schematic diagram illustrating a calibration process of refractive interface parameters according to an exemplary embodiment of the present disclosure;

[0034] Figure 8 is a block diagram illustrating a three-dimensional reconstruction apparatus for an underwater object according to an exemplary embodiment of the present disclosure;

[0035] Figure 9 is a block diagram illustrating an electronic device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0036] In order to enable ordinary people in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0037] It should be noted that the terms "first," "second," and the like in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the numbers used in this manner are interchangeable where appropriate so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The implementation methods described in the following examples do not represent all implementation methods consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with certain aspects of the present disclosure as detailed in the appended claims.

[0038] It should be noted that the phrase "at least one of the items" in this disclosure includes three types of parallel situations: "any one of the items", "a combination of any multiple items of the items", and "all of the items". For example, "including at least one of A and B" includes the following three parallel situations: (1) including A; (2) including B; (3) including A and B. For another example, "performing at least one of step 1 and step 2" includes the following three parallel situations: (1) performing step 1; (2) performing step 2; and (3) performing steps 1 and 2.

[0039] Figure 1 FIG. 4 is a schematic diagram illustrating the structure of a structured light vision system according to an exemplary embodiment of the present disclosure.

[0040] Reference Figure 1 The structured light vision system provided by the present disclosure may include a structured light projector, a scanning galvanometer, glass, an event camera and a filter.

[0041] A structured light projector, also known as a laser, is primarily used to project laser stripes. The laser can be, but is not limited to, blue-green light, which has low attenuation in water. For example, the laser's wavelength can be 450nm and its power can be 400mW.

[0042] Galvanometer: It can rotate at high speed, thereby achieving rapid scanning of laser stripes, and the scanning angle can be controlled by input voltage level.

[0043] Event camera: It is a bio-inspired sensor, which is mainly used to obtain event flow information during laser stripe scanning. Specifically, it can output asynchronous signals (including events, positions, and signs of brightness changes) by measuring the brightness changes of each pixel, suppressing redundant information. Compared with traditional cameras, event cameras have the characteristics of high temporal resolution, low power consumption and high dynamic range, and have a broad application space in high-speed, high dynamic range scenarios. In addition, the event camera may also include an optical lens for imaging, and a narrow-band filter with a central wavelength consistent with the wavelength of the projected structured light may be installed in front of the lens. Exemplarily, in the present disclosure, the pixels of the event frame may be: 346×260, and the central wavelength of the narrow-band filter in front of the lens may be 450nm.

[0044] It should be noted that the above-mentioned structured light vision system can also be divided into: a structured light parameter calibration module, an event information processing module and a three-dimensional reconstruction module, and the three-dimensional reconstruction of low-light, weak-texture underwater moving objects can be achieved by executing the above modules in sequence.

[0045] Figure 2 FIG. 4 is a schematic diagram illustrating an implementation flow of underwater 3D reconstruction based on event structured light according to an exemplary embodiment of the present disclosure.

[0046] Reference Figure 2 The structured light parameter calibration module can include camera parameter calibration, structured light plane equation parameter calibration in air, refraction interface parameter calibration, and scanning axis straight line equation parameter calibration; the event information processing module can include collecting event streams, preprocessing event streams, grouping event streams, and feature extraction of grouped event streams; the three-dimensional reconstruction module can include a self-scanning structured light visual measurement model, a structured light plane model considering refraction, a camera imaging model considering refraction, and three-dimensional coordinate calculation of underwater feature points.

[0047] Figure 3 FIG. 4 is a schematic diagram illustrating a calibration process of event structured light parameters according to an exemplary embodiment of the present disclosure.

[0048] Reference Figure 3The structured light parameter calibration module can include camera parameter calibration, air structured light plane equation parameter calibration, refractive interface parameter calibration, and scanning axis line equation parameter calibration. Specifically, camera parameter calibration can include focal length, principal point image coordinates, and pixel size; refractive interface parameter calibration can include the normal vector of the glass inner surface and the distance from the camera optical center to the glass inner surface.

[0049] The event information processing module can mainly include collecting event streams, preprocessing event streams, grouping event streams, and extracting features from grouped event streams. Specifically, during the rapid scanning of the laser line, the event camera can generate event stream information, which can include (x i ,y i ,t i ) information, where x i and y i Indicates that an event occurs at this location, that is, the light change exceeds the threshold, t i Indicates the time when the event occurred. Event stream preprocessing removes noise from the event stream. Typically, since the event points generated by laser line scanning are adjacent and continuous, isolated event points are considered noise and therefore need to be removed. Event stream grouping groups events by timestamp, with the event stream within a certain time interval being considered an event packet. Finally, feature extraction can be performed on the grouped event packets to obtain the image coordinates of the laser streak center point.

[0050] Figure 4 1 is a flow chart illustrating a method for 3D reconstruction of an underwater object according to an exemplary embodiment of the present disclosure, which is applied to a structured light vision system. As mentioned above, the structured light vision system may include a galvanometer, a laser, an event camera, and glass.

[0051] Reference Figure 4 In step 401, parameters of the structured light plane equation in air can be obtained when the galvanometer is at a current angle, wherein the structured light vision system is placed underwater. The parameters of the structured light plane equation in air can refer to the four coefficients A, B, C, and D contained in the structured light plane equation in air: Ax+By+Cz+D=0.

[0052] According to an exemplary embodiment of the present disclosure, the rotation angle of the galvanometer from the previous angle to the current angle can be obtained. Then, the rotation matrix can be determined based on the rotation angle and the pre-calibrated scanning axis straight line equation parameters. Next, the parameters of the structured light plane equation in the air at the current angle can be calculated based on the parameters of the structured light plane equation in the air at the previous angle and the rotation matrix. It should be noted that the parameters of the structured light plane equation in the air at the initial galvanometer angle can be obtained using the parameters of the structured light plane equation in the air at the initial angle obtained in the process of pre-calibrating the scanning axis straight line equation parameters. This acquisition process will be described in detail later.

[0053] Figure 2 The self-scanning structured light vision measurement model in can be obtained in the following ways, specifically:

[0054] Figure 5 FIG is a schematic diagram illustrating an underwater three-dimensional measurement process of event structured light according to an exemplary embodiment of the present disclosure. Figure 5 , the coordinate system of the event camera can be o c x c y c z c , where o c is the optical center of the event camera, z c The x axis is the optical axis of the event camera, c axis and y c The axis is parallel to the u-axis and v-axis of the event camera image plane. The point corresponding to a feature point on the underwater object on the image plane of the event camera can be defined as p0. c The intersection of p0 and the inner surface of the glass is p1,o c The refracted light of p0 passing through the inner surface of the glass is p1p2, and the refracted light of p1p2 passing through the outer surface of the glass is p2p3. In addition, the galvanometer coordinate system can be defined as o w x w y w z w , where x w Axis is the rotation axis of the galvanometer.

[0055] Assume that the image coordinates of point p0 are (u, v), and the coordinates of this point in the camera coordinate system are (x c ,y c ,z c ), then according to the pinhole imaging model we can have:

[0056]

[0057] Among them, (f x ,f y) are two magnification coefficients, (u0, v0) are the image coordinates of the camera principal point, M c is the camera intrinsic parameter matrix.

[0058] Assuming that the galvanometer angle is 0 at the initial moment, the plane equation of the structured light in the air can be: A0x+B0y+C0z+D0=0, where A0, B0, C0, and D0 can be parameters of the plane equation of the structured light in the air.

[0059] Figure 6 Schematic diagram showing the rotational optical path transformation of the galvanometer according to an exemplary embodiment of the present disclosure. Figure 6 , when the galvanometer rotation angle is θ / 2, the structured light plane rotation angle is θ. Therefore, the parameters of the structured light plane equation in the air after rotation can be:

[0060]

[0061] Among them, A1, B1, and C1 are the parameters of the structured light plane equation in the air after rotation: A1x+B1y+C1z+D1=0, (s x ,s y ,s z ) is the direction vector of the galvanometer's rotation axis.

[0062] Since a point (x0, y0, z0) on the rotation axis is located on the rotated structured light plane, A1x0+B1y0+C1z0+D1=0. Therefore, we can calculate: D1=-(A1x0+B1y0+C1z0). Then, combining the event camera imaging model and the equation of the structured light plane in the air after rotation, we can obtain the structured light vision measurement model in the air after the galvanometer rotates:

[0063]

[0064] According to an exemplary embodiment of the present disclosure, before obtaining the rotation angle of the galvanometer from the previous angle to the current angle, the parameters of the structured light plane equation in the air at each of the multiple angles of the galvanometer can also be obtained. Then, based on the multiple parameters of the structured light plane equation in the air corresponding to the multiple angles, the parameters of the scanning axis straight line equation can be calculated. Since the points on the scanning axis are located on the structured light plane, the least squares method can be used to fit the structured light plane, and the coordinates of a point on the scanning axis can be obtained, and then the parameters of the scanning axis straight line equation can be determined. Specifically, the calibration process of the scanning axis straight line equation parameters can be as follows:

[0065] Assume that the linear equation of the galvanometer rotation axis is:

[0066]

[0067] Among them, (s x ,s y ,s z ) is the direction vector of the rotation axis, (x0, y0, z0) is a point on the rotation axis, and t represents a straight line.

[0068] Using the air structured light plane equation parameter calibration method, the parameters of the plane equation at each galvanometer scanning angle can be obtained. The specific calibration process will be explained in detail later. Then, the galvanometer can be rotated to multiple angles, using four angles as an example. At this point, a set of structured light plane equation parameters can be determined at each angle, and the scanning axis line vector can be determined through least squares fitting.

[0069]

[0070] Among them, (A 0i ,B 0i ,C 0i ) is the normal vector of the structured light plane equation at the i-th angle. By performing singular value decomposition on the matrix X, the scanning axis line vector can be obtained.

[0071] In addition, since a point (x0, y0, z0) on the rotation axis is located on the structured light plane, we can have:

[0072]

[0073] (x0, y0, z0) can be obtained through least squares fitting.

[0074] According to an exemplary embodiment of the present disclosure, for each angle, a laser stripe image in different positions between a preset checkerboard and an event camera can be obtained, wherein the laser stripe image can be an image obtained by shooting the preset checkerboard by the event camera in image mode after the laser emitted by the laser is irradiated onto the galvanometer and the galvanometer reflects the received laser to the preset checkerboard. Then, based on the laser stripe image in different positions between the preset checkerboard and the event camera, the parameters of the structured light plane equation in the air at that angle can be calculated. It should be noted that by changing the position of the preset checkerboard relative to the event camera, the two laser stripes can be made non-collinear in space, and then a structured light plane can be uniquely determined by two non-collinear laser stripe straight lines.

[0075] The following describes the process of controlling the movement of the preset checkerboard to change the position of the preset checkerboard relative to the event camera. Specifically:

[0076] Assuming that at the initial position, the straight line of the laser stripe can be recorded as L1. After changing the relative position of the preset checkerboard and the event camera, the straight line of the laser stripe can be recorded as L2. Based on the positional relationship of the spatial lines, the positional relationship between L1 and L2 can have three situations: if L1 and L2 are collinear, the structured light plane cannot be uniquely determined; if L1 and L2 are parallel, the structured light plane can be uniquely determined; and if L1 and L2 intersect, the structured light plane can be uniquely determined. Assuming that the structured light vision sensor is fixed, when the preset checkerboard is rotated and translated within its own plane, L1 and L2 are collinear. Therefore, in order to make L1 and L2 non-collinear, the preset checkerboard should be moved along the z direction or rotated around the x and y axes.

[0077] It should be noted that during the calibration of the event camera parameters, the external parameters of the event camera can be obtained. It can be used to express the transformation relationship between the event camera coordinate system and the preset checkerboard coordinate system:

[0078]

[0079] Among them, [a x a y a z ] represents the normal vector of the preset chessboard plane, [p x p y p z ] represents the coordinate origin, [n x , n y , n z ] represents the direction vector of the x-axis of the camera coordinate system in the preset checkerboard coordinate system, [o x , o y , o z ] represents the direction vector of the y-axis of the camera coordinate system in the preset checkerboard coordinate system, [x t ,y t , z t ] represents the coordinate point in the preset chessboard coordinate system.

[0080] Assume that the coordinates of any point in the preset chessboard plane can be expressed as [x c y c z c ], then the following formula can be established:

[0081] a x (x c -p x )+a y (y c -p y )+a z (z c -p z )=0

[0082] Combined with the event camera imaging model, the coordinate values ​​of the laser stripe feature points in the event camera coordinate system can be obtained:

[0083]

[0084] x c =z c (u-u0) / k x

[0085] y c =z c (v-v0) / k y

[0086] Among them, k x 、k y is the internal parameter of the event camera, that is, the focal length of the event camera.

[0087] Next, the structured light plane equation can be solved by performing a least-squares fit on three or more non-collinear laser stripe points. Furthermore, to improve calibration accuracy, the preset checkerboard (i.e., the calibration plate and event camera) can be calibrated in multiple relative positions to obtain multiple non-collinear laser stripes. The parameters of the structured light plane equation can then be solved by performing a least-squares fit on all points on the laser stripes.

[0088] According to an exemplary embodiment of the present disclosure, the refractive interface parameters may include a normal vector n of the inner surface of the glass and a distance d0 between the optical center of the event camera and the inner surface of the glass.

[0089] In this case, the three-dimensional reconstruction method of an underwater object according to an exemplary embodiment of the present disclosure may further include the following steps. First, a first image and a second image may be obtained when the relative posture between the preset checkerboard and the event camera is the target posture. The first image may be an image obtained by shooting the preset checkerboard by the event camera in image mode when the structured light vision system and the preset checkerboard are both placed in the air; the second image may be an image obtained by shooting the preset checkerboard by the event camera in image mode when the structured light vision system and the preset checkerboard are both placed in water. Then, based on the first image and the second image, the coordinates of the corner points of the preset checkerboard may be calculated. Next, the normal vector n may be calculated based on the coordinates of the corner points of the preset checkerboard and the event camera parameters of the event camera obtained by pre-calibration. Then, based on the normal vector n, the distance d0 between the optical center of the event camera and the inner surface of the glass may be calculated. Further, based on the normal vector n combined with the event camera refraction model, the distance d0 between the optical center of the event camera and the inner surface of the glass may be calculated, specifically:

[0090] Figure 7 FIG. 1 is a schematic diagram illustrating a calibration process of refractive interface parameters according to an exemplary embodiment of the present disclosure. Figure 7 , due to the refraction of the medium, op3 and op1 should be on the same plane. Therefore, the refraction constraint of the normal vector n can be expressed as:

[0091] (op3×op1)·n=0

[0092] As mentioned above, the relative position of the event camera and the preset checkerboard grid can be fixed, and images of the preset checkerboard grid can be acquired in air and water. The image coordinates of the preset checkerboard grid corner points can then be obtained through image processing. Next, combined with the calibrated intrinsic parameters of the event camera, the refraction constraint equation for the normal vector n can be derived based on the image coordinates of each preset checkerboard grid corner point. Thus, using a series of preset checkerboard grid corner points, a linear equation system An = 0 can be constructed. Furthermore, by performing singular value decomposition on the matrix A, the normal vector n of the inner surface of the glass can be obtained.

[0093] The calculation process of the distance d0 from the optical center of the event camera to the inner surface of the glass can mainly include four steps:

[0094] First, the three-dimensional coordinates X3 of any corner point p3 on the preset chessboard in the camera coordinate system can be calculated based on the preset chessboard image taken in the air and the intrinsic and extrinsic parameters of the camera. Then, the direction vector of op1 can be calculated based on the intrinsic parameters of the event camera and the image coordinates of any corner point p3. In addition, the coordinates of points p1 and p2 can be calculated using the ray tracing method, which only contains one unknown number d0. Next, the direction vector of p2p3 can be calculated using the ray tracing method. Finally, d0 can be calculated based on the three-dimensional coordinates of point p2, the three-dimensional coordinates of point p3, and the direction vector of p2p3:

[0095]

[0096] Among them, X2 is the three-dimensional coordinate of point p2, X3 is the three-dimensional coordinate of point p3, and d1 is the thickness of the glass.

[0097] According to an exemplary embodiment of the present disclosure, before acquiring the first and second images when the relative pose between the preset checkerboard and the event camera is the target pose, a third image may be acquired at a different pose between the preset checkerboard and the event camera. The third image may be an image captured by the event camera in image mode, with the structured light vision system and the preset checkerboard both placed in air. The event camera parameters of the event camera may then be calibrated based on the third image at different poses between the preset checkerboard and the event camera. Specifically,

[0098] The calibration of event camera parameters and the calibration of the parameters of the structured light plane equation in air can be completed simultaneously. First, the structured light vision system can be installed on the end of the robot. Then, the event camera can be set to image mode, and then a third image of the preset checkerboard in different poses can be collected. Next, the camera parameters of the event camera can be calibrated using the collected third image of the preset checkerboard in different poses. It should be noted that the laser is turned off during the calibration of the camera parameters, and the intrinsic and extrinsic parameters of the camera can be obtained using the Zhang Zhengyou calibration method.

[0099] In step 402, the parameters of the structured light plane equation in water at the current angle can be calculated based on the parameters of the structured light plane equation in the air at the current angle and the pre-acquired refraction interface parameters. That is, the plane equation of the inner surface of the glass can be calculated based on the refraction interface parameters, i.e., the normal vector n of the glass surface and the distance d0 from the optical center of the event camera to the inner surface of the glass; then, based on the plane equation of the inner surface of the glass and the parameters of the structured light plane equation in the air at the current angle, the structured light plane equation after refraction from the inner surface of the glass can be calculated according to Snell's law. Similarly, the parameters of the structured light plane equation in water after refraction from the outer surface of the glass can also be calculated. Specifically, the structured light plane model considering refraction can be calculated in the following way:

[0100] Assuming the unit normal vector n of the inner surface of the glass is (a, b, c), and the distance from the optical center of the event camera to the inner surface of the glass is d0, then the plane equation of the inner surface of the glass can be: ax+by+cz-d0=0. Assuming the plane equation of the structured light obtained after refraction on the inner surface of the glass is: A2x+B2y+C2z+D2=0, then according to Snell's law, we can have:

[0101]

[0102] Among them, μ air represents the refractive index in air, μ glass represents the refractive index in the glass, θ1 represents the incident angle of light from the air through the inner surface of the glass to the glass, θ2 represents the incident angle of light from the glass through the outer surface of the glass to the water, the straight line direction vector of the intersection line of the structured light plane I in the air and the inner surface of the glass can be expressed as: (B1c-bC1, aC1-cA1, bA1-aB1), and the straight line direction vector of the intersection line of the structured light plane II and the inner surface of the glass after refraction from the inner surface of the glass can be expressed as: (B2c-bC2, aC2-cA2, bA2-aB2).

[0103] In addition, based on the fact that the direction vectors of the two straight lines mentioned above are consistent, we can have:

[0104]

[0105] Furthermore, considering that the normal vector (A2, B2, C2) of the structured light plane II is a unit vector, we can have:

[0106]

[0107] The plane beam equation of the intersection line passing through the inner surface of the glass and the structured light plane I in the air can be:

[0108] (A1x+B1y+C1z+D1)+k1(ax+by+cz-d)=0

[0109] Therefore, we can have:

[0110] Among them, let l1, m1, n1, L1, k1 be intermediate variables, which can be calculated by the following formula:

[0111]

[0112] Furthermore, we can set λ1, γ1, α1, and β1 as intermediate variables, which can be calculated using the following formula:

[0113]

[0114] At this point, the parameters of the structured light plane II after refraction through the inner surface of the glass can be calculated as:

[0115]

[0116] Similarly, assuming that the equation of the structured light plane that enters the water after being refracted by the outer surface of the glass is: A3x+B3y+C3z+D3=0, the parameters of the structured light plane III in the water after being refracted by the outer surface of the glass can be obtained as:

[0117]

[0118] Among them, d1 represents the thickness of the glass, λ2, γ2, α2, and β2 are intermediate variables, and their calculation process is similar to that of the aforementioned intermediate variables λ1, γ1, α1, and β1, which will not be repeated here.

[0119] In step 403, the pixel coordinate point p0(u, v) corresponding to the target underwater feature point on the underwater object at the current angle can be obtained. It should be noted that in the present disclosure, the pixel coordinate point p0(u, v) corresponding to the target underwater feature point on the underwater object at the current angle can be obtained based on the x that appears the most times in the event packet. i Determine the region of interest. Then, for each y in the region of interest i , you can calculate several x iThe average value of is taken as the center of the region of interest.

[0120] According to an exemplary embodiment of the present disclosure, the pixel coordinate point p0 can be the two-dimensional coordinate point of the target underwater feature point obtained by the event camera in the event mode after the laser emitted by the laser is irradiated onto the galvanometer at the current angle and the galvanometer reflects the received laser to the target underwater feature point.

[0121] It should be noted that the event camera in this disclosure can operate in two modes: image mode and event mode. In image mode, the camera captures a single color image. Furthermore, in image mode, the information transmitted back by the event camera is synchronized. Synchronization means that at a specific moment t, the camera will expose the image and transmit back all the pixels at that moment in time in a matrix. This creates a single photo, with all pixels corresponding to the same moment.

[0122] Event mode returns events asynchronously. Unlike image mode, which returns all pixel values ​​simultaneously, event mode returns different events at different times. Specifically, an event is returned whenever a pixel value changes within the camera's field of view, and all these events occur asynchronously.

[0123] In step 404, the ray tracing vector corresponding to the target underwater feature point can be calculated based on the pixel coordinate point p0 and the preset pinhole imaging model. Figure 5 The ray tracing vector can contain the Ocp1 vector, the p1p2 vector, and the p2p3 vector. The Ocp1 vector is refracted by the inner surface of the glass to obtain the p1p2 vector, and the p1p2 vector is refracted by the outer surface of the glass to obtain the p2p3 vector. p1 is the intersection of the ray Ocp0 and the inner surface of the glass, and Oc is the optical center of the event camera. θ1 represents the angle of incidence of the light from the air through the inner surface of the glass into the glass, θ2 represents the angle of incidence of the light from the glass through the outer surface of the glass into the water, and θ3 represents the angle of refraction of the light from the glass through the outer surface of the glass into the water.

[0124] That is, in the present disclosure, based on the image coordinates (u, v) of the pixel coordinate point p0 and the preset pinhole imaging model, the direction vector Ocp0 in the air and the intersection point p1 of the direction vector Ocp0 in the air and the inner surface of the glass can be calculated. Then, based on Snell's law, the direction vector p1p2 obtained after refraction from the inner surface of the glass and the intersection point p2 of the direction vector p1p2 with the outer surface of the glass can be calculated. Next, based on Snell's law, the direction vector p2p3 in the water obtained after refraction from the outer surface of the glass can be calculated. Specifically:

[0125] As mentioned above, the imaging point of an underwater feature point in the image plane of the event camera can be denoted as p0, and the ray The intersection point with the inner surface of the glass is p1, and the ray The rays refracted by the inner surface of the glass are ray The rays refracted by the outer surface of the glass are

[0126] The image coordinates of point P0 are (u, v), then according to the preset pinhole imaging model, we can calculate Direction vector =((u-u0) / f x ,(v-v0) / f y ,1), where (u0,v0) represents the optical center of the event camera.

[0127] The coordinates of point p1 in the camera coordinate system are:

[0128]

[0129] in, is the normal vector of the inner surface of the glass, and d0 represents the distance from the optical center of the event camera to the inner surface of the glass.

[0130] In addition, since all direction vectors of the refraction plane and normal vector All are in the same plane, so we can have:

[0131]

[0132] Under the above refraction constraints, the ray passing through the refractive surface is a linear combination of the direction vector of the previous ray and the normal vector of the refractive surface. Vector It can be expressed as:

[0133]

[0134] Here, u1 represents the refractive index in air, and u2 represents the refractive index in glass.

[0135] Then, the three-dimensional coordinates of point p2 in the camera coordinate system can be calculated using the following formula:

[0136]

[0137] in, is the normal vector of the inner surface of the glass, and d1 is the thickness of the glass.

[0138] Similarly, rays Vector It can be expressed as:

[0139]

[0140] Here, u2 represents the refractive index in glass, and u3 represents the refractive index in water.

[0141] In step 405, the three-dimensional coordinates of the target underwater feature point can be calculated based on the pixel coordinate point p0, the ray tracing vector, and the underwater structured light plane equation parameters at the current angle. Specifically:

[0142] Combined with the coordinates of point p2 in the camera coordinate system (x2, y2, z2) and Vector (l3,m3,n3), then the straight line equation of the refracted light p2p3 in water can be expressed as:

[0143]

[0144] In addition, since the feature point p3 on the underwater object is located on the underwater structured light plane III, the three-dimensional coordinates of the underwater feature point p3 can be obtained by calculating the intersection of the refracted light p2p3 in the water and the underwater structured light plane III. The specific calculation formula can be as follows:

[0145]

[0146] The equation for the structured light plane that enters the water after being refracted by the glass's outer surface is: A3x + B3y + C3z + D3 = 0. By utilizing a structured light plane model and an event camera imaging model that considers refraction, the 3D coordinates of each feature point on the underwater target can be calculated, enabling 3D reconstruction of the target.

[0147] In this disclosure, considering the advantages of event cameras in high-speed and low-light scenarios, event cameras can be introduced into underwater structured light vision systems, thereby enabling fast, accurate, and robust 3D reconstruction of low-light, weak-texture, and underwater moving objects. Furthermore, this disclosure proposes an underwater self-scanning structured light vision measurement model that simultaneously considers camera refraction and structured light plane refraction, and proposes a simple and efficient model parameter calibration method, eliminating the need for a complex underwater calibration process, simplifying the complexity of model parameter calibration and improving calibration efficiency.

[0148] Figure 83D reconstruction apparatus 800 for underwater objects according to an exemplary embodiment of the present disclosure is a block diagram showing the 3D reconstruction apparatus 800 for underwater objects. The 3D reconstruction apparatus 800 for underwater objects can be applied to a structured light vision system, which can include a galvanometer, a laser, an event camera, and glass.

[0149] Reference Figure 8 The three-dimensional reconstruction device 800 for underwater objects may include a parameter acquisition module 801, an underwater equation parameter calculation module 802, a pixel coordinate point acquisition module 803, a ray tracing vector calculation module 804 and a three-dimensional coordinate calculation module 805.

[0150] The parameter acquisition module 801 can obtain parameters of the structured light plane equation in air at the current angle of the galvanometer, wherein the structured light vision system is placed underwater. The parameters of the structured light plane equation in air can refer to the four coefficients A, B, C, and D included in the structured light plane equation in air: Ax+By+Cz+D=0.

[0151] According to an exemplary embodiment of the present disclosure, the parameter acquisition module 801 can obtain the rotation angle of the galvanometer from the previous angle to the current angle. Then, the parameter acquisition module 801 can determine the rotation matrix based on the rotation angle and the pre-calibrated scanning axis straight line equation parameters. Next, the parameter acquisition module 801 can calculate the parameters of the structured light plane equation in the air at the current angle based on the parameters of the structured light plane equation in the air at the previous angle and the rotation matrix. It should be noted that the parameters of the structured light plane equation in the air at the initial galvanometer angle can use the parameters of the structured light plane equation in the air at the initial angle obtained in the process of pre-calibrating the scanning axis straight line equation parameters. This acquisition process will be described in detail later.

[0152] According to an exemplary embodiment of the present disclosure, the apparatus 800 for 3D reconstruction of underwater objects may further include an air equation parameter acquisition module and a scanning axis line equation parameter calculation module.

[0153] The air equation parameter acquisition module can obtain the parameters of the air structured light plane equation at each of the multiple angles of the galvanometer. Then, the scanning axis line equation parameter calculation module can calculate the scanning axis line equation parameters based on the multiple air structured light plane equation parameters corresponding to the multiple angles.

[0154] According to an exemplary embodiment of the present disclosure, for each angle, the equation parameter acquisition module in the air can obtain the laser stripe image in different postures between the preset checkerboard and the event camera, wherein the laser stripe image can be an image obtained by the event camera shooting the preset checkerboard in image mode after the laser emitted by the laser is irradiated on the galvanometer and the galvanometer reflects the received laser to the preset checkerboard. Then, the equation parameter acquisition module in the air can calculate the parameters of the structured light plane equation in the air at this angle based on the laser stripe image in different postures between the preset checkerboard and the event camera. It should be noted that by changing the posture of the preset checkerboard relative to the event camera, the two laser stripes can be made non-collinear in space, and then a structured light plane can be uniquely determined by two non-collinear laser stripe straight lines.

[0155] According to an exemplary embodiment of the present disclosure, the refractive interface parameters may include a normal vector n of the inner surface of the glass and a distance d0 between the optical center of the event camera and the inner surface of the glass. The above-mentioned underwater object 3D reconstruction apparatus 800 may further include a first image acquisition module and a second image acquisition module, a corner coordinate calculation module, a normal vector calculation module, and a distance calculation module.

[0156] The first image and second image acquisition modules can acquire first and second images when the relative position between the preset checkerboard and the event camera is the target position. The first image can be an image obtained by photographing the preset checkerboard with the event camera in image mode, with the structured light vision system and the preset checkerboard both placed in air; the second image can be an image obtained by photographing the preset checkerboard with the event camera in image mode, with the structured light vision system and the preset checkerboard both placed in water. The corner point coordinate calculation module can then calculate the coordinates of the corner points of the preset checkerboard based on the first and second images. Next, the normal vector calculation module can calculate the normal vector n based on the coordinates of the corner points of the preset checkerboard and the pre-calibrated event camera parameters of the event camera. The distance calculation module can then calculate the distance d0 between the optical center of the event camera and the inner surface of the glass based on the normal vector n. Furthermore, the distance d0 between the optical center of the event camera and the inner surface of the glass can be calculated based on the normal vector n in combination with the event camera refraction model.

[0157] According to an exemplary embodiment of the present disclosure, the above-mentioned 3D reconstruction device 800 for underwater objects may further include a third image acquisition module and a camera parameter calibration module.

[0158] The third image acquisition module can acquire third images at different positions between the preset checkerboard and the event camera. The third images can be images captured by the event camera in image mode, with the structured light vision system and the preset checkerboard both placed in air. The camera parameter calibration module can then calibrate the event camera parameters of the event camera based on the third images at different positions between the preset checkerboard and the event camera.

[0159] The underwater equation parameter calculation module 802 may calculate the underwater structured light plane equation parameters at the current angle based on the air structured light plane equation parameters at the current angle and the pre-acquired refractive interface parameters.

[0160] The pixel coordinate point acquisition module 803 can acquire the pixel coordinate point p0 (u, v) corresponding to the target underwater feature point on the underwater object at the current angle.

[0161] According to an exemplary embodiment of the present disclosure, the pixel coordinate point p0 can be the two-dimensional coordinate point of the target underwater feature point obtained by the event camera in the event mode after the laser emitted by the laser is irradiated onto the galvanometer at the current angle and the galvanometer reflects the received laser to the target underwater feature point.

[0162] The ray tracing vector calculation module 804 may calculate the ray tracing vector corresponding to the target underwater feature point based on the pixel coordinate point p0 and a preset pinhole imaging model.

[0163] The three-dimensional coordinate calculation module 805 can calculate the three-dimensional coordinates of the target underwater feature point based on the pixel coordinate point p0, the ray tracing vector and the underwater structured light plane equation parameters at the current angle.

[0164] Figure 9 is a block diagram illustrating an electronic device 900 according to an exemplary embodiment of the present disclosure.

[0165] Reference Figure 9 The electronic device 900 includes at least one memory 901 and at least one processor 902. The at least one memory 901 stores instructions. When the instructions are executed by the at least one processor 902, a three-dimensional reconstruction method of an underwater object according to an exemplary embodiment of the present disclosure is performed.

[0166] As an example, the electronic device 900 can be a PC, a tablet device, a personal digital assistant, a smart phone, or other device capable of executing the above instructions. Here, the electronic device 900 is not necessarily a single electronic device, but can also be any device or circuit that can execute the above instructions (or instruction sets) individually or in combination. The electronic device 900 can also be part of an integrated control system or system manager, or can be configured as a portable electronic device that is interconnected with a local or remote (e.g., via wireless transmission) interface.

[0167] In electronic device 900, processor 902 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller, or a microprocessor. By way of example and not limitation, the processor may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, and the like.

[0168] The processor 902 can execute instructions or codes stored in the memory 901, wherein the memory 901 can also store data. Instructions and data can also be sent and received over a network via a network interface device, wherein the network interface device can use any known transmission protocol.

[0169] The memory 901 may be integrated with the processor 902, for example, by placing RAM or flash memory within an integrated circuit microprocessor or the like. Furthermore, the memory 901 may comprise a separate device, such as an external disk drive, a storage array, or any other storage device usable by a database system. The memory 901 and the processor 902 may be operatively coupled or may communicate with each other, for example, via an I / O port, a network connection, or the like, such that the processor 902 can access files stored in the memory.

[0170] In addition, the electronic device 900 may further include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.) All components of the electronic device 900 may be connected to each other via a bus and / or a network.

[0171] According to an exemplary embodiment of the present disclosure, a computer-readable storage medium may also be provided, which, when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to perform the above-mentioned three-dimensional reconstruction method of an underwater object. Examples of computer-readable storage media include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-RLTH , BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as, multimedia card, secure digital (SD) card or ultra-fast digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, any other device configured to store the computer program and any associated data, data files and data structures in a non-transitory manner and provide the computer program and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the above-mentioned computer-readable storage medium can be run in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. In addition, in one example, the computer program and any associated data, data files and data structures are distributed on a networked computer system so that the computer program and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers.

[0172] According to an exemplary embodiment of the present disclosure, a computer program product may further be provided, including a computer program, which implements the three-dimensional reconstruction method of the underwater object according to the present disclosure when executed by a processor.

[0173] The disclosed method, apparatus, electronic device, storage medium, and computer program product for 3D reconstruction of underwater objects take into account the advantages of event cameras in high-speed and low-light scenarios. This allows for the integration of event cameras into underwater structured light vision systems, enabling rapid, accurate, and robust 3D reconstruction of low-light, weak-texture, and moving underwater objects. Furthermore, the disclosed structured light vision system is simple in construction, consisting of only a laser, an event camera, and a galvanometer, without any additional hardware, resulting in low cost.

[0174] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the following claims.

[0175] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A three-dimensional reconstruction method for underwater objects, applied to a structured light vision system, characterized in that: The structured light vision system includes a galvanometer, a laser, an event camera, and glass, and the three-dimensional reconstruction method includes: Obtaining parameters of a structured light plane equation in air when the galvanometer is at a current angle, wherein the structured light vision system is placed underwater; Calculate the parameters of the structured light plane equation in water at the current angle based on the parameters of the structured light plane equation in air at the current angle and the pre-acquired refractive interface parameters; Obtain the pixel coordinate point p0(u, v) corresponding to the target underwater feature point on the underwater object at the current angle; Based on the pixel coordinate point p0 and a preset pinhole imaging model, a ray tracing vector corresponding to the target underwater feature point is calculated, wherein the ray tracing vector includes an Ocp1 vector, a p1p2 vector, and a p2p3 vector. The Ocp1 vector is refracted by the inner surface of the glass to obtain the p1p2 vector, and the p1p2 vector is refracted by the outer surface of the glass to obtain the p2p3 vector. p1 is the intersection point of the ray Ocp0 and the inner surface of the glass, and Oc is the optical center of the event camera. The three-dimensional coordinates of the target underwater feature point are calculated based on the pixel coordinate point p0, the ray tracing vector and the underwater structured light plane equation parameters at the current angle.

2. The three-dimensional reconstruction method according to claim 1, wherein: The obtaining of the parameters of the structured light plane equation in the air when the galvanometer is at the current angle includes: Obtaining the rotation angle of the galvanometer from the previous angle to the current angle; Determining a rotation matrix based on the rotation angle and pre-calibrated scan axis line equation parameters; The parameters of the structured light plane equation in air at the current angle are calculated based on the parameters of the structured light plane equation in air at the previous angle and the rotation matrix.

3. The three-dimensional reconstruction method according to claim 2, wherein: Before obtaining the rotation angle of the galvanometer from the previous angle to the current angle, the three-dimensional reconstruction method further includes: Obtaining parameters of a structured light plane equation in air when the galvanometer is at each of a plurality of angles; The scanning axis straight line equation parameters are calculated based on a plurality of plane equation parameters of the structured light in air corresponding one-to-one to the plurality of angles.

4. The three-dimensional reconstruction method according to claim 3, wherein: The obtaining of the parameters of the structured light plane equation in the air when the galvanometer is at each of the multiple angles includes: For each angle, a laser stripe image is obtained at different positions between the preset checkerboard and the event camera, wherein the laser stripe image is an image obtained by the event camera photographing the preset checkerboard in image mode after the laser emitted by the laser is irradiated onto the galvanometer mirror and the galvanometer mirror reflects the received laser light to the preset checkerboard; Based on the laser stripe images at different positions between the preset chessboard and the event camera, the parameters of the structured light plane equation in the air at the angle are calculated.

5. The three-dimensional reconstruction method according to claim 1, wherein: The refractive interface parameters include a normal vector n of the inner surface of the glass and a distance d0 between the optical center of the event camera and the inner surface of the glass; The three-dimensional reconstruction method further includes: Acquire a first image and a second image when the relative posture between the preset checkerboard and the event camera is a target posture, wherein the first image is an image obtained by photographing the preset checkerboard by the event camera in an image mode when the structured light vision system and the preset checkerboard are both placed in air, and the second image is an image obtained by photographing the preset checkerboard by the event camera in the image mode when the structured light vision system and the preset checkerboard are both placed in water; Calculating the coordinates of the corner points of the preset checkerboard based on the first image and the second image; Calculating the normal vector n based on the coordinates of the corner points of the preset checkerboard and the camera parameters of the event camera obtained by pre-calibration; Based on the normal vector n, the distance d0 between the optical center of the event camera and the inner surface of the glass is calculated.

6. The three-dimensional reconstruction method according to claim 5, wherein: Before acquiring the first image and the second image when the relative posture between the preset chessboard and the event camera is the target posture, the three-dimensional reconstruction method further includes: Acquire a third image at a different posture between the preset checkerboard and the event camera, wherein the third image is an image obtained by the event camera photographing the preset checkerboard in the image mode when the structured light vision system and the preset checkerboard are both placed in the air; Based on the third image in different postures between the preset checkerboard and the event camera, camera parameters of the event camera are calibrated.

7. The three-dimensional reconstruction method according to claim 1, wherein: The pixel coordinate point p0 is the two-dimensional coordinate point of the target underwater feature point in the imaging plane coordinate system of the event camera obtained by photographing the target underwater feature point in the event mode after the laser emitted by the laser is irradiated on the galvanometer at the current angle and the galvanometer reflects the received laser to the target underwater feature point.

8. A three-dimensional reconstruction device for underwater objects, applied to a structured light vision system, characterized in that: The structured light vision system includes a galvanometer, a laser, an event camera, and glass, and the three-dimensional reconstruction device includes: a parameter acquisition module configured to acquire parameters of a structured light plane equation in air when the galvanometer is at a current angle, wherein the structured light vision system is placed underwater; an underwater equation parameter calculation module, configured to calculate the parameters of the underwater structured light plane equation at a current angle based on the parameters of the structured light plane equation in air at a current angle and pre-acquired refractive interface parameters; A pixel coordinate point acquisition module is configured to obtain a pixel coordinate point p0 (u, v) corresponding to a target underwater feature point on the underwater object at a current angle; a ray tracing vector calculation module configured to calculate a ray tracing vector corresponding to the target underwater feature point based on the pixel coordinate point p0 and a preset pinhole imaging model, wherein the ray tracing vector includes an Ocp1 vector, a p1p2 vector, and a p2p3 vector, wherein the Ocp1 vector is refracted by the inner surface of the glass to obtain the p1p2 vector, and the p1p2 vector is refracted by the outer surface of the glass to obtain the p2p3 vector, p1 is the intersection point of the ray Ocp0 and the inner surface of the glass, and Oc is the optical center of the event camera; The three-dimensional coordinate calculation module is configured to calculate the three-dimensional coordinates of the target underwater feature point based on the pixel coordinate point p0, the ray tracing vector and the underwater structured light plane equation parameters at the current angle.

9. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the three-dimensional reconstruction method of an underwater object according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the three-dimensional reconstruction method of an underwater object according to any one of claims 1 to 7.

Images