Structured light three-dimensional reconstruction method for removing turbidity under polarized water based on focal plane

Abstract

The invention relates to the technical field of computer vision, in particular to a polarized underwater turbidity-removing structured light three-dimensional reconstruction method based on a focal plane, and the method comprises the following steps: building a polarized structured light system and underwater system model based on the focal plane, and carrying out the structured light three-dimensional reconstruction; four-channel polarization images of objects in the fuzzy water body and the clear water body are shot respectively, and polarization features are obtained based on the four-channel polarization images; inputting the polarization characteristics into a stripe recovery network, and recovering the high-frequency grating boundary of the fuzzy stripe image to obtain a recovered stripe image; and based on the recovered stripe image, structured light three-dimensional reconstruction is carried out to obtain a recovered point cloud. According to the method, stripe pictures of different channels are obtained through the polarization structured light camera, the polarization characteristics are calculated and then input into the stripe recovery network, noise is removed, the grating boundary is recovered, and three-dimensional reconstruction is more accurate.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and particularly to a structured light three-dimensional reconstruction method for underwater de-turbidity based on a split focal plane and polarization. Background Art

[0002] With the continuous deepening of ocean exploration, underwater three-dimensional structured light reconstruction technology has become an important support in the fields of ocean exploration, ocean robots, and underwater assembly, playing an important role in many fields such as underwater archaeological surveys, seabed terrain mapping, ecological protection, and inspection and maintenance of ocean equipment. However, the presence of a large number of suspended particles in the underwater environment will scatter and absorb the signal light, significantly reducing the image quality and resulting in poor effects of traditional image-based three-dimensional reconstruction methods. Achieving image clarity in turbid water is a key step in improving the accuracy of underwater three-dimensional structured light reconstruction and is also of great significance for ocean exploration. Therefore, it is particularly necessary to propose an efficient method for underwater de-turbidity.

[0003] Currently, three-dimensional reconstruction methods based on turbid water are mainly divided into two categories: one is the method based on traditional polarized structured light, which improves the clarity of turbid water reconstruction by using higher-frequency structured light projection, different filtering methods, and multi-camera systems. Specifically, due to the scattering effect of water on light, higher-frequency fringes have stronger anti-interference ability compared to lower-frequency fringes and can retain more information. Therefore, higher-frequency structured light can be used for projection. And image noise reduction and edge enhancement are performed through spatial filtering, frequency-domain filtering, etc., or the information of objects from different perspectives is captured through a multi-camera system, and the information from different perspectives is mutually complementary to obtain a more complete three-dimensional structure of the object. Since the light is blocked in turbid water, the boundaries of the projected image fringes are inevitably blurred, and it is difficult for traditional methods to effectively restore the blurred part, resulting in difficulty in qualitatively improving the accuracy of three-dimensional reconstruction of turbid water based on traditional methods. The other is the method based on deep learning, which extracts feature information from blurred images and restores the detailed features of the images through image enhancement to improve the reconstruction accuracy of objects. However, the current mainstream methods are all based on monocular cameras and color image restoration, and the restoration effect for gray-scale fringe patterns is not ideal, and it is difficult for monocular cameras to effectively capture the detailed information of objects in turbid water. Summary of the Invention

[0004] The embodiments of this application provide a structured light three-dimensional reconstruction method for underwater de-turbidity based on a split focal plane and polarization, which obtains the object fringe images in turbid and clear water through a polarized structured light camera and realizes the clarity of the fringe grating boundary through a self-supervised method. At the same time, the convolution superposition method is used to ensure the execution efficiency.

[0005] To solve the above technical problems, an embodiment of the present application provides a structured light three-dimensional reconstruction method for polarization underwater dehazing based on a sub-focal plane, including the following steps: First, a polarization structured light system and an underwater system model based on the sub-focal plane are respectively built for structured light three-dimensional reconstruction; Then, four-channel polarization images of objects in blurred water and clear water are respectively captured, and polarization features are obtained based on the four-channel polarization images; Next, the polarization features are input into a fringe restoration network to restore the high-frequency grating boundaries of the blurred fringe images, obtaining the restored fringe images; Finally, based on the restored fringe images, structured light three-dimensional reconstruction is performed to obtain the restored point cloud.

[0006] In some exemplary embodiments, the four-channel polarization images are fringe images at four different polarization angles; the four-channel polarization images include 18 groups of binary fringe images; obtaining polarization features based on the four-channel polarization images includes: using the fringe images at four different polarization angles to calculate the second Stokes parameter and the polarization angle to obtain the polarization features.

[0007] In some exemplary embodiments, the polarization structured light system includes a projector and a camera; an adjustable angular polarizer is provided between the projector and the target object, and the projector projects the four-channel polarization images onto the surface of the target object in sequence, triggering the camera to capture the fringe images.

[0008] In some exemplary embodiments, a linear polarizer is further provided between the projector and the target object, and the linear polarizer is used to convert the projection light into linearly polarized light and obtain the polarization components in different directions of the pixel by using the responses of the current pixel and its surrounding pixels.

[0009] In some exemplary embodiments, the fringe restoration network includes a fringe capture module and a global contour capture block; both the fringe capture module and the global contour capture block include a plurality of residual dense units and multi-scale sensing units; the residual dense units are used for deep feature extraction, and the multi-scale sensing units perform downsampling on the image after the residual dense units extract information by using bilinear interpolation; the multi-scale sensing units are used to further extract the detailed information of the picture from different scales, which helps to restore the fringe boundary information of the image; the fringe capture module includes an encoding part and a decoding part, and the fringe capture module is used to restore the high-frequency region of the image; the global contour capture module is used to capture the contour edges of the object and compensate for the influence of bilinear interpolation on the object contour.

[0010] In some exemplary embodiments, when the stripe capture module recovers the high-frequency region of the image, it first superimposes the stripe images of four different polarization channels to form a high-dimensional channel, and uses the pixel shuffling operation from the high-resolution field to convert the spatial information into high-dimensional channel information; then, it uses a convolutional layer to initially process the high-dimensional channel information, extracts the effective information, and removes the blurred noise; a balance is achieved between maintaining the image perception quality and reducing the file size or processing requirements to obtain detailed images of different scales; then the images of different scales are convolved to obtain finer features.

[0011] In some exemplary embodiments, when the global contour capture module captures the contour edge of an object, it uses canny edge detection to extract the initial contour information, then superimposes the second Stokes parameter and the polarization angle image with the initial contour information, and sends the superimposed image information into the stripe capture module for hierarchical feature extraction, so as to obtain more accurate features and eliminate the blurred noise; finally, a pixel shuffling upsampling operation is performed on the result of the stripe capture module, and the image size is restored, and then the restored result is fused with the filtered contour features, and through convolutional dimensionality reduction, a clearer polarization image is obtained.

[0012] In some exemplary embodiments, based on the polarization structured light system and the underwater system model of the sub-focal plane, when performing structured light three-dimensional reconstruction, a geometric model between the camera and the projector is derived from the perspective projection theory, and the geometric model is encapsulated by the following equations:

[0013]

[0014] where the subscript C / P represents the parameters of the specified camera or projector; M c / p = [X c / p Y c / p Z c / p ]′ represents the 3D space coordinates specified relative to the corresponding camera coordinate system; m c / p = [u c / p v c / p ]′ represents the projected pixel in the camera plane; the matrix K c / p integrates the focal length the principal point coordinates the tilt coefficient

[0015] In some exemplary embodiments, the transformation relationship between the camera and the projector is described as:

[0016]

[0017] where,

[0018]

[0019] Among them, R and T represent the rotation matrix and translation vector between the projector and camera coordinate systems.

[0020] In some exemplary embodiments, for a pixel (u c , v c ) in the camera image, the corresponding phase u p is obtained by decoding; the depth value z c is calculated by triangulation as follows:

[0021]

[0022] where u p is the phase obtained by decoding for the pixel (u C , v C ) in the camera image.

[0023] The technical solutions provided by the embodiments of the present application have at least the following advantages:

[0024] The embodiments of the present application provide a structured light three-dimensional reconstruction method for polarization underwater dehazing based on a split focal plane, the method comprising the following steps: First, a polarization structured light system and an underwater system model based on a split focal plane are respectively built for structured light three-dimensional reconstruction; Then, four-channel polarization images of an object in a blurred water body and a clear water body are respectively captured, and polarization features are obtained based on the four-channel polarization images; Next, the polarization features are input into a fringe recovery network to recover the high-frequency grating boundary of the blurred fringe image to obtain a recovered fringe image; Finally, based on the recovered fringe image, structured light three-dimensional reconstruction is performed to obtain a recovered point cloud.

[0025] The present application proposes a three-dimensional reconstruction method for turbid underwater based on a polarization structured light system, which combines the principle of polarization structured light with a deep learning model to achieve high-precision three-dimensional reconstruction of turbid underwater objects. Different-channel fringe pictures are obtained through a polarization structured light camera, the polarization characteristics are calculated and then input into a neural network to remove noise and recover the grating boundary, making its three-dimensional reconstruction more accurate. Different from the recovery of ordinary underwater color blurred pictures, the present application realizes a method for clarifying a dedicated underwater grayscale fringe picture, which is applicable to real-time fringe image recovery. Through an effective encoding and decoding method, it is possible to ensure the image accuracy of the recovery while ensuring the execution efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] One or more embodiments are illustrated by way of example in the accompanying drawings, which illustrations do not constitute a limitation on the embodiments, unless otherwise stated, the figures in the drawings do not constitute a scale limitation.

[0027] Figure 1 Flowchart of a structured light three-dimensional reconstruction method for polarized underwater de-turbidity based on a split focal plane provided by an embodiment of the present application.

[0028] Figure 2 Schematic diagram of modules of the structured light three-dimensional reconstruction method provided by an embodiment of the present application.

[0029] Figure 3 Schematic diagram of a polarized structured light acquisition device provided by an embodiment of the present application.

[0030] Figure 4 Geometric structure diagram of a structured light system provided by an embodiment of the present application.

[0031] Figure 5 Result of the fringe pattern provided by an embodiment of the present application.

[0032] Figure 6 Network architecture diagram of a convolutional neural network provided by an embodiment of the present application.

[0033] Figure 7 De-turbidity reconstruction result diagram provided by an embodiment of the present application. Detailed implementation manners

[0034] As can be seen from the background art, since the light is blocked in the turbid water body, the boundaries of the projected image fringes are inevitably blurred. It is difficult for traditional three-dimensional reconstruction methods to effectively recover the blurred part, resulting in the difficulty in qualitatively improving the three-dimensional reconstruction accuracy of turbid water bodies based on traditional methods. In addition, current deep learning-based methods are all based on monocular cameras and color image restoration, and the restoration effect for grayscale fringe patterns is not ideal, and it is difficult for monocular cameras to effectively capture the detailed information of objects in turbid water bodies.

[0035] Underwater three-dimensional reconstruction is an important technology in fields such as marine surveying and mapping, underwater archaeology, ecological monitoring, and underwater robot navigation. However, in a turbid water environment, due to complex optical phenomena such as light scattering, absorption, and refraction, the application of traditional three-dimensional reconstruction technology underwater faces serious challenges. Therefore, how to effectively perform high-precision three-dimensional reconstruction in turbid water has become an important research direction at the intersection of multiple disciplines such as computer vision, optical measurement, and artificial intelligence. Compared with three-dimensional reconstruction in an air environment, the underwater environment (especially turbid water) brings many challenges: Suspended particles in the water will cause light to scatter multiple times, resulting in blurred images, decreased contrast, and affecting the clarity of structured light stripes. The refractive indices of water and air are different, causing traditional camera calibration methods (such as the pinhole model) to fail, distorting the projection structured light pattern, and affecting the reconstruction accuracy. Water flow, light changes, and uneven distribution of suspended matter will cause noise changes, making it difficult for traditional filtering methods to effectively remove noise. Currently, the main solutions are mainly divided into underwater image enhancement methods based on deep learning and methods for optimizing traditional structured light devices and algorithms.

[0036] A related technology proposes an underwater three-dimensional reconstruction method based on multi-view binocular structured light, which includes the following method steps: Camera parameter calibration. First, adjust the focal length and aperture of the camera until the acquired images are clear, fix the focal length and aperture of the camera so that they remain unchanged during image acquisition, take several groups of calibration board images, and use Zhang Zhengyou's calibration method to calibrate the internal parameters of the camera. The entire parameter calibration process is completed in the air. It provides a new three-dimensional reconstruction method for underwater target objects, eliminating the need for waterproof processing of the camera and reducing the equipment manufacturing cost; This technology can reconstruct a complete three-dimensional model of underwater objects using a fixed multi-view method and reconstruct the surface of low-texture objects using binocular structured light.

[0037] Another related technology proposes an underwater target three-dimensional reconstruction method based on deep learning. It uses an attention mechanism to obtain the features that are focused on in underwater pictures, performs a homography transformation on the pictures to generate a matching feature volume, calculates the matching cost between the feature volume of this picture and the feature volumes of other pictures to obtain a four-dimensional matching cost volume, uses a multi-scale three-dimensional convolutional neural network for matching cost volume regularization, filters the cost volume to obtain a depth value probability volume, obtains a depth map through a neural network, and maps the depth values to three-dimensional space to obtain a three-dimensional point cloud map. This technology makes full use of the feature extraction ability of the convolutional neural network, further improves the representation ability of the model, greatly improves the stereo matching effect, dynamically balances the weights of each channel, and is beneficial to optimizing local feature information.

[0038] Another related technology proposes an underwater point structured light three-dimensional scanning system and method applicable to turbid water bodies, which is used to obtain the surface three-dimensional information of underwater target objects. It consists of an underwater point structured light emission unit, an underwater imaging unit, a control unit, a bracket and a carrier. The continuous reciprocating scanning of the point laser is controlled by a double galvanometer system. The image sequence of the laser points on the surface of the target object is collected through the underwater imaging unit. The control unit processes the image to extract the laser points in the image. Based on the principle of laser triangulation and combined with the collected inertial navigation information, the three-dimensional space coordinates of the surface of the target object are calculated, and the contour extraction of the space object within the scanning range is completed.

[0039] Another related technology proposes a method for restoring polarized images of turbid underwater based on a generative adversarial network, including the following steps: Step 1, build an underwater active imaging system to capture clear underwater intensity images and polarized images of turbid underwater; Step 2, establish a data set: divide the data set into a training set, a validation set, and a test set according to a ratio of 8:1:1; Step 3, construct a generative network; Step 4, pre-train: use the data set in Step 2 to pre-train the generative network; Step 5, construct a discriminant network; Step 6, construct a generative adversarial network; form a generative adversarial network with the pre-trained generative network in Step 4; use cross-entropy as the loss function to train the generative adversarial network; Step 7, use the trained generative adversarial network for image restoration.

[0040] Another related technology proposes a method and device for enhancing turbid underwater images based on a self-generative adversarial network. The method includes: obtaining a turbid underwater image and a clear reference image, and constructing an image data set; constructing a turbid underwater image enhancement model, including: a cascaded enhancement network, a color correction network, a self-adversarial module, and a feature reconstruction module; training the turbid underwater image enhancement model through the image data set; obtaining the turbid underwater image to be reconstructed; reconstructing the turbid underwater image to be reconstructed through the trained turbid underwater image enhancement model.

[0041] The main problems existing in the existing underwater de-turbidity three-dimensional reconstruction include: (1) For the three-dimensional reconstruction of turbid water bodies based on traditional methods, the clarity of the scanned images is improved as much as possible through optimization algorithms. However, if the turbid particles cannot be penetrated by light, the traditional method cannot effectively restore the clarity. (2) Although the underwater image enhancement method based on deep learning can improve the clarity of underwater images to a certain extent, this method restores attributes such as the illumination contrast of underwater images, and the restoration effect on the fringe images of structured light three-dimensional reconstruction is not ideal. (3) The reconstruction method of the device based on polarization only uses at most four images with different polarization angles as input data. (4) The method for enhancing turbid water body images based on an adversarial network is difficult to restore the grating boundary of the fringe image, which may cause decoding failure of the image and thus lead to reconstruction failure.

[0042] To solve the above technical problems, the present application provides a structured light three-dimensional reconstruction method for polarization underwater de-turbidity based on a split focal plane, including the following steps: First, a polarization structured light system and an underwater system model based on the split focal plane are respectively built for structured light three-dimensional reconstruction; Then, four-channel polarization images of objects in turbid water and clear water are respectively taken, and polarization features are obtained based on the four-channel polarization images; Next, the polarization features are input into a fringe restoration network to restore the high-frequency grating boundaries of the blurred fringe images, obtaining the restored fringe images; Finally, based on the restored fringe images, structured light three-dimensional reconstruction is performed to obtain the restored point cloud. The present application provides a structured light three-dimensional reconstruction method for polarization underwater de-turbidity based on a split focal plane, which obtains the object fringe images in turbid and clear water through a polarization structured light camera, and realizes the clarification of the fringe grating boundaries through a self-supervised method. At the same time, the convolution superposition method is used to ensure the execution efficiency.

[0043] The following will elaborate on the embodiments of the present application with reference to the accompanying drawings. However, those of ordinary skill in the art can understand that in the embodiments of the present application, many technical details are proposed to help readers better understand the present application. However, even without these technical details and various changes and modifications based on the following embodiments, the technical solutions claimed in the present application can still be implemented.

[0044] See Figure 1 , the embodiments of the present application provide a structured light three-dimensional reconstruction method for polarization underwater de-turbidity based on a split focal plane, including the following steps:

[0045] Step S101: Respectively build a polarization structured light system and an underwater system model based on the split focal plane for structured light three-dimensional reconstruction.

[0046] Step S102: Respectively take four-channel polarization images of objects in turbid water and clear water, and obtain polarization features based on the four-channel polarization images.

[0047] Step S103: Input the polarization features into a fringe restoration network to restore the high-frequency grating boundaries of the blurred fringe images, obtaining the restored fringe images.

[0048] Step S104: Based on the restored fringe images, perform structured light three-dimensional reconstruction to obtain the restored point cloud.

[0049] In some embodiments, the four-channel polarization images in step S102 are fringe images with four different polarization angles; the four-channel polarization images include 18 groups of binary fringe images; obtaining polarization features based on the four-channel polarization images includes: using the fringe images with four different polarization angles to calculate the second Stokes parameter and the polarization angle to obtain the polarization features.

[0050] This application proposes a structured light three-dimensional reconstruction method for polarization underwater de-turbidity based on a split focal plane. First, a polarization structured light system and an underwater system model based on the split focal plane are built. The hardware device of this application is developed based on the principle of encoded structured light three-dimensional scanning and is mainly used to obtain a complete sequence of three-dimensional object models in real time. To ensure obtaining sufficient information about objects in turbid water, this application uses a polarization structured light system. By placing a linear polarizer in front of the projector, the projected light is converted into linearly polarized light. Using the responses of the current pixel and its surrounding pixels, the polarization components of this pixel in the four directions of 0°, 45°, 90°, and 135° can be obtained, and the polarization characteristics, that is, the second Stokes parameter and the polarization angle of the incident light, are calculated. Among them, the second Stokes parameter emphasizes the linear component related to the light polarization angle and is therefore often used to highlight the geometric details of objects. The second Stokes parameter can reveal the texture and direction characteristics of linear features presented on the observed object or surface, especially in the presence of linear contrast. The polarization angle can more clearly display the boundaries or irregular parts of the surface and is usually used to emphasize the boundaries of objects. Especially in complex scenes, areas with rough edges or surfaces exhibit different degrees of polarization, allowing the polarization angle to more clearly display patterns and boundaries. Therefore, this application uses polarization characteristics (the second Stokes parameter and the polarization angle) as the input of the fringe recovery network, and at the same time, based on the GPU module built into the camera (an imaging device can also be used), real-time fringe image acquisition and three-dimensional point cloud reconstruction are realized.

[0051] The present application proposes a three-dimensional reconstruction method for turbid underwater based on a polarization structured light system, which combines the principle of polarization structured light with a deep learning model to achieve high-precision three-dimensional reconstruction of turbid underwater objects. Different-channel fringe images are obtained through a polarization structured light camera, and the polarization characteristics are calculated and then input into a neural network to remove noise and restore the grating boundary, making the three-dimensional reconstruction more accurate. For the structured light three-dimensional reconstruction method provided by the present application, first, the second Stokes parameter and the polarization angle are calculated using four fringe images with different polarization angles, and then the four different polarization fringe images are superimposed to form a high-dimensional channel. Pixel transformation operations of a high-resolution field are used to convert spatial information into channel information. This process helps ensure that subsequent convolutions can fully extract effective information and remove blurred noise. Then, the detailed characteristics of the image are extracted from different scales using a multi-scale sampling method, and a multi-scale sensing unit is introduced to restore the internal fringe pattern information of polarization images at different scales. Then, it is combined with a residual dense unit to solve potential performance degradation problems that may occur in deep neural networks. At the same time, a method for restoring the object contour is proposed. By superimposing and extracting the second Stokes parameter, the polarization angle, and the canny edge features, the features between different input data are made complementary through multi-scale feature extraction to restore the object edge information. And the L1 loss is used as the basic loss architecture, and a multi-scale Sobel loss is proposed to calculate the loss of feature information at different scales.

[0052] The following provides a detailed introduction to the structured light three-dimensional reconstruction method for polarization underwater de-turbidity based on the sub-focal plane provided by the present application through specific embodiments.

[0053] First, based on the polarization structured light device of the sub-focal plane, polarization data is obtained.

[0054] In current three-dimensional imaging methods, structured light technology has been widely used in many fields such as industrial inspection, virtual reality, and medicine due to its characteristics of simple hardware structure, large measurement range, large field of view, and dense point cloud. Compared with traditional binocular vision, structured light technology replaces one camera with a projector. The projector projects one or a set of pre-set light stripes onto the surface of the measured object, and the distorted stripe pattern on the surface of the object is photographed by a camera at another angle. Combining the calibration parameters of the projector and the camera, the surface of the object to be measured can be reconstructed.

[0055] Such as Figure 2As shown, the polarization structured light system of the present application includes a projector and a camera. An adjustable angular polarizer is provided between the projector and the target object. The projector sequentially projects four-channel polarization images onto the surface of the target object, triggering the camera to capture fringe images. In the present application, an adjustable angular polarizer is installed in front of the projector, and the projector sequentially projects 18 binary fringe patterns onto the target surface, triggering the camera to capture these patterns. The polarization system uses focal plane division polarization imaging, where the micropolarizer array is directly coupled to the photosensitive pixels of the detector. As Figure 2 shown, each pixel on the focal plane corresponds to a micropolarizer element, thus enabling multi-directional polarization imaging simultaneously. Every four pixels form a 2×2 array, and each array matches one of the first four micropolarizers (0°, 45°, 90°, 135°) in front of the focal plane. Each 2×2 array can obtain polarization information from four directions simultaneously. In this way, every four pixels on the focal plane can be combined into a polarization imaging detection pixel to achieve synchronous polarization imaging detection of the target.

[0056] In some embodiments, a linear polarizer is also provided between the projector and the target object. The linear polarizer is used to convert the projection light into linearly polarized light and utilize the responses of the current pixel and its surrounding pixels to obtain the polarization components of the pixel in different directions. Specifically, in the present application, a linear polarizer is placed in front of the projector to convert the projection light into linearly polarized light, and the polarization components of the pixel in different directions are obtained. This allows for the calculation of the incident Stokes vector D and the polarization angle. The present application uses a blue water tank with a radius of 80 cm to simulate an underwater scene, and places the polarization SL system in a 4-mm-thick fish tank to simulate the working environment of a real underwater camera.

[0057] In structured light polarization imaging, the second Stokes parameter is usually used to highlight the geometric details of an object because it emphasizes the linear component related to the light polarization angle. It can reveal the texture and direction features on the observed object or surface that exhibit linear characteristics, especially in the presence of linear contrast. Since the boundary regions of an object are often accompanied by changes in the polarization of the reflected light, the polarization angle can more clearly reveal the boundaries or irregular parts of the surface and is usually used to emphasize the boundaries of the object. Especially in complex scenes, areas with rough edges or surfaces exhibit different degrees of polarization, making the polarization angle image able to more clearly display patterns and boundaries. Therefore, the present application uses the Stokes parameters and the polarization angle as the input to the network. The Stokes parameters are a set of values that describe the polarization state of electromagnetic radiation. Specifically, the second Stokes parameter represents the difference in intensity between the polarization at 45° and 135° to the reference axis. In simpler terms, it measures the degree to which light is polarized at these angles.

[0058]

[0059] where I0, I5, I90 and I 135 are the light intensities at different polarization angles. The polarization angle Dop quantifies the fraction of the polarized electromagnetic wave. It ranges from 0% (completely unpolarized light) to 100% (perfectly polarized light). Mathematically, it is defined as the ratio of the power of the polarized part of the light to the total power of the light, as follows:

[0060]

[0061] Next, real-time high-precision three-dimensional reconstruction is performed. The geometric structure of the structured light system proposed in this application is as Figure 4 shown. The camera and projector models derived from perspective projection theory are encapsulated by equations:

[0062]

[0063] where the subscript C / P represents the parameters of the specified camera or projector; M c / p = [X c / p Y c / p Z c / p ′ represents the 3D spatial coordinates specified with respect to the corresponding camera coordinate system; m c / p = [u c / p v c / p ′ represents the projection pixel in the camera plane; the matrix K c / p integrates the focal length the principal point coordinates the tilt coefficient

[0064] In some embodiments, the transformation relationship between the camera and the projector is described as:

[0065]

[0066] where

[0067]

[0068] where R and T represent the rotation matrix and the translation vector between the projector and the camera coordinate systems.

[0069] In some embodiments, for the pixel (u c , v c ) in the camera image, the corresponding phase u p is obtained by decoding; the depth value z C is calculated by triangulation, as follows:

[0070]

[0071] where u pFor the phase obtained by decoding the pixels (u C , v C ) in the camera image.

[0072] Figure 5 In (a) of ,

[0072] , Figure 5 , there are 18 fringe patterns of the adopted coding strategy. In (b) of ,

[0072] , Figure 5 , there is a set of fringe images captured at a polarization angle of 0°. As Figure 5 shown in (a) of Figure 5 , the present application combines Gray-code with a row shift pattern. The first two fringe images Ω0 are used to extract the target object. The middle eight fringe patterns Ω1 are Gray-code patterns that construct 256 sub-regions, each with a unique codeword. The last eight patterns Ω2 are thin fringe patterns with a width of 4 pixels, which move 8 times to encode the positions within each sub-region. The combination of the phase value and the local phase value is obtained. Figure 5 In (b) of Figure 5 , there is shown the 3D polarization structured light system of the present application, and a set of fringe patterns is captured at a polarization angle of 0° for an underwater object. For the pixels (u c , v c ) in the camera image, the corresponding phase u p is obtained by decoding.

[0073] In some embodiments, the fringe recovery network includes a fringe capture module and a global contour capture block; both the fringe capture module and the global contour capture block include a plurality of residual dense units and multi-scale sensing units; the residual dense units are used for deep feature extraction, and the multi-scale sensing units perform downsampling on the image after the residual dense units extract information using bilinear interpolation; the multi-scale sensing units are used to further extract the detailed information of the picture from different scales, which helps to recover the fringe boundary information of the image; the fringe capture module includes an encoding part and a decoding part, and the fringe capture module is used to recover the high-frequency region of the image; the global contour capture module is used to capture the contour edge of the object and compensate for the influence of bilinear interpolation on the object contour.

[0074] In some embodiments, when the fringe capture module recovers the high-frequency region of the image, first, the fringe images of four different polarization channels are superimposed to form a high-dimensional channel, and the spatial information is converted into high-dimensional channel information using a pixel shuffle operation from the high-resolution field; then, the convolutional layer is used to perform initial processing on the high-dimensional channel information to extract effective information and remove blurred noise; a balance is achieved between maintaining the image perception quality and reducing the file size or processing requirements to obtain detailed images of different scales; then, the images of different scales are convolved to obtain finer features.

[0075] In some embodiments, when the global contour capture module captures the contour edge of an object, it uses canny edge detection to extract the initial contour information, then superimposes the Stokes second parameter and the polarization angle image on the initial contour information, and sends the superimposed image information to the stripe capture module for hierarchical feature extraction, so as to obtain more accurate features and eliminate blurred noise; finally, a pixel shuffling upsampling operation is performed on the result of the stripe capture module, and the image size is restored, and then the restored result is fused with the filtered contour features, and through convolution dimensionality reduction, a clearer polarization image is obtained.

[0076] Specifically, as Figure 6 shown, the overall network architecture of the stripe restoration network is mainly composed of a stripe capture module and a global contour capture block. Among them, the stripe capture module is mainly used to restore the high-frequency region of the image, and the global contour capture module is used to capture the edge contour of the object. Both the stripe capture module and the global contour capture module are composed of multiple residual dense units and multi-scale sensing units. The residual dense units are used for deep feature extraction, and the multi-scale sensing units are used to further extract the detailed information of the picture from different scales, which helps to restore the stripe boundary information of the image.

[0077] Specifically, first, the fringe images of four different polarization channels are superimposed to form a high-dimensional channel, and the spatial information is converted into channel information using a pixel shuffling operation from the high-resolution field. This process helps to ensure that subsequent convolutions can fully extract effective information and remove blurred noise, which has been proven beneficial for image restoration. Then, a convolutional layer is used to initially process the high-dimensional channel information, extract effective information, and remove blurred noise. The fringe capture module is divided into an encoding and a decoding part and consists of multiple residual dense units and multi-scale sensing units. The residual dense units ensure that the performance of the model does not degrade due to the extraction of more subtle features as the network deepens. The multi-scale sensing unit downsamples the image after the residual dense units extract information using bilinear interpolation. A balance is achieved between maintaining the image perception quality and reducing the file size or processing requirements to obtain detailed images at different scales. Then, the images at different scales are convolved to obtain finer features, which plays a crucial role in the restoration of polarized fringe images. Considering that bilinear interpolation averages the pixel values in the neighborhood, thus blurring the edge information of the object, this application adds a global contour capture module to capture the contour edges of the object and compensate for the impact of bilinear interpolation on the object contour. First, it uses canny edge detection to extract the initial contour information, then superimposes the second Stokes parameter and the polarization angle image with this contour information, and feeds it into a module composed of residual dense units and multi-scale sensing units for hierarchical feature extraction, thereby obtaining more accurate features and removing blurred noise. Finally, a pixel shuffling upsampling operation is performed on the result of the fringe capture module to restore the image size, and then it is fused with the filtered contour features. Through convolutional dimensionality reduction, this application obtains a clearer polarized image. It should be noted that although the fringe capture module can well restore the detailed information of the grating, the edge information may cause the grating boundary to be more blurred due to insufficient light, and the global contour capture module makes up for the blurred contour information.

[0078] The final image restoration results are as Figure 7 shown. On the left is the fringe image of the turbid water body and the reconstruction result, in the middle is the effect after the model restoration, and on the right is the effect in the clear water body. It can be Figure 7 seen that the underwater de-turbidity structured light three-dimensional reconstruction effect of this application is obvious.

[0079] Different from the restoration of ordinary underwater color blurred pictures, this application realizes a dedicated method for clarifying underwater gray fringe pictures, which is applicable to real-time fringe image restoration. Through an effective encoding and decoding method, it can ensure the image accuracy of the restoration while guaranteeing the execution efficiency.

[0080] Compared with the prior art, the advantages of the polarized underwater de-turbidity structured light three-dimensional reconstruction method based on the sub-focal plane provided by this application are as follows:

[0081] First, a polarization camera is used to obtain more sufficient object information compared to a normal camera, and polarization features that can further improve the image contrast are obtained using optical prior knowledge.

[0082] Secondly, the present application adopts an encoding and decoding method and transforms the module into an execution unit, improving the execution efficiency of the algorithm and enabling it to process fringe images in real time.

[0083] Finally, the present application proposes a network model for restoring grayscale fringe images. This model can effectively restore the boundary details of grating fringes, which is essentially different from the above-mentioned technology for restoring blurred underwater color images. Moreover, the present application has been certified by an experimental system, and the effect is very ideal, consistent with the design expectation.

[0084] With the above technical solutions, the embodiments of the present application provide a method for three-dimensional reconstruction of structured light for polarization underwater dehazing based on a split focal plane. The method includes the following steps: First, a polarization structured light system and an underwater system model based on the split focal plane are respectively built for three-dimensional reconstruction of structured light; then, four-channel polarization images of objects in a blurred water body and a clear water body are respectively taken, and polarization features are obtained based on the four-channel polarization images; next, the polarization features are input into a fringe restoration network to restore the high-frequency grating boundary of the blurred fringe image to obtain a restored fringe image; finally, based on the restored fringe image, three-dimensional reconstruction of structured light is performed to obtain a restored point cloud.

[0085] The present application proposes a method for three-dimensional reconstruction of turbid underwater based on a polarization structured light system, which combines the principle of polarization structured light with a deep learning model to achieve high-precision three-dimensional reconstruction of turbid underwater objects. Different-channel fringe images are obtained through a polarization structured light camera, and the polarization characteristics are calculated and then input into a neural network to remove noise and restore the grating boundary, making the three-dimensional reconstruction more accurate. Different from the restoration of ordinary underwater color blurred pictures, the present application realizes a method for clarifying dedicated underwater grayscale fringe pictures, which is suitable for real-time fringe image restoration. Through an effective encoding and decoding method, it is possible to ensure the execution efficiency while ensuring the accuracy of the restored image.

[0086] Those of ordinary skill in the art can understand that the above embodiments are specific embodiments for implementing the present application. In actual applications, various changes can be made in form and details without departing from the spirit and scope of the present application. Any person skilled in the art can make their own changes and modifications without departing from the spirit and scope of the present application. Therefore, the protection scope of the present application should be determined by the scope defined by the claims.

Claims

1. A structured light three-dimensional reconstruction method for underwater polarization de-turbidity based on a split focal plane, characterized in that It includes the following steps: Build a polarization structured light system and an underwater system model based on a split focal plane respectively, and perform structured light three-dimensional reconstruction; Capture four-channel polarization images of objects in blurred water and clear water respectively, and obtain polarization characteristics based on the four-channel polarization images; Input the polarization characteristics into a fringe restoration network to restore the high-frequency grating boundary of the blurred fringe image and obtain the restored fringe image; Based on the restored fringe image, perform structured light three-dimensional reconstruction to obtain the restored point cloud.

2. The method for three-dimensional reconstruction of structured light for underwater turbidity removal based on a split focal plane according to claim 1, wherein The four-channel polarization image is a fringe image with four different polarization angles; the four-channel polarization image includes 18 groups of binary fringe images; Obtaining polarization characteristics based on the four-channel polarization image includes: Using fringe images with four different polarization angles to calculate the second Stokes parameter and the polarization angle to obtain polarization characteristics.

3. The structured light three-dimensional reconstruction method for underwater de-turbidity based on a split focal plane according to claim 1, wherein The polarization structured light system includes a projector and a camera; an adjustable angular polarizer is provided between the projector and the target object, and the projector projects four-channel polarization images onto the surface of the target object in sequence to trigger the camera to capture fringe images.

4. The structured light three-dimensional reconstruction method for underwater de-turbidity based on a split focal plane according to claim 3, wherein A linear polarizer is also provided between the projector and the target object, and the linear polarizer is used to convert the projection light into linearly polarized light and obtain the polarization components in different directions of the pixel by using the responses of the current pixel and its surrounding pixels.

5. The structured light three-dimensional reconstruction method for underwater de-turbidity based on a split focal plane according to claim 1, wherein The fringe restoration network includes a fringe capture module and a global contour capture block; both the fringe capture module and the global contour capture block include a plurality of residual dense units and multi-scale sensing units; The residual dense unit is used for deep feature extraction, and the multi-scale sensing unit downsamples the image after the residual dense unit extracts information by using bilinear interpolation; the multi-scale sensing unit is used to further extract the detailed information of the picture at different scales, which helps to restore the fringe boundary information of the image; The fringe capture module includes an encoding part and a decoding part, and the fringe capture module is used to restore the high-frequency region of the image; the global contour capture module is used to capture the contour edge of the object and compensate for the influence of bilinear interpolation on the object contour.

6. The method for three-dimensional reconstruction of structured light for polarization underwater turbidity removal based on a split focal plane according to claim 5, wherein When the fringe capture module restores the high-frequency region of the image, first, the fringe images of four different polarization channels are superimposed to form a high-dimensional channel, and the spatial information is converted into high-dimensional channel information by using the pixel shuffle operation from the high-resolution field; then, the convolutional layer is used to perform initial processing on the high-dimensional channel information, extract effective information, and remove blurred noise; a balance is achieved between maintaining the perceptual quality of the image and reducing the file size or processing requirements to obtain detailed images at different scales; then the images at different scales are convolved to obtain finer features.

7. The structured light three-dimensional reconstruction method for underwater de-turbidity based on a split focal plane according to claim 5, characterized in that, When the global contour capture module captures the contour edge of an object, it uses canny edge detection to extract the initial contour information, then superimposes the Stokes second parameter and the polarization angle image on the initial contour information, and sends the superimposed image information into the fringe capture module for hierarchical feature extraction, so as to obtain more accurate features and eliminate fuzzy noise; Finally, perform pixel shuffling upsampling operation on the result of the fringe capture module, restore the image size, then fuse the restored result with the filtered contour features, and reduce the dimension through convolution to obtain a clearer polarization image.

8. The method for three-dimensional reconstruction of structured light for underwater turbidity removal based on a split focal plane according to claim 1, characterized in that, Based on the polarization structured light system and the underwater system model of the sub-focal plane, when performing structured light three-dimensional reconstruction, a geometric model between the camera and the projector is derived from the perspective projection theory, and the geometric model is encapsulated by the following equations: where the subscript C / P represents the parameters of the specified camera or projector; M c / p = [X c / p Y c / p Z c / p ′ represents the 3D spatial coordinates specified with respect to the corresponding camera coordinate system; m c / p = [u c / p v c / p ′ represents the projection pixels in the camera plane; the matrix K c / p integrates the focal length the principal point coordinates the skew coefficient 9. The structured light three-dimensional reconstruction method for underwater de-turbidity based on a split focal plane according to claim 8, characterized in that The transformation relationship between the camera and the projector is described as: Where, Where, R and T represent the rotation matrix and translation vector between the projector and the camera coordinate systems.

10. The structured light three-dimensional reconstruction method for underwater de-turbidity based on a split focal plane according to claim 9, wherein, For a pixel (u c , v c ) in the camera image, the corresponding phase u p is obtained by decoding; the depth value z C is calculated by triangulation as follows: Among them, u p is the phase obtained by decoding the pixel (u c , v c ) in the camera image.

Images