A three-dimensional reconstruction method and device for endoscopic images

Abstract

The present invention discloses a three-dimensional reconstruction method and device for endoscopic images, including: building a polarization measurement binocular endoscope system; receiving original endoscopic images in different scenes, decoupling to obtain corresponding RGB images and polarization images with different polarization angles, and calculating corresponding polarization information codes; constructing an end-to-end RGB+polarization fusion network, using the dense surface normal information carried by the polarization mode to enhance the depth information of binocular stereo vision, and using the calculated polarization information code and RGB image group as input to obtain the corresponding depth map in real time; using the current posture parameters of the image and the depth map to reconstruct the image and obtain a three-dimensional visualization point cloud. The present invention can efficiently fuse the RGB modal information and polarization modal information in the endoscopic imaging system, perform high-precision three-dimensional model reconstruction tasks for complex endoscopic scenes, thereby significantly improving the operation speed and judgment accuracy of doctors when performing medical diagnosis.

Description

Technical Field

[0001] The present invention relates to the technical field of endoscopes, and in particular to a method and device for three-dimensional reconstruction of endoscopic images. Background Art

[0002] In clinical diagnosis, high-quality three-dimensional endoscopic images can provide depth information (precise size information), which helps doctors clearly and quickly identify and locate key anatomical structures, improving clinical efficiency and safety. In addition, high-quality digital three-dimensional information is also an important prerequisite for the new generation of image-guided minimally invasive surgery and intelligent surgical robot technology. It can provide a comprehensive stereoscopic view to perceive the surface shape of tissues, plan the surgical path in advance, ensure that surgical instruments accurately reach the lesion location, and avoid damage to surrounding normal tissues and nerves. Therefore, it is of great significance to study high-quality three-dimensional information reconstruction technology for endoscopic imaging.

[0003] At present, the 3D reconstruction technology based on endoscopic imaging is mainly realized by binocular stereo vision algorithm. The typical representative algorithm is the Semi-Global Block Matching (SGBM), which needs to obtain the difference between the left and right views by feature matching, and calculate the 3D coordinates of the point by combining triangulation. The highly texture-dependent features determine that it works well for texture-rich tissues, but the algorithm will fail in scenes such as the lack of texture, specular reflection and weak details of the stomach wall; methods such as structured light 3D reconstruction and photon flight time measurement can significantly increase specific texture information, but are limited by external hardware conditions and are difficult to embed in the endoscopy system. Shadow reconstruction technology does not rely on texture information and uses the light intensity change of a single image to restore the relative height of each point on the surface of the object; but it requires the reflection function of the tissue surface to be assumed in advance, and these assumptions are difficult to be the same as the true properties of each tissue, resulting in low reconstruction accuracy. Deep learning has also been introduced into the 3D reconstruction of endoscopes, which has promoted the reconstruction effect in weak texture areas and can generate smooth results. However, it may be unreliable to restore the depth in weak texture areas relying solely on the learned shape prior because the correspondence is essentially ambiguous. Summary of the invention

[0004] In view of the shortcomings of the prior art, the present invention provides a three-dimensional reconstruction method and device for endoscopic images, which realizes high-quality three-dimensional reconstruction of tissues through an end-to-end RGB+polarization fusion network; and utilizes the surface normal information carried by the polarization mode to enhance the depth information of binocular stereo vision to achieve robust dense three-dimensional reconstruction under weak texture.

[0005] The present invention provides a three-dimensional reconstruction method of an endoscopic image, comprising the following steps:

[0006] S1: Build a polarization measurement binocular endoscope system;

[0007] S2: Receive the original endoscope images of the polarization measurement binocular endoscope system under different scenes, decouple to obtain the corresponding RGB images and polarization images with different polarization angles, and calculate the corresponding polarization information encoding;

[0008] S3: Build an end-to-end RGB+polarization fusion network, use the dense surface normal information carried by the polarization mode to enhance the depth information of binocular stereo vision, and use the calculated polarization information encoding and RGB image group as input to obtain the corresponding depth map in real time;

[0009] S4: Reconstruct the image using the current pose parameters of the image and the depth map to obtain a three-dimensional visual point cloud.

[0010] Furthermore, in step S2, the corresponding polarization information codes are calculated according to the polarization images at different polarization angles, and the process of calculating the polarization information codes is as follows:

[0011] ;

[0012] Among them, I0, I 45 ,I 90 and I 135 Respectively represent polarization images when the polarizer angle is 0°, 45°, 90°, and 135°. is the average intensity, a scalar is the degree of linear polarization, angle is the linear polarization angle, Indicates polarization information encoding.

[0013] Furthermore, in step S3, the end-to-end RGB+polarization fusion network consists of four parts: feature extraction, cost volume construction, cost aggregation and depth prediction.

[0014] Furthermore, the feature extraction is specifically as follows: using a set of 3×3 kernel convolution layers to downsample the input image, and then using a convolution layer with sixteen residual layers and two convolution layers with three residual layers to further expand the receptive field; the output feature map size is 1 / 4 × 1 / 4 of the input image size; all feature maps are connected to form a 320-channel unary feature map;

[0015] The cost volume is constructed as follows: the relevant volumes are constructed in the RGB domain and the polarization domain respectively; first, two convolutions are applied to compress the 320-channel feature map into a 32-channel feature map, and then the left feature map of each disparity level is The corresponding right feature map Connect them together to build a 4D body ;

[0016] ;

[0017] in, represents parallax, represents coordinates, represents the fourth dimension, Indicates series connection;

[0018] The cost summary is specifically as follows: and polarization-dependent volume The adaptive weighted fusion is used as the input of the 3D aggregation network. The adaptive weighted fusion module consists of several 3×3 convolutional layers and ReLU activation function layers;

[0019] ;

[0020] Among them, AWF represents the adaptive weighted fusion module, is the fusion volume, Represents matrix multiplication operation;

[0021] The consistency constraint between the three-dimensional shape of polarization information and the depth estimation of stereo vision is used to guide the network to adaptively learn and purify the effective depth information, thereby improving the accuracy of depth estimation; the 3D aggregation network is composed of four 3D convolutions with batch normalization and ReLU and two stacked 3D hourglass networks; each hourglass network is composed of four 3D convolution layers and two 3D deconvolution layers;

[0022] The depth prediction is specifically as follows: one output before the hourglass network and after each hourglass network corresponds to a 4D volume with one channel through two 3D convolutions; the volume is upsampled and converted into a probability volume using a softmax function, and the predicted depth It is calculated as the sum of each depth d weighted by its probability, expressed as:

[0023] ;

[0024] in, Indicates the depth value, represents probability;

[0025] The smooth L1 loss function is used to train the network, and the loss function The expression is:

[0026] ;

[0027] in, represents the coefficient of the i-th disparity prediction, represents the real depth map, represents the smooth L1 loss function;

[0028] The RGB+polarization fusion network is trained in combination with the loss function, and the trained network is used to obtain the corresponding depth map in real time.

[0029] Furthermore, the surface normal information implicitly contained in the polarization-dependent volume complements the RGB information to enhance the depth information of binocular stereo vision.

[0030] Furthermore, the step S4 is specifically as follows: obtaining pixel coordinates of the current image; calculating the world space coordinates of the target based on the current posture parameters and pixel coordinates of the image and the corresponding depth map obtained in step S3; obtaining the color information of each pixel in the image, and reconstructing the point cloud based on the color information and the world space coordinates of the target to obtain a three-dimensional point cloud.

[0031] The present invention also provides a three-dimensional reconstruction device for endoscopic images, comprising:

[0032] An image acquisition module, used for receiving original endoscopic images of a polarization measurement binocular endoscope system in different scenes;

[0033] An image processing module is used to correct the original endoscopic image, decouple the corrected image to obtain the corresponding RGB image and polarization images of different polarization angles, and calculate the corresponding polarization information code;

[0034] Image depth module, used to input the calculated polarization information encoding and RGB image group into the pre-trained 3D reconstruction network model to capture the corresponding depth map in real time;

[0035] The image display module is used to calculate the world space coordinates of the target based on the current posture parameters and depth map of the image, and display the three-dimensional visualization interface of the target in real time.

[0036] Furthermore, the image processing module includes:

[0037] An image preprocessing unit is used to correct the collected original endoscopic image according to the internal parameters and distortion parameters obtained by calibrating the polarization measurement binocular endoscope system, and ensure that the binocular image remains consistent in the horizontal direction;

[0038] The polarization information encoding unit is used to decouple polarization images of different polarization angles and calculate corresponding polarization information encoding.

[0039] Furthermore, the image depth module includes:

[0040] A feature extraction unit, used to perform deep feature extraction on the RGB image group and the polarization information encoded image group respectively using an RGB feature encoder and a polarization feature encoder;

[0041] A cost volume construction unit, used to construct relevant volumes for RGB domain and polarization domain respectively;

[0042] 3D aggregation unit, used to adaptively weight and fuse RGB-related volume and polarization-related volume, stimulating effective complementary information of the two for enhancement;

[0043] The depth prediction unit is used to output the effective depth information of the target.

[0044] Furthermore, the image display module is also used to perform observations at different viewing angles according to user needs and to achieve real-time freezing and storage.

[0045] Compared with the prior art, the advantages and beneficial effects of the present invention are as follows: the present invention proposes an end-to-end RGB+polarization fusion network, which uses the dense surface normal information carried by the polarization mode to enhance the depth information of binocular stereo vision, and uses the consistency constraint of the three-dimensional shape of polarization information and the stereoscopic vision depth estimation to guide the network to adaptively learn and purify the effective depth information, which effectively reduces the limitation of difficult matching of feature points in the complex working environment of the endoscope system, and significantly improves the accuracy of three-dimensional reconstruction under weak texture, so that doctors can more easily locate lesions in complex working environments, improve the ability to observe tiny structures, and thus improve the safety and accuracy of surgery. At the same time, the time complexity and space complexity of the method of the present invention are very low, and only very low hardware cost is required to achieve it, which is easy to promote and apply in engineering. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 is a flow chart of a three-dimensional reconstruction method of an endoscopic image according to Embodiment 1 of the present invention;

[0047] Figure 2 is a simplified schematic diagram of a polarization measurement binocular endoscope system according to Embodiment 1 of the present invention;

[0048] Figure 3 This is a structural diagram of the RGB+polarization fusion network framework of Example 1 of the present invention;

[0049] Figure 4 is a three-dimensional reconstruction result diagram under a working condition of Example 1 of the present invention;

[0050] Figure 5 is a three-dimensional reconstruction result diagram under another working condition of Example 1 of the present invention;

[0051] Figure 6 It is a schematic structural diagram of a three-dimensional reconstruction device for endoscopic images according to Embodiment 2 of the present invention. DETAILED DESCRIPTION

[0052] The present invention will be described in detail below based on the accompanying drawings and preferred embodiments, and the purpose and effects of the present invention will become more clear. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0053] Embodiment 1:

[0054] like Figure 1 As shown, an embodiment of the present invention provides a three-dimensional reconstruction method of an endoscopic image, comprising:

[0055] S1: Build a polarization measurement binocular endoscope system. Modify the existing commercial binocular endoscope to establish a polarization measurement binocular endoscope system that can use the system's own illumination channel to provide non-polarized illumination. The simplified schematic diagram is as follows: Figure 2 shown.

[0056] S2: Receive the original endoscopic images of the polarization measurement binocular endoscope system in different scenes, perform image correction processing on the images, decouple to obtain the corresponding RGB images and polarization images with different polarization angles, and calculate the corresponding polarization information encoding; the specific steps are as follows:

[0057] First, the endoscope system is calibrated using the Zhang Zhengyou calibration method to obtain the system internal parameter matrix M and distortion parameter matrix D, as well as the rotation matrix R and translation matrix T between the left and right eyes. Then, the collected original image is corrected based on these parameters to determine the final effective field of view. And ensure that the binocular images are consistent in the horizontal direction.

[0058] The endoscope system is equipped with two polarization cameras (CCD1 and CCD2) at the back end. The polarization camera is a focal plane camera that integrates four polarizers in different directions (angles) on the CMOS. Figure 2 As shown, the polarization images I0, I1 and I2 at four different polarizer angles can be obtained by decoupling according to the arrangement order of the polarizer array through one shot. 45 ,I 90 and I 135 Finally, the polarization information encoding is calculated as the input data of the subsequent network.

[0059]

[0060] Among them, I0, I 45 ,I 90 and I 135 Respectively represent polarization images when the polarizer angle is 0°, 45°, 90°, and 135°. is the average intensity, a scalar It is called the degree of linear polarization, and the angle is called the linear polarization angle, Indicates polarization information encoding.

[0061] S3: Build an end-to-end RGB+polarization fusion network, use the dense surface normal information carried by the polarization mode to enhance the depth information of binocular stereo vision, and use the calculated polarization information encoding and RGB image group as input to obtain the corresponding depth map in real time; the network architecture is as follows Figure 3 As shown, the specific steps are as follows:

[0062] Feature extraction. The present invention uses a ResNet-like architecture. First, a set of 3×3 kernel convolutional layers are used to downsample the input image. Then, convolutional layer 2 with sixteen residual layers, convolutional layer 3 with three residual layers, and convolutional layer 4 are used to further expand the receptive field. The output feature map size is 1 / 4 × 1 / 4 of the input image size. Finally, all feature maps are connected to form a 320-channel unary feature map.

[0063] Cost volume construction. Considering that RGB images and polarization images belong to different levels of modal information, it is difficult to effectively stimulate the effective complementary information of the two modalities by directly connecting them in series in the input image dimension. For this reason, the present invention constructs relevant volumes for the RGB domain and the polarization domain respectively. The surface normal information implicitly contained in the polarization-related volume can be used as a supplement to conventional RGB information to enhance the depth information of binocular stereo vision. Especially in the case that most endoscopic images have low texture and overexposed areas, the introduced polarization cues play an important role. Specifically, two convolutions are first applied to compress the 320-channel feature map into a 32-channel feature map, and then the left feature map of each disparity level is compressed. The corresponding right feature map Connect them together to build a 4D body .

[0064]

[0065] in, represents parallax, represents coordinates, represents the fourth dimension, Indicates series connection.

[0066] Cost summary. RGB related volume and polarization-dependent volume The adaptive weighted fusion is used as the input of the 3D aggregation network. The adaptive weighted fusion module consists of multiple 3×3 convolutional layers and ReLU activation function layers.

[0067]

[0068] Among them, AWF represents the adaptive weighted fusion module, is the fusion volume, Represents a matrix multiplication operation.

[0069] The consistency constraint of the three-dimensional shape of polarization information and stereoscopic vision depth estimation is used to guide the network to adaptively learn and purify effective depth information, thereby improving the accuracy of depth estimation. The 3D aggregation network consists of four 3D convolutions with batch normalization and ReLU and two stacked 3D hourglass networks. Each hourglass network consists of four 3D convolutional layers and two 3D deconvolutional layers.

[0070] Depth prediction. Due to the direct relationship between polarization normal clues and depth, this invention is different from conventional binocular disparity prediction and chooses to let the network directly predict depth information. There is one output before the hourglass network and after each hourglass network. For each output module, a 4D volume with 1 channel is first generated through two 3D convolutions. Then, this volume is upsampled and converted into a probability volume using the softmax function. The predicted depth It is calculated as the sum of each depth d weighted by its probability,

[0071]

[0072] in, Indicates the depth value, Represents probability.

[0073] Loss function. The smooth L1 loss function is used to train the network. Given by:

[0074]

[0075] in, represents the coefficient of the i-th disparity prediction, represents the real depth map, represents the smooth L1 loss function, Represents the independent variable.

[0076] Combined with the loss function, the RGB+polarization fusion network is trained, and the cosine annealing strategy is used to adjust the network learning rate during the training process. Finally, the trained network is used to obtain the corresponding depth map in real time.

[0077] S4: Reconstruct the image using the current image pose parameters and depth map to obtain a 3D visualization point cloud; the specific steps are:

[0078] Get the pixel coordinates of the effective field of view of the current image;

[0079] According to the current pose parameters and pixel coordinates of the image and the corresponding depth map obtained in step S3, the world space coordinates of the target are calculated, that is, the conversion from two-dimensional coordinates to three-dimensional coordinates. According to the depth value obtained by accurate depth estimation, the accuracy of the calculated world space coordinates of the target will also be improved;

[0080] The RGB value of the color information of each pixel in the image is obtained, and the point cloud is reconstructed according to the color information and the world space coordinates of the target to obtain a three-dimensional point cloud (X, Y, Z, R, G, B).

[0081]

[0082] Among them, X, Y, Z represent world space coordinates; Represents depth information; u and v are arbitrary coordinate points in the image coordinate system; , are the center coordinates of the image respectively; , They are all posture parameters.

[0083] Figure 4 , Figure 5 The following are comparison diagrams of the 3D reconstruction results of the method of the present invention and other methods under different working conditions (orthographic projection perspective diagram and depth perception perspective diagram), in which from left to right are the true value (GT), the result of the SGBM algorithm, the result of the network without adding a polarization module (Only RGB), and the result of the network adding a polarization module (RGB+polarization). It can be seen from the figure that the point cloud reconstructed by SGBM has obvious faults in the hole area, and there are also a lot of undulating noises of varying depths distributed on the surface of the rest of the area. The point cloud reconstructed by deep learning (Only RGB without polarization) not only eliminates the influence of the hole, but also has almost no undulating noise on the surface of the point cloud, and the distribution is more continuous and smooth. However, from the depth perception perspective, it can be seen that the point cloud reconstructed by the Only RGB method has obvious differences in some areas (such as Figure 4 , Figure 5 The arrows in the figure show that the depth restored in the weak texture area is still inaccurate. In contrast, the method proposed in this invention (RGB+polarization) is closer to the true value at different viewing angles and can restore the accurate 3D morphology of low-texture and featureless areas as expected, providing important theoretical support and practical value for clinical high-quality 3D endoscopic imaging.

[0084] Embodiment 2:

[0085] like Figure 6 As shown, an embodiment of the present invention provides a three-dimensional reconstruction device for endoscopic images, comprising:

[0086] The image acquisition module 10 is used to receive original endoscopic images of the polarization measurement binocular endoscope system in different scenes.

[0087] The image processing module 20 is used to perform correction processing on the original image, decouple the corrected image to obtain the corresponding RGB image and polarization images of different polarization angles, and calculate the corresponding polarization information code.

[0088] An image depth module 30 is used to input the calculated polarization information encoding and the RGB image group into a pre-trained 3D reconstruction network model to capture the corresponding depth map in real time;

[0089] The image display module 40 is used to calculate the world space coordinates of the target according to the current posture parameters of the image and the depth map, and to display the three-dimensional visualization interface of the target in real time.

[0090] Specifically, the image acquisition module 10 includes: receiving original images of the endoscope system in different scenes, temporarily storing them, judging the image clarity and the region of interest, and if yes, inputting them to the next module. If no, re-acquiring the image.

[0091] Specifically, the image processing module 20 includes:

[0092] The image preprocessing unit corrects the collected original endoscopic image according to the internal parameters and distortion parameters obtained by calibrating the polarization measurement binocular endoscope system, and ensures that the binocular image remains consistent in the horizontal direction.

[0093] The polarization information encoding unit decouples polarization images of different polarization angles and calculates corresponding polarization information encoding.

[0094] The image depth module comprises:

[0095] The feature extraction unit uses an RGB feature encoder and a polarization feature encoder to perform deep feature extraction on the RGB image group and the polarization information encoded image group respectively.

[0096] Cost volume construction unit, which constructs relevant volumes for RGB domain and polarization domain respectively.

[0097] Specifically, the image depth module 30 includes:

[0098] The feature extraction unit uses an RGB feature encoder and a polarization feature encoder to perform deep feature extraction on the RGB image group and the polarization information encoded image group respectively.

[0099] Cost volume construction unit, which constructs relevant volumes for RGB domain and polarization domain respectively.

[0100] The 3D aggregation unit adaptively weights and fuses the RGB-related volume and the polarization-related volume, stimulating the effective complementary information of the two for enhancement.

[0101] The depth prediction unit outputs the effective depth information of the target.

[0102] Specifically, the image display module 40 includes: a three-dimensional visualization interface that displays in real time in accordance with the doctor's visual perception, can perform observations from different viewing angles according to user needs, and can achieve real-time freezing and storage.

[0103] It should be noted that, since the information interaction, execution process, etc. between the units in the above-mentioned devices and systems are based on the same concept as the embodiments of the method of the present invention, the specific contents can be found in the description of the embodiments of the method of the present invention and will not be repeated here.

[0104] Those skilled in the art can understand that the above are only preferred examples of the invention and are not intended to limit the invention. Although the invention is described in detail with reference to the above examples, those skilled in the art can still modify the technical solutions recorded in the above examples or replace some of the technical features therein with equivalents. Any modification, equivalent replacement, etc. made within the spirit and principle of the invention shall be included in the protection scope of the invention.

Claims

1. A three-dimensional reconstruction method for endoscopic images, characterized in that: The steps include: S1: Build a polarization measurement binocular endoscope system; S2: Receive the original endoscope images of the polarization measurement binocular endoscope system under different scenes, decouple to obtain the corresponding RGB images and polarization images with different polarization angles, and calculate the corresponding polarization information encoding; S3: Build an end-to-end RGB+polarization fusion network, use the dense surface normal information carried by the polarization mode to enhance the depth information of binocular stereo vision, and use the calculated polarization information encoding and RGB image group as input to obtain the corresponding depth map in real time; The end-to-end RGB+polarization fusion network consists of four parts: feature extraction, cost volume construction, cost aggregation and depth prediction; The feature extraction includes: using a set of 3×3 kernel convolution layers to downsample the input image, and then using a convolution layer with sixteen residual layers and two convolution layers with three residual layers to further expand the receptive field; the output feature map size is 1 / 4 × 1 / 4 of the input image size; all feature maps are connected to form a 320-channel unary feature map; The cost volume construction includes: constructing relevant volumes for RGB domain and polarization domain respectively; firstly applying two convolutions to compress the 320-channel feature map into a 32-channel feature map, and then The corresponding right feature map Connect them together to build a 4D body; The cost summary includes: converting the RGB related volume and polarization-dependent volume Adaptive weighted fusion is used as the input of the 3D aggregation network. The adaptive weighted fusion module consists of several 3×3 convolutional layers and ReLU activation function layers. The consistency constraint of the three-dimensional shape of polarization information and the stereoscopic vision depth estimation is used to guide the network to adaptively learn and purify the effective depth information to improve the depth estimation accuracy. The 3D aggregation network consists of four 3D convolutions with batch normalization and ReLU and two stacked 3D hourglass networks. Each hourglass network consists of four 3D convolutional layers and two 3D deconvolutional layers. The depth prediction includes: generating a 4D volume with 1 channel through two 3D convolutions for one output before the hourglass network and after each hourglass network; upsampling the volume and converting it into a probability volume using a softmax function, and predicting the depth Calculate the weighted sum of each depth d according to its probability; use the smooth L1 loss function to train the network; train the RGB+polarization fusion network in combination with the loss function, and use the trained network to obtain the corresponding depth map in real time; S4: Reconstruct the image using the current pose parameters of the image and the depth map to obtain a three-dimensional visual point cloud.

2. The three-dimensional reconstruction method of endoscopic images according to claim 1, characterized in that: In step S2, the corresponding polarization information codes are calculated according to the polarization images at different polarization angles. The process of calculating the polarization information codes is as follows: ； Among them, I0, I 45 ,I 90 and I 135 Respectively represent polarization images when the polarizer angle is 0°, 45°, 90°, and 135°. is the average intensity, a scalar is the degree of linear polarization, angle is the linear polarization angle, Indicates polarization information encoding.

3. The three-dimensional reconstruction method of endoscopic images according to claim 1, characterized in that: The feature extraction is specifically as follows: using a set of 3×3 kernel convolutional layers to downsample the input image, and then using a convolutional layer with sixteen residual layers and two convolutional layers with three residual layers to further expand the receptive field; the output feature map size is 1 / 4 × 1 / 4 of the input image size; all feature maps are connected to form a 320-channel unary feature map; The cost volume is constructed as follows: the relevant volumes are constructed in the RGB domain and the polarization domain respectively; first, two convolutions are applied to compress the 320-channel feature map into a 32-channel feature map, and then the left feature map of each disparity level is The corresponding right feature map Connect them together to build a 4D body ; ； in, represents parallax, represents coordinates, represents the fourth dimension, Indicates series connection; The cost summary is specifically as follows: and polarization-dependent volume The adaptive weighted fusion is used as the input of the 3D aggregation network. The adaptive weighted fusion module consists of several 3×3 convolutional layers and ReLU activation function layers; ； Among them, AWF represents the adaptive weighted fusion module, is the fusion volume, Represents matrix multiplication operation; The consistency constraint between the three-dimensional shape of polarization information and the depth estimation of stereo vision is used to guide the network to adaptively learn and purify the effective depth information, thereby improving the accuracy of depth estimation; the 3D aggregation network is composed of four 3D convolutions with batch normalization and ReLU and two stacked 3D hourglass networks; each hourglass network is composed of four 3D convolution layers and two 3D deconvolution layers; The depth prediction is specifically as follows: one output before the hourglass network and after each hourglass network corresponds to a 4D volume with one channel through two 3D convolutions; the volume is upsampled and converted into a probability volume using a softmax function, and the predicted depth It is calculated as the sum of each depth d weighted by its probability, expressed as: ； in, Indicates the depth value, represents probability; The smooth L1 loss function is used to train the network, and the loss function The expression is: ； in, represents the coefficient of the i-th disparity prediction, represents the real depth map, represents the smooth L1 loss function; The RGB+polarization fusion network is trained in combination with the loss function, and the trained network is used to obtain the corresponding depth map in real time.

4. The three-dimensional reconstruction method of endoscopic images according to claim 3, characterized in that: The surface normal information implicitly contained in the polarization-dependent volume complements the RGB information to enhance the depth information of binocular stereo vision.

5. The three-dimensional reconstruction method of endoscopic images according to claim 1, characterized in that: The step S4 is specifically as follows: obtaining the pixel coordinates of the current image; calculating the world space coordinates of the target based on the current posture parameters and pixel coordinates of the image and the corresponding depth map obtained in step S3; obtaining the color information of each pixel in the image, and reconstructing the point cloud based on the color information and the world space coordinates of the target to obtain a three-dimensional point cloud.

6. A three-dimensional reconstruction device for endoscopic images implementing the method as claimed in claim 1, characterized in that: include: An image acquisition module, used for receiving original endoscopic images of a polarization measurement binocular endoscope system in different scenes; An image processing module is used to correct the original endoscopic image, decouple the corrected image to obtain the corresponding RGB image and polarization images of different polarization angles, and calculate the corresponding polarization information code; Image depth module, used to input the calculated polarization information encoding and RGB image group into the pre-trained 3D reconstruction network model to capture the corresponding depth map in real time; The image display module is used to calculate the world space coordinates of the target based on the current posture parameters and depth map of the image, and display the three-dimensional visualization interface of the target in real time.

7. The three-dimensional reconstruction device of endoscopic images according to claim 6, characterized in that: The image processing module comprises: An image preprocessing unit is used to correct the collected original endoscopic image according to the internal parameters and distortion parameters obtained by calibrating the polarization measurement binocular endoscope system, and ensure that the binocular image remains consistent in the horizontal direction; The polarization information encoding unit is used to decouple polarization images of different polarization angles and calculate corresponding polarization information encoding.

8. The three-dimensional reconstruction device of endoscopic images according to claim 6, characterized in that: The image depth module comprises: A feature extraction unit, used to perform deep feature extraction on the RGB image group and the polarization information encoded image group respectively using an RGB feature encoder and a polarization feature encoder; A cost volume construction unit, used to construct relevant volumes for RGB domain and polarization domain respectively; 3D aggregation unit, used to adaptively weight and fuse RGB-related volume and polarization-related volume, stimulating effective complementary information of the two for enhancement; The depth prediction unit is used to output the effective depth information of the target.

9. The three-dimensional reconstruction device of endoscopic images according to claim 6, characterized in that: The image display module is also used to perform observations at different viewing angles according to user needs and to achieve real-time freezing and storage.

Images