Monocular depth estimation network establishment method based on dynamic view selection and application thereof

Abstract

The invention discloses a monocular depth estimation network establishment method based on dynamic view selection and application thereof, and belongs to the field of monocular endoscope depth estimation.The monocular depth estimation network establishment method comprises the steps that a training data set is constructed, and joint training is conducted on a pose estimation network PoseNet and a depth estimation network DepthNet through the training data set with luminosity loss as a constraint, so that the depth estimation network PoseNet and the depth estimation network DepthNet are obtained; taking the DepthNet after joint training as a monocular depth estimation network; the construction mode of the training data comprises the following steps: respectively taking N historical frames before a target image as source images, and calculating a time consistency score corresponding to each source image; the time consistency score reflects the difference between the depth map and an ideal reference depth map; and selecting a part of historical frames with the highest time consistency score as candidate source images, selecting the candidate source image with the largest luminosity effective area as a final source image, and forming training data together with the target image. According to the method, the accuracy of monocular endoscope image depth estimation can be effectively improved.

Description

Technical Field

[0001] The present invention belongs to the field of monocular endoscope depth estimation, and more specifically, relates to a method for establishing a monocular depth estimation network based on dynamic view selection and an application thereof. Background Art

[0002] Monocular endoscopes play an important role in gastrointestinal diagnosis and surgical operations. Automatic 3D depth estimation is the key for endoscopic robots to fully understand the surgical scene and expand the limited field of view through 3D scene reconstruction. Since the field of view of videos captured by endoscopes is usually very narrow, doctors need to repeatedly observe the entire tissue to form a judgment. 3D scene reconstruction helps to expand the field of view and assist other downstream tasks, such as surgical navigation through model alignment with preoperative computed tomography (CT). Depth estimation of monocular endoscopic images is a prerequisite for reconstructing 3D scenes. In a living environment, accurate monocular endoscope depth estimation is extremely challenging due to the lack of depth labels.

[0003] At present, monocular depth estimation based on deep learning mainly relies on self-supervised learning, and its core idea is the photometric constraint between the real image and the distorted image. The goal of self-supervised monocular depth estimation is to achieve photometric consistency, which is defined as the pixel similarity between the target image and the aligned source image. Specifically, two convolutional neural networks (CNNs) need to be constructed, one called the depth estimation network (DepthNet), which can recover the depth information, based on which the source image can be projected into the 3D space. The other is called the pose estimation network (PoseNet), which can obtain the relative pose from the source view to the target view. The relative pose is used to transform the 3D spatial coordinates of the source image into the 3D space of the target view, and the distorted image is reconstructed by backprojection and bilinear interpolation. In this process, DepthNet and PoseNet achieve joint training of DepthNet and PoseNet by minimizing the photometric loss (essentially the pixel-level difference between the distorted image and the target image), thereby achieving joint optimization of the parameters in DepthNet and PoseNet. After that, the optimized DepthNet can be used to achieve depth estimation of monocular endoscopic images.

[0004] In order to achieve joint optimization of DepthNet and PoseNet, it is necessary to use source image-target image pairs as training data. In the stereo depth estimation task of binocular images, the left and right views are fixedly used as source image-target image pairs, and their relative poses are obtained in advance through calibration. In monocular scenes, there is only a single camera sensor, and its depth estimation is more challenging. Most researchers use adjacent frames in the video data captured by a monocular endoscope as source image-target image pairs. The time interval between adjacent image frames is often tens of milliseconds, so the repeated visual field areas in the image frames can be used to assist network training. However, this common static training method of using adjacent frames as source image-target image pairs inevitably brings a problem, that is, these image frames captured in a very short time contain small camera motions and provide very limited visual clues, which brings great difficulties to pose estimation. In other words, the source image-target image pairs constructed using adjacent image frames are not conducive to pose alignment of the pose estimation network, which also leads to large errors in the depth of the monocular endoscope image estimated by the depth estimation network obtained through joint training. Summary of the invention

[0005] In view of the defects of the prior art and the need for improvement, the present invention provides a method for establishing a monocular depth estimation network based on dynamic view selection and its application, with the aim of effectively solving the problems of difficult pose alignment and large depth estimation error when statically using adjacent frames as image pairs, thereby effectively improving the accuracy of depth estimation of monocular endoscopic images.

[0006] To achieve the above object, according to one aspect of the present invention, a method for establishing a monocular depth estimation network based on dynamic view selection is provided, comprising:

[0007] Step S1: construct a training data set; in the training data set, each piece of training data is a source image-target image pair, and the construction method of each training data includes:

[0008] A frame in a monocular video frame sequence is selected as the target image. For the N historical frames before the target image, each historical frame is used as the source image, and the cost cube from the source image to the target image is calculated. Each cost cube is converted into a depth map corresponding to each historical frame to obtain a depth map set.

[0009] Based on the depth map set, the depth values ​​that appear at each pixel position and the number of occurrences of each depth value are counted, and the depth value with the largest number of occurrences at each position is used as the depth value at the corresponding position to obtain the reference depth image D mv , and calculate the temporal consistency score of each historical frame; the depth map corresponding to the historical frame and the reference depth image D mvThe smaller the difference, the higher the temporal consistency score;

[0010] Select some historical frames with the highest temporal consistency score as candidate source images, and select the candidate source image with the largest photometric effective area as the final source image, which together with the target image constitute a piece of training data; the photometric effective area of ​​the image is the intersection of the non-occluded area of ​​the candidate image and the common field of view area between the image and the target image;

[0011] Step S2: Taking photometric loss as a constraint, the pose estimation network PoseNet and the depth estimation network DepthNet are jointly trained using the constructed training data set, and the jointly trained depth estimation network DepthNet is used as a monocular depth estimation network;

[0012] Among them, N is an integer greater than 1; the pose estimation network PoseNet estimates the relative pose of the target image to the source image, and the depth estimation network DepthNet is used to estimate the depth map of the image.

[0013] Furthermore, I (0) Represents the target image, with I (-n) represents the nth historical frame image before the target image, then image I (-n) With the target image I (0) Methods for determining the common visual field between the two include:

[0014] Use the pre-trained pose estimation network PoseNet to estimate the target image I (0) To Image I (-n) The relative position t (-n) ;

[0015] According to the relative posture T (-n) and the reference depth image D mv Image I (-n) To the target image I (0) The viewpoints are aligned to obtain the pixel p in the distorted image. (-n) and I (0) Medium pixel p (0) The corresponding relationship;

[0016] The pixel p in the distorted image that is in the range [0,W-1]×[0,H-1] (-n) The region formed is determined as image I (-n) With the target image I (0) The common visual field between

[0017] Where W and H are the width and height of the image, respectively, n∈{1,2…,N}.

[0018] Furthermore, image I (-n)The non-occluded area is determined by:

[0019] Use the pre-trained depth estimation network DepthNet to estimate image I (-n) The depth map D (-n) ;

[0020] Use the pre-trained pose estimation network PoseNet to estimate image I (-n) To the target image i (0) The relative position t (-n) ′;

[0021] According to the relative posture t (-n) ′ and the depth map d (-n) The target image i (0) To image i (-n) The viewpoints are aligned to obtain the distorted image i (-n) ′;

[0022] Use the pre-trained pose estimation network PoseNet to estimate the target image i (0) To the distorted image I (-n) The relative position T of (-n) ″;

[0023] According to the relative posture T (-n) '" and the reference depth image D mv The distorted image I (-n) ′ to the target image I (0) The viewpoints are aligned to obtain the distorted image I (0) ″;

[0024] The distorted image I (0) ″ and the target image I (0) The area where the difference is not greater than the preset threshold is determined as image I (-n) non-occluded area.

[0025] Furthermore, image I (-n) To the target image I (0) The cost cube is calculated by:

[0026] In the parallax range [d min ,d max ] to obtain N d Depth Plane d min and d max Respectively represent the minimum depth value and the maximum depth value;

[0027] For each depth plane D i , according to the relative posture T (-n) and the depth plane Di , image I (-n) To the target image I (0) The viewpoints are aligned to obtain the distorted image And calculate As the matching cost at this depth value; i = 0, 1, 2...N d -1;

[0028] The matching costs at each depth value are concatenated to obtain image I (-n) To the target image I (0) The cost cube C (-n) .

[0029] Further, Represents image I (-n) The corresponding depth map, then the depth map In the example, the depth value at any position (x, y) is:

[0030]

[0031] Among them, C (-n) (x, y, i) represents the matching cost corresponding to the i-th depth value at position (x, y) in the cost cube; Indicates the depth value with the minimum matching cost at the position (x, y).

[0032] Furthermore, with S (-n) Represents image I (-n) The temporal consistency score of

[0033]

[0034] Among them, D mv (x,y) represents the reference depth image D mv The depth value at position (x,y) in the image.

[0035] Furthermore, I s and I t Denote the source image and the target image respectively, with T t→s Represents the target image I t To source image I s The relative position of t Represents the target image I t The depth map of t→s and the depth map D t The source image I s To the target image I t After the viewpoints are aligned, the distorted image I s→t for:

[0036] Is→t (p s→t )=I s <proj(D t ,T t→s ,K,p s→t )>

[0037] Among them, p s→t represents pixel homogeneous coordinates, K represents the given camera intrinsic parameter matrix, proj(·) is the projection function, and <> represents differentiable bilinear sampling.

[0038] Furthermore, the expression of luminosity loss is as follows:

[0039]

[0040] in, represents photometric loss, SSIM represents structural similarity index metric, and α is the weight parameter.

[0041] According to another aspect of the present invention, a monocular endoscope depth estimation method is provided, comprising:

[0042] Inputting the monocular endoscopic image to be estimated into a monocular depth estimation network to obtain a depth map of the estimated monocular image;

[0043] Among them, the monocular depth estimation network is established by the above-mentioned monocular depth estimation network establishment method based on dynamic view selection provided by the present invention.

[0044] According to another aspect of the present invention, a computer-readable storage medium is provided, comprising a stored computer program; when the computer program is executed by a processor, the device where the computer-readable storage medium is located is controlled to execute the above-mentioned method for establishing a monocular depth estimation network based on dynamic view selection provided by the present invention, and / or the above-mentioned monocular depth estimation method provided by the present invention.

[0045] Through the above technical scheme conceived by the present invention, on the basis of the trained pose estimation network PoseNet and depth estimation network DepthNet, a higher quality training data set is further constructed to jointly train and optimize the two. When constructing the training data set, for the target image, the present invention does not directly use the adjacent frames as the source images, but selects the most suitable historical frames from the previous multiple historical frames as the source images, so as to realize the dynamic matching of the most suitable source images within a larger time span, and greatly reduce the interference of the static use of adjacent frames on the pose alignment. When selecting the source image, the time consistency score of each historical frame is first calculated, and the time consistency score reflects the time consistency of the corresponding historical frame as the source image. When imaging, the difference between the estimated target image depth map and the ideal depth map is calculated, and then some frames with high consistency scores are selected as candidate source images, which can ensure the accuracy of depth estimation to a certain extent; in the process of photometric optimization loss, sufficient effective area is required. Therefore, on the basis of selecting candidate source images, the present invention further selects the candidate source image with the largest photometric effective area as the final matching source image, and pairs it with the target image, thereby ensuring that when PoseNet and DepthNet are jointly trained based on photometric loss, PoseNet can be effectively optimized, and finally the accuracy of PoseNet for monocular endoscope depth estimation can be effectively improved.

[0046] In general, the present invention selects frames with high temporal consistency from multiple historical frames of the target image as candidate source images, and selects the candidate source image with the largest photometrically effective area as the final source image, which is paired with the target image as training data for the joint training of PoseNet and DepthNet, which can effectively improve the depth estimation accuracy of the trained depth estimation network. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 A schematic diagram of a method for establishing a monocular depth estimation network based on dynamic view selection provided by an embodiment of the present invention;

[0048] Figure 2 Synthesize schematic diagram for existing distorted images;

[0049] Figure 3 A schematic diagram of calculating a temporal consistency score provided by an embodiment of the present invention;

[0050] Figure 4 A schematic diagram of calculating the effective area of ​​luminosity provided by an embodiment of the present invention;

[0051] Figure 5 The figure is a schematic diagram comparing the estimation errors of the embodiment of the present invention and other depth estimation methods. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0053] In the present invention, the terms "first", "second", etc. (if any) in the present invention and the accompanying drawings are used to distinguish similar objects but not necessarily to describe a specific order or sequence.

[0054] In order to solve the technical problem that the existing monocular endoscopic image depth estimation method cannot effectively achieve pose alignment when directly using adjacent frames as source image-target image pairs to realize joint training of the pose estimation network PoseNet and the depth estimation network DepthNet, resulting in low depth estimation accuracy of the optimized depth estimation network DepthNet, the present invention provides a monocular depth estimation network establishment method based on dynamic view selection and its application, the overall concept of which is to improve the construction method of the training data set, for a given target image, dynamically select the most suitable source image based on time consistency and effective photometric area within a longer time span, and pair it with the target image, thereby reducing the interference of static use of adjacent frames on pose alignment, and adapting to the characteristics of joint training of the pose estimation network PoseNet and the depth estimation network DepthNet under photometric loss constraints, effectively improving the estimation accuracy of the final depth estimation network.

[0055] Based on the above concept, in order to effectively improve the training efficiency and ensure the training effect, the present invention will first obtain a pre-trained pose estimation network PoseNet and a depth estimation network DepthNet with certain estimation capabilities. The pre-trained pose estimation network PoseNet and the depth estimation network DepthNet can be directly selected from the trained network, or they can be trained using the existing training methods. Without loss of generality, the following embodiments include the pre-training process of the pose estimation network PoseNet and the depth estimation network DepthNet.

[0056] The following are examples.

[0057] Embodiment 1:

[0058] A method for establishing a monocular depth estimation network based on dynamic view selection, such as Figure 1 As shown, this embodiment includes the following steps S0-S2:

[0059] Step S0: Use adjacent frames to pre-train the pose estimation network PoseNet and the depth estimation network DepthNet;

[0060] Step S1: constructing a training dataset based on the proposed method of dynamically constructing source image-target image pairs;

[0061] Step S2: Taking photometric loss as a constraint, the pose estimation network PoseNet and the depth estimation network DepthNet are jointly trained using the constructed training dataset, and the jointly trained depth estimation network DepthNet is used as a monocular depth estimation network.

[0062] The following is a detailed explanation of each step.

[0063] In this embodiment, step S0 specifically includes:

[0064] Step S01: preprocessing the original image: first, considering the computing resources and the data size required by the network, the sizes of all video frames are scaled to a uniform size (320×156 in this embodiment); secondly, the input images are uniformly enhanced as follows:

[0065] 1) Horizontal flip;

[0066] 2) Color transformation, that is, the brightness, contrast and saturation are transformed by multiples in the range of [0.8, 1.2], and the hue is transformed in the range of [-0.1, 0.1];

[0067] Step S02: construct source image-target image pairs using adjacent frames, and use the constructed source image-target image pairs as training data to complete the pre-training of PoseNet and DepthNet.

[0068] In order to obtain the pose information between different views and obtain more reliable aligned views in the final self-supervised training, it is first necessary to use two adjacent frames of endoscopic images to calculate the photometric error between the target image and the distorted image.

[0069] The depth estimation network DepthNet is used to predict the depth of each pixel, and the pose estimation network PoseNet is used to predict the relative camera pose of two frames of images. Optionally, in this embodiment, both DepthNet and PoseNet are fully convolutional networks. PoseNet includes a lightweight encoder based on ResNet18, which has four convolutional layers. By inputting two images at the same time, the relative pose from one view to another can be predicted. The process of synthesizing distorted images is as follows: Figure 2 As shown, the target image is I t , using adjacent image frames as source images I sFirst, DepthNet estimates the target frame I t The depth map D t , PoseNet estimates the relative camera motion T from the target view to the source view t→s . Using D t and T t→s , so that the source view I s To target view I t The viewpoints are aligned to form a new distorted image I s→t The resulting distorted image I s→t The matching pixels corresponding to their source images satisfy the following relationship:

[0070] I s→t (p s→t )=I s <proj(D t ,T t→s ,K,p s→t )>

[0071] Among them, p s→t Represents the pixel homogeneous coordinates, K represents the given camera intrinsic parameter matrix, and proj(·) is the projection function. With the above pixel matching relationship, I s→t We can use the differentiable bilinear sampling represented by <·> from I s Get the value of each pixel position. After the transformation, DepthNet and PoseNet can be constrained by photometric loss, which is calculated as follows:

[0072]

[0073] in, represents photometric loss, SSIM represents structural similarity index measurement, and α is a weight parameter; optionally, in this embodiment, the value of α is 0.85.

[0074] It is easy to understand that the above structural description of DepthNet and PoseNet is only an optional implementation of the present invention and should not be understood as the only limitation of the present invention. In other embodiments of the present invention, other structures or even other types of network structures may also be used. In addition, in some other embodiments of the present invention, if the trained DepthNet and PoseNet are directly used, there is no need to perform step S0.

[0075] In this embodiment, in step S1, for each source image-target image pair in the training data set, the construction method is as follows:

[0076] In order to solve the problem of difficult alignment of camera poses in adjacent image frames, a cost cube calculated using frozen PoseNet is constructed to select reliable views, such as Figure 3 shown.

[0077] First, take the current frame I (0) is the target image, for I (0) For the past N historical frames, use the frozen pre-trained PoseNet to predict I (0) To the nth frame before it I (-n) The relative position T (-n) .

[0078] In order to calculate the matching cost under different depth probabilities, this embodiment uses the minimum depth d min To the maximum depth d max Some depths D are defined by mean sampling in the interval i Each D i Represents a depth plane, N d The number of depth planes determined by the mean sampling is determined by the sampling interval. Then, each depth plane D i Perform pose alignment and combine the depth planes D i The corresponding distorted image Figure 2 The method of synthesizing the distorted image is the same as shown in the figure. The coordinates and I (-n) The coordinates of satisfy the following correspondence:

[0079]

[0080] By exhaustively defining all depth planes, a set of reconstructed images can be obtained 3D cost cube C (-n) It can be expressed as:

[0081]

[0082] Among them, (x, y) represents the two-dimensional spatial coordinates of the image, and I is the new dimension of the cost cube, which represents the matching cost at different depth planes. The cost cube temporal flow can be further calculated from the past N historical frames, that is, the three-dimensional cost cube C calculated from the N historical frames (-n) These cost cubes represent the depth probability distribution of the target image viewpoint. No matter how the pose changes between different frames, the depth probability distribution of the target frame viewpoint should remain consistent.

[0083] In order to obtain the temporal consistency score, this embodiment first transforms the three-dimensional cost cube C into (-n) Convert to depth map

[0084]

[0085] Use the minimum matching cost method to obtain depth maps under different historical frames Collection Perform majority voting in the time dimension to select the reference depth image D mv The depth value at each position in the , that is, based on the depth map set The depth values ​​that have appeared at each pixel position and the number of times each depth value appears are counted, and the depth value with the largest number of occurrences at each position is taken as the depth value at the corresponding position. The reference depth image thus determined represents a typical stable and temporally consistent depth distribution.

[0086] Based on the determined reference depth image D mv , this embodiment further calculates the temporal consistency score of each historical frame; the depth map corresponding to the historical frame and the reference depth image D mv The smaller the difference is, the higher the temporal consistency score is. (-n) I (-n) The time consistency score of S is, in this embodiment, (-n) Under the expression:

[0087]

[0088] When the temporal consistency score calculated in this embodiment takes the corresponding historical frame as the source image, the estimated target image depth map is consistent with the ideal depth map (i.e., the reference depth image D mv ). After obtaining the temporal consistency score of each historical frame, this embodiment selects the largest K frames as candidate source images, and these candidate source images are further screened to determine the source image that finally matches the target image. This embodiment uses the reference depth image as a benchmark to determine the candidate source images, ensuring that the training data composed of the final source image and target image pairing can effectively improve the depth estimation accuracy of DepthNet.

[0089] In practical applications, the number K of the selected candidate source images can be flexibly set to a positive integer greater than 1. Optionally, in this embodiment, K is set to 0.5*N.

[0090] After selecting candidate source images with high temporal consistency, considering that sufficient effective areas are required in the photometric optimization loss process, this embodiment further selects the candidate source image with the largest photometric effective area from the candidate source images as the final matching source image. This process involves the calculation of the common field of view area and the non-occluded area, such as Figure 4 shown.

[0091] Specifically, there are usually two reasons for the generation of invalid regions: 1) due to the non-common field of view caused by camera motion, that is, in the target image I (0) Within the viewpoint but in candidate source image i (-n) 1) Areas outside the viewpoint; 2) Areas blocked by tissues and instruments.

[0092] First, in order to calculate the common field of view between the target image and each candidate source image, according to the reference depth image D mv and relative pose T (-n) , I (-n) To I (0) The viewpoints are aligned to obtain the pixel p in the distorted image. (-n) and I (0) Medium pixel p (0) The coordinate matching relationship of Figure 2 The distorted image synthesis relationship shown in the figure can be obtained by (-n) The expression is as follows:

[0093] p (-n) =proj(D mv ,T (-n) ,K,P (0) )

[0094] For I (0) Not in I (-n) The homogeneous coordinates of the corresponding points p (-n) The value will exceed I (-n) By checking I (0) Zhong and I (-n) Whether the coordinates of the corresponding points are within the range of [0, W-1] × [0, H-1] can be used to determine whether the corresponding points are in the common field of view. W and H are the width and height of the image, respectively. Refer to step S01 and set them to 320 and 256. Finally, the distorted image p (-n) The area formed by the points in the range [0,W-1]×[0,H-1] is determined as I (-n) with I (0) The common visual area between the two is obtained by

[0095] In addition, since the depth estimation of the occluded area is different at the same position due to different geometric structures, this embodiment uses a cyclic warping method to filter out inconsistent areas and locate non-occluded areas. (0) Reprojection yields I (-n) ′, and then use I (-n) ′Reprojection to obtain I (0) ″, I (0) To I (-n) The viewpoints are aligned to obtain the distorted image I (-n) ′; then I (-n) ' to I (-n) ′ performs pose alignment to obtain the distorted image I (0) ″, the depth information and relative pose needed in this process are estimated by using the pre-trained DepthNet and PoseNet respectively. Since the depth of the occluded area in the two images is inconsistent, the I (0) "With I (0) The area with a large difference is the occlusion area. (0) "With I (0) The area formed by the points with smaller differences is the non-blocking area. Optionally, in this embodiment, the following method is used to generate the non-blocking area mask

[0096]

[0097] Wherein, γ is a preset percentage threshold, and optionally, the percentage threshold γ is set to 0.2.

[0098] Therefore, combined with and The intersection of the common field of view area and the non-occluded area can be obtained, that is, the photometric effective area, and the relevant expression is as follows:

[0099]

[0100] For the selected K candidate source images, the photometric effective area is calculated according to the above method, and the candidate source image with the largest photometric effective area is selected as the most suitable source image for photometric optimization and compared with the target image I (0) The pairs constitute the training data.

[0101] Step S2 of this embodiment is similar to the above step S02, except that in step S2, the training data is the source image-target image pair constructed by the above step S1.

[0102] In general, the depth estimation network trained by the dynamic view selection method proposed in this embodiment matches the most appropriate source image frame within a large time span, greatly reducing the interference of static use of adjacent frames on pose alignment, and the proposed photometric effective area calculation method simultaneously detects the mutual view area and the unobstructed area, ensuring that the selected source image frame has a reliable photometric effective area size. Compared with other methods that statically use adjacent frames, this embodiment dynamically uses historical frames to train the network, and the monocular depth estimation network finally constructed has higher accuracy than networks trained by other methods on a dataset without depth labels. In addition, while improving the estimation accuracy of the depth estimation network through joint training, this embodiment can also improve the estimation accuracy of the pose estimation network for relative poses.

[0103] Embodiment 2:

[0104] A monocular endoscope depth estimation method, comprising:

[0105] Inputting the monocular endoscopic image to be estimated into a monocular depth estimation network to obtain a depth map of the estimated monocular image;

[0106] Among them, the monocular depth estimation network is established by the monocular depth estimation network establishment method based on dynamic view selection provided in the above embodiment 1.

[0107] Since the depth estimation accuracy of the monocular depth estimation network established in the above-mentioned embodiment 1 is relatively high, based on the monocular depth estimation network, this embodiment has relatively high estimation accuracy.

[0108] The beneficial effects that can be achieved by this embodiment are further analyzed and verified in combination with specific comparative experimental data.

[0109] This embodiment selects 8 mainstream depth estimation methods (Monodepth2, HR-Depth, DIFFNet, Endo-SfMLearner, AF-SfMLearner, MonoViT, GasMono and Lite-Mono) as the comparison methods of this embodiment, among which Endo-SfMLearner and AF-SfMLearner are customized depth estimation methods for endoscopes, and the rest are depth estimation methods in natural scenes. These methods are all trained using static adjacent frames.

[0110] This embodiment and the comparison methods are tested on the laparoscopic dataset (SCARED), and compared with other source frame selection methods. The evaluation indicators used are Abs Rel, Sq Rel, RMSE, RMSE log and δ, and the calculation formulas of each evaluation indicator are as follows:

[0111]

[0112] Among them, d i represents the predicted depth value, Indicates the actual depth value.

[0113] Among the evaluation indicators, the smaller the Abs Rel, Sq Rel, RMSE, and RMSE log, the better the corresponding effect; δ is the accuracy, and the larger the value, the better the corresponding effect.

[0114] Before measuring, the predicted depth map needs to be scaled because it is scale-unknown, and the scaling factor is the median gold standard depth divided by the median predicted depth.

[0115] Table 1 shows the evaluation indicators of this embodiment and other methods. According to the results shown in Table 1, this embodiment surpasses other methods in five indicators. Among them, in the indicator of Sq Rel, this method is 15.89% lower than the second best method AF-SfMLearner.

[0116] Table 1 Evaluation indicators of test results of each method

[0117]

[0118] Figure 5 The error comparison graphs of various methods are shown. Each row is a case in the SCARED dataset, and each column from left to right is divided into RGB images and local error graphs of Monodepth2, HR-Depth, DIFFNet, Endo-SfMLearner, AF-SfMLearner, MonoViT, GasMono, Lite-Mono, and this example; the error graph is obtained by mapping the Abs Rel value of the pixel to different colors. Figure 5 It can be seen from the results that the error of this embodiment is significantly smaller than that of other methods.

[0119] Embodiment 3:

[0120] A computer-readable storage medium includes a stored computer program; when the computer program is executed by a processor, the device where the computer-readable storage medium is located is controlled to execute the method for establishing a monocular depth estimation network based on dynamic view selection provided in the above-mentioned embodiment 1, and / or the monocular depth estimation method provided in the above-mentioned embodiment 2.

[0121] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for establishing a monocular depth estimation network based on dynamic view selection, characterized in that: include: Step S1: construct a training data set; In the training data set, each piece of training data is a source image-target image pair, and the construction method of each training data includes: A frame in a monocular video frame sequence is selected as the target image. For the N historical frames before the target image, each historical frame is used as the source image, and the cost cube from the source image to the target image is calculated. Each cost cube is converted into a depth map corresponding to each historical frame to obtain a depth map set. Based on the depth map set, the depth values ​​that appear at each pixel position and the number of occurrences of each depth value are counted, and the depth value with the largest number of occurrences at each position is used as the depth value at the corresponding position to obtain a reference depth image D mv , and calculate the temporal consistency score of each historical frame; the depth map corresponding to the historical frame and the reference depth image D mv The smaller the difference, the higher the temporal consistency score; Select some historical frames with the highest temporal consistency score as candidate source images, and select the candidate source image with the largest photometric effective area as the final source image, which together with the target image constitute a piece of training data; the photometric effective area of ​​the image is the intersection of the non-occluded area of ​​the candidate image and the common field of view area between the image and the target image; Step S2: Taking photometric loss as a constraint, jointly training the pose estimation network PoseNet and the depth estimation network DepthNet using the constructed training data set, and using the jointly trained depth estimation network DepthNet as the monocular depth estimation network; Among them, N is an integer greater than 1; the pose estimation network PoseNet is used to estimate the relative pose of the target image to the source image, and the depth estimation network DepthNet is used to estimate the depth map of the image.

2. The method for establishing a monocular depth estimation network based on dynamic view selection according to claim 1, characterized in that: Take I (0) Represents the target image, with I (-n) represents the nth historical frame image before the target image, then image I (-n) With the target image I (0) Methods for determining the common visual field between the two include: Use the pre-trained pose estimation network PoseNet to estimate the target image I (0) To Image I (-n) The relative position T (-n) ; According to the relative posture T (-n) and the reference depth image D mv Image I (-n) To the target image I (0) The viewpoints are aligned to obtain the pixel p in the distorted image. (-n) and I (0) Medium pixel p (0) The corresponding relationship; The pixel p in the distorted image that is in the range [0,W-1]×[0,H-1] (-n) The region formed is determined as image I (-n) With the target image I (0) The common visual field between Where W and H are the width and height of the image, respectively, n∈{1,2…,N}.

3. The method for establishing a monocular depth estimation network based on dynamic view selection according to claim 2, characterized in that: Image I (-b) The non-occluded area is determined by: Use the pre-trained depth estimation network DepthNet to estimate image I (-n) The depth map D (-n) ; Use the pre-trained pose estimation network PoseNet to estimate image I (-n) To the target image I (0) The relative position T (-n)′ ; According to the relative posture T (-n)′ and the depth map D (-n) The target image I (0) To Image I (-n) The viewpoints are aligned to obtain the distorted image I (-n) ′; Use the pre-trained pose estimation network PoseNet to estimate the target image I (0) To the distorted image I (-n) The relative position T of (-n)′′ ; According to the relative posture T (-n)″″ and the reference depth image D mv The distorted image I (-n) ′ to the target image I (0) The viewpoints are aligned to obtain the distorted image I (0) ″; The distorted image I (0) ″ and the target image I (0) The area where the difference is not greater than the preset threshold is determined as image I (-n) non-occluded area.

4. The method for establishing a monocular depth estimation network based on dynamic view selection according to claim 2 or 3, characterized in that: Image I (-n) To the target image I (0) The cost cube is calculated by: In the parallax range [d min ,d max ] to obtain N d Depth Plane d min and d max Respectively represent the minimum depth value and the maximum depth value; For each depth plane D i , according to the relative posture T (-n) and the depth plane D i , image I (-n) To the target image I (0) The viewpoints are aligned to obtain the distorted image And calculate As the matching cost at this depth value; i = 0, 1, 2...N d -1; The matching costs at each depth value are concatenated to obtain image I (-n) To the target image I (0) The cost cube C (-n) .

5. The method for establishing a monocular depth estimation network based on dynamic view selection according to claim 4, characterized in that: by Represents image I (-n) The corresponding depth map, then the depth map In the example, the depth value at any position (x, y) is: Among them, C (-n) (x, y, i) represents the matching cost corresponding to the i-th depth value at position (x, y) in the cost cube; Indicates the depth value with the minimum matching cost at the position (x, y).

6. The method for establishing a monocular depth estimation network based on dynamic view selection according to claim 5, characterized in that: S (-n) Represents image I (-n) The temporal consistency score of Among them, D mv (x, y) represents the reference depth image D mv The depth value at position (x,y) in the image.

7. The method for establishing a monocular depth estimation network based on dynamic view selection according to claim 1, characterized in that: Take I s and I t Denote the source image and the target image respectively, with T t→s Represents the target image I t To source image I s The relative position of t Represents the target image I t The depth map of t→s and the depth map D t The source image I s To the target image I t After the viewpoints are aligned, the distorted image I s→t for: I s→t (p s→t )=i s <proj(D t ,T t→s ,K,p s→t )> Among them, p s→t represents pixel homogeneous coordinates, K represents the given camera intrinsic parameter matrix, proj(·) is the projection function, and <> represents differentiable bilinear sampling.

8. The method for establishing a monocular depth estimation network based on dynamic view selection according to claim 7, characterized in that: The expression of the luminosity loss is as follows: in, represents photometric loss, SSIM represents structural similarity index metric, and α is the weight parameter.

9. A monocular endoscope depth estimation method, characterized in that: include: Inputting the monocular endoscopic image to be estimated into a monocular depth estimation network to obtain a depth map of the estimated monocular image; Wherein, the monocular depth estimation network is established by the method for establishing a monocular depth estimation network based on dynamic view selection described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: Including a stored computer program; when the computer program is executed by a processor, the device where the computer-readable storage medium is located is controlled to execute the method for establishing a monocular depth estimation network based on dynamic view selection as described in any one of claims 1 to 8, and / or the monocular depth estimation method as described in claim 9.

Images