Self-supervised monocular depth estimation method and device for laparoscopic surgery video

Abstract

The invention relates to a self-supervised monocular depth estimation method for a laparoscopic surgery video, which is applied to medical engineering. The method specifically comprises the following steps: acquiring an initial video frame sequence corresponding to a laparoscopic surgery video; identifying a target filling area in the initial video frame sequence through a gated convolutional network, and filling the target filling area to obtain a filled video frame sequence; inputting the filled video frame sequence into a preset depth estimation model, obtaining a multi-scale depth map through a depth network included in the preset depth estimation model, and obtaining a relative pose matrix of adjacent frames in the filled video frame sequence through a pose network included in the preset depth estimation model, and generating a depth estimation result and a corresponding target depth map according to the multi-scale depth map and the relative pose matrix.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of medical engineering technology, and more specifically, to a method, device, electronic device, and storage medium for self-supervised monocular depth estimation of laparoscopic surgery videos. Background Art

[0002] Compared to open surgery, minimally invasive laparoscopic surgery offers the advantages of less surgical bleeding and shorter recovery time, and has been widely used in recent years. However, due to the narrow field of view and lack of depth perception, laparoscopic surgery places high demands on the surgeon's experience and skills. With the recent development of virtual reality (VR) / augmented reality (AR) technology, surgical navigation based on AR can address these challenges. Computer vision-based AR navigation systems are a viable and low-cost solution and can enhance surgical visualization if the position of the laparoscope is known. These AR systems overlay preoperative data with intraoperative endoscopic data through registration techniques. However, to ensure the reliability of AR navigation systems, a high-precision registration process must be performed. The accuracy of video registration algorithms depends primarily on the quality of intraoperative anatomical reconstruction from endoscopic videos. Reconstructing three-dimensional structures from monocular videos has been a long-standing research topic, and depth estimation is an essential step in achieving this reconstruction. Furthermore, many tasks, such as surgical robotics, medical image segmentation, surgical planning assistance, and surgical instrument recognition, can benefit from depth estimation.

[0003] Image sequence depth estimation methods in related technologies are based on multi-view geometry principles, such as Structure from Motion (SfM) and Simultaneous Localization and Mapping (SLAM). Although the depth estimation task has been developed for many years in natural scenes, the problem is more difficult in laparoscopic scenes due to inconsistent lighting, sparse texture features, and non-Lambertian reflectance characteristics of soft tissue. Geometry-based methods rely heavily on feature extraction and matching. Smooth and repetitive soft tissue textures often lead to sparse features and feature matching errors. Therefore, traditional methods still cannot achieve ideal performance. In harsh natural environments, methods such as depth estimation, segmentation, and detection based on deep learning have developed rapidly due to the release of large data sets. However, it is difficult to obtain a large amount of real ground truth data in laparoscopic scenes.

[0004] Therefore, how to improve the accuracy of depth estimation in laparoscopic surgery scenarios becomes a problem that needs to be solved. Summary of the Invention

[0005] An object of the embodiments of the present disclosure is to provide a new technical solution for a self-supervised monocular depth estimation method, electronic device, and storage medium for laparoscopic surgery videos.

[0006] According to a first aspect of the present disclosure, there is provided a self-supervised monocular depth estimation method for a laparoscopic surgery video, comprising: obtaining an initial video frame sequence corresponding to the laparoscopic surgery video;

[0007] Identifying a target filling area in the initial video frame sequence through a gated convolutional network, and filling the target filling area to obtain a padded video frame sequence;

[0008] The padded video frame sequence is input into a preset depth estimation model, a multi-scale depth map is obtained through the depth network included in the preset depth estimation model, and a relative pose matrix of adjacent frames in the padded video frame sequence is obtained through the pose network included in the preset depth estimation model, so as to generate a depth estimation result and a corresponding target depth map based on the multi-scale depth map and the relative pose matrix.

[0009] Optionally, the deep network includes an encoding subnetwork and a decoding subnetwork. Before obtaining a multi-scale depth map through the deep network included in the preset depth estimation model, the method also includes: performing multi-scale feature extraction on the padded image sequence through the encoding subnetwork, and the encoder generates a multi-level feature representation by parallel processing local areas of different sizes and establishing global feature relationships; fusing the multi-level feature representation across layers and scales through the decoding subnetwork, and adjusting the weights of the fused features in combination with the spatial attention mechanism and the channel attention mechanism; based on the weight-adjusted features, outputting depth maps of different resolutions through multiple independent prediction modules to obtain the multi-scale depth map.

[0010] Optionally, the multi-scale feature extraction of the encoding subnetwork includes: performing local feature extraction of different ranges on the input image to generate feature maps of multiple scales; performing global feature relationship modeling on the feature maps of the multiple scales to generate feature representations containing long-range dependencies; and gradually fusing the feature representations through cascading multiple feature aggregation stages to generate the multi-level feature representation.

[0011] Optionally, the cross-layer and cross-scale fusion of the decoding subnetwork includes: connecting the output features of each stage in the encoding subnetwork with the features of the corresponding scale of the decoding subnetwork; assigning weights to the connected features in the spatial dimension and channel dimension respectively; inputting the weighted features into the multiple independent prediction modules to generate a depth map with a resolution decreasing from the input image as the multi-scale depth map.

[0012] Optionally, before obtaining the relative pose matrix of adjacent frames in the padded video frame sequence through the pose network included in the preset depth estimation model, the method also includes: obtaining a multi-scale feature map of each video frame and its adjacent frames in the padded video frame sequence through convolution, the multi-scale feature map being a feature map of multiple resolutions generated after downsampling and feature extraction of the spliced ​​adjacent frame images through multiple convolution layers of different step sizes; extracting the channel attention vector of the multi-scale feature map through the pyramid squeezing attention subnetwork in the pose network to calculate the soft allocation weight of the multi-scale feature map based on the channel attention vector; calculating the weighted multi-scale feature map according to the correspondence between the multi-scale feature map and the soft allocation weight; splicing the weighted multi-scale feature map to obtain a fusion feature, and obtaining the relative pose matrix according to the fusion feature.

[0013] Optionally, before generating the depth estimation result and the corresponding target depth map based on the multi-scale depth map and the relative pose matrix, the method further includes: based on the relative pose matrix and the camera intrinsic parameters, reprojecting the temporally adjacent source frame images in the padded picture sequence to the target frame coordinate system to generate a reprojected image, the source frame image being the previous frame and / or the next frame image temporally adjacent to the target frame image in the padded picture sequence, and the target frame image being the video frame in the padded picture sequence to which the depth map is currently to be generated; calculating the edge-perceived loss of the average normalized inverse depth of the target frame image, the edge-perceived loss being obtained by summing the gradient of the inverse depth and the exponentially weighted product of the image gradient, and the photometric loss being the pixel intensity difference between the reprojected image and the target frame image; weightedly summing the photometric loss and the edge-perceived loss at multiple preset scales, and combining the filtering processing of invalid pixel areas with automatic masking to obtain a total loss; updating the parameters of the depth network and the pose network according to the total loss to generate the depth estimation result and the target depth map.

[0014] Optionally, obtaining an initial video frame sequence corresponding to the laparoscopic surgery video includes: converting the video frames included in the laparoscopic surgery video into grayscale images, and smoothing the grayscale images using a Gaussian blur filter to obtain a target grayscale image; converting the target grayscale image into a binary image based on a preset grayscale threshold, and sequentially applying an erosion operation to the binary image to remove noise and a dilation operation to expand a highlight area to obtain a target binary image;

[0015] Connected component analysis is performed on the target binary image to generate a mask sequence; and the initial video frame sequence is generated according to a correspondence between the mask sequence and the video frame.

[0016] According to the second aspect of the present disclosure, a self-supervised monocular depth estimation device for laparoscopic surgery videos is also provided, including: an acquisition module for acquiring an initial video frame sequence corresponding to the laparoscopic surgery video; a filling module for identifying a target filling area in the initial video frame sequence through a gated convolutional network, and filling the target filling area to obtain a padded video frame sequence; a generation module for inputting the padded video frame sequence into a preset depth estimation model, obtaining a multi-scale depth map through the depth network included in the preset depth estimation model, and obtaining a relative pose matrix of adjacent frames in the padded video frame sequence through the pose network included in the preset depth estimation model, so as to generate a depth estimation result and a corresponding target depth map based on the multi-scale depth map and the relative pose matrix.

[0017] According to a third aspect of the present disclosure, an electronic device is further provided, comprising a memory and a processor, wherein the memory is used to store a computer program; and the processor is used to execute the computer program to implement the method according to the first aspect of the present disclosure.

[0018] According to a fourth aspect of the present disclosure, a computer-readable storage medium is further provided, on which a computer program is stored. When the computer program is executed by a processor, the method according to the first aspect of the present disclosure is implemented.

[0019] One beneficial effect of the embodiments of the present disclosure is that by obtaining an initial video frame sequence and using a gated convolutional network to process the target filled area therein, it is possible to repair image defects caused by lighting or reflection and improve the quality of the video frames. The padded video frame sequence is then input into a preset depth estimation model, and a multi-scale depth map and a relative pose matrix of adjacent frames are obtained through a depth network and a pose network. Accurate depth estimation results and target depth maps can be generated, thereby providing accurate depth information for laparoscopic surgery.

[0020] Other features and advantages of the embodiments of the present disclosure will become apparent from the following detailed description of exemplary embodiments of the present disclosure with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the embodiments of the present disclosure.

[0022] Figure 1 is a flow chart illustrating a method for self-supervised monocular depth estimation of laparoscopic surgery videos according to one embodiment;

[0023] Figure 2 is a schematic diagram of the effect of filling the target filling area according to an embodiment;

[0024] Figure 3 is a schematic diagram of a target depth map corresponding to a depth estimation result according to an embodiment;

[0025] Figure 4 is a schematic diagram of a preset depth estimation model in an embodiment processing a laparoscopic surgery video;

[0026] Figure 5 is a block diagram of a self-supervised monocular depth estimation apparatus for laparoscopic surgery videos according to one embodiment;

[0027] Figure 6 is a schematic diagram of the hardware structure of an electronic device according to an embodiment. DETAILED DESCRIPTION

[0028] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that unless otherwise specifically stated, the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure.

[0029] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.

[0030] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered part of the specification.

[0031] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0032] It should be noted that like reference numerals and letters refer to like items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0033] Figure 1 The present invention is a flowchart of a method for self-supervised monocular depth estimation of laparoscopic surgery videos according to one embodiment. The method can be applied to any device capable of implementing the method for self-supervised monocular depth estimation of laparoscopic surgery videos.

[0034] like Figure 1 As shown, the self-supervised monocular depth estimation method for laparoscopic surgery videos of this embodiment may include the following steps S110 to S130:

[0035] Step S110: obtaining an initial video frame sequence corresponding to the laparoscopic surgery video.

[0036] The initial video frame is used to represent the two-dimensional image sequence continuously collected during the operation. Each frame records the visual information of the surgical field at a specific time point, including details such as tissue morphology and instrument position. It is the raw data basis for depth estimation and posture analysis.

[0037] In one embodiment, step S110 is specifically implemented through steps S1101 to S1103.

[0038] Step S1101 : converting video frames included in the laparoscopic surgery video into grayscale images, and smoothing the grayscale images using a Gaussian blur filter to obtain a target grayscale image.

[0039] It should be noted that monocular depth estimation refers to estimating the depth information of each object in a scene using a sequence of images captured by a single camera. Since laparoscopic surgery videos usually only provide a single perspective, monocular depth estimation is particularly important for such applications.

[0040] Optionally, after the surgical video is disassembled into independent video frames, the disassembled video frames can also be converted into a frame sequence of a target format (such as PNG format) to retain image transparency information and reduce compression distortion, providing standardized image input for subsequent preprocessing.

[0041] A grayscale image is an image obtained by converting the RGB values ​​of each pixel in a color video frame into grayscale values. A Gaussian blur filter is used to smooth grayscale images to reduce noise and detail interference, resulting in a clearer target grayscale image. Converting video frames to grayscale images simplifies data processing. The Gaussian blur filter also helps remove high-frequency noise from the image, making subsequent image processing more stable and efficient.

[0042] Step S1102 : converting the target grayscale image into a binary image based on a preset grayscale threshold, applying an erosion operation to the binary image to remove noise and a dilation operation to expand the highlight area, thereby obtaining a target binary image.

[0043] The preset grayscale threshold is used to distinguish the critical grayscale value of the highlight area and the normal area in the image, and the image pixels can be divided into foreground (highlight area) and background (normal area).

[0044] A binary image is the result of dividing the pixels in an image into two categories: foreground and background. The foreground represents the highlighted area (the target area to be filled), and the background represents the other areas.

[0045] For example, the erosion operation can remove small noise points in a binary image, while the dilation operation can expand the highlight area to make it more obvious for subsequent connected component analysis. The erosion operation reduces noise by eliminating small spots in the image and disconnecting narrow connections, while the dilation operation strengthens the connectivity of the target area by expanding the foreground area.

[0046] Step S1103 : performing connected component analysis on the target binary image to generate a mask sequence.

[0047] Connected components are used to represent interconnected groups of pixels in a binary image. They can be identified using connectivity analysis algorithms (such as those based on the four-connected or eight-connected criteria). Connected component analysis can identify independent target regions in a binary image and generate a mask sequence.

[0048] The mask sequence is used to represent a series of mask images that identify the target area in the video frame sequence. Each mask image corresponds to a video frame and is used to indicate the area that needs to be processed in the video frame.

[0049] Step S1104 : generating an initial video frame sequence according to the correspondence between the mask sequence and the video frame.

[0050] By mapping the mask sequence to the original video frame, each mask image in the mask sequence is combined with the corresponding video frame to identify the area in the video frame that needs to be processed. This combination ensures that subsequent processing steps, such as filling or further analysis, can accurately operate on the target area, thereby improving the accuracy and efficiency of the entire processing process.

[0051] In step S120 , a target filling area in the initial video frame sequence is identified through a gated convolutional network, and the target filling area is filled to obtain a padded video frame sequence.

[0052] The gated convolutional network is used to automatically learn the areas that need to be filled in the initial video frame sequence and generate the corresponding filling content. The padded video frame sequence is used to represent the video frame sequence processed by the gated convolutional network, in which the target filling area has been properly filled, making the video frame more visually complete and coherent, and providing more accurate input data for subsequent depth estimation. By learning features from the data, the gated convolutional network can intelligently identify the areas that need to be filled and generate textures and content that match the surrounding environment, thereby improving the quality and usability of the video frames.

[0053] For example, target filling areas usually refer to bright reflective areas caused by inconsistent lighting or non-Lambertian reflective properties of soft tissue, which may affect the accuracy of depth estimation. Figure 2 The first row is shown.

[0054] The gated convolutional network identifies and fills these regions by:

[0055] The formula of the gated convolutional network can be expressed as:

[0056] O y,x =φ(I·∑∑W f )⊙σ(I·∑∑W g )

[0057] Among them, in the formula of the gated convolutional network, O y,x The y and x in the output feature map represent the position coordinates of the pixel, y represents the vertical index, and x represents the horizontal index; y,x Represents the value of the output feature map at position (y, x); I represents the input image, that is, each frame image in the initial video frame sequence that needs to be padded; σ is an activation function (such as Sigmoid), which is used to map the input to between 0 and 1 and to generate a gating signal so that the output gating value is between 0 and 1; φ is an activation function, which can be any activation function, such as ReLU, ELU, and LeakyReLU, which is used to introduce nonlinearity so that the model can learn more complex feature representations; W f and W g There are two different convolution filters, which are used to generate the gate signal and the feature signal respectively. f and ∑∑W g It means that the sum of the convolution kernel at each position on the feature map is performed to calculate each element of the output feature map.

[0058] In this process, the gated convolutional network can dynamically adjust the weights of the convolution operation to better process the features of different areas. In this way, the network can identify the highlighted reflective areas and generate reasonable filling content based on the information of the surrounding pixels. For example, in a surgical video, if an area becomes too bright due to reflections, the network can fill the area by analyzing the information of adjacent pixels and frames to generate content that matches the texture of the surrounding tissue. The resulting padded video frame sequence is not only more visually complete, but also more accurately reflects the actual state of the surgical scene in terms of content, which is helpful for subsequent depth estimation and image analysis. For example Figure 2 As shown in the second row.

[0059] In step S130, the padded video frame sequence is input into a preset depth estimation model, a multi-scale depth map is obtained through the depth network included in the preset depth estimation model, and a relative pose matrix of adjacent frames in the padded video frame sequence is obtained through the pose network included in the preset depth estimation model, so as to generate a depth estimation result and a corresponding target depth map based on the multi-scale depth map and the relative pose matrix.

[0060] Depth estimation results are used to represent the depth position and distance relationship of objects and structures in the surgical scene in three-dimensional space. In this disclosure, depth estimation results are depth information obtained by fusing the multi-scale depth maps output by the depth network, and are used to reflect the depth relationship between different objects and structures in the scene.

[0061] The target depth map is used to represent the specific depth value of each pixel. It is a depth map generated based on the depth estimation result and the relative pose matrix, and corresponds to the original video frame. It contains the depth value of each pixel and provides a depth basis for target-driven applications. In this disclosure, the target depth map is a visual representation of the depth estimation result, which intuitively integrates the abstract depth information into the visual scene corresponding to the original video frame. For details, please refer to Figure 3 The figure shows the target depth map corresponding to the depth estimation results in the embodiment of the present disclosure. The top row shows the original laparoscopic surgery video frame, and the bottom row shows the corresponding depth estimation results and target depth map. The depth information is represented by color coding, with colors from light to dark representing the change in depth from near to far, demonstrating the effective distinction and accurate estimation of different depth areas.

[0062] By implementing a self-supervised monocular depth estimation method for laparoscopic surgery videos, by obtaining an initial video frame sequence and using a gated convolutional network to process the target filled area, it is possible to repair image defects caused by lighting or reflection and improve the quality of the video frames. The padded video frame sequence is then input into a preset depth estimation model. The multi-scale depth map and the relative pose matrix of adjacent frames are obtained through the depth network and pose network, which can generate accurate depth estimation results and target depth maps, thereby providing accurate depth information for laparoscopic surgery.

[0063] In one embodiment, the deep network includes an encoding subnetwork and a decoding subnetwork, and before step S130 , further includes steps S1301 to S1303 .

[0064] In step S1301, the encoding sub-network performs multi-scale feature extraction on the padded image sequence. The encoding sub-network generates a multi-level feature representation by processing local regions of different sizes in parallel and establishing a global feature relationship.

[0065] The encoding subnetwork is used to extract features from the padded image sequence. By processing local regions of different sizes in parallel and establishing global feature relationships, it generates a feature representation that contains multi-level semantic and spatial information. The multi-level feature representation refers to a set of multi-resolution feature maps (such as original resolution, 1 / 2, 1 / 4, and 1 / 8 resolution) output by the encoding subnetwork at different stages.

[0066] In one embodiment, the multi-scale feature extraction of the encoding subnetwork includes: performing local feature extraction of different ranges on the input image to generate feature maps of multiple scales; modeling the global feature relationship of the feature maps of multiple scales to generate feature representations containing long-range dependencies; and gradually fusing the feature representations through cascading multiple feature aggregation stages to generate a multi-level feature representation.

[0067] Specifically, the encoding subnetwork uses multiple convolutional layers and Transformer structures to process local regions of different sizes in parallel, extract local features of the image, establish global feature relationships, and generate feature representations that include long-range dependencies. Through multiple cascaded feature aggregation stages, features of different scales are gradually integrated to generate multi-level feature representations, providing more comprehensive and accurate feature information for depth estimation. Optionally, combined with Figure 4 As shown in Figure 2, the specific structure of the encoding subnetwork is as follows:

[0068] The encoding subnetwork constructs a deep encoder in five stages. In the first stage, the padded image sequence is input and first passes through a convolutional block consisting of two convolutional layers (with a kernel size of 3×3). The first convolution has a stride of 2 to reduce the resolution, and the second convolution has a stride of 1 to preserve details, outputting the initial feature map.

[0069] In the second to fifth stages, multi-path Transformer blocks are stacked in each stage, called "Fusion CNN & Transformer" layers. Each "Fusion CNN & Transformer" layer contains three layers:

[0070] The first layer is multi-scale patch embedding, which uses 3×3, 5×5, and 7×7 convolution kernels in parallel to extract local features of different sizes and generate feature maps of multiple scales (such as high-resolution detail features and low-resolution global features).

[0071] The second layer is the multi-scale feature interaction layer, which models the global feature relationship of the feature maps of each scale separately, captures the long-range dependency through the Transformer block (replacing the traditional convolution), and generates a feature representation containing contextual associations. Specifically, the second layer of the encoding subnetwork is the multi-scale feature interaction layer, and the extracted features pass through a layer composed of convolution blocks and Transformers in parallel. The layer contains three parallel Transformer blocks and one convolution block. The convolution block consists of 1×1 convolution, 3×3 depthwise convolution and 1×1 convolution, which constructs the local relationship between features. Due to the self-attention mechanism, the three Transformer blocks model the spatial information interaction in the input, thereby generating features P1, P2 and P3. In each Transformer block, the MLP layer of MHSA is replaced by KAN (Kolmogorov-Arnold Network). The visual tags embedded using the multi-scale patch embedding module are projected into the query vector (Q), key vector (K) and value vector (V∈R N×C , R represents the dimension of the value vector, N is the number of features, and C is the dimension of the embedding) vector, where C is used to normalize the attention scores to ensure numerical stability: This formula represents the factor attention function FactorAtt(Q,K,V) which scales the query vector Q by dividing it by the square root of the embedding dimension C and then multiplies it by the transpose K of the key vector K. T The result of softmax normalization of the product with the value vector V is used to calculate the attention weight in this way, capture the dependencies between different parts of the input data, and realize weighted aggregation of the value vector V, so as to dynamically adjust the degree of attention to different information in feature processing, and highlight the influence of image feature information on generating feature representations containing long-range dependencies.

[0072] In this way, the network can effectively capture the dependencies between different parts of the image and realize weighted aggregation of value vectors, thereby dynamically adjusting the degree of attention to different information during feature processing and highlighting the influence of image feature information on generating feature representations containing long-range dependencies.

[0073] The third layer uses the global to local feature interaction module to fuse the extracted local features with the global features:

[0074] A i =Concat([F i ,P i,0 ,P i,1 ,P i,2 ])

[0075]

[0076] Among them, Fi represents the feature map extracted in the Fusion CNN&Transformer layer of the i-th stage; P i,0 represents the feature map extracted by the first multi-scale feature interaction layer in the i-th stage; P i,1 represents the feature map extracted by the first multi-scale feature interaction layer in the i-th stage; P i,2 A represents the feature map extracted by the first multi-scale feature interaction layer in the i-th stage; i Indicates that in the i-th stage, F i 、P i,0 、P i,1 、P i,2 The aggregated feature maps after connection contain feature information of different scales and levels; Represents the aggregated feature map A i The result after convolution processing; X i+1 Represents the feature map after convolution processing, which is used for feature extraction and processing in the next stage.

[0077] In step S1302, the multi-level feature representation is fused across layers and scales through the decoding sub-network, and the weights of the fused features are adjusted by combining the spatial attention mechanism and the channel attention mechanism.

[0078] In one embodiment, the cross-layer and cross-scale fusion of the decoding subnetwork includes: connecting the output features of each stage in the encoding subnetwork with the features of the corresponding scale of the decoding subnetwork; assigning weights to the connected features in the spatial dimension and channel dimension respectively; and inputting the weighted features into multiple independent prediction modules to generate a depth map with a resolution decreasing from the input image as a multi-scale depth map.

[0079] Cross-layer and cross-scale fusion integrates the multi-level feature representations output by the encoding subnetwork to achieve more accurate depth estimation. Specifically, the decoding subnetwork uses a cross-layer connection mechanism to fuse features from different levels of the encoding process, thereby preserving the image's detail and structure. Cross-scale fusion combines feature maps of different resolutions, enabling the decoding subnetwork to capture depth variations at different scales, providing richer information for subsequent depth estimation.

[0080] In step S1301, the encoding subnetwork has generated a multi-level feature representation by processing local regions of different sizes in parallel and establishing global feature relationships, which contains feature information extracted from different levels and provides a basis for cross-layer and cross-scale fusion of the decoding subnetwork. When performing cross-layer and cross-scale fusion, the decoding subnetwork will connect the output features of each stage in the encoding subnetwork with the features of the corresponding scale in the decoding subnetwork to ensure the integrity and continuity of the features.

[0081] Each of the multiple independent prediction modules is responsible for predicting a depth map at a specific resolution and can perform depth estimation on features of different scales through an independent prediction mechanism to generate multi-scale depth maps. Specifically, each independent prediction module receives the weighted features as input, performs convolution operations and nonlinear transformations, and outputs a depth map of the corresponding resolution. The resolutions of the depth maps decrease in turn, such as 1, 1 / 2, 1 / 4, and 1 / 8 resolution depth maps, which correspond to depth information of different scales. In this way, the decoding subnetwork can provide diverse depth estimation results for different application scenarios and needs, improving the flexibility and practicality of depth estimation.

[0082] In step S1303, based on the weight-adjusted features, different resolution depth maps are output by the multiple independent prediction modules to obtain multi-scale depth maps.

[0083] By way of example, each independent prediction module consists of two convolution layers and a Sigmoid activation function. Specifically, the weighted features are first input to the first convolution layer, which uses a 3x3 convolution kernel to extract spatial information of the feature map. After convolution operation, the feature map is passed to the second convolution layer, which uses a 1x1 convolution kernel to adjust the number of channels of the feature map. Finally, the feature map passes through the Sigmoid activation function to limit the output value to between 0 and 1, generating a depth map of the corresponding resolution. The multiple independent prediction modules output depth maps at different resolutions, such as 1, 1 / 2, 1 / 4, and 1 / 8 resolution depth maps. These depth maps reflect the depth information of the image from different scales, where high-resolution depth maps can provide more detailed depth details and are suitable for local depth analysis, while low-resolution depth maps can provide global depth trends and are suitable for overall depth estimation. By combining depth maps of different resolutions, more comprehensive and accurate depth estimation results can be obtained to meet the needs of different application scenarios for depth information.

[0084] In one embodiment, before obtaining the relative pose matrix of adjacent frames in the padded video frame sequence through the pose network included in the preset depth estimation model, the method also includes: obtaining a multi-scale feature map of each video frame and its adjacent frames in the padded video frame sequence through convolution, the multi-scale feature map being a feature map of multiple resolutions generated after downsampling and feature extraction of the spliced ​​adjacent frame images through multiple convolution layers of different step sizes; extracting the channel attention vector of the multi-scale feature map through the pyramid squeezing attention subnetwork in the pose network to calculate the soft allocation weight of the multi-scale feature map based on the channel attention vector; calculating the weighted multi-scale feature map according to the correspondence between the multi-scale feature map and the soft allocation weight; splicing the weighted multi-scale feature map to obtain a fused feature, and obtaining the relative pose matrix according to the fused feature.

[0085] The convolutional backbone uses a multi-branch structure to process adjacent frames in parallel: after concatenating the current frame with the adjacent frame in the channel dimension, it is fed into a convolutional group consisting of multiple convolution kernel sizes (e.g., 3×3, 5×5, 7×7) and strides (e.g., 1, 2, 4). Each branch generates feature maps of different resolutions. For example, a branch with a stride of 1 retains high-resolution details (e.g., instrument tip motion), while a branch with a stride of 4 extracts low-resolution semantics (e.g., overall organ displacement), forming a feature pyramid containing multi-scale information from fine to coarse.

[0086] Adjacent frames refer to temporally adjacent video frames in a video sequence, including the previous, current, and next frames. Pose calculation involves the relative pose between the previous and current frames and / or the current and next frames. During feature extraction, two adjacent frames (such as [previous and current frames] or [current and next frames]) are concatenated along the channel dimension and then fed into a convolutional network. This captures motion cues and structural differences between frames, providing temporally correlated features for pose estimation.

[0087] The Pyramid Squeeze Attention subnetwork (PSA) is a network structure used to extract the channel attention vector of the feature map (F1, F2, F3, F4). It can generate the channel attention vector by analyzing the importance of each channel in the multi-scale feature map, thereby realizing the adaptive weighting of different channel features. In this disclosure, the SEWeight module based on PSA is used to perform the weighting of the multi-scale feature map F1 output by the convolution stem. i Processing is done through the formula:

[0088] Z i =SEWeight(F i ),i=0,1,2…S-1

[0089] Extract the channel-wise attention vector, where S=4, Z i is the attention weight.

[0090] For example, two adjacent frames of spliced ​​images are convolved to generate a multi-scale feature map F i , calculate the channel attention vector Z through the SEWeight module i Finally, use the soft allocation weight formula:

[0091]

[0092] To Z i Normalize and get the soft allocation weight att of each scale feature map i . Based on the formula:

[0093] Y i =F i ⊙att i ,i=1,2,3,…S-1

[0094] Att i And the corresponding scale feature map F i Perform channel multiplication, where ⊙ represents channel multiplication, Y i refers to the feature map with the obtained multi-scale channel attention weights, and finally based on the formula:

[0095] Out=Concat([Y0,Y1,…,Y S-1 ])

[0096] A splicing operation is performed to fuse the multi-scale weighted feature maps. Based on this fused feature Out, it is further converted into a six-degree-of-freedom (6-DoF) relative pose matrix T, achieving accurate pose estimation between adjacent frames. Based on these steps, channel attention and soft allocation mechanisms can be used to adaptively select spatial scale features that are sensitive to inter-frame motion, improving the accuracy of pose estimation.

[0097] In one embodiment, before generating a depth estimation result and a corresponding target depth map based on the multi-scale depth map and the relative pose matrix, the method further includes: based on the relative pose matrix and the camera intrinsic parameters, reprojecting the temporally adjacent source frame images in the padded image sequence to the target frame coordinate system to generate a reprojected image, the source frame image being the previous frame and / or the next frame image temporally adjacent to the target frame image in the padded image sequence, and the target frame image being the video frame in the padded image sequence to which the depth map is currently to be generated; calculating the edge-perceived loss of the average normalized inverse depth of the target frame image, the edge-perceived loss being obtained by summing the gradient of the inverse depth and the exponentially weighted product of the image gradient, and the photometric loss being the pixel intensity difference between the reprojected image and the target frame image; weightedly summing the photometric loss and the edge-perceived loss at multiple preset scales, and combining the automatic masking to filter the invalid pixel area to obtain a total loss; updating the parameters of the depth network and the pose network according to the total loss to generate a depth estimation result and a target depth map.

[0098] Camera intrinsic parameters are the camera's intrinsic parameter matrix, describing the camera's optical properties and the geometric relationships during the imaging process. These parameters can include focal length, principal point coordinates, and lens distortion coefficients. During the reprojection process, the camera intrinsic parameters work together with the relative pose matrix to map pixels in the source frame to the target frame coordinate system based on depth information, achieving geometric transformation of the image and providing geometric constraints for subsequent image reconstruction loss calculations.

[0099] The reprojected image is generated by reprojecting the source frame image (previous frame or next frame) into the target frame coordinate system based on the relative pose matrix and depth estimation results. Specifically, the pixel points in the source frame image (such as the previous frame or next frame) are projected into the target frame coordinate system based on the depth estimation results through the relative pose matrix and camera intrinsic parameters to generate the reprojected image. For example, the source frame The point in The reprojected image in the target frame coordinate system is calculated and compared with the target frame image for similarity to supervise the accuracy of depth estimation.

[0100] Automatic masking is used to filter out pixel areas in the image where reprojection is invalid due to factors such as occlusion. It can be any technology used to filter invalid pixel areas, and can automatically identify and block out unreliable pixels due to occlusion, motion blur, or depth discontinuity. When calculating the loss function, automatic masking is implemented by introducing a binary mask (μ). When μ takes the value of 0 or 1, it indicates whether a pixel is occluded or invalid. Optionally, the calculation formula for automatic masking is:

[0101]

[0102] in, represents the image similarity, Represents the pixel of the reprojected image, I i (p) represents the pixel of the source frame image, and I(p) represents the pixel of the target frame image.

[0103] Total loss is by converting the luminosity loss and edge-aware loss The weighted summation at multiple preset scales is combined with automatic masking. Optionally, the total loss is calculated as:

[0104]

[0105] in, represents four different scales, μ is the binary mask of automatic masking, and λ = 10 -3 is the weight coefficient of edge-aware loss.

[0106] For example, using pose estimation and camera intrinsic parameters, the source frame image Reproject the depth estimation result into the target frame coordinate system to generate a reprojected image The reprojection operation is implemented by the following formula:

[0107]

[0108] Calculate the original image and the reprojected image The image similarity between Image similarity is defined as follows:

[0109]

[0110] Among them, SSIM (Structural Similarity Index) is the structural similarity index, α = 0.85; photometric loss In two adjacent images The minimum value of is used to measure the photometric difference between the reprojected image and the target image; for each pixel p, the minimum value of the loss calculated from the previous and next adjacent frames is calculated by the following formula:

[0111]

[0112] “-1” and “1” represent the previous frame and the next frame relative to the target image, respectively. In order to maintain the edge characteristics, the edge-aware loss is also used. The edge-aware loss is defined as: Where d represents the average normalized inverse depth of I and Represent the gradients in the x and y directions respectively; then the photometric loss and edge perception loss are calculated at four scales and weighted summed, combined with the automatic masking to filter the invalid pixel area, to obtain the total loss Optionally, automatic masking is performed using the formula:

[0113]

[0114] The total loss is given by the formula:

[0115]

[0116] where λ = 10 -3 ; Finally, according to the total loss Update the parameters of the depth network and pose network to minimize the difference between the reprojected image and the target image, thereby generating a depth estimation result and a corresponding target depth map.

[0117] Figure 5 FIG. 1 is a block diagram of a self-supervised monocular depth estimation apparatus for laparoscopic surgery videos according to an embodiment. Figure 5 As shown, the apparatus 500 may include:

[0118] An acquisition module 510 is used to acquire an initial video frame sequence corresponding to the laparoscopic surgery video;

[0119] a filling module 520 for identifying a target filling region in the initial video frame sequence through a gated convolutional network, and filling the target filling region to obtain a padded video frame sequence;

[0120] A generation module 530 is used to input the padded video frame sequence into a preset depth estimation model, obtain a multi-scale depth map through the depth network included in the preset depth estimation model, and obtain a relative pose matrix of adjacent frames in the padded video frame sequence through the pose network included in the preset depth estimation model, so as to generate a depth estimation result and a corresponding target depth map based on the multi-scale depth map and the relative pose matrix.

[0121] In an optional embodiment, the generation module 530 is further used to perform multi-scale feature extraction on the padded image sequence through the encoding subnetwork, and the encoder generates a multi-level feature representation by parallel processing local areas of different sizes and establishing global feature relationships; the multi-level feature representation is fused across layers and scales through the decoding subnetwork, and the fused features are weighted by combining the spatial attention mechanism and the channel attention mechanism; based on the weight-adjusted features, depth maps of different resolutions are output through multiple independent prediction modules to obtain the multi-scale depth map.

[0122] In an optional embodiment, the generation module 530 is further used to extract local features of different ranges on the input image to generate feature maps of multiple scales; perform global feature relationship modeling on the feature maps of the multiple scales to generate feature representations containing long-range dependencies; and gradually fuse the feature representations through multiple cascaded feature aggregation stages to generate the multi-level feature representation.

[0123] In an optional embodiment, the generation module 530 is further used to connect the output features of each stage in the encoding subnetwork with the features of the corresponding scale of the decoding subnetwork; assign weights to the connected features in the spatial dimension and channel dimension respectively; and input the weighted features into the multiple independent prediction modules to generate a depth map with a resolution decreasing from the input image as the multi-scale depth map.

[0124] In an optional embodiment, the generation module 530 is further used to obtain a multi-scale feature map of each video frame and its adjacent frames in the padded video frame sequence through convolution, where the multi-scale feature map is a feature map of multiple resolutions generated after downsampling and feature extraction of the spliced ​​adjacent frame images through multiple convolution layers of different step sizes; extracting the channel attention vector of the multi-scale feature map through the pyramid squeezing attention subnetwork in the pose network to calculate the soft allocation weight of the multi-scale feature map based on the channel attention vector; calculating the weighted multi-scale feature map according to the correspondence between the multi-scale feature map and the soft allocation weight; splicing the weighted multi-scale feature map to obtain a fusion feature, and obtaining the relative pose matrix according to the fusion feature.

[0125] In an optional embodiment, the generation module 530 is further used to reproject the temporally adjacent source frame images in the padded image sequence to the target frame coordinate system based on the relative pose matrix and the camera intrinsic parameters to generate a reprojected image, wherein the source frame image is the previous frame and / or the next frame image temporally adjacent to the target frame image in the padded image sequence, and the target frame image is the video frame in the padded image sequence to which the depth map is currently to be generated; calculate the edge-perceived loss of the average normalized inverse depth of the target frame image, wherein the edge-perceived loss is obtained by summing the gradient of the inverse depth and the exponential weighted product of the image gradient, and the photometric loss is the pixel intensity difference between the reprojected image and the target frame image; weightedly sum the photometric loss and the edge-perceived loss at multiple preset scales, and combine the automatic masking to filter the invalid pixel area to obtain the total loss; update the parameters of the depth network and the pose network according to the total loss to generate the depth estimation result and the target depth map.

[0126] In an optional embodiment, the acquisition module 510 is specifically used to convert the video frames included in the laparoscopic surgery video into grayscale images, and smooth the grayscale images through a Gaussian blur filter to obtain a target grayscale image; based on a preset grayscale threshold, the target grayscale image is converted into a binary image, and corrosion operations are applied to the binary image in sequence to remove noise and dilation operations are applied to expand the highlight area to obtain a target binary image; connected component analysis is performed on the target binary image to generate a mask sequence; and the initial video frame sequence is generated according to the correspondence between the mask sequence and the video frame.

[0127] Figure 6 is a schematic diagram of the hardware structure of an electronic device according to another embodiment.

[0128] like Figure 6 As shown, the electronic device 600 includes a processor 610 and a memory 620, wherein the memory 620 is used to store an executable computer program, and the processor 610 is used to execute a method as any of the above method embodiments under the control of the computer program.

[0129] In some embodiments, the processor 610 can be used to control the overall operation of the electronic device 600. For example, the processor 610 can execute instructions to implement all or part of the steps of the method in any of the foregoing embodiments of the present disclosure, thereby realizing one or more operations such as voice communication, data communication, database operation, display control, component control, multimedia processing, etc., wherein the above-mentioned components may include sensors, cameras, headphones, input and output devices, etc., and the component may be an internal component of the electronic device itself, or an external component connected to the electronic device wirelessly or wired. The above-mentioned multimedia may include one or more of voice, image, video, and text.

[0130] The electronic device 600 may be any device that can implement the self-supervised monocular depth estimation method for laparoscopic surgery videos.

[0131] Each module of the above electronic device 600 can be implemented by the processor 610 in this embodiment executing a computer program stored in the memory 610, or can be implemented by other structures, which is not limited here.

[0132] The present disclosure may be a system, method and / or computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0133] A computer-readable storage medium can be a tangible device that can hold and store instructions for use by an instruction execution device. A computer-readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination thereof. As used herein, a computer-readable storage medium is not to be construed as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through an electrical wire.

[0134] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in the computer-readable storage medium in each computing / processing device.

[0135] The computer program instructions for performing the operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, and conventional procedural programming languages ​​such as "C" language or similar programming languages. Computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., utilizing an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of the computer-readable program instructions. The electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0136] Various aspects of the present disclosure are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0137] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0138] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0139] The flowcharts and block diagrams in the accompanying drawings show the possible implementation architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment or part of an instruction, and the module, program segment or part of the instruction contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that implementation by hardware, implementation by software, and implementation by a combination of software and hardware are all equivalent.

[0140] The embodiments of the present disclosure have been described above. The above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terms used herein are selected to best explain the principles of the embodiments, their practical applications, or technical improvements in the marketplace, or to enable other persons skilled in the art to understand the embodiments disclosed herein. The scope of the present disclosure is defined by the appended claims.

Claims

1. A self-supervised monocular depth estimation method for laparoscopic surgery videos, characterized by: include: Obtaining an initial video frame sequence corresponding to a laparoscopic surgery video; Identifying a target filling area in the initial video frame sequence through a gated convolutional network, and filling the target filling area to obtain a padded video frame sequence; The padded video frame sequence is input into a preset depth estimation model, a multi-scale depth map is obtained through the depth network included in the preset depth estimation model, and a relative pose matrix of adjacent frames in the padded video frame sequence is obtained through the pose network included in the preset depth estimation model, so as to generate a depth estimation result and a corresponding target depth map based on the multi-scale depth map and the relative pose matrix.

2. The method according to claim 1, characterized in that The depth network includes an encoding subnetwork and a decoding subnetwork, and obtaining a multi-scale depth map through the depth network included in the preset depth estimation model includes: Performing multi-scale feature extraction on the padded image sequence through the encoding sub-network, wherein the encoding sub-network generates a multi-level feature representation by processing local regions of different sizes in parallel and establishing a global feature relationship; The multi-level feature representation is fused across layers and scales through the decoding subnetwork, and the weights of the fused features are adjusted by combining the spatial attention mechanism and the channel attention mechanism; Based on the weighted features, depth maps of different resolutions are outputted by multiple independent prediction modules to obtain the multi-scale depth map.

3. The method according to claim 2, characterized in that The multi-scale feature extraction of the encoding sub-network includes: Extract local features of different ranges from the input image to generate feature maps of multiple scales; Performing global feature relationship modeling on the feature maps of the multiple scales to generate feature representations containing long-range dependencies; The feature representations are gradually fused through a plurality of cascaded feature aggregation stages to generate the multi-level feature representation.

4. The method according to claim 2, characterized in that The cross-layer and cross-scale fusion of the decoding sub-network includes: Connecting the output features of each stage in the encoding subnetwork with the features of the corresponding scale of the decoding subnetwork; Assign weights to the spatial dimension and channel dimension of the connected features respectively; The weighted features are input into the multiple independent prediction modules to generate a depth map with a resolution decreasing from the input image as the multi-scale depth map.

5. The method according to claim 1, wherein The step of obtaining relative pose matrices of adjacent frames in the padded video frame sequence through a pose network included in the preset depth estimation model includes: Obtaining a multi-scale feature map of each video frame and its adjacent frames in the padded video frame sequence by convolution, wherein the multi-scale feature map is a feature map of multiple resolutions generated by downsampling and feature extraction of the spliced ​​adjacent frame images through multiple convolution layers with different step sizes; Extracting a channel attention vector of the multi-scale feature map through a pyramid squeeze attention sub-network in the pose network to calculate a soft assignment weight of the multi-scale feature map based on the channel attention vector; Calculating a weighted multi-scale feature map according to a correspondence between the multi-scale feature map and the soft assignment weights; The weighted multi-scale feature maps are spliced ​​to obtain fused features, and the relative pose matrix is ​​obtained according to the fused features.

6. The method according to claim 1, characterized in that Generating a depth estimation result and a corresponding target depth map according to the multi-scale depth map and the relative pose matrix includes: Based on the relative pose matrix and the camera intrinsic parameters, reprojecting the temporally adjacent source frame images in the padded image sequence to a target frame coordinate system to generate a reprojected image, wherein the source frame image is a previous frame and / or a subsequent frame image temporally adjacent to the target frame image in the padded image sequence, and the target frame image is a video frame in the padded image sequence for which a depth map is currently to be generated; Calculating an average normalized inverse depth edge perception loss and a photometric loss of the target frame image, wherein the edge perception loss is obtained by summing the exponentially weighted product of the inverse depth gradient and the image gradient, and the photometric loss is the pixel intensity difference between the reprojected image and the target frame image; The photometric loss and the edge perception loss are weightedly summed at multiple preset scales, and the total loss is obtained by combining the filtering process of invalid pixel areas with automatic masking; The parameters of the depth network and the pose network are updated according to the total loss to generate the depth estimation result and the target depth map.

7. The method according to claim 1, characterized in that The obtaining of an initial video frame sequence corresponding to the laparoscopic surgery video includes: Converting the video frames included in the laparoscopic surgery video into grayscale images, and smoothing the grayscale images using a Gaussian blur filter to obtain a target grayscale image; Converting the target grayscale image into a binary image based on a preset grayscale threshold, applying an erosion operation to the binary image to remove noise and a dilation operation to expand highlight areas, to obtain a target binary image; performing connected component analysis on the target binary image to generate a mask sequence; The initial video frame sequence is generated according to the correspondence between the mask sequence and the video frames.

8. A self-supervised monocular depth estimation device for laparoscopic surgery videos, characterized in that: include: An acquisition module is used to acquire an initial video frame sequence corresponding to the laparoscopic surgery video; a filling module, configured to identify a target filling area in the initial video frame sequence through a gated convolutional network, and fill the target filling area to obtain a padded video frame sequence; A generation module is used to input the padded video frame sequence into a preset depth estimation model, obtain a multi-scale depth map through the depth network included in the preset depth estimation model, and obtain a relative pose matrix of adjacent frames in the padded video frame sequence through the pose network included in the preset depth estimation model, so as to generate a depth estimation result and a corresponding target depth map based on the multi-scale depth map and the relative pose matrix.

9. An electronic device comprising a memory and a processor, wherein the memory is used to store a computer program; and the processor is used to execute the computer program to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the computer program implements the method according to any one of claims 1 to 7.

Images