Endoscope image disparity prediction model establishment method based on bidirectional supervision and application

Abstract

The application discloses a kind of endoscope image disparity prediction model establishment method and application based on two-way supervision, belong to image processing technical field, including: constructing initial neural network, including two branches, respectively branch A and branch B, each branch is binocular image as input, predicts corresponding left eye image disparity probability distribution and disparity map;Partial labeled dataset is used to train initial neural network, for unlabeled sample, with the disparity map predicted by branch A as the pseudo-label of branch B, supervise the disparity probability distribution and disparity map of branch B, and with the disparity map predicted by branch B as the pseudo-label of branch A, supervise the disparity probability distribution and disparity map of branch A;The initial neural network or one branch after training is used as disparity prediction model.The application can effectively solve the technical problem that the depth estimation accuracy of endoscope image is further improved when the model for endoscope disparity prediction is trained in the prior art, and the single direction knowledge flow is limited.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and more specifically, relates to a method and application for establishing a parallax prediction model for endoscopic images based on bidirectional supervision. Background Technology

[0002] As surgical procedures become increasingly minimally invasive, endoscopes are being widely used in clinical surgery. Surgeons only need to make a few incisions on the patient's body surface, insert an endoscope into the patient's body, and operate instruments to complete the surgery under the guidance of images captured by the endoscope. Compared to traditional open surgery, minimally invasive surgery has the advantages of smaller incisions, less pain, faster recovery, and a lower infection rate.

[0003] Traditional endoscopes used in minimally invasive surgery capture two-dimensional images. These images often suffer from significant distortion and a lack of depth perception. Therefore, they cannot provide surgeons with intuitive depth information, easily leading to misjudgments of the relative positions of critical areas and instruments during surgery, potentially resulting in operational errors and internal organ bleeding.

[0004] For the reasons mentioned above, binocular endoscopes have been widely used in minimally invasive surgery in recent years. This type of endoscope uses two image acquisition units to capture images of the target and transmits these images to a special display device, providing doctors with three-dimensional images. Simultaneously, providing depth information of the surgical scene helps doctors better determine the relative positions of instruments and tissues, significantly reducing surgical risks and shortening operation time. Furthermore, the reconstructed surgical scene can be used for registration with preoperative models, surgical training, and augmented reality applications.

[0005] The current principle for depth estimation using binocular cameras is as follows: Matching points are found in two images captured by the binocular camera. The disparity is calculated by determining the pixel position difference between the corresponding points in the two images, and then the depth value of that point is calculated based on the camera's inherent parameters. Traditional methods based on this principle have achieved good results in natural images. However, these methods are highly dependent on the target texture and assume that the target surface is smooth. Therefore, when reconstructing images captured by an endoscope using traditional methods, significant errors in the estimation are often caused by factors such as reflections, instrument occlusion, and limited texture features.

[0006] In recent years, Convolutional Neural Networks (CNNs) have achieved remarkable results in many complex problems, including binocular depth estimation. For binocular depth estimation, supervised CNNs fuse contextual information at multiple scales, thus achieving better performance than traditional methods. However, training depth estimation networks requires a large amount of pseudo-labeled depth data, which is difficult to obtain in practice. Especially in the in vivo environment, typically only a small number of endoscopic images have corresponding depth labels.

[0007] To address the sparse labeling problem of endoscopic images, the latest approach involves semi-supervised training of a binocular depth estimation network. This involves training the network using both sparsely labeled and unlabeled images. Specifically, a teacher model is first trained in a supervised manner using sparse labels. Then, the teacher model predicts data from unlabeled images, and these predictions are used as pseudo-labels to further train a student model alongside the sparse labels. The student model achieves better depth estimation accuracy by extracting the pseudo-labels predicted by the teacher model. However, in this semi-supervised training method, knowledge can only flow unidirectionally from the teacher model to the student model. Since the teacher model has already been trained when the student model is being trained, the fixed pseudo-labels limit further improvements in the student model's accuracy. Summary of the Invention

[0008] To address the shortcomings and improvement needs of existing technologies, this invention provides a method and application for establishing an endoscope image disparity prediction model based on bidirectional supervision. It aims to solve the technical problem that the unidirectional knowledge flow limits the further improvement of the depth estimation accuracy of endoscope images when training models for endoscope disparity prediction.

[0009] To achieve the above objectives, according to one aspect of the present invention, a method for establishing an endoscopic image disparity prediction model based on bidirectional adaptive supervision is provided, comprising:

[0010] Initialization steps: Construct an initial neural network based on a pre-trained cost cube prediction network for estimating cost cubes in stereo images. This network includes two branches, branch A and branch B. Each branch includes: a cost cube prediction network; a disparity probability distribution prediction module, which predicts the probability of each pixel at each candidate disparity value based on the cost cube output by the cost cube prediction network, thus obtaining the disparity probability distribution; and a disparity prediction module, which calculates the disparity value of each pixel based on the disparity probability distribution, thus obtaining the disparity map.

[0011] Model training steps: The initial neural network is trained using a dataset of binocular endoscope images with partial disparity map labels. During the training process, the samples in each training batch are either all labeled samples or all unlabeled samples. For unlabeled samples, the disparity map predicted by branch A is used as the pseudo-label of branch B, and the disparity map predicted by branch B is used as the pseudo-label of branch A.

[0012] Model extraction steps: Use the trained initial neural network as the first endoscopic image disparity prediction model, or use the model obtained by connecting the cost cube prediction network, disparity probability distribution prediction module and disparity prediction module in one branch of the trained initial neural network as the second endoscopic disparity prediction model.

[0013] Furthermore, the branch also includes: a confidence network, used to estimate the accuracy of the disparity map output by the disparity prediction module to obtain a confidence map;

[0014] Furthermore, during the model training process, for unlabeled samples, the training loss function includes the adaptive cross-supervision loss L. ACS Its expression is as follows:

[0015]

[0016]

[0017]

[0018] Where R represents the preset disparity range, R>0; s represents the disparity value within the disparity range; (x,y) represents the pixel coordinates; P a and P b Let A and B represent the disparity probability distributions predicted by branch A and branch B, respectively. and The pseudo-labels represent the disparity probability distributions of branches A and B, respectively. and These represent the disparity map pseudo-labels for branches A and B, respectively. D a and D b These represent the disparity maps predicted by branch A and branch B, respectively. K a and K b These represent confidence plots for branch A and branch B, respectively, where M>1.

[0019] Furthermore, in the model training step, for unlabeled samples, the training loss function also includes an adaptive parallel supervision loss L. APS Its expression is as follows:

[0020]

[0021] Where H and W represent the height and width of the image, respectively; Smooth L1 This represents smoothed L1 loss.

[0022] Furthermore, in the model training step, for labeled samples, the training loss function includes the disparity probability distribution constraint L. dist Its expression is:

[0023]

[0024]

[0025]

[0026] Where N represents the total number of labeled pixels; Indicates the parallax map label; and These are pseudo-labels representing the disparity probability distributions of branches A and B, respectively.

[0027] Furthermore, in the model training steps, for labeled samples, the training loss function also includes disparity map constraints L. value Its expression is:

[0028]

[0029] Furthermore, in the model training step, for labeled samples, the training loss function also includes confidence graph constraints L. conf Its expression is:

[0030]

[0031] in, The confidence map label for branch A is generated by taking the disparity map D predicted by branch A. a With disparity map labels Perform pixel-by-pixel comparison; if the pixel difference is within a preset range, then label the confidence map. The corresponding pixel in the confidence map is marked as "1"; otherwise, the confidence map label... The corresponding pixel in the middle is marked as "0"; The confidence map label for branch B is generated by taking the disparity map D predicted by branch B. b With disparity map labels Perform pixel-by-pixel comparison; if the pixel difference is within a preset range, then label the confidence map. The corresponding pixel in the confidence map is marked as "1"; otherwise, the confidence map label... The corresponding pixel is marked as "0".

[0032] Furthermore, the cost cube prediction network includes:

[0033] The first feature extraction module is used to extract features at different scales in the left eye image and fuse them with contextual information to obtain the feature map of the left eye image.

[0034] The second feature extraction module is used to extract features at different scales in the right eye image and fuse them with contextual information to obtain the feature map of the right eye image.

[0035] The pixel feature stitching module is used to stitch the feature maps of the left and right eyes images pixel by pixel to obtain the cost cube C. concat ;

[0036] The inter-group feature stitching module is used to group the features of corresponding pixels in the feature maps of the left and right images, calculate the inner product of the corresponding groups, and obtain the cost cube C. group ;

[0037] The cube stitching module is used to stitch together the cost cube C. concat With the cost cube C group Perform pixel-by-pixel stitching to obtain the cost cube C feat ;

[0038] and one or more cascaded 3D Attenuated CNN modules to augment the cost cube C using channel attention mechanisms. feat The correlation between the middle channels is used to obtain the cost cube of the stereo image.

[0039] According to another aspect of the present invention, a method for predicting disparity of endoscopic images based on bidirectional adaptive supervision is provided, comprising: inputting a binocular endoscopic image into a second endoscopic disparity prediction model established by the above-mentioned method for establishing a bidirectional adaptive supervision endoscopic image disparity prediction model provided by the present invention, to obtain a disparity map of the binocular endoscopic image.

[0040] According to another aspect of the present invention, a method for predicting disparity in endoscopic images based on bidirectional adaptive supervision is provided, comprising: inputting a binocular endoscope image into a first endoscopic image disparity prediction model established by the above-mentioned method for establishing a bidirectional adaptive supervision-based endoscopic image disparity prediction model provided by the present invention, obtaining two disparity maps of the binocular endoscope image and a confidence map corresponding to each disparity map, calculating the average confidence of all pixels based on the confidence map, and taking the disparity map with the higher average confidence of pixels as the disparity prediction result.

[0041] According to another aspect of the present invention, a computer-readable storage medium is provided, comprising: a stored computer program; when the computer program is executed by a processor, it controls the device where the computer-readable storage medium is located to execute the above-mentioned method for establishing an endoscopic image disparity prediction model based on bidirectional adaptive supervision provided by the present invention, and / or the above-mentioned method for predicting endoscopic image disparity based on bidirectional adaptive supervision provided by the present invention.

[0042] In summary, the above-described technical solutions conceived in this invention can achieve the following beneficial effects:

[0043] (1) The method for establishing a disparity prediction model for endoscopic images based on bidirectional adaptive supervision provided in this invention establishes an initial neural network comprising two branches, namely branch A and branch B. These two branches can independently predict the cost cube, disparity probability distribution, and disparity map of binocular endoscopic images. During the training of the model using unlabeled data, the disparity maps predicted by the two branches serve as pseudo-labels for the disparity maps of the other branch. Thus, knowledge can flow bidirectionally between the two branches during training, and the two branches can be continuously updated during training, thereby providing higher quality disparity map pseudo-labels and effectively improving the utilization efficiency of unlabeled images. Ultimately, the established model has high disparity prediction accuracy.

[0044] (2) In the preferred embodiment of the bidirectional adaptive supervision-based method for establishing an endoscopic image disparity prediction model provided by the present invention, each branch of the established model further includes a confidence network for evaluating the accuracy of the disparity map predicted by that branch, and an adaptive cross-supervision loss L is designed for unlabeled samples based on the confidence map. ACS Specifically, while using the disparity map predicted by one branch as the pseudo-label of the disparity map of another branch, a corresponding disparity probability distribution pseudo-label is generated based on the disparity map pseudo-label. Furthermore, the probability distribution in the disparity probability distribution pseudo-label is adjusted using the confidence map estimated by the branch, so that pixels with low confidence have a smoother distribution. Based on this adaptive cross-supervision loss, the intensity of supervision can be adaptively adjusted, thereby further improving the utilization efficiency of unlabeled images.

[0045] (3) The method for establishing an endoscopic image disparity prediction model based on bidirectional adaptive supervision provided in this invention, in its preferred embodiment, also designs an adaptive parallel supervision loss L for unlabeled samples based on the confidence map of branch estimation. APS Specifically, while using the disparity map predicted by one branch as the pseudo-label of the disparity map of another branch, the confidence map estimated by the branch is used as the weight to calculate the loss between the disparity map prediction result and the disparity map pseudo-label. This can effectively suppress possible errors in the disparity map pseudo-label and further improve the prediction accuracy of the established model.

[0046] (4) In the preferred embodiment of the method for establishing a disparity prediction model for endoscopic images based on bidirectional adaptive supervision provided by the present invention, for labeled samples, a disparity probability distribution constraint L is designed in the loss function based on the confidence map of the branch estimation. distSpecifically, it generates corresponding disparity probability distribution pseudo-labels based on the labeled disparity map labels, and adjusts the probability distribution in the disparity probability distribution pseudo-labels using the confidence map of branch estimation, so that pixels with low confidence have a smoother distribution, thereby making more effective use of sparse labels in labeled data.

[0047] (5) In the preferred embodiment of the method for establishing an endoscope image disparity prediction model based on bidirectional adaptive supervision provided by the present invention, for labeled samples, a disparity map constraint L is designed in the loss function based on the confidence map of the branch estimation. value The confidence graph estimated by the branch is used as the weight to calculate the corresponding constraints, which can effectively suppress possible errors in the label and further improve the prediction accuracy of the established model.

[0048] (6) The method for establishing an endoscopic image disparity prediction model based on bidirectional adaptive supervision provided in this invention includes a cost cube prediction network for estimating the cost cube of binocular images. The inter-group feature stitching module groups the features of corresponding pixels in the feature maps of the left and right images to calculate the inner product, thus obtaining the cost cube C. group The cost cube C group These features will be concatenated into the final cost cube. Since the calculated inner product includes the similarity between features, the cost cube output by the cost cube prediction network also carries the similarity information between the features of the left and right eyes. This information can serve as auxiliary information to speed up the training of the model and improve the prediction accuracy of the model.

[0049] (7) The endoscopic image disparity prediction method based on bidirectional adaptive supervision provided by the present invention can quickly complete the disparity prediction using the first endoscopic image disparity prediction model and has high disparity accuracy; it can also use the second endoscopic image disparity prediction model, combined with the confidence estimation results, to further improve the accuracy of disparity prediction. Attached Figure Description

[0050] Figure 1 This is a schematic diagram of a dual-branch network framework provided in an embodiment of the present invention;

[0051] Figure 2 This is a schematic diagram of the cost cube prediction network provided in an embodiment of the present invention;

[0052] Figure 3 This is a schematic diagram of the structure of the 3D Attenuated CNN module provided in an embodiment of the present invention;

[0053] Figure 4The diagrams provided in this embodiment of the invention are adaptive cross-supervision and adaptive parallel supervision; wherein, (a) is an adaptive cross-supervision diagram and (b) is an adaptive parallel supervision diagram;

[0054] Figure 5 A comparison chart of prediction results between the endoscopic image parallax prediction method provided in this embodiment of the invention and other prediction methods;

[0055] Figure 6 This is a schematic diagram comparing the image reconstructed from the disparity map predicted by the endoscopic image disparity prediction method provided in this embodiment of the invention with the original left eye image. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0057] In this invention, the terms "first," "second," etc. (if present) in the invention and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0058] To address the technical challenge of unidirectional knowledge flow limiting the accuracy of depth estimation in endoscopic images during training models for disparity prediction, this invention provides a method and application for establishing an endoscopic image disparity prediction model based on bidirectional supervision. The overall approach involves establishing a two-branch network and training it on an endoscopic dataset containing both labeled and unlabeled training data. During training, for the unlabeled training data, the disparity maps predicted by the two branches are used as pseudo-labels for each other, enabling bidirectional knowledge flow between the two branches. This continuous updating of the two branches during training effectively improves the quality of the disparity map pseudo-labels, thereby enhancing the model's prediction accuracy. Furthermore, by incorporating the confidence map of the disparity map, appropriate loss functions are designed for both unlabeled and labeled data to further improve the model's prediction accuracy, ultimately enhancing the accuracy of subsequent disparity predictions.

[0059] The following is an example.

[0060] Example 1:

[0061] A method for establishing an endoscopic image disparity prediction model based on bidirectional adaptive supervision includes:

[0062] Initialization steps: Construct an initial neural network based on a pre-trained cost cube prediction network used for estimating cost cubes of stereo images; the initial neural network is a two-branch network, such as... Figure 1 As shown, the two branches are branch A and branch B, respectively. The branches include: a cost cube prediction network; a disparity probability distribution prediction module, which is used to predict the probability of each pixel in each candidate disparity value based on the cost cube output by the cost cube prediction network to obtain the disparity probability distribution; and a disparity prediction module, which is used to calculate the disparity value of each pixel based on the disparity probability distribution to obtain the disparity map.

[0063] Model training steps: The initial neural network is trained using a dataset of binocular endoscope images with partial disparity map labels. During the training process, the samples in each training batch are either all labeled samples or all unlabeled samples. For unlabeled samples, the disparity map predicted by branch A is used as the pseudo-label of branch B, and the disparity map predicted by branch B is used as the pseudo-label of branch A.

[0064] Model extraction steps: The trained initial neural network is used as the first endoscopic image disparity prediction model.

[0065] Cost cube, or cost-volume, in this embodiment, is used to estimate the cost cube of binocular endoscope images. The cost cube prediction network is as follows: Figure 2 As shown, it specifically includes:

[0066] The first feature extraction module is used to extract the left eye image I. l Features at different scales are analyzed, and their contextual information is fused to obtain the feature map F of the left eye image. l ;

[0067] The second feature extraction module is used to extract the right eye image I. r Features at different scales are analyzed, and their contextual information is fused to obtain the feature map F of the right eye image. r ;

[0068] The pixel feature stitching module is used to stitch the feature map F of the left eye image. l Feature map F of the right eye image r Perform pixel-by-pixel stitching to obtain the cost cube C concat ;

[0069] The inter-group feature stitching module is used to stitch together the feature map F of the left eye image. l Feature map F of the right eye image r In the process, after grouping the features of corresponding pixels, the inner product of the corresponding groups is calculated to obtain the cost cube C. group ;

[0070] The cube stitching module is used to stitch together the cost cube C. concat With the cost cube C group Perform pixel-by-pixel stitching to obtain the cost cube C feat ;

[0071] And three cascaded 3D Attenuated CNN modules to augment the cost cube C using channel attention mechanisms. feat The correlation between the channels is used to obtain the cost cube C of the binocular image; in this embodiment, the structure of the 3D Attenuated CNN module is as follows: Figure 3 As shown, it includes: an encoder-decoder structure consisting of multiple 3D convolutional layers and multiple 3D deconvolutional layers, and a channel attention mechanism module located in the middle of the encoder-decoder structure; the 3D Atten CNN module also includes skip connections to prevent gradient vanishing; the 3D Atten CNN can aggregate feature cubes C feat Regarding information in the parallax dimension and image space dimension, it should be noted that in practical applications, the number of 3D Attenuated CNNs in the Cost Cube prediction network can be flexibly adjusted.

[0072] In this embodiment, since the inner product calculated by the inter-group feature stitching module in the cost cube prediction network includes the similarity between the features of the left and right images, the cost cube output by the cost cube prediction network also carries the similarity information between the features of the left and right images. This information can be used as auxiliary information to speed up the training of the model and improve the prediction accuracy of the model.

[0073] Optionally, in this embodiment, the disparity probability distribution prediction module uses the softmax function to transform the cost cube C into the probability of that pixel at each candidate disparity value, calculated as follows:

[0074]

[0075] Where R represents the preset disparity range, R>0; s represents the disparity value within the disparity range R; (x,y) represents the pixel coordinates; P(x,y,s) represents the probability that the disparity of the pixel at (x,y) is s.

[0076] Optionally, in this embodiment, the disparity prediction module calculates the disparity map using the following formula:

[0077]

[0078] For ease of description, in this embodiment, the cost cube prediction network is abbreviated as DEnet. Since DEnet has many parameters, the publicly available SceneFlow synthetic stereo dataset is used for pre-training to reduce the difficulty of transferring the data to the endoscope dataset in subsequent steps. During pre-training, a smooth L1 loss (i.e., smooth L1 loss) is used. L1 Instead of the traditional L1 loss for supervision, the expression for the smoothed L1 loss is as follows:

[0079]

[0080]

[0081] in This represents the label disparity value.

[0082] In the above initialization steps, the cost cube prediction network used to build the initial neural network is the pre-trained DEnet.

[0083] like Figure 1 As shown, to further improve the model's accuracy, in this embodiment, each branch also includes a confidence network, which takes the cost cube C predicted by DENet as input and is used to estimate the accuracy of the disparity map output by the disparity prediction module to obtain a confidence map. Optionally, in this embodiment, the confidence network specifically includes two convolutional layers, one batch normalization layer, and one sigmoid function layer. After the cost cube C predicted by DENet is input into the confidence network, the final value of the confidence map output by the confidence network is restricted to the range [0,1] by the sigmoid function. Lower values ​​in the confidence map represent a higher probability of errors in the corresponding disparity in the disparity map.

[0084] Figure 1 Medium, DEnet a and Confnet a Let C represent the cost cube prediction network and the confidence network in branch A, respectively. a P a D a and K a These represent the cost cube, disparity probability distribution, disparity map, and confidence map generated in branch A, respectively. Correspondingly, DEnet... b and Confnet b Let C represent the cost cube prediction network and the confidence network in branch B, respectively. b P b D b and K bThese represent the cost cube, disparity probability distribution, disparity map, and confidence map generated in branch B, respectively.

[0085] In the above initialization steps, DEnet a and DEnet b Pre-trained weights will be loaded, but the weights of the last two layers of each 3D AttenCNN module will be randomly initialized separately to ensure that the cost cubes generated by the two branches, i.e., C, are equal. a and C b There are differences; Confnet a and Confnet b All weights are randomly initialized.

[0086] In the above model training steps, in order to improve the training efficiency of the model, as a preferred implementation method, this embodiment will first execute the training batch of labeled samples. In order to make the network more effective in utilizing the sparse labels of the labeled data, this embodiment proposes to simultaneously impose three constraints on the disparity probability distribution, disparity map and confidence map of the labeled image.

[0087] In order to design the disparity probability distribution constraint, this embodiment will first generate the disparity distribution of the gold standard label. and To constrain branches A and B respectively, and The generation formulas are as follows:

[0088]

[0089]

[0090] Where R represents the preset disparity range, R>0; s represents the disparity value within the disparity range; (x,y) represents the pixel coordinates; This represents the pseudo-label in the generated disparity probability distribution. In the diagram, the probability that the disparity of the pixel at (x,y) is s is similarly... This represents the pseudo-label in the generated disparity probability distribution. In the equation, the probability that the disparity of the pixel at (x,y) is s; and It reaches its maximum when the distance s is closest, and the probability decreases with increasing distance at other disparity values.

[0091] Further research revealed that points with low confidence generally appear in areas that are difficult to match, such as textureless tissue surfaces. For these challenging regions, constraining and training with a sharp pseudo-label distribution is extremely difficult. Based on this consideration, this embodiment calculates ρ based on the confidence level when generating disparity probability distribution pseudo-labels. The specific calculation expression is as follows:

[0092]

[0093]

[0094] Among them, K a and K b These represent the confidence plots for branch A and branch B estimates, respectively, with K... a (x,y) and K b (x, y) represent the disparity map D, respectively. a and D b The accuracy of the disparity value of the pixel at (x,y); M>1, optionally, in this embodiment, M=2. Based on the above calculation formula, when the confidence level decreases, ρ decreases accordingly, leading to a wider distribution. Conversely, when the confidence level increases, the distribution narrows. Thus, the distribution can be adaptively adjusted by the confidence level, so that pixels with low confidence levels have a smoother distribution, which is equivalent to softening the label. According to the expression of ρ, it can be seen that the disparity probability distribution generated in this embodiment is that the label is related to both the disparity map and the confidence map.

[0095] Based on the generated disparity probability distribution pseudo-labels and In this embodiment, for labeled samples, the training loss function includes the disparity probability distribution constraint L. dist Its expression is:

[0096]

[0097] Where N represents the total number of labeled pixels; The label represents the disparity map. Since the disparity probability distribution is related to both the label and the confidence map, this embodiment designs the disparity probability distribution constraint L in the loss function for labeled samples. dist It can achieve joint optimization of DEnet and Confnet; disparity probability distribution constraint L dist In the example, the first item is only used to update branch A, and the second item is only used to update branch B;

[0098] In this embodiment, for labeled data, the disparity map constraint L designed in the loss function is... value Its expression is:

[0099]

[0100] Among them, smooth L1 This represents smoothed L1 loss; similarly, L value In the example, the first item is only used to update branch A, and the second item is only used to update branch B;

[0101] In this embodiment, for labeled data, in order to design confidence map constraints L in the loss function... conf Treating confidence assessment as a binary classification task, the disparity map predicted by the branch is compared pixel-by-pixel with the disparity map label. If the difference is within 3 pixels, the disparity map prediction is considered accurate, and the pixel is marked as "1" in the confidence map; otherwise, it is marked as "0". This labeling result serves as the confidence map label information for training the confidence network. It should be noted that when generating the confidence map label, the difference range between the disparity map predicted by the branch and the disparity map label for the same pixel can be set to other ranges depending on the actual situation. Finally, in this embodiment, for labeled data, the confidence map constraint L designed in the loss function... conf Its expression is:

[0102]

[0103] in, and These represent the confidence plot labels for branches A and B, respectively; similarly, L conf In the example, the first item is only used to update branch A, and the second item is only used to update branch B;

[0104] The above disparity map constraint L value and confidence plot constraints L conf In the process, the confidence graph estimated by the branch is used as the weight to calculate the corresponding constraints, which can effectively suppress possible errors in the label and further improve the prediction accuracy of the established model.

[0105] Based on the above disparity probability distribution constraint L dist Disparity map constraint L value and confidence plot constraints L conf In this embodiment, when training the dual-branch network using labeled samples, there is a supervised loss L. full The specific expression is as follows:

[0106] L full =L dist +L value +L conf

[0107] For unlabeled samples, this embodiment designs two supervision methods—adaptive cross-supervision and adaptive parallel supervision—to constrain the disparity probability distribution, disparity map, and confidence map in each branch, thereby enabling self-supervised training of the network. Correspondingly, for unlabeled samples, the training loss function includes the adaptive cross-supervision loss L. ACS and adaptive parallel supervision loss L APS ;

[0108] Taking the disparity probability distribution of branch A constraining branch B as an example, such as Figure 4 As shown in (a), the disparity map D predicted by branch A is... a As the pseudo-label of the disparity map for branch B, i.e. And based on the disparity map pseudo-label, a disparity probability distribution pseudo-label is generated. Its expression is as follows:

[0109]

[0110] Similarly, in the process of constraining branch A from branch B, the pseudo-label of the disparity map of branch A... Based on the disparity probability distribution pseudo-label generated by the disparity map pseudo-label Its expression is as follows:

[0111]

[0112] Adaptive cross-supervision loss L ACS The expression is as follows:

[0113]

[0114] Adaptive cross-supervision loss L ACS In the example, the first item is only used to update branch B, and the second item is only used to update branch A.

[0115] Similarly, taking branch A constraining branch B as an example, such as Figure 4 As shown in (b), the disparity map D predicted by branch A is... a As the pseudo-label of the disparity map for branch B, i.e. Disparity map D used to guide branch B prediction b To improve the pseudo-labels of the parallax map For reliability, D is also used a Corresponding confidence plot K a The loss is calculated as a weight and used to suppress... Possible errors;

[0116] Adaptive parallel supervision loss L APS The expression is as follows:

[0117]

[0118] Where H and W represent the height and width of the image, respectively; Smooth L1 Smoothing L1 loss; Adaptive parallel supervision loss L APS In the example, the first item is used only to update branch A, and the second item is used only to update branch B.

[0119] Ultimately, the self-supervised loss L of the two-branch network self as follows:

[0120] L self =L APS +L ACS

[0121] After the model training is completed, in the model extraction step of this embodiment, the entire trained network is used as a model to predict the disparity of binocular endoscope images, which is denoted as the first endoscope image disparity prediction model. After the binocular endoscope image is input into the model, each branch will output a disparity map and a corresponding confidence map.

[0122] In summary, the model established by the bidirectional adaptive supervision-based endoscopic image disparity prediction model provided in this embodiment is a disparity estimation network based on a two-branch convolutional neural network. It can accurately estimate disparity on sparsely labeled datasets, achieving higher accuracy compared to other networks. During model training, parallel and cross-supervision of unlabeled images are proposed, making full use of unlabeled images for training. Simultaneously, a confidence network, Confnet, is proposed to evaluate the confidence of pseudo-labels, and the strength of the two types of supervision is adaptively adjusted through confidence, thereby further improving the utilization efficiency of unlabeled images. Furthermore, three constraints are proposed simultaneously on the disparity map, disparity probability distribution, and confidence map of labeled images, enabling the network to more effectively utilize the sparse labels of labeled data.

[0123] Example 2:

[0124] A method for establishing a disparity prediction model for endoscopic images based on bidirectional adaptive supervision is presented in this embodiment, which is similar to the above embodiment 1. The difference is that after the model training is completed, in the model extraction step of this embodiment, only the cost cube prediction network, the disparity probability distribution prediction module, and the model obtained by connecting the disparity prediction module of one branch of the network are extracted as the model for predicting the disparity of binocular endoscopic images. This model is denoted as the second endoscopic disparity prediction model. After the binocular endoscopic image is input into the model, the model output is the disparity prediction result of the binocular endoscopic image.

[0125] In this embodiment, the specific implementation of the remaining steps can be referred to the description in Embodiment 1 above, and will not be repeated here.

[0126] Example 3:

[0127] A method for predicting disparity in endoscopic images based on bidirectional adaptive supervision includes: inputting a binocular endoscope image into a first endoscopic image disparity prediction model established by the bidirectional adaptive supervision-based endoscopic image disparity prediction model establishment method provided in Embodiment 1 above, obtaining two disparity maps of the binocular endoscope image and a confidence map corresponding to each disparity map, calculating the average confidence of all pixels based on the confidence map, and taking the disparity map with the higher average confidence of pixels as the disparity prediction result.

[0128] This embodiment utilizes two branches in the first endoscopic image disparity prediction model to predict the disparity map and the confidence map corresponding to each disparity map, and further uses the disparity map with higher confidence as the final disparity prediction result, which can further improve the prediction accuracy of the disparity map.

[0129] Example 4:

[0130] A method for predicting disparity in endoscopic images based on bidirectional adaptive supervision includes: inputting binocular endoscopic images into a second endoscopic disparity prediction model established by the bidirectional adaptive supervision-based endoscopic image disparity prediction model establishment method provided in Embodiment 2 above, to obtain a disparity map of the binocular endoscopic images.

[0131] Example 5:

[0132] A computer-readable storage medium includes: a stored computer program; when the computer program is executed by a processor, it controls the device where the computer-readable storage medium is located to execute the bidirectional adaptive supervision-based endoscopic image disparity prediction model establishment method provided in Embodiment 1 or 2 above, and / or the bidirectional adaptive supervision-based endoscopic image disparity prediction method provided in Embodiment 3 or 4 above.

[0133] To further verify the disparity prediction accuracy of this invention for binocular endoscopic images, the invention was tested on a sparsely labeled laparoscopic dataset. Common prediction methods (HSMNet, CFnet, Soft MT, Improved TSN) were compared with the binocular endoscopic disparity prediction method (denoted as Ours) provided in Example 3 above. The error comparison results of various prediction methods are as follows: Figure 5 As shown. Figure 5 In the diagram, the leftmost column shows the left eye image input to the network, and the following five columns are the prediction error diagrams of HSMNet, CFnet, Soft MT, Improved TSN, and Ours, respectively. By comparing the prediction errors of the following five columns, it can be seen that the binocular endoscopic disparity prediction method proposed in this invention has fewer prediction errors compared with other methods, especially at the edge of organs, where the binocular endoscopic disparity prediction method proposed in this invention estimates more accurately.

[0134] Figure 6 The left and right images in the image are the left eye image input into the model and the model reconstructed based on the disparity map predicted by the model, respectively. The comparison shows that the reconstructed model is smooth and has no noise points, which has clinical application significance.

[0135] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for establishing an endoscopic image disparity prediction model based on bidirectional adaptive supervision, characterized in that, The method comprises: An initialization step: an initial neural network is constructed based on a pre-trained cost volume prediction network for estimating a cost volume of a binocular image, the initial neural network comprising two branches, branch A and branch B; the branches comprise: a cost volume prediction network; a disparity probability distribution prediction module for predicting a probability of each pixel at each candidate disparity value based on a cost volume output by the cost volume prediction network to obtain a disparity probability distribution; a disparity prediction module for calculating a disparity value of each pixel based on the disparity probability distribution to obtain a disparity map; A model training step: the initial neural network is trained using a binocular endoscopic image dataset partially labeled with a disparity map, and in the training process, each training batch of samples is either labeled or unlabeled, and for unlabeled samples, the disparity map predicted by branch A is used as a pseudo label for branch B, and the disparity map predicted by branch B is used as a pseudo label for branch A; A model extraction step: the trained initial neural network is used as a first endoscopic image disparity prediction model, or a model obtained by connecting the cost volume prediction network, the disparity probability distribution prediction module and the disparity prediction module in one of the branches of the trained initial neural network is used as a second endoscopic disparity prediction model; The branches further comprise: a confidence network for estimating the accuracy of the disparity map output by the disparity prediction module to obtain a confidence map; And, in the model training step, for the unlabeled sample, the training loss function includes an adaptive cross-supervision loss L ACS The expression is as follows: wherein, R denotes a preset disparity range, R > 0; s denotes a disparity value within the disparity range; x , y ) denotes a pixel coordinate; P a and P b denote disparity probability distributions predicted by branch A and branch B, respectively; and denote disparity probability distribution pseudo labels of branch A and branch B, respectively; and denote disparity map pseudo labels of branch A and branch B, respectively, , , D a and D b denote disparity maps predicted by branch A and branch B, respectively; , , K a and K b denote confidence maps estimated by branch A and branch B, respectively, M > 1. 2.The endoscopic image disparity prediction model establishment method based on bidirectional adaptive supervision of claim 1, wherein, In the model training step, for the unlabeled sample, the training loss function further includes an adaptive parallel supervision loss L APS The expression is as follows: where H and W represent the height and width of the image, respectively; Smooth L1 denotes the smooth L1 loss. 3.The endoscopic image disparity prediction model establishment method based on bidirectional adaptive supervision according to claim 1 or 2, characterized in that, The model training step, for the labeled samples, the training loss function includes a disparity probability distribution constraint L dist The expression is: wherein N represents the total number of labeled pixel points; denotes the disparity map label; and denote the disparity probability distribution pseudo-labels of branch A and branch B, respectively. 4.The method of claim 3, wherein, The model training step, for the labeled samples, the training loss function further comprises a disparity map constraint L value The expression is: 。 5.The endoscopic image disparity prediction model establishment method based on bidirectional adaptive supervision of claim 4, wherein, The model training step, for a labeled sample, the training loss function further comprises a confidence map constraint L conf The expression is: wherein, a confidence map label of branch A is generated by comparing the disparity map predicted by branch A D a with the disparity map label pixel by pixel, if the pixel difference is within a preset range, the corresponding pixel in the confidence map label is marked as "1", otherwise, the corresponding pixel in the confidence map label is marked as "0"; a confidence map label of branch B is generated by comparing the disparity map predicted by branch B D b with the disparity map label pixel by pixel, if the pixel difference is within a preset range, the corresponding pixel in the confidence map label is marked as "1", otherwise, the corresponding pixel in the confidence map label is marked as "0". 6.The endoscopic image disparity prediction model establishment method based on bidirectional adaptive supervision of claim 1, wherein, The cost volume prediction network comprises: A first feature extraction module for extracting features of different scales in a left eye image and fusing the inter-context information thereof to obtain a feature map of the left eye image; A second feature extraction module for extracting features of different scales in a right eye image and fusing the inter-context information thereof to obtain a feature map of the right eye image; The pixel feature splicing module is configured to splice the feature map of the left-eye image and the feature map of the right-eye image pixel by pixel to obtain a cost cube. C concat ; The inter-group feature splicing module is configured to group the features of corresponding pixels in the feature map of the left-eye image and the feature map of the right-eye image, calculate the inner product of the corresponding groups, and obtain a cost cube C group ; A cubic patching module is used to patch a cost cube C concat with a cost cube C group pixel by pixel to obtain a cost cube C feat ; and one or more cascaded 3D Atten CNN modules for enhancing the cost volume with channel attention mechanism C feat The correlation between the middle channels is obtained to obtain the cost volume of the binocular image.

7. A method for endoscopic image disparity prediction based on bidirectional adaptive supervision, characterized in that, The method comprises: Inputting the binocular endoscopic image into the second endoscopic disparity prediction model established by the endoscopic image disparity prediction model establishment method based on bidirectional adaptive supervision according to any one of claims 1-6 to obtain a disparity map of the binocular endoscopic image.

8. An endoscopic image disparity prediction method based on bidirectional adaptive supervision, characterized by, The method comprises: Inputting the binocular endoscopic image into the first endoscopic image disparity prediction model established by the endoscopic image disparity prediction model establishment method based on bidirectional adaptive supervision according to any one of claims 1-6 to obtain two disparity maps of the binocular endoscopic image and a confidence map corresponding to each disparity map, calculating the average confidence of all pixels according to the confidence map, and taking the disparity map with a higher average pixel confidence as the disparity prediction result.

9. A computer-readable storage medium, characterized in that, The computer program is stored in the computer readable storage medium and is executed by a processor to control the device where the computer readable storage medium is located to execute the endoscopic image disparity prediction model establishment method based on bidirectional adaptive supervision according to any one of claims 1-6 and / or the endoscopic image disparity prediction method based on bidirectional adaptive supervision according to claim 7 or 8. ​

Images