Parallax estimation method, system and media

Abstract

The application discloses a parallax estimation method, system and medium, the method comprises the following steps: acquiring a training data set, and extracting the scale feature of each pair of binocular training view in the training data set; according to the size from big to small, the first dimension cost and the second dimension cost of the scale feature are calculated alternately, wherein the first dimension cost and the second dimension cost have a correlation relationship of the calculation range according to the alternate calculation sequence; the cost volume corresponding to the scale is generated based on the result of the alternate calculation, and the model training is performed according to the scale feature and the cost volume, so as to obtain a parallax estimation model, which is used for parallax estimation of the binocular view photographed in real time. The parallax map output by the neural network model contains more comprehensive and rich information.

Description

Technical Field

[0001] This invention belongs to the field of medical devices, and in particular relates to a parallax estimation method, system and medium. Background Technology

[0002] 3D reconstruction of binocular endoscopic images helps doctors or assistive systems fully understand the current surgical scene and is an essential key technology in modern minimally invasive surgery. It has wide applications in medical surgical robots, surgical navigation, and augmented reality. Traditional 3D reconstruction based on binocular image pairs (left and right images) first finds matching pixels between the left and right images, then calculates the spatial offset (i.e., parallax) of the matching pixels in the two images. The depth information of the image can be obtained through the parallax and the inherent parameters of the lens, thus obtaining the 3D reconstruction result.

[0003] Disparity estimation is the most critical and challenging step in 3D reconstruction. Traditional disparity estimation methods often produce very sparse disparity maps, with only valid values ​​found at the locations of matching pixels in two view images. Furthermore, binocular endoscopic images are typically high-resolution images generated at close range, containing rich detail information. Accurately recovering good image detail remains a significant challenge for existing deep learning algorithms. In summary, existing disparity estimation methods suffer from the problem of insufficient information contained in the disparity estimation map. Summary of the Invention

[0004] To address the problem that existing disparity estimation methods lack sufficient information in the disparity estimation map, this invention proposes the following technical solution:

[0005] A first aspect of the present invention provides a disparity estimation method, comprising: acquiring a training dataset and extracting scale features of each pair of binocular training views in the training dataset; alternately calculating a first dimension cost and a second dimension cost of the scale features in descending order of scale, wherein the first dimension cost and the second dimension cost have a correlation in their calculation range according to the alternating calculation order; generating a cost body for the corresponding scale based on the results of the alternating calculation, and performing model training according to the scale features and the cost body to obtain a disparity estimation model for use in estimating disparity of binocular views captured in real time.

[0006] Preferably, the step of alternately calculating the first dimension cost and the second dimension cost of the scale features from largest to smallest includes: calculating the first dimension cost corresponding to the largest scale feature among the scale features; determining a first calculation range based on the first dimension cost of the same scale features among the scale features, and calculating the second dimension cost of the same scale features based on the first calculation range, wherein the same scale features include the largest scale feature; determining a second calculation range based on the second dimension cost of a larger scale feature among the scale features, and calculating the first dimension cost of a smaller scale feature among the scale features based on the second calculation range, wherein the larger scale feature includes the largest scale feature.

[0007] Preferably, determining the second calculation range based on the second dimension cost of the larger scale feature in the scale features includes: determining each pair of pixels with the highest correlation in the binocular training view based on the second dimension cost of the larger scale feature in the scale features, wherein each pair of pixels includes a first pixel and a second pixel; determining multiple third pixels that the first pixel maps from the larger scale feature to the smaller scale feature in the scale features, and determining multiple fourth pixels that the second pixel maps from the larger scale feature to the smaller scale feature; determining the correspondence between each third pixel and each fourth pixel, and determining the second calculation range based on the correspondence.

[0008] Preferably, the step of calculating the first dimension cost of the smaller scale feature in the scale feature based on the second calculation range includes: determining each pixel in the preset first dimension neighborhood of each fourth pixel based on the second calculation range; calculating the correlation between the third pixel and each pixel in the corresponding first dimension neighborhood according to the smaller scale feature; and determining the first dimension cost corresponding to the smaller scale feature based on the pixel with the highest correlation in the first dimension neighborhood.

[0009] Preferably, the first dimensional neighborhood includes a first directional neighborhood and a second directional neighborhood that are perpendicular to each other in the first dimension. The step of determining each pixel of the fourth pixel in the preset first dimensional neighborhood based on the second calculation range includes: determining each pixel of the fourth pixel in the first directional neighborhood and / or the second directional neighborhood based on the second calculation range.

[0010] Preferably, determining the correspondence between each third pixel and each fourth pixel includes: determining the mapping position of each third pixel relative to the first pixel and the mapping position of each fourth pixel relative to the second pixel, and establishing a correspondence between third pixels and fourth pixels with the same mapping position; or, calculating the average correlation of pixels in a preset second-dimensional neighborhood between each third pixel and each fourth pixel, and establishing a correspondence between each third pixel and each fourth pixel based on the magnitude of the average correlation.

[0011] Preferably, after determining the correspondence between each of the third pixels and each of the fourth pixels, the method further includes: randomly generating an offset vector for at least one of the fourth pixels, and updating the fourth pixels that correspond to the third pixels with the offset vectors to pixels offset based on the offset vectors.

[0012] Preferably, the step of performing model training based on the scale features and the cost volume to obtain a disparity estimation model includes: fusing the scale features and the cost volume with the largest scale, and performing disparity estimation at the largest scale on the binocular training view through a preset decoder based on the fusion result; fusing the disparity estimation results corresponding to the smaller scale features and cost volumes in the scale features and the cost volume, as well as the larger scale features and cost volumes, and performing disparity estimation at the smaller scale on the binocular training view through the decoder based on the fusion result, until a disparity estimation result at the smallest scale is obtained, wherein the larger scale includes the largest scale; and performing model optimization based on the disparity estimation result of the binocular training view at the smallest scale until convergence, thereby obtaining a disparity estimation model.

[0013] Preferably, the minimum scale is the same as the scale of the binocular training view.

[0014] A second aspect of the present invention provides a disparity estimation system, comprising: a binocular endoscope and a processor, the binocular endoscope and the processor being communicatively connected, the processor having a disparity estimation model deployed therein, and the processor being configured to: receive a binocular view captured in real time by the binocular endoscope and input the binocular view into the disparity estimation model; extract scale features of the binocular view, and alternately calculate a first dimension cost and a second dimension cost of the scale features from high to low scale, wherein the first dimension cost and the second dimension cost have a correlation in their calculation range according to the alternating calculation order; generate a cost body for the corresponding scale based on the alternating calculation results, and perform disparity estimation on the binocular view according to the scale features and the cost body to obtain an estimation result.

[0015] Preferably, the step of alternately calculating the first dimension cost and the second dimension cost of the scale features from largest to smallest includes: calculating the first dimension cost corresponding to the largest scale feature among the scale features; determining a first calculation range based on the first dimension cost of the same scale features among the scale features, and calculating the second dimension cost of the same scale features based on the first calculation range, wherein the same scale features include the largest scale feature; determining a second calculation range based on the second dimension cost of a larger scale feature among the scale features, and calculating the first dimension cost of a smaller scale feature among the scale features based on the second calculation range, wherein the larger scale feature includes the largest scale feature.

[0016] Preferably, determining the second calculation range based on the second dimension cost of the larger scale feature in the scale features includes: determining each pair of pixels with the highest correlation in the binocular training view based on the second dimension cost of the larger scale feature in the scale features, wherein each pair of pixels includes a first pixel and a second pixel; determining multiple third pixels that the first pixel maps from the larger scale feature to the smaller scale feature in the scale features, and determining multiple fourth pixels that the second pixel maps from the larger scale feature to the smaller scale feature; determining the correspondence between each third pixel and each fourth pixel, and determining the second calculation range based on the correspondence.

[0017] Preferably, the step of calculating the first dimension cost of the smaller scale feature in the scale feature based on the second calculation range includes: determining each pixel in the preset first dimension neighborhood of each fourth pixel based on the second calculation range; calculating the correlation between the third pixel and each pixel in the corresponding first dimension neighborhood according to the smaller scale feature; and determining the first dimension cost corresponding to the smaller scale feature based on the pixel with the highest correlation in the first dimension neighborhood.

[0018] Preferably, the first dimensional neighborhood includes a first directional neighborhood and a second directional neighborhood that are perpendicular to each other in the first dimension. The step of determining each pixel of the fourth pixel in the preset first dimensional neighborhood based on the second calculation range includes: determining each pixel of the fourth pixel in the first directional neighborhood and / or the second directional neighborhood based on the second calculation range.

[0019] Preferably, determining the correspondence between each third pixel and each fourth pixel includes: determining the mapping position of each third pixel relative to the first pixel and the mapping position of each fourth pixel relative to the second pixel, and establishing a correspondence between third pixels and fourth pixels with the same mapping position; or, calculating the average correlation of pixels in a preset second-dimensional neighborhood between each third pixel and each fourth pixel, and establishing a correspondence between each third pixel and each fourth pixel based on the magnitude of the average correlation.

[0020] Preferably, after determining the correspondence between each of the third pixels and each of the fourth pixels, the method further includes: randomly generating an offset vector for at least one of the fourth pixels, and updating the fourth pixels that correspond to the third pixels with the offset vectors to pixels offset based on the offset vectors.

[0021] Preferably, the step of performing model training based on the scale features and the cost volume to obtain a disparity estimation model includes: fusing the scale features and the cost volume with the largest scale, and performing disparity estimation at the largest scale on the binocular training view through a preset decoder based on the fusion result; fusing the disparity estimation results corresponding to the smaller scale features and cost volumes in the scale features and the cost volume, as well as the larger scale features and cost volumes, and performing disparity estimation at the smaller scale on the binocular training view through the decoder based on the fusion result, until a disparity estimation result at the smallest scale is obtained, wherein the larger scale includes the largest scale; and performing model optimization based on the disparity estimation result of the binocular training view at the smallest scale until convergence, thereby obtaining a disparity estimation model.

[0022] Preferably, the minimum scale is the same as the scale of the binocular training view.

[0023] A third aspect of the present invention provides a readable storage medium on which a computer program is stored, which, when executed by a processor, implements the steps of the above disparity estimation method.

[0024] The beneficial effects of this invention are as follows: By extracting features at multiple scales from the training dataset, and then alternately calculating the first and second dimension costs of the scale features from smallest to largest scale, a cost volume representing more detailed information is gradually searched locally from a larger range with less computation, thus representing the global pixel matching between binocular views. Finally, the model is trained based on the scale features and cost volumes searched from global to local, resulting in a disparity estimation model that outputs a disparity map containing more comprehensive and richer information when used for disparity estimation. Attached Figure Description

[0025] Figure 1 This is a schematic diagram of the structure of an embodiment of the deep learning network model in this invention;

[0026] Figure 2 This is a schematic diagram of the structure of an embodiment of the feature extractor in this invention;

[0027] Figure 3 This is a flowchart illustrating an embodiment of the disparity estimation method in this invention;

[0028] Figure 4 This is a schematic diagram of the first embodiment of multi-scale cost calculation in this invention;

[0029] Figure 5 This is a schematic diagram of the second embodiment of multi-scale cost calculation in this invention;

[0030] Figure 6 This is a schematic diagram of the third embodiment of multi-scale cost calculation in this invention;

[0031] Figure 7 This is a schematic diagram of the fourth embodiment of multi-scale cost calculation in this invention;

[0032] Figure 8 This is a schematic diagram of the fifth embodiment of multi-scale cost calculation in this invention;

[0033] Figure 9 This is a schematic diagram of the sixth embodiment of multi-scale cost calculation in this invention;

[0034] Figure 10 This is a schematic diagram of the seventh embodiment of multi-scale cost calculation in this invention;

[0035] Figure 11 This is a schematic diagram of the eighth embodiment of multi-scale cost calculation in this invention;

[0036] Figure 12 This is a schematic diagram of the structure of an embodiment of the robot system in this invention;

[0037] Figure 13 This is a flowchart illustrating the application of the deep learning network model in this invention. Detailed Implementation

[0038] To facilitate understanding of the present invention, a more detailed description is provided below with reference to the accompanying drawings and specific embodiments. Preferred embodiments of the invention are shown in the drawings. However, the invention can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to provide a thorough and complete understanding of the disclosure of the invention.

[0039] It should be noted that, unless otherwise defined, all technical and scientific terms used in this specification have the same meaning as understood by one of ordinary skill in the art to which this invention pertains. The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. For example, the term "a plurality of" includes two or more.

[0040] I. Building the LDCVNet Model

[0041] For easier understanding, please refer to Figure 1 and Figure 2 The structure of the LDCVNet model will be explained below:

[0042] This invention originally designed a multi-scale deep learning network model based on a locally deformable cost volume (LDCVNet model). Figure 1 The LDCVNet model includes at least: a feature extractor 11, locally deformable cost body modules (LDCV modules 12, 13, and 14), and decoders (decoders 15, 16, and 17). In addition, depending on completeness or scalability, the LDCVNet model may include other basic components of a deep learning network model, such as an input layer and a loss layer, and its components may be optimized and improved; these will not be elaborated upon here.

[0043] In this embodiment, the images input to the LDCVNet model appear in pairs, such as the left and right views captured by a binocular endoscope. The input layer receives the binocular training views, such as the left and right views, and performs preprocessing, including at least one of the following preprocessing steps: (1) resizing the left and right views to meet the input requirements of the feature extractor; (2) performing data augmentation on the left and right views based on the required scene type to increase the view scenes for subsequent model training and expand the applicable scenarios of the model application. After preprocessing is completed at the input layer, the left and right views are output to the feature extractor 11.

[0044] In this embodiment, the feature extractor 11 includes multiple convolutional layers, which are used to perform the same scale feature extraction process on the input left and right views respectively to obtain scale features. The scale feature extraction process is described below using the left view as an example.

[0045] In one implementation, the left view can be scaled according to a preset scale to obtain multiple left views at different scales. These are then input into the feature extractor 11 to perform a scale feature extraction process, resulting in features at different scales. This indicates that the scale features include multiple different scale features. The feature extractor 11 includes i convolutional layers, where the parameters of convolutional layers 1 to 1 can be set as follows: the kernel size is a*a; the number of channels is n1, n2, ..., ni respectively; and the stride is m.

[0046] In another implementation, the same feature extraction process can be performed multiple times on the left view using feature extractor 11. Each time the feature extraction process is executed, different stride combinations are set for convolutional layer i to output features at different scales. The parameters of convolutional layers 1 to i can be set as follows: kernel size is a*a; number of channels is n1, n2, ..., ni respectively; stride combination is Mj. Here, Mj is the stride combination set when performing the j-th feature extraction process.

[0047] In a preferred embodiment, such as Figure 2 As shown, the feature extractor 11 is configured to include three convolutional kernels (convolutional kernel 21, convolutional kernel 22, and convolutional kernel 23), and the feature extraction process is performed three times for the left view and the right view respectively. The specific parameters of the three convolutional layers are shown below.

[0048] First feature extraction process (e.g.) Figure 2 In (a): For parameters {kernel size, number of channels, stride}, the first convolutional layer 21 is set to {3*3, 64, 1}; the second convolutional layer 22 is set to {3*3, 128, 2}; the third convolutional layer 23 is set to {3*3, 256, 2}; and the output view is a 1 / 16 scale feature.

[0049] Second feature extraction process (e.g.) Figure 2 In (b)): For parameters {kernel size, number of channels, stride}, the first convolutional layer 21 is set to {3*3, 64, 1}; the second convolutional layer 22 is set to {3*3, 128, 1}; the third convolutional layer 23 is set to {3*3, 256, 2}; and the output view is a 1 / 4 scale feature.

[0050] The third feature extraction process (such as) Figure 2 In (c): For the parameters {kernel size, number of channels, stride}, the first convolutional layer 21 is set to {3*3, 64, 1}; the second convolutional layer 22 is set to {3*3, 128, 1}; the third convolutional layer 23 is set to {3*3, 256, 1}; the original scale features of the output view.

[0051] In this embodiment, the feature extractor 11 extracts scale features and outputs 1 / 16 scale features to LDCV module 12, 1 / 4 scale features to LDCV module 13, and the original scale features to LDCV module 14, according to their scale size. LDCV modules 12, 13, and 14 are used to calculate the cost of features at different scales. Among these, besides... Figure 1 The diagram shows a method where multiple LDCV modules are used to calculate costs for features at different scales. Alternatively, a single LDCV module can be used to calculate costs for features at different scales sequentially; this will not be elaborated upon here. Furthermore, the cost is used to measure the similarity between the left and right views.

[0052] Specifically, each LDCV module performs the same cost calculation process for each scale feature. Simultaneously, the output of the LDCV module for larger scale features is used as the input of the LDCV module for smaller scale features to assist in cost calculation. This process, from a larger search range to a smaller search range, achieves progressive pixel matching of the binocular training view, resulting in cost volumes at multiple scales. For example, the output of LDCV module 12 is used as the input of LDCV module 13 to assist in cost calculation; the output of LDCV module 13 is used as the input of LDCV module 14 to assist in cost calculation.

[0053] Multiple scale features include features with various resolutions. For example, the original scale feature has a resolution of 960*540, the 1 / 4 scale feature has a resolution of 240*140, and the 1 / 16 scale feature has a resolution of 60*35. Larger and smaller scale features are relative concepts within the multiple scale features. For instance, if the larger scale feature is the original scale feature, then the smaller scale features are the 1 / 4 scale feature and the 1 / 16 scale feature; if the larger scale feature is the 1 / 4 scale feature, then the smaller scale feature is the 1 / 16 scale feature, and so on. It's important to note that smaller scale features do not include the largest scale feature, and larger scale features do not include the smallest scale feature. The cost volume refers to the fusion of cost calculation results from multiple output channels of the same scale feature, specifically expressed as: W*H*D*F, where W*H is the resolution (scale), D is the cost calculation result, and F is the number of feature channels.

[0054] In one implementation, the cost calculation process performed by the LDCV module includes one-dimensional cost calculation. To overcome the large amount of computation required in existing one-dimensional cost calculations, which necessitates calculating the feature correlation between each pixel in the left view and each pixel in the same row of the right view, the LDCV module further incorporates a one-dimensional neighborhood constraint of 1*q in the one-dimensional cost calculation. Utilizing the known correspondence between pixels in the left and right views, the one-dimensional cost calculation is completed by calculating the feature correlation between each pixel in the left view and the corresponding pixels in the right view within the q pixels of the one-dimensional neighborhood of 1*q.

[0055] In one implementation, the cost calculation process performed by the LDCV module also includes two-dimensional cost calculation. To overcome the problem of large disparity estimation errors caused by the assumption that the one-dimensional cost calculation is based on perfect stereo correction of the left and right views, the LDCV module adds two-dimensional cost calculation, further incorporating a two-dimensional neighborhood constraint of q*p. Utilizing the known pixel correspondence between the left and right views, the feature correlation of each pixel in the left view with corresponding pixels in the right view within the q*p two-dimensional neighborhood is calculated, thus completing the two-dimensional cost calculation.

[0056] Specifically, the formulas for calculating one-dimensional cost and two-dimensional cost are as follows:

[0057]

[0058] Where C is the number of channels in the feature map, F1 is the feature map of the left view, F2 is the feature map of the right view, x`=x+f(d), y`=y+g(d), where d represents the dimension. When calculating the one-dimensional cost, d=1, let f(1)∈[r,-r], g(1)=0, such as when r=2, a 1*5 one-dimensional neighborhood is determined; when calculating the two-dimensional cost, d=2, let f(2)∈[r,-r], g(2)∈[r,-r], such as when r=2, a 5*5 two-dimensional neighborhood is determined.

[0059] Among them, the feature correlation calculation between pixels can calculate the similarity distance of feature values ​​between pixels, or the squared error cost function, etc.

[0060] In one implementation, the cost calculation process performed by the LDCV module includes the alternating calculation of the aforementioned one-dimensional and two-dimensional costs, while defining a computational range correlation between the one-dimensional and two-dimensional costs according to the alternating calculation order. One-dimensional cost calculation reduces the computational load, while two-dimensional cost calculation addresses the issue of imperfect stereo correction in the left and right views. Furthermore, the LDCV module defines the alternating cost calculation from large to small scales, enabling the calculation of pixel correlations within a larger search range with a smaller computational load.

[0061] In this embodiment, the scale features output by the feature extractor and the cost volume output by the LDCV module are input into the decoder to predict the disparity map between the left and right views. Figure 1 As shown, based on the scale size, the features of each scale are output to decoder 15, decoder 16, and decoder 17 respectively, and the cost volume generated by LDCV module 12 is output to decoder 15, the cost volume generated by LDCV module 12 is output to decoder 16, and the cost volume generated by LDCV module 14 is output to decoder 17.

[0062] Specifically, each decoder includes: a downsampling layer, a multi-layer residual network (Residual Network, or ResNet for short), an upsampling layer, and a fully connected layer. The number of decoders can be one or more. Features and costs at each scale are input to the decoder, and disparity maps are gradually output from large to small scales until the original disparity map of the output view is reached, serving as the final disparity estimation result. The residual network is preferably four layers with channels of 128, 128, 64, and 32 respectively.

[0063] Specifically, the disparity map is predicted by inputting the features and cost volume at the largest scale into the decoder. Then, the disparity map is input into the decoder along with the features and cost volume at the next scale to predict the disparity map at the next scale. This process is repeated until the disparity map at the smallest scale is obtained, which is then used as the final disparity estimation result.

[0064] For example, such as Figure 1 As shown, there are multiple decoders. Decoder 15 predicts the disparity based on the input features and cost volume. Figure 1 and parallax Figure 1 The output is sent to decoder 16; decoder 16 calculates the input features and cost volume, as well as the disparity. Figure 1 The parallax was predicted. Figure 2 and parallax Figure 2 Output value decoder 17; decoder 17 is based on the input features and cost volume, as well as disparity Figure 2 The parallax was predicted. Figure 3 and parallax Figure 3 This serves as the final disparity estimation result.

[0065] In this embodiment, the LDCVNet model, as a supervised learning model, calculates a loss function in the loss layer based on the gold standard disparity in the annotation file and the disparity estimation result output by the decoder. The accuracy of the disparity estimation is measured by the loss value. The loss value is then input into the optimizer to iteratively optimize the model. The optimizer is preferably the Adam (Adaptive Moment Estimation) optimizer.

[0066] II. LDCVNet Model Training

[0067] The structure of the LDCVNet model has been explained above; please refer to the following section. Figures 3-11 The execution flow of the disparity estimation method is explained below, specifically the training flow of the LDCVNet model:

[0068] 31. Obtain the training dataset and extract the scale features of each pair of binocular training views in the training dataset;

[0069] 32. Calculate the first dimension cost and the second dimension cost of the scale feature alternately, from largest to smallest, wherein the first dimension cost and the second dimension cost are related in terms of their calculation range according to the alternating calculation order.

[0070] 33. Generate a cost volume for the corresponding scale based on the results of alternating calculations, and perform model training according to the scale features and the cost volume to obtain a disparity estimation model for use in estimating disparity of real-time captured binocular views.

[0071] In this embodiment, during the LDCVNet model training phase, the acquired training dataset includes multiple pairs of binocular training views. Each pair of binocular training views includes a left view and a right view. The feature extractor extracts multiple different scale features from the left and right views respectively. These multiple different scale features, arranged from largest to smallest scale, represent semantic information of different richness and precision in their respective views using different receptive fields. On the one hand, they are used to estimate the disparity between the left and right views by referring to the visual field range from a larger size to a smaller size. On the other hand, they are used to search for pixel correlations between the left and right views from a larger search range to a smaller search range to generate the cost volume.

[0072] In this embodiment, the alternating calculation of the first and second dimension costs of scale features refers to: calculating the first and second dimension costs of larger scale features, and then returning to calculate the first and second dimension costs of the next smaller scale feature, achieving an alternating calculation such as "first dimension cost, second dimension cost, first dimension cost...". There is a correlation between the first and second dimension costs in terms of their calculation range. This correlation means that within the same scale feature, the first dimension cost can determine the calculation range of the second dimension cost; within different scale features, the second dimension cost of a larger scale feature can determine the calculation range of the first dimension cost of the next smaller scale feature, and so on.

[0073] Here, the first-dimensional cost and the second-dimensional cost refer to the costs between the binocular views searched from different dimensional ranges, respectively, measuring the similarity between the binocular training views from different dimensional ranges. For example, the first-dimensional cost can be one-dimensional and the second-dimensional cost can be two-dimensional; or the first-dimensional cost can be two-dimensional and the second-dimensional cost can be one-dimensional.

[0074] In this embodiment, during model training, the scale features and the largest scale features and cost volume in the cost volume are fused. Based on the fusion result, the maximum scale disparity estimation is performed on the binocular training view using a preset decoder. The disparity estimation results corresponding to the smaller scale features and cost volumes in the cost volume, as well as the larger scale features and cost volumes, are fused. Based on the fusion result, the smaller scale disparity estimation is performed on the binocular training view using the decoder until the minimum scale disparity estimation result is obtained, wherein the larger scale includes the maximum scale. Model optimization is performed based on the minimum scale disparity estimation result of the binocular training view until convergence, resulting in a disparity estimation model.

[0075] In another implementation, besides alternately calculating the first and second dimension costs according to scale from large to small, cost calculation can also be performed according to scale from small to large. In this embodiment, based on the scale ratio between the smaller and larger scale features, the number of pixels required in the smaller scale feature to map to a single pixel of the larger scale feature is determined; based on the number of pixels, multiple pixels of the smaller scale feature are sequentially mapped to a single pixel of the larger scale feature; based on the matching relationship between multiple pixels of the smaller scale feature in the left and right views, the correspondence between the mapped single pixel of the larger scale feature is determined; based on the correspondence, the calculation range of the larger scale feature is determined, and the calculation of the first dimension cost of the larger scale feature is performed. This process is repeated to achieve alternating calculation of the first and second dimension costs of scale features from small to large scale. Compared to alternating calculation from large to small scale, the alternating calculation process from small to large scale is the same except that the method of determining the calculation range between the first and second dimension costs among multiple scale features is different.

[0076] The following uses the left view as an example to illustrate the scale feature extraction process:

[0077] When generating a cost volume of the corresponding scale based on the results of alternating computation, in one implementation, the left view can be scaled according to a preset scale to obtain multiple left views of different scales. Then, feature extraction is performed sequentially on the multiple left views of different scales in the feature extractor to obtain feature maps of the same scale. Among the multiple left views of different scales, at least one left view of the original scale is included, which means that the extracted scale features include at least the features of the original scale of the left view.

[0078] In another implementation, the same feature extraction process is executed multiple times on the left view using a feature extractor, with different preset parameters called each time, so that the feature extractor extracts features at different scales each time. At least one of the multiple feature extraction processes includes a feature extraction process that obtains features at the original scale of the left view.

[0079] Furthermore, the training dataset used in this embodiment includes a large amount of synthetic data and a small amount of real data. A transfer learning strategy from synthetic data to real data is employed, using synthetic data for model pre-training and real data for model fine-tuning. The large amount of synthetic data includes, for example, 40,000-500,000 pairs of binocular training views, while the small amount of real data includes, for example, 25-35 pairs of binocular training views. Synthetic data refers to binocular views captured by a simulated binocular camera generated by software, while real data refers to binocular views captured using a binocular camera in real-world scenarios, specifically binocular views captured by a binocular endoscope after it has been extended into the body.

[0080] Specifically, the synthetic data can be from the Scene Flow datasets, including 35,454 training pairs and 4,370 test pairs, with an image size of 960*540. The real data can be from the StereoCorrespondence and Reconstruction of Endoscopic Data (SCARED) dataset, including 7 sets of abdominal endoscopy data, each set containing 5 pairs of binocular views. The reliability of the 7 sets of data can be evaluated, and 5 sets are selected for model fine-tuning. These 5 sets include 20 training pairs (1 set) and 5 test pairs (4 sets), with an image size of 1024*1280. Using synthetic data to perform model pre-training solves the problem of difficulty in annotating binocular views captured in real-world scenes.

[0081] In addition, if the feature extractor of the LDCVNet model pre-sets the input image size requirement to be 960*540, and the synthesized data meets this size requirement, then the input layer does not need to adjust the image size of the synthesized data. The image size of the real data is 1024*1280, so the input layer needs to randomly crop the image size of the real data to 960*540. Then, one or more of the following data augmentation processes are performed on the synthesized data and the real data: adding random noise, random contrast adjustment, random chroma adjustment, vertical image misalignment by 1-3 rows, and image normalization.

[0082] Specifically, the training dataset also includes labeled data for binocular training views, used to annotate the gold standard disparity between each pair of binocular training views. The disparity of the binocular training views is estimated using scale features and cost volumes, and the difference between the disparity estimation results and the gold standard disparity is compared to evaluate the accuracy of the model's disparity estimation, thus enabling model optimization. Finally, the optimized disparity estimation model is used to estimate the disparity of the real-time captured binocular views.

[0083] The following explains the alternating calculation process for the first-dimensional cost and the second-dimensional cost:

[0084] In one implementation, the step of alternately calculating the first dimension cost and the second dimension cost of the scale features from largest to smallest includes: calculating the first dimension cost corresponding to the largest scale feature among the scale features; determining a first calculation range based on the first dimension cost of the same scale features among the scale features, and calculating the second dimension cost of the same scale features based on the first calculation range, wherein the same scale features include the largest scale feature; determining a second calculation range based on the second dimension cost of a larger scale feature among the scale features, and calculating the first dimension cost of a smaller scale feature among the scale features based on the second calculation range, wherein the larger scale feature includes the largest scale feature.

[0085] For example, such as Figure 4 As shown in (a), after the 1 / 16 scale feature undergoes one-dimensional cost calculation 31, the calculation range of two-dimensional cost calculation 32 is determined based on the calculation result of one-dimensional cost calculation 31. Then, after two-dimensional cost calculation 32, a 1 / 16 scale cost volume is generated. Simultaneously, the calculation result of two-dimensional cost calculation 32 is output to the one-dimensional cost calculation 33 used for the 1 / 4 scale feature; as shown in (a), the 1 / 16 scale feature undergoes one-dimensional cost calculation 32. Figure 4 As shown in (b), based on the calculation results of the 1 / 4 scale features and the two-dimensional cost calculation 32, the one-dimensional cost calculation 33 and the two-dimensional cost calculation 34 are performed to generate the 1 / 4 scale cost volume. Simultaneously, the calculation result of the two-dimensional cost calculation 34 is output to the one-dimensional cost calculation 35 used for the original scale features; as shown in (b), Figure 4 As shown in (c), based on the original scale features and the calculation results of two-dimensional cost calculation 34, one-dimensional cost calculation 35 and two-dimensional cost calculation 36 are performed to generate the original scale cost volume.

[0086] In this embodiment, the alternating calculation order includes a first calculation order and a second calculation order. The first calculation order refers to the calculation order from the first dimension cost to the second dimension cost in features of the same scale. The second calculation order refers to the calculation order from the second dimension cost of a larger scale feature to the first dimension cost of a smaller scale feature. The first dimension cost refers to the matching pixel obtained by searching within the first dimension range for each pixel in one view (e.g., the left view) relative to another view (e.g., the right view) in the binocular training view. Similarly, the second dimension cost refers to the matching pixel obtained by searching within the second dimension range for each pixel in the left view (e.g., the right view). The calculation range (e.g., the first calculation range, the second calculation range) is specifically represented as the pixel correspondence determined by the matching pixels between the left and right views.

[0087] In one implementation, determining the second calculation range based on the second dimension cost of the larger scale feature in the scale features includes: determining each pair of pixels with the highest correlation in the binocular training view based on the second dimension cost of the larger scale feature in the scale features, wherein each pair of pixels includes a first pixel and a second pixel; determining a plurality of third pixels from the larger scale feature to the smaller scale feature in the scale features, and determining a plurality of fourth pixels from the larger scale feature to the smaller scale feature; determining the correspondence between each third pixel and each fourth pixel, and determining the second calculation range based on the correspondence.

[0088] In this embodiment, the first pixel and the second pixel are located in different views of the binocular training view, for example, the first pixel is in the left view and the second pixel is in the right view, or the first pixel is in the right view and the second pixel is in the left view. The third pixel and the fourth pixel are mapped to the first pixel and the second pixel, respectively, meaning that the third pixel and the fourth pixel are also located in different views of the binocular training view.

[0089] In this embodiment, based on the scale ratio between the larger and smaller scale features, the number of pixels of each pixel of the larger scale feature mapped to the smaller scale feature is determined; based on the number of pixels, a single first pixel is sequentially mapped to multiple third pixels, and a single second pixel is sequentially mapped to multiple fourth pixels; based on the matching first and second pixels, the correspondence between each mapped third pixel and each fourth pixel is determined.

[0090] Specifically, for example, if the scale ratio between a 1 / 16 scale feature and a 1 / 4 scale feature is 4:1, then a single first pixel of the 1 / 16 scale feature can be mapped to four third pixels of the 1 / 4 scale feature, and a single second pixel of the 1 / 16 scale feature can be mapped to four fourth pixels of the 1 / 4 scale feature; further, the correspondence between these four third pixels and four fourth pixels is determined.

[0091] In this embodiment, calculating the first dimension cost of the smaller scale feature in the scale feature based on the second calculation range includes: determining each pixel in the preset first dimension neighborhood of each fourth pixel based on the second calculation range; calculating the correlation between the third pixel and each pixel in the corresponding first dimension neighborhood according to the smaller scale feature; and determining the first dimension cost corresponding to the smaller scale feature based on the pixel with the highest correlation in the first dimension neighborhood.

[0092] In one implementation, the training dataset contains built-in correction parameters. Before being input into the feature extractor, each pair of stereo training views in the training dataset is first stereo-corrected. For example, this ensures that matching pixels in the left and right view pixel maps are in the same row number, and that the image display effect is identical, thus confirming that the stereo training views are perfectly stereo-corrected. Assuming perfect stereo correction makes calculating the cost of the stereo training views more convenient. For instance, since each pair of related pixels is in the same row in the pixel map of the stereo training views, only the correlation of each pixel in the same row between the left and right views needs to be calculated to obtain the cost between the stereo views.

[0093] In this embodiment, the cost calculation requires calculating the correlation of each pixel in the same row between the left and right views. To further reduce the computational cost, this embodiment also defines a first-dimensional neighborhood, such as a one-dimensional neighborhood 1*q. The fourth pixel is determined to have q pixels within the one-dimensional neighborhood 1*q. Then, using the correspondence between each third pixel and each fourth pixel, the correlation between a single third pixel and these q pixels is calculated. The pixel with the highest correlation among the q pixels is selected to complete the calculation of the first-dimensional cost. If the resolution of the left and right views is n*m, then the computational cost of the one-dimensional cost in this embodiment is only q / (n*m) of the original cost calculation.

[0094] Similarly, this cost also assumes that the binocular training views are perfectly stereo corrected, which can introduce significant errors in subsequent disparity estimation. Therefore, to reduce the disparity estimation error caused by this cost calculation, this embodiment adds a second-dimensional cost calculation, such as a two-dimensional cost calculation. For features of the same scale, not only is the feature correlation between pixels in the left view and pixels in the same row in the right view calculated, but the feature correlation between pixels in the left view and pixels in one or more adjacent rows in the right view is also calculated. A second-dimensional neighborhood, such as a two-dimensional neighborhood q*p, is also defined to reduce the computational cost of the second-dimensional cost. The q*p pixels in the right view within the two-dimensional neighborhood q*p are determined. Then, using the known pixel correspondence between the left and right views, the correlation between a single pixel in the left view and these q*p pixels is calculated, thus completing the calculation of the second-dimensional cost.

[0095] One-dimensional cost calculation reduces computational load, while two-dimensional cost calculation addresses the issue of imperfect stereo correction in the left and right views. By alternating between one-dimensional and two-dimensional cost calculations, and ensuring a correlation in computational range between them according to the alternating calculation order, cost volumes can be generated sequentially, enabling the calculation of pixel correlations over a larger search range with a smaller computational load.

[0096] By generating cost volumes through alternating calculations of one-dimensional and two-dimensional costs, pixel correlations over a larger search range can be calculated with less computation. For example, if the scale features relative to the original view scale range from large to small (1 / 16, 1 / 4, 1), and the search range is based on the 1 / 16 scale feature, then the cost volumes for the 1 / 4 and 1 scale features need to be searched. If the cost volume for each scale feature is calculated non-alternatingly, the computational cost required to calculate the correlation of a single pixel within each scale feature is:

[0097] 1 / 16 scale features: one-dimensional neighborhood is 1*5; two-dimensional neighborhood is 5*5; computational cost: 30.

[0098] 1 / 4 scale features: 1*20 for one-dimensional neighborhood; 20*20 for two-dimensional neighborhood; computational cost: 420.

[0099] 1. Scale features: One-dimensional neighborhood is 1*90; two-dimensional neighborhood is 90*90; computational cost is 8190.

[0100] If the cost volume of each scale feature is calculated in an alternating manner, since the output of the 1 / 16 scale feature can determine the correspondence between the pixels of the 1 / 4 scale feature, and the output of the 1 / 4 scale feature can determine the correspondence between the pixels of the 1 scale feature, the search range under the 1 / 4 scale feature is associated with the search range under the 1 / 16 scale feature, and the search range under the original scale feature is associated with the search range under the 1 / 4 scale feature. Therefore, the one-dimensional neighborhood of each scale feature can be set to 1*5; the two-dimensional neighborhood can be set to 5*5; the final computation amount is 30. It is also possible to search the cost volume of the 1 / 4 scale feature and the 1 scale feature using the search range under the 1 / 16 scale feature.

[0101] Preferably, such as Figure 5 As shown, the first-dimensional neighborhood is set to 1*5, and the second-dimensional neighborhood is set to 5*5. The first-dimensional cost is the one-dimensional cost, and the second-dimensional cost is the two-dimensional cost. The alternating calculation process of the one-dimensional cost and the two-dimensional cost is shown below:

[0102] 1. For features at the 1 / 16 scale, the calculation of the one-dimensional cost is as follows: Figure 5 As shown in (a), it is known that there is a correspondence between a pixel q3 in the left view and a pixel p3 in the right view. The five pixels {p1, p2, p3, p4, p5} in the 1*5 neighborhood of pixel p3 are determined, and the correlation between pixel q3 and the five pixels {p1, p2, p3, p4, p5} is calculated. The pixel p4 with the highest correlation is taken as the result of the calculation of the one-dimensional cost of pixel p3.

[0103] 2. For features of the same 1 / 16 scale, the calculation of the two-dimensional cost is as follows: Figure 5 As shown in (b), based on the calculation result of the one-dimensional cost, it can be seen that pixel q3 and pixel p4 have a corresponding relationship. 25 pixels {p12, p13, p14, p15, p16; p22, p23, p24, p25, p26; p2, p3, p4, p5, p6; p42, p43, p44, p45, p46; p52, p53, p54, p55, p56} in the 5*5 neighborhood of pixel p4 are determined. The correlation between pixel q3 and these 25 pixels is calculated, and the pixel p44 with the highest correlation is taken as the calculation result of the two-dimensional cost of pixel p3.

[0104] 3. For the 1 / 4 scale features of the next scale, the calculation of the one-dimensional cost is as follows: Figure 5 As shown in (c) and (d):

[0105] 3.1 In Figure 5 In (c), based on the two-dimensional cost calculation results of the 1 / 16 scale feature, it can be seen that pixel q3 and pixel p44 have a corresponding relationship. In the left view, pixel q3 under the 1 / 16 scale feature is mapped to four pixels {q31, q32, q33, q34} under the 1 / 4 scale feature, and in the right view, pixel p44 under the 1 / 16 scale feature is mapped to four pixels {p441, p442, p443, p444} under the 1 / 4 scale feature. Based on the relative positional relationship of the four pixels in the left and right views, the correspondence between pixel q31 and pixel p441, pixel q32 and pixel p442, pixel q33 and pixel p443, and pixel q34 and pixel p444 is established.

[0106] 3.2 In Figure 5 In (d), taking pixel q31 and pixel p441 as examples, the one-dimensional cost of the 1 / 4 scale feature is calculated. Five pixels {p445, p446, p441, p442, p447} within the 1*5 neighborhood of pixel p441 are determined, and the correlation between pixel q31 and these five pixels is calculated. The pixel p446 with the highest correlation is then taken as the result of calculating the one-dimensional cost of pixel p31.

[0107] Similarly, the one-dimensional cost and two-dimensional cost of the scale feature are calculated.

[0108] In one implementation, the first dimensional neighborhood includes a first directional neighborhood and a second directional neighborhood that are perpendicular to each other in the first dimension. The step of determining each pixel point of the fourth pixel point in the preset first dimensional neighborhood based on the second calculation range includes: determining each pixel point of the fourth pixel point in the first directional neighborhood and / or the second directional neighborhood based on the second calculation range.

[0109] In this embodiment, considering that the left and right views are imperfect stereo corrections, the first-dimensional neighborhood includes two one-dimensional first-direction neighborhoods and a second-direction neighborhood, such as the first-direction neighborhood 1*q and the second-direction neighborhood q*1, to initially filter relevant pixels in one or more rows above or below the fourth pixel. Figure 6 As shown, based on the second calculation range, it can be determined that the third pixel q3 in the left view and the fourth pixel p3 in the right view have a corresponding relationship. Furthermore, the pixels {p1, p2, p3, p4, p5} in the first direction neighborhood of the fourth pixel p3 and the pixels {p6, p7, p3, p8, p9} in the second direction neighborhood are determined. The result of the one-dimensional cost can be the pixel p7 in the row above the fourth pixel p3.

[0110] In one embodiment, determining the correspondence between each third pixel and each fourth pixel includes: determining the mapping position of each third pixel relative to the first pixel and the mapping position of each fourth pixel relative to the second pixel, and establishing a correspondence between third pixels and fourth pixels with the same mapping position; or, calculating the average correlation of pixels in a preset second-dimensional neighborhood between each third pixel and each fourth pixel, and establishing a correspondence between each third pixel and each fourth pixel based on the magnitude of the average correlation.

[0111] In this embodiment, from high-scale features to low-scale features, the first pixel is mapped to multiple third pixels, and the second pixel is mapped to multiple fourth pixels. Since the correlation between the first and second pixels is the strongest, the pixels corresponding to each third pixel should be determined from the multiple fourth pixels. When establishing the correspondence between each fourth pixel and each third pixel, any of the following determination methods can be used:

[0112] A. When the first pixel is mapped to the third pixel, and when the second pixel is mapped to the fourth pixel, a mapping matrix is ​​formed. The row and column numbers of the mapping matrix where the third pixel is located are determined, as are the row and column numbers of the mapping matrix where the fourth pixel is located. Third and fourth pixels with the same row and column number are identified as being in the same mapping position, and a mapping relationship is established between third and fourth pixels at the same mapping position. For example... Figure 7 As shown, the third pixels q31, q32, q33, and q34 are located at row and column positions 11, 12, 21, and 22 of the mapping matrix of the first pixel q3, respectively; the fourth pixels p441, p442, p443, and p444 are located at row and column positions 11, 12, 21, and 22 of the mapping matrix of the second pixel p44, respectively. Therefore, the correspondence between the third pixel q31 and the fourth pixel p441, the third pixel q32 and the fourth pixel p442, the third pixel q33 and the fourth pixel p443, and the third pixel q34 and the fourth pixel p444 is established.

[0113] B. Pre-determine that each third and fourth pixel's second-dimensional neighborhood does not contain other third and fourth pixels; then calculate the average correlation between each third pixel's pixels within its second-dimensional neighborhood and each fourth pixel's pixels within its second-dimensional neighborhood. This assesses the correlation between each third and fourth pixel from a larger receptive field, while avoiding cross-influence and improving the accuracy of correlation assessment. Figure 8 As shown, the second-dimensional neighborhood is preset to 2*2. The pixel matrix 81, 82, 83, and 84 of the third pixel points q31, q32, q33, and q34 in the 2*2 neighborhood are determined, and the pixel matrix 85, 86, 87, and 88 of the fourth pixel points p441, p442, p443, and p444 in the 2*2 neighborhood are determined. The average correlation between pixel matrix 81, 82, 83, and 84 and pixel matrix 85, 86, 87, and 88 are cross-calculated. The third pixel points and fourth pixel points are selected in descending order of the average correlation value to establish a correspondence, until all correspondences between the third pixel points and fourth pixel points are established. If there is a third pixel point or fourth pixel point among the obtained third pixel points and fourth pixel points that already have a correspondence, then no correspondence is established between them. The mean correlation value refers to the average value of the correlation values ​​obtained by calculating the correlation between the pixels in the two pixel matrices.

[0114] C. Only cross-calculate the correlation between each third pixel and each fourth pixel to reduce the computational cost of correlation evaluation. That is, Figure 8As shown, only the correlation between the third pixel points q31, q32, q33, and q34 and the fourth pixel points p441, p442, p443, and p444 is calculated. The third pixel points and fourth pixel points are selected in descending order of correlation to establish a correspondence until all correspondences between the third pixel points and fourth pixel points are established. If there is a third pixel point or fourth pixel point among the obtained third pixel points and fourth pixel points that already have a correspondence, then no correspondence is established between the two.

[0115] In one embodiment, after determining the correspondence between each third pixel and each fourth pixel, the method further includes: randomly generating an offset vector for at least one fourth pixel, and updating the fourth pixel with the corresponding third pixel to a pixel offset based on the offset vector.

[0116] In this embodiment, at least one fourth pixel that needs to be offset is randomly determined, and the corresponding adjacent pixels are determined, wherein the adjacent pixels do not include other fourth pixels; the offset vector between the fourth pixel and each adjacent pixel is determined, and one of the target offset vectors is randomly selected from each offset vector; the fourth pixel is offset based on the target offset vector, and its position is swapped with the offset pixel; the swapped pixel is then associated with the third pixel.

[0117] like Figure 9 As shown, for example, the third pixels q31, q32, q33, and q34 correspond to the fourth pixels p441, p442, p443, and p444, respectively; the fourth pixels p441, p442, p443, and p444 determine the adjacent pixels (901, 902, 903), (904, 905, 906), (907, 908, 909), and (910, 911, 912), respectively; the first offset vector from the fourth pixel p441 to the adjacent pixel 902, the second offset vector from the fourth pixel p442 to the adjacent pixel 906, and the second offset vector from the fourth pixel p443 to the adjacent pixel 902 are randomly selected. The third offset vector of 909, and the fourth offset vector from the fourth pixel p444 to the adjacent pixel 912; based on the first offset vector, the fourth pixel p441 is swapped with the adjacent pixel 902; based on the second offset vector, the fourth pixel p442 is swapped with the adjacent pixel 906; based on the third offset vector, the fourth pixel p443 is swapped with the adjacent pixel 909; based on the fourth offset vector, the fourth pixel p444 is swapped with the adjacent pixel 912; establish the correspondence between the third pixels q31, q32, q33, and q34 and the adjacent pixels 902, 906, 909, and 912, respectively.

[0118] Furthermore, when offsetting pixels, the fourth pixel that needs to be offset can be determined randomly. Alternatively, a preset number of fourth pixels that need to be offset can be determined based on the correlation between each fourth pixel and the third pixel, from smallest to largest.

[0119] In addition, when calculating the first-dimensional cost, such as the one-dimensional cost, the pixels within the one-dimensional neighborhood can also be adjusted by offsetting. For example... Figure 10 As shown, pixel q3 and pixel p4 have a corresponding relationship. Then, the pixels within a 1*5 neighborhood of pixel p4 are determined. The adjacent pixels (101, 102), (103, 104), (105, 106), and (107, 108) in the next and following rows of pixels p2, p3, p5, and p6 within the 1*5 neighborhood are determined. Pixels p2 and p5 that need to be offset are randomly determined, and offsets of 110 from p2 to p1 and 120 from p5 to p6 are randomly selected. Pixel p2 is moved to the position of pixel p101 according to offset 110, and pixel p5 is moved to the position of pixel p106 according to offset 120, resulting in new pixels within the 1*5 neighborhood of p4. It should be noted that pixel p4, which has a corresponding relationship with pixel p3, cannot be used as the object of offset.

[0120] In addition, when calculating the second-dimensional cost, such as the two-dimensional cost, the pixels within the two-dimensional neighborhood can also be adjusted by offsetting. For example... Figure 11 As shown, pixel q3 and pixel p4 have a corresponding relationship. Then, the pixels within a 5x5 neighborhood of pixel p4 are determined. Pixels p12, p16, p6, and p55 within the 5x5 neighborhood that need to be offset are randomly selected. Offsets 111 (p12 to p22), 112 (p16 to p15), 113 (p6 to p25), and 114 (p55 to p46) are randomly selected. Based on offset 111, pixels p12 and p22 are swapped; based on offset 112, pixels p16 and p15 are swapped; based on offset 113, pixels p6 and p25 are swapped; and based on offset 114, pixels p55 and p46 are swapped, resulting in new pixels within the 5x5 neighborhood of p4. It should be noted that pixel p4, which has a corresponding relationship with pixel p3, cannot be used as the target for offsetting.

[0121] The training process of the model will be explained next based on scale features and cost volume:

[0122] In this embodiment, the features and cost volumes of the left and right views at the maximum scale are fused and input into the decoder to estimate the disparity between the left and right views, thus obtaining the disparity estimation result. The disparity estimation result at the maximum scale is then fused with the features and cost volumes at the next smaller scale, and the decoder is used to further estimate the disparity at the next smaller scale. This process is repeated until the disparity estimation result at the minimum scale is obtained, which is then used as the output of the decoder.

[0123] Specifically, when fusing the disparity estimation result at the largest scale with the features and cost volume at the next smaller scale, the disparity estimation result at the largest scale needs to be upsampled to the next smaller scale before being fused with the features and cost volume at the next smaller scale. The fusion of features and cost volumes can be processed using one of the following methods: weighting, combination, or stacking.

[0124] In this embodiment, when performing model optimization based on the disparity estimation results of the binocular training view at the minimum scale, the following loss function is applied to evaluate the accuracy of the disparity estimation results:

[0125]

[0126] Where V(p) is the mask for the gold standard disparity of p pixels on the binocular training view, S() is the smoothing L1 loss, and D pred (p), with D gt (p) represents the disparity estimation result and the gold standard disparity for p pixels, respectively.

[0127] Then, during model training, Adam (Adaptive Moment Estimation) can be used as the optimizer, with an initial learning rate of 10 during the pre-training phase. -4 Every 100 epochs, the learning rate is adjusted to half of the previous learning rate. The learning rate for the fine-tuning phase is set to 10%. -6 The pre-training cycle consists of 300 epochs, and the fine-tuning cycle consists of 20 epochs. After each cycle, the loss gradient is observed to determine whether the model has converged.

[0128] In one implementation, the minimum scale is the same as the scale of the binocular training view. This ensures that the disparity estimation result of the minimum-scale binocular training view is generated based on the global field of view of the view.

[0129] In addition, the trained disparity estimation model needs to be tested using test data to perform disparity estimation and model testing. The test data consists of three parts: the first part is 4370 test data pairs from the Scene Flow datasets; the second part is 5 test data pairs from the SCARED dataset, both of which include gold standard disparity; the third part is a private dataset, with the view acquisition device being a binocular endoscope on which the disparity estimation model will be applied, and this part of the data does not include gold standard disparity.

[0130] Specifically, the test data for the first and second parts can be evaluated using endpoint disparity error (EPE), which is calculated as follows:

[0131]

[0132] Where N is the number of pixels, D pre (p) and D gt (p) represents the disparity estimation result and the gold standard disparity for p pixels, respectively.

[0133] Specifically, for the test data in the third part, since there is no gold standard for this part of the test data, we use a subjective evaluation method to assess its results.

[0134] III. Application of LDCVNet Model

[0135] The training process for the LDCVNet model has been explained above. Please refer to the following section. Figure 12 and Figure 13 The application of the LDCVNet model is explained below:

[0136] A disparity estimation system can be built based on the LDCVNet model. This system includes a binocular endoscope and a processor, which are communicatively connected. The processor deploys the disparity estimation model and is configured to: receive real-time binocular views captured by the endoscope and input these views into the disparity estimation model; extract scale features from the binocular views and, according to scale from high to low, alternately calculate the first and second dimension costs of these scale features, where the first and second dimension costs are correlated in their computational range according to the alternating calculation order; generate a cost volume for the corresponding scale based on the alternating calculation results; and, based on the scale features and the cost volume, perform disparity estimation on the binocular views, obtaining the estimation result and outputting it from the disparity estimation model. In this embodiment, compared to LDCVNet model training, a loss layer is not executed; the execution process is essentially the same as LDCVNet model training. That is, the real-time binocular views captured by the endoscope are directly input into the trained LDCVNet model, and the disparity estimation result is output.

[0137] In one implementation, such as Figure 12 As shown, the LDCVNet model is applied to a robotic system, such as a surgical robot system, which includes a binocular endoscope 121, a slave control device 122, a master control device 123, and an imaging cart 124. The LDCVNet model can be deployed on the processor of any one of the slave control device 122, the master control device 123, and the imaging cart 124 to perform disparity estimation on the binocular view captured by the binocular endoscope 121.

[0138] Specifically, the slave control device 122 is detachably connected to the binocular endoscope 121 and is used to clamp and brake the movement of the binocular endoscope 121; the master control device 123 is used to communicate with the slave control device 122 and to control the slave control device 122 to perform various surgical operations, thereby indirectly controlling the movement of the binocular endoscope 121; the image trolley 124 is communicated with the binocular endoscope 121 via the endoscope data cable and is used to display the images captured in real time by the binocular endoscope 121, and is also communicated with the master control device 123 via the master image communication bus to forward the images captured in real time by the binocular endoscope 121 to the display interface built into the master control device 123 for display.

[0139] For example, the doctor can send control signals to the slave control device 122 by operating the main control device 123; the slave control device 122 performs corresponding surgical operations according to the received control signals, such as braking the 6-DOF movement of the binocular endoscope, adjusting the shooting range and field of view of the binocular endoscope 121, and capturing binocular views in real time and transmitting them to the imaging carriage 124 and the main control device 123; the imaging carriage 124 or the main control device 123 performs disparity estimation on the binocular views according to the deployed LDCVNet model, and then performs three-dimensional reconstruction of the binocular views based on the disparity estimation results, and then generates a display view based on the three-dimensional reconstruction results, and displays it in real time on the imaging carriage 124 or the main control device 123.

[0140] In a preferred embodiment, such as Figure 13 As shown, a CT scan is performed on the patient before surgery to obtain CT images. The target tissue area is then segmented from the images and modeled. The modeling results are then sampled to construct a CT point cloud model, which is stored for later use. During the surgery, a binocular endoscope is inserted into the patient's body to capture real-time binocular views of at least a portion of the target tissue area. On one hand, stereoscopic correction is performed on the binocular views, and the results are input into the LDCVNet model for disparity estimation. The disparity estimation results are then used to predict the depth information of the binocular views. On the other hand, image segmentation is performed on the binocular views to obtain the target tissue area. Based on the depth information and the target tissue area, 3D reconstruction is performed to construct a point cloud model for binocular endoscopic imaging. Finally, the CT point cloud model and the endoscope point cloud model are combined to perform point cloud registration. Based on the registration results, a display view of the target tissue area is output and displayed on the interface for the doctor to observe the real-time in vivo images captured by the binocular endoscope.

[0141] It should be noted that other sorting schemes that can be easily conceived by those skilled in the art within the technical scope disclosed in this invention should also be within the protection scope of this invention, and will not be elaborated here.

[0142] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional modules is merely an example. In practical applications, the above functions can be assigned to different functional units or modules as needed, that is, the internal structure of the mobile terminal can be divided into different functional units or modules to complete all or part of the functions described above. The functional modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the modules in the mobile terminal can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0143] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps described in the above-described method embodiments.

[0144] This application provides a computer program product that, when run on a mobile terminal, enables the mobile terminal to implement the steps described in the above-described method embodiments.

[0145] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0146] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0147] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determination" or "if the described condition or event is detected" may be interpreted, depending on the context, as "once determination," "in response to determination," "once the described condition or event is detected," or "in response to the detection of the described condition or event."

[0148] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0149] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0150] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this invention. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0151] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0152] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0153] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0154] If an integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals.

[0155] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A disparity estimation method, characterized in that, The disparity estimation method includes: Obtain the training dataset and extract the scale features of each pair of binocular training views in the training dataset; The first and second dimension costs of the scale features are calculated alternately, from largest to smallest. Based on the results of alternating computation, a cost volume corresponding to the scale is generated, and model training is performed according to the scale features and the cost volume to obtain a disparity estimation model, which is used to estimate the disparity of real-time captured binocular views.

2. The disparity estimation method according to claim 1, characterized in that, The step of alternately calculating the first-dimensional cost and the second-dimensional cost of the scale feature according to the scale from large to small includes: Calculate the first dimension cost corresponding to the largest scale feature among the scale features; Based on the first dimension cost of the same scale features in the scale features, a first calculation range is determined, and a second dimension cost of the same scale features is calculated based on the first calculation range, wherein the same scale features include the largest scale feature; A second calculation range is determined based on the second dimension cost of the larger scale feature among the scale features, and the first dimension cost of the smaller scale feature among the scale features is calculated based on the second calculation range, wherein the larger scale feature includes the largest scale feature.

3. The disparity estimation method according to claim 2, characterized in that, The step of determining the second calculation range based on the second dimension cost of the larger scale feature in the scale features includes: Based on the second dimension cost of the larger scale feature in the scale features, determine each pair of pixels with the highest correlation in the binocular training view, wherein each pair of pixels includes a first pixel and a second pixel. Determine multiple third pixels from the larger scale feature to the smaller scale feature in the scale feature, and determine multiple fourth pixels from the larger scale feature to the smaller scale feature; The correspondence between each of the third pixels and each of the fourth pixels is determined, and the second calculation range is determined based on the correspondence.

4. The disparity estimation method according to claim 3, characterized in that, The cost of calculating the first dimension of the smaller scale feature in the scale feature based on the second calculation range includes: Based on the second calculation range, each pixel in the preset first dimension neighborhood of each fourth pixel is determined; Based on the smaller scale features, calculate the correlation between the third pixel and each pixel in the corresponding first-dimensional neighborhood. The first-dimensional cost corresponding to the smaller-scale feature is determined based on the pixel with the highest correlation in the first-dimensional neighborhood.

5. The disparity estimation method according to claim 4, characterized in that, The first dimensional neighborhood includes a first directional neighborhood and a second directional neighborhood that are perpendicular to each other in the first dimension. The step of determining each pixel in the preset first dimensional neighborhood of each fourth pixel based on the second calculation range includes: Based on the second calculation range, each of the fourth pixel points is determined in the first directional neighborhood and / or the second directional neighborhood.

6. The disparity estimation method according to claim 3, characterized in that, Determining the correspondence between each third pixel and each fourth pixel includes: Determine the mapping position of each third pixel relative to the first pixel, and determine the mapping position of each fourth pixel relative to the second pixel, and establish a correspondence between third pixels and fourth pixels with the same mapping position; or, Calculate the average correlation between each third pixel and each fourth pixel in a preset second-dimensional neighborhood, and establish a correspondence between each third pixel and each fourth pixel based on the magnitude of the average correlation.

7. The disparity estimation method according to claim 3, characterized in that, After determining the correspondence between each of the third pixels and each of the fourth pixels, the method further includes: Randomly generate an offset vector for at least one of the fourth pixels, and update the corresponding fourth pixels of the third pixel to pixels offset based on the offset vector.

8. The disparity estimation method according to any one of claims 1-7, characterized in that, The step of training the model based on the scale features and the cost volume to obtain the disparity estimation model includes: The scale features and the largest scale features and cost volume are fused together, and based on the fusion result, the disparity estimation of the largest scale is performed on the binocular training view through a preset decoder. The disparity estimation results corresponding to the smaller scale features and cost volume, as well as the larger scale features and cost volume, are fused together. Based on the fusion result, the smaller scale disparity estimation is performed on the binocular training view through the decoder until the minimum scale disparity estimation result is obtained. The larger scale includes the maximum scale. Based on the disparity estimation results of the binocular training view at the minimum scale, model optimization is performed until convergence, resulting in a disparity estimation model.

9. The disparity estimation method according to claim 8, characterized in that, The minimum scale is the same as the scale of the binocular training view.

10. A disparity estimation system, characterized in that, The disparity estimation system includes a binocular endoscope and a processor, the binocular endoscope and the processor being communicatively connected, the processor deploying a disparity estimation model, and the processor being configured to: Receive the binocular view captured in real time by the binocular endoscope and input the binocular view into the disparity estimation model; The scale features of the binocular view are extracted, and the first dimension cost and the second dimension cost of the scale features are calculated alternately from high to low scale. The first dimension cost and the second dimension cost have a correlation relationship in terms of calculation range according to the alternating calculation order. Based on the results of alternating computation, a cost volume for the corresponding scale is generated. Then, based on the scale features and the cost volume, disparity estimation is performed on the binocular view to obtain the estimation result, which is output from the disparity estimation model.

11. A readable storage medium, characterized in that, The readable storage medium stores a computer program that, when executed by a processor, implements the steps of the disparity estimation method as described in claims 1-9.

Images