Three-dimensional model building method, endoscope and storage medium

Abstract

The present invention discloses a three-dimensional model building method, an endoscope and a storage medium. The three-dimensional model building method comprises: obtaining a plurality of original structure images obtained by projecting different structure light patterns onto the area to be measured, and an original image obtained by not projecting onto the area to be measured; calculating a precision disparity map using a preset stereo matching algorithm based on each of the original structure images; obtaining a three-dimensional scene model of the area to be measured using a first preset rule based on the precision disparity map and the original image, thereby accelerating the stereo matching speed and achieving a real-time effect while ensuring the three-dimensional stereo accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of medical devices, and in particular to a three-dimensional model building method, an endoscope and a storage medium. Background Art

[0002] To achieve the three-dimensional effect of minimally invasive abdominal surgery assisted by endoscope, the doctor is generally required to wear 3D glasses and watch the 3D display screen while performing the surgery; or, a binocular endoscope can capture two abdominal images, which are displayed on the left and right monitors respectively. The contents of the left and right monitors enter the doctor's left and right eyes respectively, thus achieving a three-dimensional effect.

[0003] The traditional binocular stereo matching algorithm needs to match every feature point in the left and right camera views. The matching feature calculation time is long and cannot achieve real-time effect during the operation. In addition, in the process of finding feature points by the binocular stereo matching algorithm, rich texture images are required, but the tissues and organs in the abdominal cavity do not have strong texture information, and the three-dimensional stereo accuracy is not high. Summary of the invention

[0004] Based on this, it is necessary to provide a three-dimensional model building method, an endoscope and a storage medium to address the problems in the above-mentioned background technology, propose a new preset stereo matching algorithm, establish a three-dimensional scene model of the area to be tested illuminated by the endoscope, and speed up the stereo matching speed while ensuring the three-dimensional stereo accuracy to achieve real-time effect.

[0005] In order to solve the above technical problems, the first aspect of the present application proposes a three-dimensional model establishment method for establishing a three-dimensional scene model of a region to be measured, the method comprising:

[0006] Acquire a plurality of original structured images obtained by projecting different structured light patterns onto the area to be measured, and an original image obtained by not projecting different structured light patterns onto the area to be measured;

[0007] A precision disparity map is calculated based on each of the original structure images using a preset stereo matching algorithm;

[0008] A three-dimensional scene model of the area to be measured is acquired according to the precision disparity map and the original image using a first preset rule.

[0009] In the three-dimensional model building method provided in the above embodiment, multiple original structural images obtained by projecting different structured light patterns are obtained at the same time to increase the surface texture of the area to be measured, and the original image obtained without projection; wherein the original structural images corresponding to different structured light patterns are different; a precision disparity map is calculated based on each original structural image using a preset stereo matching algorithm, and a three-dimensional scene model is obtained by using a first preset rule for the precision disparity map and the original image, thereby ensuring the three-dimensional stereo accuracy and accelerating the stereo matching speed to achieve a real-time effect.

[0010] In one embodiment, the step of calculating the precision disparity map using a preset stereo matching algorithm according to each of the original structure images includes:

[0011] Performing dedistortion processing and stereo correction processing on each of the original structural images in sequence to obtain a plurality of structural images;

[0012] The precision disparity map is calculated based on each of the structural images using the preset stereo matching algorithm.

[0013] In one of the embodiments, the structural image includes a first structural image and a second structural image in the same plane, and a feature point where a preset point is projected on the first structural image and a matching point where the preset point is projected on the second structural image are in the same row.

[0014] In one embodiment, the preset stereo matching algorithm includes a first preset algorithm and a second preset algorithm; the step of calculating the precision disparity map using the preset stereo matching algorithm according to each of the structural images includes:

[0015] According to the first structural image and the second structural image, the first preset algorithm is used to calculate a first matching disparity map;

[0016] The precision disparity map is obtained by calculating the second preset algorithm according to the first structural image, the second structural image and the first matching disparity map.

[0017] In one embodiment, the step of calculating the first matching disparity map using a first preset algorithm according to the first structural image and the second structural image includes:

[0018] According to the first structural image and the second structural image, the first matching algorithm is used to calculate a low-resolution disparity map;

[0019] A first optimization process is performed on the low-resolution disparity map to obtain the first matching disparity map.

[0020] In one embodiment, the step of calculating the low-resolution disparity map using the first matching algorithm according to the first structural image and the second structural image includes:

[0021] Acquire a plurality of first matching grid points on the first structural image, wherein the first matching grid points are spaced apart by a first preset number of pixels;

[0022] Acquire a preset first matching grid point on the first structural image, and an original disparity search range of the preset first matching grid point on the second structural image;

[0023] Calculating a matching cost value for each pixel point within the original disparity search range according to a preset energy function;

[0024] Comparing the matching cost value of each pixel point within the original disparity search range, determining the minimum matching cost value within the original disparity search range, and determining the pixel point corresponding to the minimum matching cost value within the original disparity search range as a preset first matching pixel point; the preset first matching pixel point and the preset first matching grid point are in the same row;

[0025] According to the comparison result of the minimum matching cost value of the preset first matching pixel point and the preset first matching cost threshold, the mapped first matching pixel point in the first structure image is determined.

[0026] In one embodiment, determining the mapped first matching pixel point in the first structure image includes:

[0027] Determine whether the minimum matching cost value of the preset first matching pixel point is less than a preset first matching cost threshold;

[0028] If not, determining the preset first matching grid point as a parallax hole;

[0029] If so, obtain the first disparity search range of the preset first matching pixel point on the first structural image, and calculate the matching cost value of each pixel point in the first disparity search range according to the preset energy function, compare the matching cost value of each pixel point in the first disparity search range, determine the minimum matching cost value in the first disparity search range, and determine the pixel point corresponding to the minimum matching cost value in the first disparity search range as the mapped first matching pixel point; the mapped first matching pixel point and the preset first matching pixel point are in the same row.

[0030] In one of the embodiments, after determining the mapped first matching pixel point in the first structure image, the method further includes:

[0031] The low-resolution disparity map is acquired according to a comparison result between a distance between the preset first matching grid point and the mapped first matching pixel point and a preset first matching distance.

[0032] In one of the embodiments, obtaining the low-resolution disparity map includes:

[0033] Determine whether the distance between the preset first matching grid point and the mapped first matching pixel point is less than a preset first matching distance;

[0034] If it is greater than or equal to the preset first matching distance, determining the preset first matching grid point as a parallax hole;

[0035] If it is less than the preset first matching spacing, the disparity value of the preset first matching grid point is calculated according to the horizontal coordinate of the preset first matching grid point and the horizontal coordinate of the preset first matching pixel point, and the first matching grid point on the first structural image is re-acquired, and the disparity value of each of the first matching grid points on the first structural image is calculated to obtain the low-resolution disparity map.

[0036] In one embodiment, performing a first optimization process on the low-resolution disparity map to obtain the first matching disparity map includes:

[0037] Remove discrete grid points;

[0038] The disparity holes and the grid point positions where the discrete grid points are located are filled and sampled according to a bilinear interpolation method to obtain the first matching disparity map.

[0039] In one embodiment, the step of calculating the precision disparity map using a second preset algorithm according to the first structural image, the second structural image and the first matching disparity map includes:

[0040] According to the first structural image, the second structural image and the first matching disparity map, the second matching algorithm is used to calculate a second matching disparity map;

[0041] The second matching disparity map is optimized for the second time to obtain the precision disparity map.

[0042] In one embodiment, the step of calculating the second matching disparity map using the second matching algorithm according to the first structural image, the second structural image and the first matching disparity map includes:

[0043] Acquire a plurality of second matching grid points on the first structural image, wherein the second matching grid points are spaced apart by a second preset number of pixels;

[0044] Acquire a preset second matching grid point on the first structural image, and acquire a second disparity search range of the preset second matching grid point on the second structural image according to the first matching disparity map;

[0045] Calculating a matching cost value for each pixel point within the second disparity search range according to a preset energy function;

[0046] Comparing the matching cost value of each pixel point within the second disparity search range, determining the minimum matching cost value within the second disparity search range, and determining the pixel point corresponding to the minimum matching cost within the second disparity search range as a preset second matching pixel point; the preset second matching pixel point and the preset second matching grid point are in the same row;

[0047] According to the comparison result of the minimum matching cost value of the preset second matching pixel point and the preset second matching cost threshold, the mapped second matching pixel point in the first structure image is determined.

[0048] In one embodiment, determining the mapped second matching pixel point in the first structure image according to the comparison result of the minimum matching cost value of the preset second matching pixel point and the preset second matching cost threshold includes:

[0049] Determine whether the minimum matching cost value of the preset second matching pixel point is less than a preset second matching cost threshold;

[0050] If not, determining the preset second matching grid point as a parallax hole;

[0051] If so, obtain the third disparity search range of the preset second matching pixel point on the first structural image, and calculate the matching cost value of each pixel point in the third disparity search range according to the preset energy function, compare the matching cost value of each pixel point in the third disparity search range, determine the minimum matching cost value in the third disparity search range, and determine the pixel point corresponding to the minimum matching cost value in the third disparity search range as the mapped second matching pixel point; the mapped second matching pixel point and the preset second matching pixel point are in the same row.

[0052] In one of the embodiments, after determining the mapped second matching pixel point in the first structure image, the method further includes:

[0053] The second matching disparity map is acquired according to a comparison result between the distance between the preset second matching grid point and the mapped second matching pixel point and a preset second matching distance.

[0054] In one of the embodiments, obtaining the second matching disparity map according to a comparison result between the distance between the preset second matching grid point and the mapped second matching pixel point and a preset second matching distance includes:

[0055] Determine whether the distance between the preset second matching grid point and the mapped second matching pixel point is less than a preset second matching distance;

[0056] If it is greater than or equal to the preset second matching distance, determining the preset grid point as a parallax hole;

[0057] If it is less than the preset second matching spacing, the disparity value of the preset second matching grid point is calculated according to the horizontal coordinate of the preset second matching grid point and the horizontal coordinate of the preset second matching pixel point, and the second matching grid point on the first structural image is re-acquired, and the disparity value of each of the second matching grid points on the first structural image is calculated to obtain the second matching disparity map.

[0058] In one embodiment, performing a second optimization process on the second matching disparity map to obtain the precision disparity map includes:

[0059] Remove discrete grid points;

[0060] Filling the disparity hole and the grid point positions where the discrete grid points are located according to a bilinear interpolation method;

[0061] The precision disparity map is obtained according to the sub-pixel optimization technology and the local tilted plane model.

[0062] In one of the embodiments, the method of acquiring the three-dimensional scene model of the area to be measured by using a first preset rule according to the precision disparity map and the original image further includes:

[0063] Obtain the camera calibration parameters of the endoscope;

[0064] According to the camera calibration parameters, the precision disparity map formula is converted to obtain a three-dimensional point cloud map;

[0065] Preprocessing the three-dimensional point cloud image to obtain a three-dimensional surface model;

[0066] Point cloud texture mapping is performed on the three-dimensional surface model and the original image to obtain the three-dimensional scene model.

[0067] In one embodiment, the preprocessing includes point cloud denoising and point cloud meshing.

[0068] In one embodiment, the step of acquiring a plurality of original structured images obtained by projecting different structured light patterns onto the area to be measured, and an original image obtained before projecting different structured light patterns onto the area to be measured, further includes:

[0069] The structured light pattern is acquired, where the structured light pattern includes a first structured light pattern and a second structured light pattern that is different from the first structured light pattern.

[0070] In one embodiment, acquiring the structured light pattern includes:

[0071] Acquire a structural canvas, wherein the structural canvas includes a plurality of grids, each of which is provided with spots; and the grids correspond to the spots one by one;

[0072] Obtaining a random code for each of the spots;

[0073] According to the random code, the code of the spot in each grid is obtained according to a second preset rule to obtain a spot pattern;

[0074] The structured light pattern is acquired according to a determination result of the coding of the spots in the spot pattern.

[0075] In one embodiment, obtaining the structured light pattern according to the result of determining the encoding of the spots in the spot pattern includes:

[0076] Determining whether the codes of the spots in the spot pattern are unique;

[0077] If yes, determining the speckle pattern as the structured light pattern;

[0078] If not, the codes of the spots in the spot pattern are retrieved again.

[0079] A second aspect of the present application provides an endoscope, comprising:

[0080] A projection module, used to project different structured light patterns onto the area to be measured;

[0081] An imaging module, used for acquiring a plurality of original structured images obtained by projecting different structured light patterns onto the area to be measured, and an original image obtained by not projecting onto the area to be measured; different structured light patterns correspond to different original structured images;

[0082] The image processing device is connected to both the imaging module and the projection module and is configured as follows:

[0083] A precision disparity map is calculated based on each of the original structure images using a preset stereo matching algorithm;

[0084] A three-dimensional scene model of the area to be measured is acquired according to the precision disparity map and the original image using a first preset rule.

[0085] In one embodiment, it also includes:

[0086] Lighting module;

[0087] Camera module;

[0088] A cold light source, used to provide cold light source illumination to the projection module;

[0089] The control circuit is configured as follows: a first end is connected to both the projection module and the imaging module, and a second end is connected to the image processing device, and is used to switch between the cold light source illumination mode and the structured light pattern projection mode.

[0090] A third aspect of the present application provides a storage medium having a computer program stored thereon, wherein the computer program implements the steps of the above method when executed by a processor.

[0091] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention and implement it according to the contents of the specification, the following is a detailed description of the preferred embodiments of the present invention in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0092] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, drawings of other embodiments can be obtained based on these drawings without paying any creative work.

[0093] Figure 1 A schematic diagram of a process of establishing a three-dimensional model provided in an embodiment of the present application;

[0094] Figure 2 A schematic diagram of a process for obtaining a structured light pattern provided in an embodiment of the present application;

[0095] Figure 3 A schematic diagram of two different structured light patterns provided in an embodiment of the present application;

[0096] Figure 4 A schematic diagram of an original image and an original structure image provided in an embodiment of the present application;

[0097] Figure 5 A schematic diagram of a process for obtaining a precision disparity map provided in an embodiment of the present application;

[0098] Figure 6 A schematic diagram of the principle of a de-distortion and stereo correction process provided in one embodiment of the present application;

[0099] Figure 7 A schematic diagram of a process for obtaining a precision disparity map provided in another embodiment of the present application;

[0100] Figure 8 A schematic diagram of a flow chart of a first matching algorithm provided in an embodiment of the present application;

[0101] Fig. 9A schematic diagram of a flow chart of a second matching algorithm provided in an embodiment of the present application;

[0102] Fig.10 A schematic diagram of a precision disparity map provided in an embodiment of the present application;

[0103] Fig.11 A partial flow chart of a three-dimensional model building method provided in one embodiment of the present application;

[0104] Fig.12 A schematic diagram of a three-dimensional point cloud image provided in an embodiment of the present application;

[0105] Fig.13 A schematic diagram of a three-dimensional scene model provided in an embodiment of the present application;

[0106] Fig.14 This is a schematic diagram of the structure of an endoscope provided in one embodiment of the present application.

[0107] Explanation of reference numerals: 10, projection module; 20, imaging module; 30, image processing device; 4, first lighting lamp; 42, second lighting lamp; 5, first camera; 52, second camera; 60, cold light source; 70, control circuit. DETAILED DESCRIPTION

[0108] In order to facilitate the understanding of the present application, the present application will be described more fully below with reference to the relevant drawings. The preferred embodiments of the present application are provided in the drawings. However, the present application can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive.

[0109] Unless otherwise noted, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application. The term "and / or" used herein includes any and all combinations of one or more of the related listed items.

[0110] In the case of using “including”, “having”, and “comprising” described herein, another component may be added unless a clear limiting term such as “only”, “consisting of”, etc. is used. Unless mentioned otherwise, a term in the singular form may include a plural form and should not be understood as being one in number.

[0111] It should be understood that although the terms "first", "second", etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element without departing from the scope of the present application.

[0112] In this application, unless otherwise clearly specified and limited, the terms "connected", "connection" and the like should be understood in a broad sense, for example, it can be directly connected or indirectly connected through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0113] In order to illustrate the above technical solution of the present application, a specific embodiment is provided below for illustration.

[0114] Existing technologies use binocular stereo matching algorithms, such as global stereo matching algorithms Graph Cut and Belief Propagation, etc., which need to establish a global energy function and obtain the optimal disparity value by minimizing the global energy function, but the memory usage is large and the processing speed is extremely slow; such as local stereo matching algorithms (Sum of absolute differences, SAD), etc., which use the energy minimization method to estimate the disparity, and each pixel is calculated in parallel, which can be real-time, but the solution in the local range will lead to poor matching effect; there is also a most classic semi-global stereo matching algorithm SGM (Semi-global Matching), which still uses a global framework, but uses a highly efficient one-dimensional path aggregation method to replace the two-dimensional minimization algorithm in the global algorithm when calculating the step of minimizing the energy function, and uses the one-dimensional optimum to approximate the two-dimensional optimum, so that a better disparity map can be obtained, and the algorithm efficiency is also greatly improved, but it still takes one or two seconds to complete the disparity calculation.

[0115] Therefore, the present application proposes a three-dimensional model building method, an endoscope and a storage medium to build a three-dimensional scene model of the area to be measured illuminated by the endoscope, thereby ensuring the three-dimensional stereo accuracy and accelerating the stereo matching speed to achieve a real-time effect.

[0116] In a three-dimensional model building method provided in one embodiment of the present application, the three-dimensional model building method is used to build a three-dimensional scene model of the area to be measured illuminated by the endoscope, such as Figure 1 As shown, including:

[0117] Step S10: acquiring a plurality of original structured images obtained by projecting different structured light patterns onto the area to be tested, and an original image obtained by not projecting the different structured light patterns onto the area to be tested;

[0118] Step S20: Calculating a precision disparity map using a preset stereo matching algorithm according to each original structure image;

[0119] Step S30: acquiring a three-dimensional scene model of the area to be measured using a first preset rule according to the precision disparity map and the original image.

[0120] In the three-dimensional model building method provided in the above embodiment, multiple original structural images obtained by projecting different structured light patterns are obtained at the same time to increase the surface texture of the area to be measured, and the original image obtained without projection; wherein the original structural images corresponding to different structured light patterns are different; a precision disparity map is calculated based on each original structural image using a preset stereo matching algorithm, and a three-dimensional scene model is obtained by using a first preset rule for the precision disparity map and the original image, thereby ensuring the three-dimensional stereo accuracy and accelerating the stereo matching speed to achieve a real-time effect.

[0121] As an example, different structured light patterns correspond to different original structured images; the structured light pattern includes a speckle light pattern, and the structured projection includes a speckle projection.

[0122] As an example, the area to be measured is the irradiation area of ​​the endoscope, and the original structural image includes a first original structural image and a second original structural image. The first original structural image and the second original structural image have different structured light patterns, which facilitates the correspondence between the first preset algorithm and the second preset algorithm during the process to generate a high-precision three-dimensional scene model.

[0123] In one embodiment, before step S10: acquiring a plurality of original structured images obtained by projecting different structured light patterns onto the area to be measured, and an original image obtained without projecting the different structured light patterns onto the area to be measured, the step further includes:

[0124] Step S101: acquiring a structured light pattern, where the structured light pattern includes a first structured light pattern and a second structured light pattern having a structure pattern different from that of the first structured light pattern.

[0125] Specifically, Figure 2 As shown, step S101: obtaining a structured light pattern, including:

[0126] Step S1011: obtaining a structure canvas, the structure canvas including a plurality of grids, each of which is provided with spots; the grids correspond to the spots one by one;

[0127] Step S1012: obtaining a random code of each spot;

[0128] Step S1013: Obtain the code of the spot in each grid according to the random code and the second preset rule to obtain the spot pattern;

[0129] Step S1014: acquiring a structured light pattern according to the determination result of the coding of the spots in the spot pattern.

[0130] As an example, each time the second preset rule detects that the coding of the spots in the spot pattern is not unique, the preset rule is reset to ensure that the second preset rule is different each time it is executed to speed up the allocation of spot coding in each grid, thereby improving the efficiency of building the three-dimensional model.

[0131] In one embodiment, step S1014: obtaining a structured light pattern according to a result of determining the encoding of spots in the spot pattern, includes:

[0132] Step S1014a: determining whether the code of the spot in the spot pattern is unique;

[0133] Step S1014b: If yes, the speckle pattern is determined as a structured light pattern;

[0134] If not, the encoding of the spots in the spot pattern is retrieved.

[0135] As an example, the random encoding includes the shape, orientation, size, color, grayscale, and at least one of an offset distance and an offset direction within a corresponding grid of each spot.

[0136] As an example, the encoding of each spot in the first structured light pattern corresponding to the first original structured image and the encoding of each spot in the second structured light pattern corresponding to the second original structured image are different from each other; the encoding of all spots in the first structured light pattern is different and has global uniqueness; the encoding of all spots in the second structured light pattern is different and has global uniqueness.

[0137] As an example, each grid has M*M pixels, where M is a positive integer; a spot in the grid may occupy multiple pixels.

[0138] To facilitate understanding of the encoding in structured light patterns, Figure 3 The middle left picture shows the first structured light pattern. Figure 3 The middle right picture shows the second structured light pattern. Please note that Figure 3 For illustration only, there can be countless types of structured light patterns; it is only necessary to ensure that the codes of all spots in the first structured light pattern are different from the codes of all spots in the second structured light pattern. Figure 4 The area to be tested is the abdominal cavity. Figure 4 The middle left image is the original image that is not projected onto the area to be tested. Figure 4The middle right image is the original structural image with a structured light pattern; structured light patterns with different encodings can be used to increase the characteristic texture of the area to be tested, such as tissues and organs in the abdominal cavity in a dark environment.

[0139] In one embodiment, Figure 5 As shown, step S20: using a preset stereo matching algorithm to calculate the precision disparity map according to each original structure image, including:

[0140] Step S21: performing dedistortion processing and stereo correction processing on each original structural image in sequence to obtain multiple structural images;

[0141] Step S22: Calculate a precision disparity map using a preset stereo matching algorithm based on each original structure image.

[0142] As an example, Figure 6 As shown, the first original structural image and the second original structural image are subjected to dedistortion and stereo correction processing in sequence, and the first original structural image and the second original structural image that are not aligned in coplanar rows are corrected to be aligned in coplanar rows, that is, the structural image includes the first structural image and the second structural image in the same plane, and the feature point PI projected by the preset point P on the first structural image and the matching point Pr projected by the preset point P on the second structural image are in the same row, so that in the process of presetting the stereo matching algorithm, only the matching points on the same row need to be searched, thereby speeding up the efficiency of the algorithm. Among them, the preset point can be any point in space, and is not on the first original structural image and the second original structural image.

[0143] In one embodiment, Figure 7 As shown, the preset stereo matching algorithm includes a first preset algorithm and a second preset algorithm; step S22: using the preset stereo matching algorithm to calculate the precision disparity map according to each structural image, including the following steps:

[0144] Step S221: obtaining a first matching disparity map by calculating using a first preset algorithm according to the first structural image and the second structural image;

[0145] Step S222: according to the first structural image, the second structural image and the first matching disparity map, a second preset algorithm is used to calculate and obtain a precision disparity map.

[0146] The first preset algorithm is described in detail below:

[0147] In one embodiment, step S221: calculating a first matching disparity map using a first preset algorithm according to the first structural image and the second structural image includes:

[0148] Step S2211: using a first matching algorithm to calculate a low-resolution disparity map according to the first structural image and the second structural image;

[0149] Step S2212: performing a first optimization process on the low-resolution disparity map to obtain a first matching disparity map.

[0150] As an example, the first matching algorithm includes but is not limited to a coarse matching algorithm.

[0151] As an example, the size of the low resolution image is 1 / N times that of the first structure image, where N is an integer; the size of the first structure image is the same as that of the second structure image.

[0152] In one embodiment, Figure 8 The flowchart of the first matching algorithm shown in FIG. 1 is as follows: Step S2211: According to the first structural image and the second structural image, a low-resolution disparity map is calculated using the first matching algorithm, including:

[0153] Step S22111: acquiring a plurality of first matching grid points on the first structural image, wherein the first matching grid points are spaced apart by a first preset number of pixels;

[0154] Step S22112: obtaining a preset first matching grid point p on the first structural image and an original disparity search range of the preset first matching grid point on the second structural image;

[0155] Step S22113: Calculate the matching cost value of each pixel point within the original disparity search range according to a preset energy function;

[0156] Step S22114: comparing the matching cost value of each pixel point within the original disparity search range, and determining a preset first matching pixel point q corresponding to the minimum matching cost value within the original disparity search range; the preset first matching pixel point q and the preset first matching grid point p are in the same row;

[0157] Step S22115: Determine the mapped first matching pixel point p' in the first structure image according to the comparison result between the preset minimum matching cost value of the first matching pixel point q and the preset first matching cost threshold.

[0158] As an example, the preset energy function includes a preset matching cost function.

[0159] As an example, a preset first matching grid point p may be randomly selected, and the original disparity search range is relatively large. In the subsequent process of searching for the minimum matching cost value, the range is gradually narrowed according to the disparity value of each grid point.

[0160] As an example, the disparity search range refers to point A on the first structural image and point A' on the second structural image. Point A and point A' correspond to each other. On the same line, the 40th pixel is 40 pixels away from point A' as the range between the start search point and point A'. On the second structural image, in the process of determining the minimum matching cost value within the original disparity search range, the matching cost value of each pixel point within the original disparity search range is calculated from left to right.

[0161] As an example, the preset first matching pixel point q and the preset first matching grid point p may be in the same column or in different columns; for example, the preset first matching grid point p is located on a 10*10 grid point, and the preset first matching pixel point q is located on a 10*4 grid point.

[0162] In one embodiment, the preset energy function is a zero-mean normalized cross-correlation function C ZNCC (x,y,d),C ZNCC The expression of (x,y,d) is as follows:

[0163]

[0164] Among them, x, y are the coordinates of the pixel point, d is the disparity value, is the pixel value intensity of the nth first structure image at the position (x+k, y+p), m is the number of structure image pairs, the range of k and p is [-l, l], the range of n is [1, m], is the pixel change intensity within the area to be tested within a preset time with (x, y) as the center point of the first structural image, is the average pixel intensity of the first structural image in the preset space, is the pixel change intensity of the second structural image with (xd, y) as the center point within the preset time and within the area to be tested, is the average pixel intensity of the second structural image in the preset space.

[0165] As an example, the area to be tested is kept unchanged, and multiple first structure images and multiple second structure images are continuously acquired. For example, within the preset time of 3 milliseconds, 3 first structure images and 3 second structure images are acquired in sequence every 1 millisecond, so as to combine the pixel information in the time domain and the space domain to calculate the correlation value C. The larger the correlation value, the smaller the minimum matching cost.

[0166] It should be noted that the matching cost value corresponding to each pixel point within the original disparity search range in the second structured image can be calculated in parallel, thereby increasing the calculation efficiency.

[0167] For details, please continue to refer to Figure 8, step S22115: determining the mapped first matching pixel point in the first structure image according to the comparison result of the preset minimum matching cost value of the first matching pixel point and the preset first matching cost threshold, including:

[0168] Step S22115a: Determine whether the minimum matching cost value of the preset first matching pixel point q is less than a preset first matching cost threshold;

[0169] Step S22115b: If not, determining the preset first matching grid point p as a parallax hole;

[0170] Step S22115c: If so, obtain the first disparity search range of the preset first matching pixel point q on the first structural image, and calculate the matching cost value of each pixel point in the first disparity search range according to the preset energy function, compare the matching cost value of each pixel point in the first disparity search range, determine the minimum matching cost value in the first disparity search range, and determine the pixel point corresponding to the minimum matching cost value in the first disparity search range as the mapped first matching pixel point p'; the mapped first matching pixel point p' is in the same row as the preset first matching pixel point q.

[0171] As an example, in the present application, a parallax hole refers to a parallax value of a grid point on an image being a preset parallax value, and the preset parallax value may be 0.

[0172] As an example, the original disparity search range is larger than the first disparity search range; the original disparity search range and the first disparity search range are on the same line; the first disparity search range is dynamically reduced according to the disparity values ​​of the surrounding grid points of the preset first matching grid point p.

[0173] As above, the matching cost value of each pixel point within the first disparity search range on the first structured image can be calculated in parallel to improve the calculation efficiency.

[0174] In one embodiment, please continue to refer to Figure 8 , step S22116: after determining the mapped first matching pixel point in the first structure image according to the comparison result of the preset minimum matching cost value of the first matching pixel point and the preset first matching cost threshold, the method further includes:

[0175] Step S22116: acquiring a low-resolution disparity map according to a comparison result of a distance between a preset first matching grid point p and a mapped first matching pixel point p' and a preset first matching distance.

[0176] Specifically, step S22116: obtaining a low-resolution disparity map according to a comparison result of the distance between the preset first matching grid point p and the mapped first matching pixel point p' and the preset first matching distance, includes:

[0177] Step S22116a: determining whether the distance between the preset first matching grid point p and the mapped first matching pixel point p' is less than the preset first matching distance;

[0178] Step S22116b: if it is greater than or equal to the preset first matching interval, determining the preset first matching grid point as a parallax hole;

[0179] Step S22116c: If it is less than the preset first matching spacing, the disparity value of the preset first matching grid point p is calculated according to the horizontal coordinate Xp of the preset first matching grid point p and the horizontal coordinate Xq of the preset first matching pixel point, and the first matching grid point on the first structural image is re-acquired, and the disparity values ​​of each first matching grid point on the first structural image are calculated to obtain a low-resolution disparity map.

[0180] As an example, the disparity values ​​of all first matching grid points on the first structured image are calculated, and all first matching grid points are combined to form a low-resolution disparity map.

[0181] As an example, a difference between a horizontal coordinate Xp of a preset first matching grid point p and a horizontal coordinate Xq of a preset first matching pixel point is calculated to obtain a disparity value d=Xp-Xq of the preset first matching grid point p.

[0182] As an example, if the distance p' between the preset first matching grid point p and the mapped first matching pixel point is less than the preset first matching distance, it is determined that the disparity values ​​of the preset first matching grid point p of the first structure image and the preset first matching pixel point q of the second structure image are consistent, and point p corresponds to point q.

[0183] In one embodiment, a grid point with a disparity value of 0 in the low-resolution disparity map is a disparity hole; Step S2212: performing a first optimization process on the low-resolution disparity map to obtain a first matching disparity map, including:

[0184] Step S22121: remove discrete grid points;

[0185] Step S22122: fill the disparity holes and the positions of the discrete grid points according to the bilinear interpolation method and sample them to obtain a first matching disparity map.

[0186] As an example, after the disparity holes are filled using the bilinear interpolation method, the filled disparity map is sampled, and the size of the filled disparity map is restored to the size of the first structural image or the second structural image, thereby obtaining a first matching disparity map.

[0187] It should be noted that the calculation method of the preset energy function in the second preset algorithm is the same as the calculation method of the preset energy function in the first preset algorithm, which will not be repeated below; the second preset algorithm is described in detail below:

[0188] In one embodiment, step S222: using a second preset algorithm to calculate a precision disparity map according to the first structural image, the second structural image and the first matching disparity map, includes:

[0189] Step S2221: using a second matching algorithm to calculate a second matching disparity map according to the first structural image, the second structural image and the first matching disparity map;

[0190] Step S2222: performing a second optimization process on the second matching disparity map to obtain a precision disparity map.

[0191] As an example, the second matching algorithm includes but is not limited to a fine matching algorithm.

[0192] Specifically, Fig. 9 As shown, step S2221: according to the first structural image, the second structural image and the first matching disparity map, a second matching algorithm is used to calculate a second matching disparity map, including:

[0193] Step S22211: acquiring a plurality of second matching grid points on the first structural image, wherein the second matching grid points are spaced apart by a second preset number of pixels;

[0194] Step S22212: Obtaining a preset second matching grid point on the first structure image And obtain the preset second matching grid points according to the first matching disparity map a second disparity search range on the second structured image;

[0195] Step S22213: calculating the matching cost value of each pixel point in the second disparity search range according to a preset energy function;

[0196] Step S22214: Compare the matching cost value of each pixel point within the second disparity search range, determine the minimum matching cost value within the second disparity search range, and determine the pixel point corresponding to the minimum matching cost value within the second disparity search range as the preset second matching pixel point Preset the second matching pixel Matching grid points with preset second are in the same row;

[0197] Step S22215: According to the preset second matching pixel point The comparison result of the minimum matching cost value of the first structure image and the preset second matching cost threshold is used to determine the second matching pixel point mapped in the first structure image.

[0198] In one embodiment, the expression of the second disparity search range is:

[0199]

[0200] Among them, x q is the preset coordinate of the second matching pixel point q, is the preset narrowband search radius, disp(q) is the preset second matching grid point The parallax value of .

[0201] As an example, the preset narrowband search radius is set according to actual needs and is not limited in this application.

[0202] In one embodiment, step S22215: according to the preset second matching pixel point The comparison result of the minimum matching cost value of the first structure image and the preset second matching cost threshold is used to determine the second matching pixel point mapped in the first structure image. include:

[0203] Step S22215a: Determine the preset second matching pixel point Whether the minimum matching cost value is less than a preset second matching cost threshold;

[0204] Step S22215b: If not, preset the second matching grid point Determined as a parallax hole;

[0205] Step S22215c: If yes, obtain the preset second matching pixel point A third disparity search range on the first structured image, and calculating a matching cost value of each pixel point within the third disparity search range according to a preset energy function, comparing the matching cost value of each pixel point within the third disparity search range, determining a minimum matching cost value within the third disparity search range, and determining a pixel point corresponding to the minimum matching cost value within the third disparity search range as a mapped second matching pixel point Map the second matching pixel Match the preset second pixel In the same row.

[0206] In one embodiment, please continue to refer to Fig. 9 , step S22215: according to the preset second matching pixel point The comparison result of the minimum matching cost value of the first structure image and the preset second matching cost threshold is used to determine the second matching pixel point mapped in the first structure image. After that, it also includes:

[0207] Step S22216: According to the preset second matching grid point The second matching pixel with the mapping The comparison result of the spacing between the pixels and the preset second matching spacing is used to obtain a second matching disparity map.

[0208] For details, please continue to refer to Fig. 9 , step S22216: according to the preset second matching grid point The second matching pixel with the mapping The method compares the spacing of the first and second matching spacings with each other to obtain a second matching disparity map, including:

[0209] Step S22216a: Determine the preset second matching grid point The second matching pixel with the mapping Whether the distance between them is less than a preset second matching distance;

[0210] Step S22216b: If it is greater than or equal to the preset second matching interval, the preset grid point Determined as a parallax hole;

[0211] Step S22216c: If it is less than the preset second matching distance, then according to the preset second matching grid point The horizontal coordinate of and preset second matching pixel The horizontal coordinate of the second matching grid point is calculated, the disparity value of the preset second matching grid point is calculated, and the second matching grid point on the first structure image is re-acquired, the disparity value of each second matching grid point on the first structure image is calculated, and the second matching disparity map is obtained.

[0212] As an example, as an example, the disparity values ​​of all second matching grid points on the first structured image are calculated, and all second matching grid points are combined to form a low-resolution disparity map.

[0213] As an example, calculate the preset second matching grid point The horizontal coordinate of and the horizontal coordinate of the preset second matching pixel point The difference between the preset second matching grid points is obtained. The parallax value

[0214] In one embodiment, step S2222: performing a second optimization process on the second matching disparity map to obtain a precision disparity map includes:

[0215] Step S22221: remove discrete grid points;

[0216] Step S22222: filling the disparity holes and the positions of the discrete grid points according to the bilinear interpolation method;

[0217] Step S22223: Obtain a precision disparity map based on the sub-pixel optimization technology and the local tilted plane model, such as Fig.10 shown.

[0218] As an example, a grid point with a disparity value of 0 in the second matching disparity map is a disparity hole.

[0219] As an example, sub-pixel optimization technology is adopted to obtain sub-pixels using a quadratic curve interpolation method. The principle is that the parallax point and the two parallax points around it determine a quadratic curve according to three points, and a parabola passing through these three points is obtained. The vertex of the parabola is the optimal sub-pixel.

[0220] As an example, the local tilted plane model smoothes the disparity map, and each local tilted plane is fitted using the least squares method. Since each local area has no correlation, the least squares fitting of each local tilted plane can be improved in efficiency by parallel computing.

[0221] In one embodiment, Fig.11 As shown, step S30: obtaining a three-dimensional scene model of the area to be measured by adopting a first preset rule according to the precision disparity map and the original image, including:

[0222] Step S31: obtaining the camera calibration parameters of the endoscope;

[0223] Step S32: According to the camera calibration parameters, the precision disparity map formula is converted to obtain a three-dimensional point cloud map, such as Fig.12 As shown;

[0224] Step S33: preprocessing the three-dimensional point cloud image to obtain a three-dimensional surface model;

[0225] Step S34: Perform point cloud texture mapping on the 3D surface model and the original image to obtain a 3D scene model, such as Fig.13 shown.

[0226] As an example, Fig.13 The left picture in the figure shows a 3D scene model of a surgical instrument suspended above the tissue, ready to clamp or shear the lesion. Fig.13 The right picture in the figure is a three-dimensional top view of the in vitro simulation scene.

[0227] Specifically, the camera calibration parameters are the calibration parameters of the endoscope before leaving the factory, and the calibration parameters include the camera intrinsic parameter matrix, distortion coefficient matrix, intrinsic matrix, fundamental matrix, rotation matrix and translation matrix. The steps to obtain the camera calibration parameters are as follows:

[0228] Step S311: obtaining a grid plate pattern;

[0229] Step S312: collecting multiple calibration postures of the grid plate;

[0230] Step S313: Find the corner points of the grid plate and use matrix calculation to obtain camera calibration parameters.

[0231] As an example, the working distance of the endoscope is 3cm-12.5cm from the lens, so a black and white grid of 3mm in size is designed, with a width of 12 squares and a length of 9 squares, and calibrated between 4cm-12cm in front of the lens. This distance is the working distance of the endoscope and the image is clearest; during the calibration process, the calibration plate is rotated, tilted, and moved forward and backward so that the calibration plate appears in different postures in the camera's field of view, and about 15 images are taken.

[0232] Specifically, according to the camera calibration parameters and combined with the parallel binocular vision geometric relationship, the precision disparity map is converted into spatial xyz coordinate information. The precision disparity map can be visualized in the form of a heat map. The bluer the color, the smaller the disparity value of the precision disparity map, and the yellower the color, the larger the disparity value of the precision disparity map.

[0233] As an example, the unit of disparity is pixel, the horizontal coordinate of the disparity value on the image is marked as X, the vertical coordinate of the disparity value on the image is marked as Y, and the depth information is recorded as Z, in millimeters. X = (x-cx) * Z / f, Y = (y-cy) * Z / f, Z = (f * baseline) / disp; baseline is the distance between the optical centers of the two cameras, which is also the baseline distance; disp is the disparity value; f represents the normalized focal length; cx is the horizontal coordinate of the optical center of the camera on the image; cy is the vertical coordinate of the optical center of the camera on the image;

[0234] As an example, preprocessing includes point cloud denoising and point cloud meshing. For point cloud denoising, firstly downsample the point cloud to perform data compression to improve the efficiency of the algorithm, then perform statistical analysis on the neighborhood of each point to remove neighborhood points that do not meet certain standards. For point cloud meshing, the greedy triangulation method is used to triangulate the point cloud, that is, the point cloud is first projected onto a two-dimensional coordinate plane through the normal, and then the projected point cloud is triangulated in the plane to obtain the topological connection relationship of each point. Finally, the topological connection between the original three-dimensional points is determined based on the topological connection relationship of the projected points in the plane. The resulting triangular mesh is the reconstructed three-dimensional surface model.

[0235] As an example, point cloud texture mapping processing stores all color information of the original image on the texture map, and when displayed, it is rendered according to the texture coordinates and texture map of each grid on the three-dimensional surface model, and finally a three-dimensional scene model is obtained.

[0236] In one embodiment of the present application, Fig.14As shown, an endoscope is also proposed, which executes the above-mentioned three-dimensional model building method, including: a projection module 10, an imaging module 20, and an image processing device 30. The projection module 10 is used to project different structured light patterns on the area to be measured; the imaging module 20 is used to obtain a plurality of original structure images obtained by projecting different structured light patterns on the area to be measured, and the original image obtained by not projecting on the area to be measured; the image processing device 30 is connected to both the imaging module 20 and the projection module 10, and is configured to: calculate the precision disparity map according to each original structure image using a preset stereo matching algorithm; and obtain the three-dimensional scene model of the area to be measured using a first preset rule according to the precision disparity map and the original image.

[0237] In the endoscope provided in the above embodiment, multiple original structural images obtained by projecting different structured light patterns are obtained at the same time to increase the surface texture of the area to be measured and the original image obtained without projection; wherein the original structural images corresponding to different structured light patterns are different; a precision disparity map is calculated based on each original structural image using a preset stereo matching algorithm, and a three-dimensional scene model is obtained for the precision disparity map and the original image using a first preset rule, thereby accelerating the stereo matching speed while ensuring the three-dimensional stereo accuracy and achieving a real-time effect; and a tiny projection module is embedded in the endoscope without increasing the volume of the existing endoscope, with a simple structure and convenient equipment.

[0238] In one embodiment, please continue to refer to Fig.14 The endoscope further includes: an illumination module, a camera module, a cold light source 60 and a control circuit 70. The cold light source 60 is used to provide cold light source illumination to the projection module 10 via a light guide; the control circuit 70 is configured as follows: a first end is connected to both the projection module 10 and the imaging module 20, and a second end is connected to the image processing device 30, and is used to switch between the cold light source illumination mode and the structured light pattern projection mode, control the projection frequency, and the camera imaging frequency, and the device is compact and can realize complex control.

[0239] As an example, the lighting module includes a first lighting lamp 41 and a second lighting lamp 42 which are symmetrically distributed on the front end surface of the endoscope; the camera module includes a first camera 51 and a second camera 52; wherein the first camera 51 and the second camera 52 are symmetrically distributed on the front end surface of the endoscope, and are both located between the first lighting lamp 41 and the second lighting lamp 42.

[0240] As an example, the first illumination lamp 41 and the second illumination lamp 42 emit visible light.

[0241] In one embodiment, the image processing device 30 is further configured to:

[0242] Performing dedistortion processing and stereo correction processing on each original structural image in turn to obtain a plurality of structural images; the structural images include a first structural image and a second structural image in the same plane;

[0243] The precision disparity map is calculated based on each structural image using a preset stereo matching algorithm.

[0244] In one embodiment, the preset stereo matching algorithm includes a first preset algorithm and a second preset algorithm; the image processing device 30 is also configured to: according to the first structural image and the second structural image, use the first preset algorithm to calculate a first matching disparity map; according to the first structural image, the second structural image and the first matching disparity map, use the second preset algorithm to calculate a precision disparity map.

[0245] Specifically, the image processing device 30 is further configured to: calculate a low-resolution disparity map using a first matching algorithm according to the first structural image and the second structural image; and perform a first optimization process on the low-resolution disparity map to obtain a first matching disparity map.

[0246] In one embodiment, the image processing device 30 is further configured to: obtain a plurality of first matching grid points on the first structure image, wherein the first matching grid points are separated by a first preset number of pixels;

[0247] Acquire a preset first matching grid point on the first structure image, and acquire an original disparity search range of the preset first matching grid point on the second structure image;

[0248] Calculate the matching cost value of each pixel point within the original disparity search range according to a preset energy function;

[0249] Compare the matching cost value of each pixel point within the original disparity search range, determine the minimum matching cost value within the original disparity search range, and determine the pixel point corresponding to the minimum matching cost value within the original disparity search range as the preset first matching pixel point; the preset first matching pixel point and the preset first matching grid point are in the same row;

[0250] According to the comparison result of the preset minimum matching cost value of the first matching pixel point and the preset first matching cost threshold, the mapped first matching pixel point in the first structure image is determined.

[0251] In one embodiment, the image processing device 30 is further configured to: acquire a low-resolution disparity map according to a comparison result of a distance between a preset first matching grid point and a mapped first matching pixel point and a preset first matching distance.

[0252] In one embodiment, the image processing device 30 is further configured to: obtain a plurality of second matching grid points on the first structure image, wherein the second matching grid points are separated by a second preset number of pixels;

[0253] Acquire a preset second matching grid point on the first structural image, and acquire a second disparity search range of the preset second matching grid point on the second structural image according to the first matching disparity map;

[0254] Calculate the matching cost value of each pixel point within the second disparity search range according to a preset energy function;

[0255] Comparing the matching cost value of each pixel point within the second disparity search range, determining the minimum matching cost value within the second disparity search range, and determining the pixel point corresponding to the minimum matching cost within the second disparity search range as the preset second matching pixel point; the preset second matching pixel point and the preset second matching grid point are in the same row;

[0256] According to the comparison result of the preset minimum matching cost value of the second matching pixel point and the preset second matching cost threshold, the mapped second matching pixel point in the first structure image is determined.

[0257] In one embodiment, the image processing device 30 is further configured to: obtain the second matching disparity map according to a comparison result of the distance between the preset second matching grid point and the mapped second matching pixel point and the preset second matching distance.

[0258] In one embodiment of the present application, a storage medium is further provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above method are implemented.

[0259] For the specific limitations of the three-dimensional model building method in the above embodiment, please refer to the limitations of the three-dimensional model building method above, which will not be repeated here.

[0260] It should be understood that, unless otherwise specified herein, the execution of the steps described is not strictly limited in order, and the steps may be executed in other orders. Moreover, at least a portion of the steps described may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but may be executed at different times, and the execution order of these sub-steps or stages is not necessarily sequential, but may be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0261] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0262] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0263] The above-mentioned embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A three-dimensional model building method, characterized in that: For establishing a three-dimensional scene model of the area to be measured, the method comprises: Acquire a plurality of original structured images obtained by projecting different structured light patterns onto the area to be measured, and an original image obtained by not projecting different structured light patterns onto the area to be measured; According to the low-resolution disparity maps of at least two structural images of each of the original structural images, a precision disparity map is calculated using a preset stereo matching algorithm; the preset stereo matching algorithm includes a matching algorithm for obtaining the low-resolution disparity map; the matching algorithm is used to obtain a preset first matching grid point of one of the structural images, a mapped first matching pixel point having a minimum matching cost value that meets the requirements within an original disparity search range on another of the structural images, and the low-resolution disparity map is associated with the preset first matching grid point and the mapped first matching pixel point; A three-dimensional scene model of the area to be tested is acquired according to the precision disparity map and the original image using a first preset rule; the first preset rule is used to acquire the three-dimensional scene model according to the original image, and a three-dimensional surface model acquired according to the camera calibration parameters of the endoscope and the precision disparity map.

2. The method according to claim 1, characterized in that The method of calculating the precision disparity map using a preset stereo matching algorithm according to each of the original structure images includes: Performing dedistortion processing and stereo correction processing on each of the original structural images in sequence to obtain a plurality of structural images; The precision disparity map is calculated based on each of the structural images using the preset stereo matching algorithm.

3. The method according to claim 2, characterized in that The structural image includes a first structural image and a second structural image in the same plane, and feature points of a preset point projected on the first structural image and matching points of the preset point projected on the second structural image are in the same row.

4. The method according to claim 3, characterized in that The preset stereo matching algorithm includes a first preset algorithm and a second preset algorithm; the step of calculating the precision disparity map using the preset stereo matching algorithm according to each of the structural images includes: According to the first structural image and the second structural image, the first preset algorithm is used to calculate a first matching disparity map; The precision disparity map is obtained by calculating the second preset algorithm according to the first structural image, the second structural image and the first matching disparity map.

5. The method according to claim 4, characterized in that The matching algorithm includes a first matching algorithm; the first matching disparity map is calculated by using a first preset algorithm according to the first structural image and the second structural image, including: According to the first structural image and the second structural image, the first matching algorithm is used to calculate a low-resolution disparity map; A first optimization process is performed on the low-resolution disparity map to obtain the first matching disparity map.

6. The method according to claim 5, characterized in that The step of calculating a low-resolution disparity map using the first matching algorithm according to the first structural image and the second structural image includes: Acquire a plurality of first matching grid points on the first structural image, wherein the first matching grid points are spaced apart by a first preset number of pixels; Acquire a preset first matching grid point on the first structural image, and an original disparity search range of the preset first matching grid point on the second structural image; Calculate the matching cost value of each pixel point in the original disparity search range according to a preset energy function; Comparing the matching cost value of each pixel point within the original disparity search range, determining the minimum matching cost value within the original disparity search range, and determining the pixel point corresponding to the minimum matching cost value within the original disparity search range as a preset first matching pixel point; the preset first matching pixel point and the preset first matching grid point are in the same row; According to the comparison result of the minimum matching cost value of the preset first matching pixel point and the preset first matching cost threshold, the mapped first matching pixel point in the first structure image is determined.

7. The method according to claim 6, characterized in that The determining of the mapped first matching pixel point in the first structure image includes: Determine whether the minimum matching cost value of the preset first matching pixel point is less than a preset first matching cost threshold; If not, determining the preset first matching grid point as a parallax hole; If so, obtain the first disparity search range of the preset first matching pixel point on the first structural image, and calculate the matching cost value of each pixel point in the first disparity search range according to the preset energy function, compare the matching cost value of each pixel point in the first disparity search range, determine the minimum matching cost value in the first disparity search range, and determine the pixel point corresponding to the minimum matching cost value in the first disparity search range as the mapped first matching pixel point; the mapped first matching pixel point and the preset first matching pixel point are in the same row.

8. The method according to claim 7, characterized in that After determining the mapped first matching pixel point in the first structure image, the method further includes: The low-resolution disparity map is acquired according to a comparison result between a distance between the preset first matching grid point and the mapped first matching pixel point and a preset first matching distance.

9. The method according to claim 8, characterized in that The obtaining of the low-resolution disparity map comprises: Determine whether the distance between the preset first matching grid point and the mapped first matching pixel point is less than a preset first matching distance; If it is greater than or equal to the preset first matching distance, determining the preset first matching grid point as a parallax hole; If it is less than the preset first matching spacing, the disparity value of the preset first matching grid point is calculated according to the horizontal coordinate of the preset first matching grid point and the horizontal coordinate of the preset first matching pixel point, and the first matching grid point on the first structural image is re-acquired, and the disparity value of each of the first matching grid points on the first structural image is calculated to obtain the low-resolution disparity map.

10. The method according to claim 8, characterized in that The performing a first optimization process on the low-resolution disparity map to obtain the first matching disparity map includes: Remove discrete grid points; The disparity holes and the grid point positions where the discrete grid points are located are filled and sampled according to a bilinear interpolation method to obtain the first matching disparity map.

11. The method according to claim 4, characterized in that The matching algorithm includes a second matching algorithm; the precision disparity map is calculated using a second preset algorithm according to the first structural image, the second structural image and the first matching disparity map, including: According to the first structural image, the second structural image and the first matching disparity map, the second matching algorithm is used to calculate a second matching disparity map; The second matching disparity map is optimized for the second time to obtain the precision disparity map.

12. The method according to claim 11, characterized in that The step of calculating the second matching disparity map by using the second matching algorithm according to the first structural image, the second structural image and the first matching disparity map includes: Acquire a plurality of second matching grid points on the first structural image, wherein the second matching grid points are spaced apart by a second preset number of pixels; Acquire a preset second matching grid point on the first structural image, and acquire a second disparity search range of the preset second matching grid point on the second structural image according to the first matching disparity map; Calculating a matching cost value for each pixel point within the second disparity search range according to a preset energy function; Comparing the matching cost value of each pixel point within the second disparity search range, determining the minimum matching cost value within the second disparity search range, and determining the pixel point corresponding to the minimum matching cost within the second disparity search range as a preset second matching pixel point; the preset second matching pixel point and the preset second matching grid point are in the same row; According to the comparison result of the minimum matching cost value of the preset second matching pixel point and the preset second matching cost threshold, the mapped second matching pixel point in the first structure image is determined.

13. The method according to claim 12, characterized in that The determining the mapped second matching pixel point in the first structure image according to the comparison result of the minimum matching cost value of the preset second matching pixel point and the preset second matching cost threshold comprises: Determine whether the minimum matching cost value of the preset second matching pixel point is less than a preset second matching cost threshold; If not, determining the preset second matching grid point as a parallax hole; If so, obtain the third disparity search range of the preset second matching pixel point on the first structural image, and calculate the matching cost value of each pixel point in the third disparity search range according to the preset energy function, compare the matching cost value of each pixel point in the third disparity search range, determine the minimum matching cost value in the third disparity search range, and determine the pixel point corresponding to the minimum matching cost value in the third disparity search range as the mapped second matching pixel point; the mapped second matching pixel point and the preset second matching pixel point are in the same row.

14. The method according to claim 13, characterized in that After determining the mapped second matching pixel point in the first structure image, the method further includes: The second matching disparity map is obtained according to a comparison result between the distance between the preset second matching grid point and the mapped second matching pixel point and a preset second matching distance.

15. The method according to claim 14, characterized in that The obtaining of the second matching disparity map according to a comparison result between the distance between the preset second matching grid point and the mapped second matching pixel point and a preset second matching distance comprises: Determine whether the distance between the preset second matching grid point and the mapped second matching pixel point is less than a preset second matching distance; If it is greater than or equal to the preset second matching distance, determining the preset second matching grid point as a parallax hole; If it is less than the preset second matching spacing, the disparity value of the preset second matching grid point is calculated according to the horizontal coordinate of the preset second matching grid point and the horizontal coordinate of the preset second matching pixel point, and the second matching grid point on the first structural image is re-acquired, and the disparity value of each of the second matching grid points on the first structural image is calculated to obtain the second matching disparity map.

16. The method according to claim 13, characterized in that The performing a second optimization process on the second matching disparity map to obtain the precision disparity map includes: Remove discrete grid points; Filling the disparity hole and the grid point positions where the discrete grid points are located according to a bilinear interpolation method; The precision disparity map is obtained according to the sub-pixel optimization technology and the local tilted plane model.

17. The method according to any one of claims 1 to 4, characterized in that: The step of acquiring the three-dimensional scene model of the area to be measured by adopting a first preset rule according to the precision disparity map and the original image further includes: Obtain the camera calibration parameters of the endoscope; According to the camera calibration parameters, the precision disparity map formula is converted to obtain a three-dimensional point cloud map; Preprocessing the three-dimensional point cloud image to obtain a three-dimensional surface model; Point cloud texture mapping is performed on the three-dimensional surface model and the original image to obtain the three-dimensional scene model.

18. The method according to claim 17, characterized in that The preprocessing includes point cloud denoising processing and point cloud meshing processing.

19. The method according to any one of claims 1 to 4, characterized in that: Before acquiring a plurality of original structured images obtained by projecting different structured light patterns onto the area to be measured, and original images obtained before projecting them onto the area to be measured, the method further includes: The structured light pattern is acquired, where the structured light pattern includes a first structured light pattern and a second structured light pattern that is different from the first structured light pattern.

20. The method according to claim 19, characterized in that The acquiring the structured light pattern comprises: Acquire a structural canvas, wherein the structural canvas includes a plurality of grids, each of which is provided with spots; and the grids correspond to the spots one by one; Obtaining a random code for each of the spots; According to the random code, the code of the spot in each grid is obtained according to a second preset rule to obtain a spot pattern; The structured light pattern is acquired according to a determination result of the coding of the spots in the spot pattern.

21. The method according to claim 20, characterized in that The step of obtaining the structured light pattern according to the result of determining the encoding of the spots in the spot pattern comprises: Determining whether the codes of the spots in the spot pattern are unique; If yes, determining the speckle pattern as the structured light pattern; If not, the codes of the spots in the spot pattern are retrieved again.

22. An endoscope, characterized in that: include: A projection module, used to project different structured light patterns onto the area to be measured; An imaging module, used for acquiring a plurality of original structured images obtained by projecting different structured light patterns onto the area to be measured, and an original image obtained by not projecting onto the area to be measured; different structured light patterns correspond to different original structured images; The image processing device is connected to both the imaging module and the projection module and is configured as follows: According to the low-resolution disparity maps of at least two structural images of each of the original structural images, a precision disparity map is calculated using a preset stereo matching algorithm; the preset stereo matching algorithm includes a matching algorithm for obtaining the low-resolution disparity map; the matching algorithm is used to obtain a preset first matching grid point of one of the structural images, a mapped first matching pixel point having a minimum matching cost value that meets the requirements within an original disparity search range on another of the structural images, and the low-resolution disparity map is associated with the preset first matching grid point and the mapped first matching pixel point; A three-dimensional scene model of the area to be tested is acquired according to the precision disparity map and the original image using a first preset rule; the first preset rule is used to acquire the three-dimensional scene model according to the original image, and a three-dimensional surface model acquired according to the camera calibration parameters of the endoscope and the precision disparity map.

23. The endoscope according to claim 22, characterized in that Also includes: Lighting module; Camera module; A cold light source, used to provide cold light source illumination to the projection module; The control circuit is configured as follows: a first end is connected to both the projection module and the imaging module, and a second end is connected to the image processing device, and is used to switch between the cold light source illumination mode and the structured light pattern projection mode.

24. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 21 are implemented.

Images