Binocular stereo matching method of 3D vision, endoscope system and storage medium

Abstract

The invention relates to a binocular stereo matching method based on 3D vision, an endoscope system and a storage medium. The binocular stereo matching method comprises the steps that a binocular imaging system is used for obtaining a first image and a second image of a focus area; the image preprocessing system is used for processing the first image and the second image to improve the quality and adaptability of the images; the binocular stereo matching algorithm system is used for carrying out stereo matching on the first image and the second image to obtain a view difference; the three-dimensional reconstruction system is used for generating a depth map through the view difference, mapping the information of the original image to a three-dimensional grid model for three-dimensional information fusion, and obtaining a three-dimensional model of the constructed image; and the image display system is used for displaying the three-dimensional model of the constructed image. Three-dimensional imaging of a focus area is achieved through binocular imaging, image processing and three-dimensional reconstruction, the accuracy and safety of an operation are improved, the 3D vision technology is combined with a medical endoscope, three-dimensional imaging of a focus is achieved, more comprehensive focus information is provided, and therefore the diagnosis accuracy and the success rate of the operation are improved.

Description

Technical Field

[0001] The present invention relates to the field of binocular stereo matching methods, and in particular to a 3D vision binocular stereo matching method, a 3D vision endoscope system and a storage medium. Background Art

[0002] In recent years, medical endoscope technology has developed rapidly, and a variety of endoscope devices with 4K resolution have appeared on the market. These ultra-high-definition endoscopes provide doctors with clearer images of lesions, greatly improving the success rate of surgery. However, most existing medical endoscopes use 2D imaging technology. Although high-resolution endoscopes provide clear images, they lack depth information and cannot provide three-dimensional structural information of lesions, which can easily lead to misjudgment and misoperation during surgery. 3D vision technology has made significant progress in recent years, especially in the fields of computer vision and image processing. Combining 3D vision technology with medical endoscopes is expected to achieve stereoscopic imaging of lesions and provide more comprehensive lesion information, thereby improving diagnostic accuracy and surgical success rate, becoming a breakthrough technology in the medical field. Summary of the invention

[0003] The technical problem to be solved by the present invention is that most existing medical endoscopes use 2D imaging technology. Although high-resolution endoscopes provide clear images, they lack depth information and cannot provide three-dimensional structural information of lesions, which can easily lead to misjudgment and misoperation during surgery. In view of the above-mentioned defects of the prior art, a 3D vision binocular stereo matching method, an endoscope system and a storage medium are provided.

[0004] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0005] Construct a 3D vision binocular stereo matching method, including:

[0006] Filtering the first image and the second image to generate a feature map;

[0007] Based on the feature maps of the first image and the second image, matching points with higher confidence are selected as support points;

[0008] Constructing a local plane model for the support points to obtain disparity information of the entire first image;

[0009] A disparity map of the second image is obtained through the disparity information of the first image.

[0010] Preferably, selecting matching points with higher confidence as supporting points based on the feature maps of the first image and the second image includes:

[0011] First, a globally reliable disparity search range is set, and a 16-dimensional feature vector is constructed according to processed pixel points in the first image. Then, feature matching is performed to obtain a similarity measure value between the first image and the second image.

[0012] Preferably, performing feature matching to obtain a similarity measure value between the first image and the second image includes:

[0013] For each pixel in the first image, the feature vectors of the four corners in the 5x5 neighborhood of each pixel in the second image are used to match within the globally reliable disparity search range, and the similarity measure of the matching pairs is calculated.

[0014] If the ratio of the minimum value and the second minimum value of the similarity metric of the matching pair is within the set range, the disparity value is taken as the disparity value of the candidate support point.

[0015] Preferably, after obtaining the disparity value, all the support points F are obtained by using the left-right consistency principle, and the disparity values ​​of the corresponding matching points in the first image and the second image that are equal and the number of other support points in the neighborhood around the support point are determined to screen the remaining matches again to obtain all the support points F;

[0016] When performing left-right consistency detection, if the absolute value of the disparity of a pixel point of the first image and the disparity of a corresponding point of the second image are within a set range, the disparity value of the first image is considered reliable, otherwise it is set to -1 or other invalid values.

[0017] Preferably, constructing a local plane model for the support points to obtain disparity information of the entire first image includes:

[0018] Perform Delaunay triangulation on the supporting point set F in the first image to obtain a set of triangles, wherein Delaunay triangulation ensures that the circumscribed circle of the triangle does not contain other feature points, thereby generating a relatively uniform triangular mesh. For each triangle T, calculate its corresponding disparity plane.

[0019] Preferably, obtaining the disparity map of the second image through the disparity information of the first image includes:

[0020] Using the prior probability distribution and the image likelihood probability distribution, and performing stereo matching to obtain a disparity map of the second image;

[0021] Among them, according to the probability generation model, given the first image and support points, sampling is performed from the second image, and the probability model is the maximum a posteriori probability estimation:

[0022]

[0023] in Indicates that the second image is located at For the observation points on the epipolar line, the posterior probability indicates that given the first image observation point, the support point set and the second image observation point, the first image observation point takes the disparity d n The probability of , the posterior probability can be decomposed into:

[0024]

[0025] Substituting the prior probability and likelihood probability into the above formula and taking the negative logarithm, we can get an energy function:

[0026]

[0027] The first part of the energy function represents the likelihood term, and the second part is the prior term or smoothing term. By minimizing the energy function, the observation point can be obtained. to obtain a disparity map of the second image.

[0028] Preferably, after obtaining the disparity map of the second image through the disparity information of the first image, the method further includes:

[0029] Use adaptive median filtering to remove noise in the disparity map and retain the edges of the disparity map;

[0030] Among them, in the process of stereo matching images, a pixel at a certain point in the disparity map needs to be set to an invalid value through interpolation based on a reasonable disparity value, and then interpolated in the horizontal and vertical directions. When the disparity values ​​of the leftmost and rightmost sides of an invalid area are d1 and d2, the depth mutation is considered, and then the adaptive median filter is used to eliminate the noise in the disparity map to retain the edge of the disparity map.

[0031] Construct a 3D vision endoscope system, including:

[0032] Binocular imaging system: acquiring the first image and the second image of the lesion area;

[0033] Image preprocessing system: used for processing the acquired first image and the second image to improve the quality and adaptability of the first image and the second image;

[0034] Binocular stereo matching algorithm system: used to perform stereo matching on the first image and the second image to obtain the view difference;

[0035] 3D reconstruction system: generates a depth map through view difference, and maps the information of the original image to the 3D mesh model for 3D information fusion to obtain a 3D model of the constructed image;

[0036] Image display system: displays the three-dimensional model of the constructed image.

[0037] Preferably, the depth of each pixel is calculated using the disparity map and the relationship between the disparity, the depth and the focal length to generate a depth map, and each pixel value in the depth map is given to represent the distance from the point to the binocular imaging system. The depth value in the depth map is converted into a point in three-dimensional coordinates to obtain the three-dimensional coordinates of each pixel, and the color information of the original first image and the second image is mapped to the three-dimensional network model to obtain a three-dimensional model of the constructed image;

[0038] Among them: the binocular imaging system uses two sets of endoscopic cameras at a certain distance to capture the first image and the second image of the lesion area respectively, and the two sets of endoscopic cameras have the same parameter models and work synchronously.

[0039] A storage medium is constructed, on which program instructions are stored, characterized in that: the program instructions are used to execute the 3D vision binocular stereo matching method as described in any one of the above items when running.

[0040] The beneficial effects of the present invention are: through binocular imaging, image processing and three-dimensional reconstruction, stereoscopic imaging of the lesion area is achieved, and the accuracy and safety of the operation are improved. The 3D vision technology is combined with the medical endoscope to achieve stereoscopic imaging of the lesion, provide more comprehensive lesion information, thereby improving the accuracy of diagnosis and the success rate of the operation. The endoscope combined with 3D vision can capture lesion images from multiple angles, and use computer algorithms to reconstruct these images into a three-dimensional model, so that the three-dimensional model has not only a spatial structure, but also a surface texture, enhancing the visualization effect. The depth and spatial structure of the lesion are displayed by the three-dimensional model, providing more accurate spatial information, helping doctors to more accurately evaluate the size, shape and position of the lesion. The doctor's spatial perception ability is significantly improved, and the risk of misjudgment during surgery is reduced. 3D vision combined with medical endoscopes not only improves the accuracy of lesion imaging and the accuracy of surgical operations, but also realizes three-dimensional reconstruction and stereoscopic imaging of endoscopic images, provides more accurate spatial information, improves the success rate of surgery, and improves the quality of medical care and the treatment effect of patients. And doctors can observe the fine structure of the surgical area more clearly through 3D endoscope technology, so as to perform more accurate operations. It can also be used in medical education and surgical training to provide doctors with a more realistic surgical simulation environment, helping them improve their operational skills and clinical experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the present invention will be further described below in conjunction with the accompanying drawings and embodiments. The drawings described below are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work:

[0042] Figure 1A schematic diagram of a flow chart of an endoscope system method according to a preferred embodiment of the present invention;

[0043] Figure 2 A schematic diagram of a binocular stereo matching method according to a preferred embodiment of the present invention;

[0044] Figure 3 Schematic diagram of an endoscope system according to a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the following will be described clearly and completely in combination with the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are partial embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the protection scope of the present invention.

[0046] A method for operating a 3D vision endoscope according to a preferred embodiment of the present invention; Figure 1 As shown, it is a flowchart of the endoscope operation method provided by an embodiment of the present invention. The method can be executed by a device, and the device can be implemented by software and / or hardware.

[0047] Specifically, in this embodiment, the endoscope operation method includes:

[0048] S20: Acquire a left view image and a right view image of the lesion area;

[0049] In a preferred embodiment of the present invention, two endoscopic cameras at a certain distance are used to capture the left view image and the right view image of the lesion area respectively. In the process of acquiring the left view image and the right view image, it is preferred to use two sets of endoscopic cameras with the same parameter model and working synchronously to acquire the two sets of view images of the lesion area, so as to facilitate the subsequent three-dimensional reconstruction of the acquired images.

[0050] S21: preprocessing the acquired left view image and right view image to improve image quality and adaptability;

[0051] In a preferred embodiment of the present invention, the acquired left view image and right view image of the lesion area are preprocessed to improve the image quality and adaptability to subsequent processing. In the preprocessing process, the image of the lesion area is subjected to denoising, contrast enhancement, and distortion correction to process the noise points in the original image to improve the image quality and adaptability to subsequent processing. The specific denoising, contrast enhancement, and distortion correction methods are the same as the existing methods and will not be repeated here.

[0052] S22: Matching feature points of the left view image and the right view image, obtaining corresponding points in the images, and generating a disparity map;

[0053] In the preferred embodiment of the present invention, since the images acquired by the medical endoscope require high quality and low latency characteristics, ordinary binocular stereo matching algorithms such as SIFT (Scale Invariant Feature Transform), SURF (Speeded Up Robust Features) or ORB (Orbital Robust Features) are no longer suitable for acquiring images of the lesion area. Therefore, a new binocular stereo algorithm is needed to acquire high quality and low latency images of the lesion area.

[0054] Furthermore, the left view image and the right view image are first filtered to generate a feature map, and then matching is performed based on the features to screen out matching points with higher confidence as support points, and a triangulated network is constructed based on the support points to obtain the disparity information of the entire image; then, stereo matching is performed using the prior probability distribution and the image likelihood probability distribution to obtain the view difference, and some values ​​in the view difference are invalidated to obtain the view edge to obtain the final depth map.

[0055] S23: Calculate depth information based on the view difference and generate a depth map, and then construct a three-dimensional model of the component image based on the depth map;

[0056] In a preferred embodiment of the present invention, depth calculation and three-dimensional reconstruction are used for the disparity map obtained according to the binocular stereo matching algorithm, and combined with the camera parameters, the depth information of each pixel is calculated through the triangulation principle to generate a depth map, and the left view image, right view image and pixel points are mapped to the three-dimensional space through the depth map to construct a three-dimensional model of the image.

[0057] Furthermore, in the process of depth calculation and 3D reconstruction, the specific implementation method is as follows:

[0058] According to the principle of binocular vision, the relationship between disparity d and depth Z can be expressed as:

[0059]

[0060] Where f is the focal length of the camera, and B is the baseline distance between the two cameras. Using the parallax distance and formula ①, the depth of each pixel can be calculated to generate a depth map. Each pixel value in the depth map represents the distance from the point to the camera, thereby generating a pixel coordinate system in the image:

[0061]

[0062] Where (x, y) represents the pixel coordinate position in the image. The depth value in the depth map is converted into a point in the three-dimensional coordinate system to obtain the position of the three-dimensional coordinate system:

[0063]

[0064] Among them, (c x, c y ) is the coordinate of the optical center of the image, f x and f y is the focal length of the camera in the x and y directions. Through the formula ③, the three-dimensional coordinates of each pixel are obtained, and then the color information of the original image is mapped to the three-dimensional grid model, so that the three-dimensional model not only has a spatial structure, but also has a surface texture, which enhances the visualization effect.

[0065] S24: Displaying the three-dimensional model of the constructed image;

[0066] In a preferred embodiment of the present invention, a 3D image display system can be formed by using 3D glasses and a data processing device. The camera host sends the 3D model of the constructed image through a wireless transmission device, and the wireless receiving device on the 3D glasses receives the 3D model information of the image and displays it through the display device on the 3D glasses. It should be noted that the display device can also be other intelligent devices, not limited to 3D glasses. Any device or apparatus that can receive the 3D model of the constructed image sent by the camera host, and the device or apparatus can display the received 3D model of the constructed image, can be applied to the present invention and belong to the protection scope of the present invention.

[0067] Furthermore, the display of the three-dimensional model of the constructed image can also be composed of a data transceiver and a display device. The data transceiver uses high-frequency 5G communication technology to display real-time communication within a small range. The display device displays the received three-dimensional model of the constructed image, so that the doctor can clearly observe the three-dimensional structure and spatial position of the lesion.

[0068] Specifically, in the above embodiments, Figure 2 As shown, the above step S22: matching the feature points of the left view image and the right view image, obtaining corresponding points in the image, and generating a disparity map includes:

[0069] S220: performing sobel filtering on the left view image and the right view image to generate a sobel feature map;

[0070] In a preferred embodiment of the present invention, two groups of simple 3x3 kernels are used to process the brightness changes in the horizontal direction and the vertical direction respectively, as shown in the following table:

[0071]

[0072] Thus, we get the horizontal direction G x and vertical G y , taking into account the changes in these two directions, the change in direction G is calculated using formula ④:

[0073]

[0074] S221: Matching is performed based on the Sobel feature, and matching points with higher confidence are selected as support points;

[0075] In a preferred embodiment of the present invention, a globally reliable disparity search range is first set for calculating support points:

[0076] [disp min ,disp max ]

[0077] Generally, the minimum parallax is set to 0, and the maximum parallax needs to be adjusted according to the scene used. At the same time, the horizontal and vertical response values ​​G of the processed pixels in the left view image are x and G y , then for each pixel, select 12 horizontal response values ​​from the 5X5 neighborhood around it, and select 4 vertical response values, and form a 16-dimensional feature vector through formula ⑤:

[0078] f l (u, v) = (s1, s2,..., s 16 ) r ⑤

[0079] When performing feature matching, for each pixel in the left view image, the feature vectors f of the four corners in the 5X5 neighborhood of the corresponding pixel in the right view image are used. r (u-d+2,v+2),f r (u-d+2,v-2),f r (ud-2, v+2), f r (ud-2, v-2), d∈[disp min ,disp max ] to match, the similarity measurement value of the matching pair is estimated as shown in formula ⑥ and formula ⑦:

[0080] E(f1)=∑|f l (u, v)-f r (xd, y)|, x=u±2, y=v±2 ⑥

[0081] E(f2)=∑|f l (u, v)-f r (x+d, y)|, x=ud±2, y=v±2 ⑦

[0082] If the ratio of the minimum value and the second minimum value of E(f1) is within the set range, the disparity value is taken as the disparity value of the candidate support point. Applying the left-right consistency principle, that is, the disparity values ​​of the corresponding matching points in the left and right images are equal, and the number of other support points in a certain neighborhood (3x3 area) around the support point, continue to screen the remaining matches, and all the support points F can be obtained through the above steps.

[0083] Furthermore, when performing left-right consistency detection, the disparity of the left pixel (u, v) is set to d (l) , then the corresponding point (ud (l) , the parallax of v) is d (r) , if |d (l) -d (r) |<τ, then the parallax d (l) is reliable, otherwise it is set to -1 or other invalid values. τ is generally 1 or 2, and can be adjusted appropriately according to the situation.

[0084] S222: constructing a Delaunay triangulation network for the support points to obtain disparity information of the entire image;

[0085] In order to obtain the disparity information of the entire image in a preferred embodiment of the present invention, a Delaunay triangulation network is used to construct a local plane model.

[0086] Furthermore, the supporting point set F in the left image is subjected to Delaunay triangulation to obtain a set of triangles T = {T1, T2...T m Delaunay triangulation ensures that the circumscribed circle of a triangle does not contain other feature points, thereby generating a relatively uniform triangular mesh. For each triangle T, calculate its corresponding disparity plane, assuming that its vertices are (x1, y1, d1), (x2, y2, d2), (x3, y3, d3), and other d i is the disparity value, then the disparity plane can be expressed as:

[0087] d(x,y)=a·x+b·y+c⑧

[0088] From this, the linear equations of the parallax plane parameters a, b, c can be determined as:

[0089]

[0090] Thus solving the equation we can get the values ​​of a, b, and c.

[0091] S223: Using the prior probability distribution and the image likelihood probability distribution to perform dense disparity estimation to obtain a disparity map;

[0092] In a preferred embodiment of the present invention, a priori probability distribution and image likelihood probability distribution are used, and stereo matching is performed based on them. The priori distribution determines the range of disparity sampling, and the likelihood probability provides a judgment on the reliability of disparity sampling.

[0093] According to the probability generation model, given the left view image and support points, sampling can be performed from the right view image. The probability model is the maximum a posteriori probability estimation (MAP):

[0094]

[0095] in Indicates the right view image is located at For the observation points on the epipolar line, the posterior probability represents the given left-view image observation point, the support point set and the right-view image observation point. The left-view image observation point takes the disparity d n The posterior probability can be decomposed into:

[0096]

[0097] Substituting the prior probability and likelihood probability into the formula And taking the negative logarithm we can get an energy function

[0098]

[0099] The first part of the energy function represents the likelihood term, and the second part is the prior term or smoothing term. By minimizing the energy function, the observation point The disparity is divided into two parts when the energy function is minimized by pre-calculating the prior term.

[0100] Among them, based on the prior distribution, only the local range of the average disparity of the observation point after interpolation is considered, that is:

[0101] |d-μ|<sradius×σ, where sradius is generally 2 or 3.

[0102] The disparity search range composed of the disparity of all supporting points in a 20X20 neighborhood around the pixel does not consider the priori term when minimizing the energy function. The view disparity of the right view image can be obtained through the above steps.

[0103] S224: eliminating noise points in the disparity map by interpolation and adaptive median filtering, and retaining the edge of the disparity map;

[0104] In the preferred embodiment of the present invention, in the process of stereo matching, there are many situations that may cause the pixel of a certain point in the disparity map to be set to an invalid value, such as the left-right consistency check may cause it to be invalid. The disparity at these points often needs to be given a reasonable disparity value through interpolation. First consider the horizontal direction. If the disparity values ​​of the leftmost and rightmost of an invalid area are d1 and d2, considering the depth mutation, the interpolation formula of the disparity of all points in it is as follows:

[0105]

[0106] Where η represents the threshold of depth discontinuity. The vertical direction is interpolated once in the same way. After interpolation, an adaptive median filter is used to eliminate noise in the disparity map while better retaining the edge of the disparity map. This method is faster in calculation and can maintain the edge information of the disparity map.

[0107] Through the above method, 3D vision technology is combined with medical endoscopes to achieve stereoscopic imaging of lesions and provide more comprehensive lesion information, thereby improving diagnostic accuracy and surgical success rate. The endoscope combined with 3D vision can capture lesion images from multiple angles and reconstruct these images into three-dimensional models using computer algorithms. The depth and spatial structure of the lesion are displayed by a three-dimensional model, providing more accurate spatial information to help doctors more accurately assess the size, shape and position of the lesion. The doctor's spatial perception ability is significantly improved, and the risk of misjudgment during surgery is reduced. 3D vision combined with medical endoscopes not only improves the accuracy of lesion imaging and the accuracy of surgical operations, but also realizes three-dimensional reconstruction and stereoscopic imaging of endoscopic images, provides more accurate spatial information, improves the success rate of surgery, and improves medical quality and patient treatment effects. And doctors can observe the subtle structure of the surgical area more clearly through 3D endoscope technology, so as to perform more precise operations. It can also be used for medical education and surgical training to provide doctors with a more realistic surgical simulation environment to help them improve their operating skills and clinical experience.

[0108] Corresponding to a method for operating a 3D vision endoscope, the present invention also provides a 3D vision endoscope system, specifically, as Figure 3 As shown, the endoscope operating system includes: a binocular imaging system 100 , an image preprocessing system 110 , a binocular stereo matching algorithm system 120 , a three-dimensional reconstruction system 130 and an image display system 140 .

[0109] A binocular imaging system 100 acquires a left view image and a right view image of the lesion area;

[0110] Specifically, two endoscopic cameras at a certain distance are used to capture the left view image and the right view image of the lesion area respectively. In the process of obtaining the left view image and the right view image, it is preferred to use two sets of endoscopic cameras with the same parameter model and working synchronously to obtain the two sets of view images of the lesion area, so as to facilitate the subsequent three-dimensional reconstruction of the obtained images.

[0111] An image preprocessing system 110 preprocesses the acquired left-view image and right-view image of the lesion area to improve image quality and adaptability;

[0112] Specifically, the acquired left view image and right view image of the lesion area are preprocessed to improve the image quality and adaptability to subsequent processing. In the preprocessing process, the image of the lesion area is denoised, contrast enhanced, and distortion corrected to improve the image quality and adaptability to subsequent processing. The specific denoising, contrast enhancement, and distortion correction methods are the same as the existing method and will not be repeated here.

[0113] The binocular stereo matching algorithm system 120 will match the feature points of the work image and the right view image, obtain the corresponding points in the images, and generate the view difference;

[0114] Specifically, the left view image and the right view image are first filtered to generate a feature map, and then matching is performed based on the features to select matching points with higher confidence as support points, and a triangular network is constructed based on the support points to obtain the disparity information of the entire image; then stereo matching is performed using the prior probability distribution and the image likelihood probability distribution to obtain the view difference, and some values ​​in the view difference are invalidated to obtain the view edge to obtain the final depth map. The working method of the specific binocular stereo matching algorithm system is the same as the aforementioned binocular stereo matching method, and will not be repeated here.

[0115] The 3D reconstruction system 130 calculates the depth information of each pixel based on the view difference, generates a depth map, and constructs a 3D model of the image;

[0116] Specifically, according to the disparity map obtained by the binocular stereo matching algorithm and combined with the camera parameters, the depth information of each pixel is calculated through the triangulation principle to generate a depth map, and the left view image, the right view image and the pixel points are mapped to the three-dimensional space through the depth map to construct a three-dimensional model of the image. The working method of the specific three-dimensional reconstruction system is the same as the aforementioned endoscope operation method, and will not be repeated here.

[0117] An image display system 140 displays the constructed three-dimensional model;

[0118] Specifically, a 3D image display system is formed by using 3D glasses and a data processing device, the camera host sends the 3D model of the constructed image through a wireless transmission device, and the wireless receiving device on the 3D glasses receives the 3D model information of the image and displays it through the display device on the 3D glasses. It should be noted that the display device can also be other intelligent devices, not limited to 3D glasses. Any device or apparatus that can receive the 3D model of the constructed image sent by the camera host, and the device or apparatus can display the received 3D model of the constructed image, can be applied to the present invention and belong to the protection scope of the present invention.

[0119] Furthermore, the display of the three-dimensional model of the constructed image can also be composed of a data transceiver and a display device. The data transceiver uses high-frequency 5G communication technology to display real-time communication within a small range. The display device displays the received three-dimensional model of the constructed image, so that the doctor can clearly observe the three-dimensional structure and spatial position of the lesion.

[0120] Those skilled in the art will appreciate that all or part of the functions of the various methods in the above-mentioned embodiments can be implemented by hardware or by computer programs. When all or part of the functions in the above-mentioned embodiments are implemented by computer programs, the program can be stored in a computer-readable storage medium, and the storage medium can include: read-only memory, random access memory, disk, optical disk, hard disk, etc., and the program is executed by a computer to implement the above-mentioned functions. For example, the program is stored in the memory of the device, and when the program in the memory is executed by the processor, all or part of the above-mentioned functions can be implemented. In addition, when all or part of the functions in the above-mentioned embodiments are implemented by computer programs, the program can also be stored in a storage medium such as a server, another computer, disk, optical disk, flash disk or mobile hard disk, and can be downloaded or copied and saved in the memory of the local device, or the system of the local device is updated, and when the program in the memory is executed by the processor, all or part of the functions in the above-mentioned embodiments can be implemented.

[0121] Furthermore, the program instructions are used to perform the following steps when running:

[0122] Obtaining left-view images and right-view images of the lesion area;

[0123] Preprocessing the collected left-view image and right-view image to improve image quality and adaptability;

[0124] Perform Sobel filtering on the left view image and the right view image to produce a Sobel feature map;

[0125] Matching is performed based on the Sobel feature, and matching points with higher confidence are selected as support points;

[0126] Construct a Delaunay triangulation network for the support points to obtain the disparity information of the entire image;

[0127] Using the prior probability distribution and the image likelihood probability distribution to perform dense disparity estimation, a disparity map is obtained;

[0128] Eliminate noise in the disparity map by interpolation and adaptive median filtering, and retain the edge of the disparity map;

[0129] Calculate the depth information based on the view difference and generate a depth map, and then build a 3D model of the image based on the depth map;

[0130] A three-dimensional model of the image is constructed and displayed.

[0131] It should be understood that the present invention is described by some embodiments, and those skilled in the art will appreciate that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present invention. In addition, under the teachings of the present invention, these features and embodiments may be modified to adapt to specific circumstances and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited by the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the scope of protection of the present invention.

Claims

1. A 3D vision binocular stereo matching method, characterized in that: include: Filtering the first image and the second image to generate a feature map; Based on the feature maps of the first image and the second image, matching points with higher confidence are selected as support points; Constructing a local plane model for the support points to obtain disparity information of the entire first image; A disparity map of the second image is obtained through the disparity information of the first image.

2. The binocular stereo matching method according to claim 1, characterized in that: Based on the feature maps of the first image and the second image, matching points with higher confidence are selected as support points, including: First, a globally reliable disparity search range is set, and a 16-dimensional feature vector is constructed according to processed pixel points in the first image. Then, feature matching is performed to obtain a similarity measure value between the first image and the second image.

3. The binocular stereo matching method according to claim 2, characterized in that: Performing feature matching to obtain a similarity measure value between the first image and the second image includes: For each pixel in the first image, the feature vectors of the four corners in the 5x5 neighborhood of each pixel in the second image are used to match within the globally reliable disparity search range, and the similarity measure of the matching pairs is calculated. If the ratio of the minimum value and the second minimum value of the similarity metric of the matching pair is within the set range, the disparity value is taken as the disparity value of the candidate support point.

4. The binocular stereo matching method according to claim 4, characterized in that: After obtaining the disparity value, all support points are obtained by using the left-right consistency principle, and the disparity values ​​of the corresponding matching points in the first image and the second image that are equal and the number of other support points in the neighborhood around the support point are determined to screen the remaining matches again to obtain all support points; When performing left-right consistency detection, if the absolute value of the disparity of a pixel point of the first image and the disparity of a corresponding point of the second image are within a set range, the disparity value of the first image is considered reliable, otherwise it is set to -1 or other invalid values.

5. The binocular stereo matching method according to claim 1, characterized in that: Constructing a local plane model for the support points to obtain the disparity information of the entire first image includes: Perform Delaunay triangulation on the supporting point set in the first image to obtain a set of triangles, wherein the Delaunay triangulation ensures that the circumscribed circle of the triangle does not contain other feature points to generate a uniform triangular mesh, and for each triangle, calculate its corresponding disparity plane.

6. The binocular stereo matching method according to claim 1, characterized in that: Obtaining a disparity map of a second image through disparity information of a first image includes: Using the prior probability distribution and the image likelihood probability distribution, and performing stereo matching to obtain a disparity map of the second image; Among them, according to the probability generation model, given the first image and support points, sampling is performed from the second image, and the probability model is the maximum a posteriori probability estimation: in Indicates that the second image is located at For the observation points on the epipolar line, the posterior probability indicates that given the first image observation point, the support point set and the second image observation point, the first image observation point takes the disparity d n The probability of , the posterior probability can be decomposed into: Substitute the prior probability and likelihood probability into the above formula and take the negative logarithm to obtain an energy function: The first part of the energy function represents the likelihood term, and the second part is the prior term or smoothing term. By minimizing the energy function, the observation point is obtained. to obtain a disparity map of the second image.

7. The binocular stereo matching method according to claim 6, characterized in that: After obtaining the disparity map of the second image through the disparity information of the first image, the method further includes: Use adaptive median filtering to remove noise in the disparity map and retain the edges of the disparity map; Among them, in the process of stereo matching images, a pixel at a certain point in the disparity map needs to be set to an invalid value through interpolation based on a reasonable disparity value, and then interpolated in the horizontal and vertical directions. When the disparity values ​​of the leftmost and rightmost areas of an invalid area consider the depth mutation, an adaptive median filter is used to eliminate the noise in the disparity map to retain the edge of the disparity map.

8. A 3D vision endoscope system, characterized in that: include: Binocular imaging system: acquiring the first image and the second image of the lesion area; Image preprocessing system: used for processing the acquired first image and the second image to improve the quality and adaptability of the first image and the second image; Binocular stereo matching algorithm system: used to perform stereo matching on the first image and the second image to obtain the view difference; 3D reconstruction system: generates a depth map through view difference, and maps the information of the original image to the 3D mesh model for 3D information fusion to obtain a 3D model of the constructed image; Image display system: displays the three-dimensional model of the constructed image.

9. The endoscope system according to claim 8, characterized in that: Using the disparity map and the relationship between the disparity, depth and focal length, the depth of each pixel is calculated to generate a depth map, and each pixel value in the depth map is given to represent the distance from the point to the binocular imaging system. The depth value in the depth map is converted into a point in three-dimensional coordinates to obtain the three-dimensional coordinates of each pixel, and the color information of the original first image and the second image is mapped to the three-dimensional network model to obtain a three-dimensional model of the constructed image; Among them: the binocular imaging system uses two sets of endoscopic cameras at a certain distance to capture the first image and the second image of the lesion area respectively, and the two sets of endoscopic cameras have the same parameter models and work synchronously.

10. A storage medium having program instructions stored thereon, characterized in that: The program instructions are used to execute the 3D vision binocular stereo matching method as described in any one of claims 1 to 7 when running.

Images