3D electronic endoscope system based on binocular vision fusion

Abstract

The invention belongs to the field of medical equipment, and particularly relates to a binocular vision fusion-based 3D electronic endoscope system, which comprises a shell bearing module, a double-path image acquisition module, a signal transmission module, a digital signal processing module and a binocular vision construction module, the shell bearing module adopts a cylindrical shell structure and is used for accommodating and fixing related parts of the double-path image acquisition module and ensuring the stability and the position precision of each part in the working process; the double-path image acquisition module is composed of two groups of image acquisition units with the same structure, each group of image acquisition unit independently completes acquisition of one path of image, and the two groups of units work cooperatively to obtain binocular images with parallax; the system further comprises a signal transmission module, a digital signal processing module and a binocular vision construction module. The system is reasonable in design, real-time acquisition of a high-definition 3D image and accurate geometric information measurement are realized through binocular vision fusion, and the accuracy of operation diagnosis is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of medical equipment, and in particular to a 3D electronic endoscope system based on binocular vision fusion. Background Art

[0002] In the medical field, endoscopes are extremely important diagnostic and treatment tools. Traditional monocular endoscopes can only provide two-dimensional plane images. It is difficult for doctors to accurately obtain the depth information and precise size of the target object from these images, which to a certain extent limits the accuracy of diagnosis and the precision of surgical operations.

[0003] For example, monocular endoscopes have significant shortcomings in determining wound size, lesion area, and the spatial location of diseased tissue. While some 3D endoscope technologies are currently available, some suffer from complex structures, high costs, and suboptimal imaging quality, failing to meet clinical needs. Therefore, we propose a 3D electronic endoscope system based on binocular vision fusion to address these issues. Summary of the Invention

[0004] The purpose of the present invention is to solve the shortcomings of the prior art and to propose a 3D electronic endoscope system based on binocular vision fusion.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] A 3D electronic endoscope system based on binocular vision fusion includes a housing carrying module, a dual-channel image acquisition module, a signal transmission module, a digital signal processing module, and a binocular vision construction module;

[0007] The housing supporting module adopts a cylindrical housing structure, which is used to accommodate and fix the relevant components of the dual-channel image acquisition module to ensure the stability and position accuracy of each component during operation;

[0008] Dual-channel image acquisition module: It consists of two groups of image acquisition units with the same structure. Each group of image acquisition units independently completes the acquisition of one channel of images. The two groups of units work together to obtain binocular images with parallax.

[0009] Signal transmission module: responsible for stably and efficiently transmitting the digital image signals output by the dual-channel image acquisition module to the digital signal processing module, ensuring that the signals are not interfered with during transmission and that the integrity of the image information is guaranteed;

[0010] Digital signal processing module: performs a series of processing on the received digital image signals, including correction, feature extraction and matching, and calculation of three-dimensional coordinates and geometric information, providing data support for subsequent binocular vision construction;

[0011] Binocular vision construction module: based on the information output by the digital signal processing module, the mis-matching points are removed, the three-dimensional coordinates of the non-matching points are calculated, and finally the three-dimensional image of the target object is constructed.

[0012] Preferably, the image acquisition unit is subdivided into an illumination module, an optical imaging module, a photoelectric conversion module and an image preprocessing module, and the illumination module is composed of optical fibers, one end of which is connected with an external light source and the other end extends to the front end of the lens.

[0013] Optical imaging module: mainly composed of a lens, which is composed of high-precision optical lenses and can accurately image the target object and clearly focus the optical image of the target object on the photosensitive surface of the photoelectric conversion module.

[0014] Photoelectric conversion module: the core is a miniature high-definition image sensor, which can convert the optical signal transmitted by the optical imaging module into an electrical signal.

[0015] Image preprocessing module: it is an image acquisition module that receives the electrical signal from the photoelectric conversion module and performs preliminary processing, including noise reduction and A / D conversion to convert the analog electrical signal into a digital image signal.

[0016] Preferably, the digital signal processing module receives the digital image signal from the image acquisition unit, which is subdivided into an image correction module, a feature descriptor construction module, a feature matching module and a three-dimensional coordinate and geometric information calculation module.

[0017] Image correction module: according to the pre-calibrated parameters, the digital image signal is corrected for distortion and polar line correction to eliminate the distortion caused by the lens during image acquisition and improve the accuracy of the image.

[0018] Feature descriptor construction module: for a point in one of the two images, a local feature descriptor is constructed using its local feature information.

[0019] Three-dimensional coordinate and geometric information calculation module: the three-dimensional coordinates of the matching points in the world coordinate system are obtained by using the triangular geometry method.

[0020] Preferably, the distortion correction adopts a pinhole camera model, and for a point (u, v) in the image, the distortion correction formula is as follows:

[0021] Radial distortion correction

[0022] X=x(1+k1r 2 +k2r 4 +k3r 6 )

[0023] Y=y(1+k1r 2+k2r 4 +k3r 6 )

[0024] Among them, x, y are the normalized image coordinates before correction, r 2 =x 2 +y 2 , k1, k2, k3 are radial distortion coefficients, X, Y are coordinates after radial distortion;

[0025] Tangential distortion correction

[0026] X=x+[2p1xy+p2(r 2 +2x 2 )]

[0027] Y=y+[2p1xy+p2(r 2 +2x 2 )]

[0028] Among them, p1 and p2 are tangential distortion coefficients.

[0029] Preferably, the Euclidean distance calculation formula is:

[0030]

[0031] Among them, a i ,b i are the i-th components of the two feature descriptors respectively.

[0032] Preferably, let the projection matrix of the left camera be p l , the projection matrix of the right camera is p r , the matching point on the left image is xl=(ul,vl,1) T , the matching point on the right image is xr=(ur,vr,1) T , then the three-dimensional space point X=(X,Y,Z,W) T Satisfy xl=P l X and xr = P r X, the three-dimensional coordinate can be obtained by solving this system of equations;

[0033] And through multiple mutually matching points to measure the length, width, depth, area and volume, the geometric information of the target object is obtained, and the geometric information is output to the binocular vision construction module;

[0034] The length measurement formula is the Euclidean distance between two points:

[0035]

[0036] Among them, (X1, Y1, Z1) and (X2, Y2, Z2) are the three-dimensional coordinates of two points.

[0037] Preferably, the binocular vision construction module receives geometric information from the digital signal processing module, which is subdivided into a false matching point elimination module, an unmatching point three-dimensional coordinate calculation module and a three-dimensional image construction module.

[0038] False matching point elimination module: when part of one side image is blocked in the other side image, resulting in a matching point being a false matching point, the false matching point is eliminated through consistency check.

[0039] Unmatched point three-dimensional coordinate calculation module: for points that do not match successfully, the parallax value of the most adjacent matching point is used to calculate the three-dimensional coordinates by triangulation principle.

[0040] Three-dimensional image construction module: through the above processing, the three-dimensional coordinates of all points on the two side images are obtained, thereby realizing binocular vision and constructing a three-dimensional image of the target object.

[0041] Preferably, the re-projection error calculation formula is

[0042]

[0043] Where F is the fundamental matrix, x l ,x r is a matching point pair.

[0044] Preferably, the parallax d = ul-ur, the depth Where f is the camera focal length, B is the baseline distance, and the three-dimensional coordinates can be calculated according to the depth and image coordinates.

[0045] Advantages of the present application:

[0046] 1. Accurate three-dimensional information: the present application can obtain the geometric size information of the captured image and the three-dimensional image of the captured image in real time, providing accurate quantitative geometric size information of the lesion for the diagnosis and decision-making of the clinician. Compared with the prior art, not only the planar image information of the captured image can be provided, but also accurate quantitative geometric size information and three-dimensional image information can be provided, which helps the doctor to more accurately judge the disease and develop a treatment plan.

[0047] 2. Improve the accuracy of surgical operation: the doctor can obtain more intuitive and accurate three-dimensional images through the system during the operation, clearly understand the spatial position and structure of the diseased tissue, thereby improving the accuracy of the operation, reducing the risk of operation and reducing the damage to the surrounding normal tissue.

[0048] 3. The system structure is relatively simple, common image acquisition and processing components are used, the cost of the system is reduced under the premise of ensuring performance, and it is conducive to wide application in clinical practice. BRIEF DESCRIPTION OF DRAWINGS

[0049] Figure 1 This is a block diagram of the 3D electronic endoscope system based on binocular vision fusion proposed by the present invention;

[0050] Figure 2 This is a block diagram of the image acquisition unit of the 3D electronic endoscope system based on binocular vision fusion proposed by the present invention;

[0051] Figure 3 This is a block diagram of the data signal processing module of the 3D electronic endoscope system based on binocular vision fusion proposed by the present invention;

[0052] Figure 4 This is a block diagram of the binocular vision construction module of the 3D electronic endoscope system based on binocular vision fusion proposed in the present invention. DETAILED DESCRIPTION

[0053] The following is combined with Figure 1-4 This application is described in further detail.

[0054] The embodiments of the present application disclose a 3D electronic endoscope system based on binocular vision fusion.

[0055] Reference Figure 1-4 , a 3D electronic endoscope system based on binocular vision fusion, including a shell carrying module, a dual-channel image acquisition module, a signal transmission module, a digital signal processing module and a binocular vision construction module;

[0056] The housing supports the module: a cylindrical housing structure is used to accommodate and fix the relevant components of the dual-channel image acquisition module, ensuring the stability and position accuracy of each component during operation; the size of the cylindrical housing is designed according to the size of the internal components and the requirements of the medical use scenario, so that it has good insertion and operation convenience.

[0057] Dual-channel image acquisition module: It consists of two groups of image acquisition units with the same structure. Each group of image acquisition units independently completes the acquisition of one channel of images. The two groups of units work together to obtain binocular images with parallax.

[0058] Signal transmission module: responsible for stably and efficiently transmitting the digital image signals output by the dual-channel image acquisition module to the digital signal processing module, ensuring that the signals are not interfered with during transmission and that the integrity of the image information is guaranteed;

[0059] Digital signal processing module: performs a series of processing on the received digital image signals, including correction, feature extraction and matching, and calculation of three-dimensional coordinates and geometric information, providing data support for subsequent binocular vision construction;

[0060] Binocular vision construction module: based on the information output by the digital signal processing module, the mismatched points are removed, the three-dimensional coordinates of the unmatched points are calculated, and finally the three-dimensional image of the target object is constructed.

[0061] In this embodiment, the image acquisition unit is subdivided into an illumination module, an optical imaging module, a photoelectric conversion module and an image preprocessing module. The illumination module is composed of optical fibers, one end of which is connected to an external light source and the other end extends to the front end of the lens. Its function is to guide the light generated by the external light source to the target object efficiently and stably, to provide sufficient and uniform illumination for the optical imaging module, and to ensure that clear images can be acquired under different in-vivo environments. The optical fibers are made of materials with high light transmittance to reduce light loss.

[0062] Optical imaging module: mainly composed of a lens, which is composed of high-precision optical lenses and can accurately image the target object, focusing the optical image of the target object on the photosensitive surface of the photoelectric conversion module; the focal length, field of view and other parameters of the lens are optimized according to the requirements of the endoscope to meet the observation requirements of different parts.

[0063] Photoelectric conversion module: the core is a miniature high-definition image sensor, which can convert the optical signal from the optical imaging module into an electrical signal; the miniature high-definition image sensor uses high-resolution and high-sensitivity chips to ensure that it can capture the subtle features of the target object and provide high-quality raw signals for subsequent processing.

[0064] Image preprocessing module: it is an image acquisition module that receives the electrical signal from the photoelectric conversion module and performs preliminary processing, including noise reduction and A / D conversion to convert the analog electrical signal into a digital image signal; the signal output end of the image acquisition module is connected to the LVDS interface through LVDS wires to output the processed digital image signal to the digital signal processing module. The illumination module and the optical imaging module of the two image acquisition units are arranged on the cross section of the cylindrical shell. This layout can ensure that the two image acquisition units collect images of the target object from different angles, thereby obtaining binocular images with certain parallax, providing a basis for subsequent three-dimensional reconstruction.

[0065] In this embodiment, the digital signal processing module receives digital image signals from the image acquisition unit. This module is subdivided into an image correction module, a feature descriptor construction module, a feature matching module and a three-dimensional coordinate and geometric information calculation module.

[0066] Image correction module: according to the pre-calibrated parameters, the digital image signal is corrected for distortion and polar line to eliminate the distortion caused by the lens during image acquisition and improve the accuracy of the image.

[0067] Feature descriptor construction module: For a certain point in one side of the binocular image, a local feature descriptor is constructed using its local feature information. Using the SIFT feature descriptor, for each key point in the image, a 16×16 neighborhood is taken, which is divided into 4×4 sub-regions. For each sub-region, the gradient histogram in 8 directions is calculated to form a 128-dimensional feature vector as the local feature descriptor. For all points in the other side of the image that are in the same row as the point, a local feature descriptor is constructed using the same construction method.

[0068] Feature matching module: Calculates the Euclidean distance between the feature descriptors of the image points on both sides and selects the point with the minimum distance as the matching point in the other image.

[0069] Three-dimensional coordinate and geometric information calculation module: Use triangulation geometry to obtain the three-dimensional coordinates of the left and right matching points in the world coordinate system.

[0070] In this embodiment, the distortion correction adopts a pinhole camera model. For a point (u, v) in the image, the distortion correction formula is as follows:

[0071] Radial distortion correction

[0072] X=x(1+k1r 2 +k2r 4 +k3r 6 )

[0073] Y=y(1+k1r 2 +k2r 4 +k3r 6 )

[0074] Among them, x, y are the normalized image coordinates before correction, r 2 =x 2 +y 2 , k1, k2, k3 are radial distortion coefficients, X, Y are coordinates after radial distortion;

[0075] Tangential distortion correction

[0076] X=x+[2p1xy+p2(r 2 +2x 2 )]

[0077] Y=y+[2p1xy+p2(r 2 +2x 2 )]

[0078] Among them, p1 and p2 are tangential distortion coefficients.

[0079] The Euclidean distance calculation formula is:

[0080]

[0081] Among them, a i ,b i are the i-th components of the two feature descriptors respectively.

[0082] Let the projection matrix of the left camera be p l , the projection matrix of the right camera is p r , the matching point on the left image is xl=(ul,vl,1) T , the matching point on the right image is xr=(ur,vr,1) T , then the three-dimensional space point X=(X,Y,Z,W) T Satisfy xl=P l X and xr = P r X, the three-dimensional coordinate can be obtained by solving this system of equations;

[0083] And through multiple mutually matching points to measure the length, width, depth, area and volume, the geometric information of the target object is obtained, and the geometric information is output to the binocular vision construction module;

[0084] The length measurement formula is the Euclidean distance between two points:

[0085]

[0086] Among them, (X1, Y1, Z1) and (X2, Y2, Z2) are the three-dimensional coordinates of two points;.

[0087] In this embodiment, the binocular vision construction module receives geometric information from the digital signal processing module, and the module is subdivided into a mismatched point elimination module, an unmatched point three-dimensional coordinate calculation module, and a three-dimensional image construction module;

[0088] Mismatched point removal module: When a part of one image is obscured in the other image, resulting in a mismatched point, the mismatched point is removed through consistency checking. The consistency check uses the RANSAC algorithm, randomly selecting a certain number of matching point pairs to calculate the basic matrix, and then calculating the reprojection error of other matching point pairs relative to the basic matrix. Point pairs with reprojection errors less than a threshold are considered inliers. When the number of inliers reaches a certain ratio, the basic matrix is ​​considered valid and the outliers (mismatched points) are removed.

[0089] Unmatched point 3D coordinate calculation module: For unmatched points, the disparity value of the closest matched point is used to calculate the 3D coordinates using the triangulation principle;

[0090] 3D image construction module: Through the above processing, the 3D coordinates of all points on the images on both sides are obtained, thereby achieving binocular vision and constructing a 3D image of the target object.

[0091] The formula for calculating the reprojection error is:

[0092]

[0093] Among them, F is the basic matrix, x l ,x r is a matching point pair.

[0094] Parallax d = ul - ur, depth Where f is the focal length of the camera, B is the baseline distance, and the three-dimensional coordinates can be calculated based on the depth and image coordinates.

[0095] In the present invention, the geometric size information of the captured image and the three-dimensional image of the captured image can be obtained in real time, providing clinical doctors with accurate and quantitative geometric size information of the lesion for their diagnostic decisions. It can not only provide planar image information of the captured image, but also provide accurate and quantitative geometric size information and three-dimensional image information, which helps doctors to more accurately judge the condition and formulate treatment plans. During the operation, doctors can obtain more intuitive and accurate three-dimensional images through this system, and clearly understand the spatial position and structure of the diseased tissue, thereby improving the accuracy of the surgical operation, reducing the surgical risk, and reducing damage to the surrounding normal tissues.

[0096] The above are all preferred embodiments of the present application, and are not intended to limit the scope of protection of the present application. Therefore, any equivalent changes made based on the structure, shape, and principle of the present application should be included in the scope of protection of the present application.

Claims

1. A 3D electronic endoscope system based on binocular vision fusion, characterized in that: It includes a shell carrying module, a dual-channel image acquisition module, a signal transmission module, a digital signal processing module and a binocular vision construction module; The housing supporting module adopts a cylindrical housing structure, which is used to accommodate and fix the relevant components of the dual-channel image acquisition module to ensure the stability and position accuracy of each component during operation; Dual-channel image acquisition module: It consists of two groups of image acquisition units with the same structure. Each group of image acquisition units independently completes the acquisition of one channel of images. The two groups of units work together to obtain binocular images with parallax. Signal transmission module: responsible for stably and efficiently transmitting the digital image signals output by the dual-channel image acquisition module to the digital signal processing module, ensuring that the signals are not interfered with during transmission and that the integrity of the image information is guaranteed; Digital signal processing module: performs a series of processing on the received digital image signals, including correction, feature extraction and matching, and calculation of three-dimensional coordinates and geometric information, providing data support for subsequent binocular vision construction; Binocular vision construction module: Based on the information output by the digital signal processing module, it eliminates mismatched points, calculates the three-dimensional coordinates of unmatched points, and finally constructs a three-dimensional image of the target object.

2. The 3D electronic endoscope system based on binocular vision fusion according to claim 1 is characterized in that: The image acquisition unit is subdivided into an illumination module, an optical imaging module, a photoelectric conversion module and an image preprocessing module.

3. The 3D electronic endoscope system based on binocular vision fusion according to claim 2 is characterized in that: Lighting module: Made of optical fiber, one end of the optical fiber is connected to the external light source, and the other end extends to the front end of the lens; Optical imaging module: mainly consists of a lens, which is composed of a high-precision optical lens group. It can accurately image the target object and clearly focus the optical image of the target object onto the photosensitive surface of the photoelectric conversion module; Photoelectric conversion module: The core is a micro high-definition image sensor, which can convert the light signal transmitted by the optical imaging module into an electrical signal; Image preprocessing module: that is, the image acquisition module, which receives the electrical signal from the photoelectric conversion module and performs preliminary processing on it, including noise reduction and A / D conversion, and converts the analog electrical signal into a digital image signal.

4. The 3D electronic endoscope system based on binocular vision fusion according to claim 1, characterized in that: The digital signal processing module receives the digital image signal from the image acquisition unit. The module is subdivided into an image correction module, a feature descriptor construction module, a feature matching module, and a three-dimensional coordinate and geometric information calculation module; Image correction module: performs distortion correction and epipolar correction on digital image signals according to pre-calibrated parameters to eliminate distortion caused by lens factors during image acquisition and improve image accuracy; Feature descriptor construction module: For a certain point in one side of the binocular image, the local feature descriptor is constructed using its local feature information; Feature matching module: Calculates the Euclidean distance between the feature descriptors of the image points on both sides and selects the point with the minimum distance as the matching point in the other image. Three-dimensional coordinate and geometric information calculation module: Use triangulation geometry to obtain the three-dimensional coordinates of the left and right matching points in the world coordinate system.

5. The 3D electronic endoscope system based on binocular vision fusion according to claim 4 is characterized in that: Distortion correction uses a pinhole camera model. For a point (u, v) in the image, the distortion correction formula is as follows: Radial distortion correction X=x(1+k1r 2 +k2r 4 +k3r 6 ) Y=y(1+k1r 2 +k2r 4 +k3r 6 ) Among them, x, y are the normalized image coordinates before correction, r 2 =x 2 +y 2 , k1, k2, k3 are radial distortion coefficients, X, Y are coordinates after radial distortion; Tangential distortion correction X=x+[2p1xy+p2(r 2 +2x 2 )] Y=y+[2p1xy+p2(r 2 +2x 2 )] Among them, p1 and p2 are tangential distortion coefficients.

6. The 3D electronic endoscope system based on binocular vision fusion according to claim 4 is characterized in that: The Euclidean distance calculation formula is: Among them, a i ,b i are the i-th components of the two feature descriptors respectively.

7. The 3D electronic endoscope system based on binocular vision fusion according to claim 4, characterized in that: Let the projection matrix of the left camera be p l , the projection matrix of the right camera is p r , the matching point on the left image is xl=(ul,vl,1) T , the matching point on the right image is xr=(ur,vr,1) T , then the three-dimensional space point X=(X,Y,Z,W) T Satisfy xl=P l X and xr = P r X, the three-dimensional coordinate can be obtained by solving this system of equations; And through multiple mutually matching points to measure the length, width, depth, area and volume, the geometric information of the target object is obtained, and the geometric information is output to the binocular vision construction module; The length measurement formula is the Euclidean distance between two points: Among them, (X1, Y1, Z1) and (X2, Y2, Z2) are the three-dimensional coordinates of two points.

8. The 3D electronic endoscope system based on binocular vision fusion according to claim 1, characterized in that: The binocular vision construction module receives geometric information from the digital signal processing module, and the module is subdivided into a mismatched point elimination module, an unmatched point three-dimensional coordinate calculation module, and a three-dimensional image construction module; Mismatching point elimination module: When a part of one image is blocked in the other image, resulting in a mismatching point, the module eliminates the mismatching point through consistency checking. Unmatched point 3D coordinate calculation module: For unmatched points, the disparity value of the closest matched point is used to calculate the 3D coordinates using the triangulation principle; 3D image construction module: Through the above processing, the 3D coordinates of all points on the images on both sides are obtained, thereby achieving binocular vision and constructing a 3D image of the target object.

9. The 3D electronic endoscope system based on binocular vision fusion according to claim 7, characterized in that: The formula for calculating the reprojection error is: Among them, F is the basic matrix, x l ,x r is a matching point pair.

10. The 3D electronic endoscope system based on binocular vision fusion according to claim 7, characterized in that: Parallax d = ul - ur, depth Where f is the focal length of the camera, B is the baseline distance, and the three-dimensional coordinates can be calculated based on the depth and image coordinates.

Images