Accessory for endoscope, endoscope system, control method and device

Abstract

The present invention provides an accessory for an endoscope, an endoscope system, a control method, and a device. The accessory includes: a tube assembly, a liquid lens group, a wedge lens group, and a driver group; the proximal end of the tube assembly is used for detachable connection with the insertion end of the endoscope; the wedge lens group is arranged on the distal end of the tube assembly, the liquid lens group and the wedge lens are spaced apart, and the driver group is electrically connected to the liquid lens group; wherein the driver group is used to adjust the curvature of the liquid lens group to achieve focal length and magnification adjustment of the endoscope, and the wedge lens group is used to increase the imaging angle of the endoscope. The wedge lens group arranged at the front side of the insertion end of the endoscope can also effectively expand the range of the surgical field. Through the liquid lens and its corresponding driver group, rapid focusing can be achieved while freely switching the magnification, ensuring clear and continuous observation of the surgical field area.

Description

Technical Field

[0001] The present invention relates to the field of electronic technology, and in particular to an accessory for an endoscope, an endoscope system, a control method and a device. Background Art

[0002] With the continuous development of medical technology, endoscopes have been widely used in modern medicine.

[0003] Although the endoscope in the prior art can directly observe the surgical field, it cannot provide variable magnification and variable field of view during use, resulting in the doctor having to constantly move the endoscope according to the area of ​​interest.

[0004] Therefore, how to expand the field of view of the endoscope while achieving variable magnification has become an urgent problem to be solved in the industry. Summary of the Invention

[0005] The present invention provides an accessory for an endoscope, an endoscope system, a control method and a device, which are used to solve the defect in the prior art that the endoscope cannot achieve fast focusing during use.

[0006] The present invention provides an accessory for an endoscope, the accessory comprising: a tube assembly, a liquid lens assembly, a wedge lens assembly and a driver assembly;

[0007] The proximal end side of the tube assembly is adapted to be detachably connected to the insertion end of the endoscope;

[0008] The wedge mirror group is arranged at the distal end side of the tube assembly, the liquid lens group and the wedge mirror are spaced apart, and the driver group is electrically connected to the liquid lens group;

[0009] The driver group is used to adjust the curvature of the liquid lens group to adjust the focal length and magnification of the endoscope, and the wedge lens group is used to increase the imaging angle of the endoscope.

[0010] According to an accessory for an endoscope provided by the present invention, the liquid lens assembly includes a first liquid lens and a second liquid lens, and the driver assembly includes a first driver and a second driver;

[0011] The first driver is electrically connected to the first liquid lens, and the second driver is electrically connected to the second liquid lens;

[0012] The first liquid lens is used for adjusting the focal length, and the second liquid lens is used for adjusting the magnification.

[0013] According to the present invention, an accessory for an endoscope further comprises: a micromotor;

[0014] The micromotor is connected to the wedge-shaped mirror group, and the micromotor is used to control the wedge-shaped mirror group to adjust the tilt angle.

[0015] According to the present invention, an accessory for an endoscope further comprises: a concave lens group and a convex lens group;

[0016] The concave lens group is arranged between the proximal end side of the tube assembly and the liquid lens group, and the convex lens group is arranged between the concave lens group and the wedge lens group;

[0017] The concave lens group is used to eliminate chromatic aberration, and the convex lens group is used to focus light.

[0018] According to the present invention, an accessory for an endoscope further includes: a first input unit and a second input unit, both of which are connected to the liquid lens assembly;

[0019] Wherein, the first input unit is used to adjust the curvature of the liquid lens assembly in response to the magnification increase input to magnify the imaging of the endoscope;

[0020] The second input unit is used to respond to a reduction magnification input and adjust the curvature of the liquid lens assembly to reduce the imaging of the endoscope.

[0021] Optionally, the liquid lens assembly further includes a third liquid lens, and the driver assembly further includes a third driver;

[0022] The third driver is electrically connected to the third liquid lens;

[0023] The third liquid lens and the first liquid lens are used together for focal length adjustment.

[0024] The present invention also provides an endoscope system, comprising: an endoscope and the above-mentioned accessories for the endoscope.

[0025] The present invention also provides a control method based on the above-mentioned endoscope system, comprising:

[0026] Perform clarity detection on the edge area of ​​endoscopic imaging;

[0027] When the definition of the pixels in the edge area is lower than a preset threshold, the driving voltage of the driver group is searched and approximated within a preset voltage range by a golden ratio segmentation method to obtain a target driving voltage of the driver group, wherein the target driving voltage is the driving voltage corresponding to the maximum definition of the pixels in the edge area;

[0028] The curvature of the liquid lens assembly is adjusted based on the target driving voltage to form an image according to the adjusted liquid lens assembly.

[0029] According to a control method based on the above-mentioned endoscope system provided by the present invention, the clarity detection of the edge area of ​​the endoscope imaging includes:

[0030] When the endoscope detects a change in the object distance, performing a clarity detection on an edge area of ​​the endoscope imaging;

[0031] Or; when the magnification of the liquid lens assembly changes, the clarity of the edge area of ​​the endoscope imaging is detected.

[0032] Optionally, before adjusting the curvature of the liquid lens assembly based on the target driving voltage to form an image according to the adjusted liquid lens assembly, the method further includes:

[0033] Acquire multiple image groups at different wedge mirror angles collected during a process of searching and approximating the driving voltage of the driver group within a preset voltage range, wherein each image group includes at least two images at different magnifications;

[0034] Perform SIFT feature point detection and matching on each of the images to determine the translation vector and rotation vector between the images;

[0035] Each group of the images is spliced ​​based on the translation vector and the rotation direction to obtain a target image.

[0036] Optionally, after the step of determining the translation vector and the rotation vector between the images, the method further comprises:

[0037] constructing a first depth map based on the translation vector and the rotation vector;

[0038] Performing focus edge detection on each of the images using a Laplacian of Gaussian kernel to obtain a second depth map, wherein a corresponding depth value of each pixel in the second depth map is determined based on a layer where a norm of the Laplacian of Gaussian kernel is maximum;

[0039] performing normalization processing on the second depth map and performing depth optimization processing on the second depth map to obtain a third depth map;

[0040] Adding and averaging the first depth map and the third depth map to obtain a fourth fused depth map;

[0041] Performing splicing processing on the fourth fused depth map based on the translation vector and the rotation vector to obtain a target fused map;

[0042] A target three-dimensional reconstruction model is determined based on the target fused depth map and the three-dimensional model texture coverage information, wherein the three-dimensional model texture coverage information is determined based on imaging of the adjusted liquid lens group.

[0043] The present invention also provides a control device, comprising:

[0044] A clarity measurement module is used to detect the clarity of the edge area of ​​endoscopic imaging;

[0045] an extreme value search module, which compares the contrast between the current image and the previous frame image and uses the golden ratio segmentation method to search and approximate the driving voltage of the driver group within a preset voltage range to obtain a target driving voltage of the driver group, wherein the target driving voltage is the driving voltage corresponding to the highest definition of the pixel points in the edge area;

[0046] The adjustment module is used to adjust the curvature of the liquid lens group based on the target driving voltage so as to form an image according to the adjusted liquid lens group.

[0047] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any of the control methods described above when executing the program.

[0048] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which implements the steps of any of the control methods described above when executed by a processor.

[0049] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the steps of any one of the control methods described above are implemented.

[0050] The present invention provides an accessory for an endoscope, an endoscope system, a control method and a device. The wedge-shaped mirror group arranged on the front side of the insertion end of the endoscope can also effectively expand the range of the surgical field. Through the liquid lens and its corresponding driver group, rapid focusing can be achieved while freely switching the magnification, ensuring clear and continuous observation of the surgical field area. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0052] Figure 1This is a schematic structural diagram of an endoscope accessory provided in an embodiment of the present application;

[0053] Figure 2 This is one of the schematic diagrams of the endoscope system structure provided in an embodiment of the present application;

[0054] Figure 3 This is the second structural diagram of the endoscope system described in the embodiment of the present application;

[0055] Figure 4 A flow chart of the control method provided in the embodiment of the present application;

[0056] Figure 5 A schematic diagram of the control device structure provided in an embodiment of the present application;

[0057] Figure 6 The following is a schematic diagram of the physical structure of an electronic device. DETAILED DESCRIPTION

[0058] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0059] Figure 1 This is a schematic diagram of the structure of the accessories for the endoscope provided in the embodiment of the present application, such as Figure 1 As shown, it includes: a tube assembly 1, a liquid lens assembly 2, a wedge mirror assembly 3 and a driver assembly 4;

[0060] The proximal side 11 of the tube assembly 1 is used for detachable connection with the insertion end of the endoscope;

[0061] The wedge mirror group 3 is arranged at the distal end of the tube assembly 1, the liquid lens group 2 is spaced apart from the wedge mirror 3, and the driver group 4 is electrically connected to the liquid lens group 2;

[0062] The driver group 4 is used to adjust the curvature of the liquid lens group 2 to adjust the focal length of the endoscope, and the wedge lens group 3 is used to increase the imaging angle of the endoscope.

[0063] Specifically, the proximal and distal sides of the tube assembly described in the embodiment of the present application should both have openings, and the interior of the tube assembly should be through-holes. The tubular diameter of the tube assembly should be slightly larger than the insertion end of the endoscope, so that the endoscope can be inserted from the proximal side of the tube assembly.

[0064] The proximal end of the tube assembly described in the embodiment of the present application is the section connected to the endoscope during use, and the other end of the tube assembly is the distal end.

[0065] In an embodiment of the present application, the proximal side of the tube assembly is used for detachable connection with the insertion end of the endoscope, that is, the endoscope can be connected to the accessory through the proximal side of the tube assembly, and the endoscope can also be separated from the accessory after connection.

[0066] The wedge mirror assembly described in the embodiment of the present application can be composed of one or more wedge mirrors. A single wedge mirror can deflect the light θ d For example, the wedge mirror used in the wedge mirror assembly in the embodiment of the present application has a cutting angle of 18.8° and a deflection angle of 10°.

[0067] Optionally, a single wedge mirror can deflect the light by θ d , and the two wedge mirrors can deflect the light by 4θ d For example, the wedge mirror used in the wedge mirror assembly in the embodiment of the present application has a cutting angle of 18.8° and a deflection angle of 10°, which can achieve a 40° deflection of light.

[0068] The wedge-shaped mirror assembly can be specifically arranged at the distal end side of the tube assembly, so that the wedge-shaped mirror assembly here can deflect light, thereby helping the endoscope to obtain a larger field of view.

[0069] The driver group described in the embodiments of the present application can be specifically arranged outside the tube assembly, and can be specifically connected to the computer via a USB cable and a CMOS cable placed in parallel at the rear, thereby reducing the impact on the volume of the accessory. There may be one or more drivers in the driver group, each of which can be connected to one or more liquid lenses and output different driving voltages to the liquid lens according to corresponding specifications to adjust its curvature.

[0070] The liquid lens assembly described in the embodiments of the present application may specifically include one liquid lens or multiple liquid lenses, each of which is connected to one or more drivers in the driver assembly. The driver transmits a driving voltage to the liquid lens, and the curvature of the liquid lens is adjusted by different driving voltages, so that the liquid lens assembly can effectively adjust the focal length and magnification of the endoscope.

[0071] Specifically, the front end of the outer end side of the tube assembly described in the embodiment of the present application is provided with a lighting device, which can be an LED lamp or other lighting device. At the same time, the setting of the lighting device will not hinder the light acquisition of the wedge-shaped lens group.

[0072] In an embodiment of the present application, the range of the surgical field can be effectively expanded by using a wedge-shaped mirror group arranged on the front side of the insertion end of the endoscope. Through the liquid lens and its corresponding driver group, rapid focusing can be achieved while freely switching the magnification, thereby ensuring clear and continuous observation of the surgical field area.

[0073] Optionally, the liquid lens assembly includes a first liquid lens and a second liquid lens, and the driver assembly includes a first driver and a second driver;

[0074] The first driver is electrically connected to the first liquid lens, and the second driver is electrically connected to the second liquid lens;

[0075] The first liquid lens is used for adjusting the focal length, and the second liquid lens is used for adjusting the magnification.

[0076] Specifically, the liquid lens assembly described in the embodiments of the present application may include two small-aperture liquid lenses, which are a first liquid lens for focusing and a second liquid lens for magnification adjustment.

[0077] And correspondingly, the first driver is electrically connected to the first liquid lens, and the second driver is electrically connected to the second liquid lens.

[0078] The first driver outputs a driving voltage to the first liquid lens, thereby achieving focal length adjustment. That is, the first driver outputs different driving voltages to correspondingly adjust the curvature of the first liquid lens, thereby continuously adjusting the focal length of the endoscope.

[0079] The second driver outputs a driving voltage to the second liquid lens, thereby achieving adjustment of the magnification. That is, the second driver outputs different driving voltages to adjust the curvature of the second liquid lens, thereby continuously adjusting the magnification of the endoscope.

[0080] In the embodiment of the present application, the first liquid lens and the second liquid lens can effectively achieve the change of viewing angle and synchronous focusing of the object photographed by the endoscope.

[0081] Optionally, the accessory further comprises: a micromotor;

[0082] The micromotor is connected to the wedge-shaped mirror group, and the micromotor is used to control the wedge-shaped mirror group to adjust the tilt angle.

[0083] Specifically, the micro motor described in the embodiment of the present application can be electrically connected to each wedge mirror in the wedge mirror group, and the angle of each wedge mirror in the wedge mirror group is controlled by the micro motor. The addition of the wedge mirror can meet the change of the incident angle of the light to expand the viewing angle of the surgical field. A single wedge mirror can deflect the light by θ d, and the two wedge mirrors can deflect the light by 4θ d .

[0084] For example, the wedge mirror used in the wedge mirror assembly in the embodiment of the present application has a cutting angle of 18.8° and a deflection angle of 10°, which can achieve a 40° deflection of light.

[0085] In the embodiment of the present application, the micromotor is controlled to control the tilt angle of the middle wedge mirror of the wedge mirror group, so as to meet the change of the incident angle of the endoscope light and provide the endoscope with various wide fields of view.

[0086] The motor drives the wedge mirror to rotate and capture multiple images at different viewing angles. The focal length parameters of the liquid lens are read when each image is captured, and the corresponding camera intrinsic parameter matrix is ​​obtained as follows:

[0087]

[0088] Where f is the focal length, dx is the horizontal pixel size, dy is the vertical pixel size, (u o , v o ) is the intersection of the optical axis and the camera sensor.

[0089] By performing SIFT-based feature point detection and matching on the two images, we can obtain the homogeneous coordinates of the corresponding feature points in the two images (x1, y1, 1) and (x2, y2, 1). According to the epipolar geometry relationship, we can obtain x′Fx=0, where x′ and x correspond to the coordinates of the feature points in the two images. According to the equation, the basic parameter value F of the camera can be solved.

[0090] According to the camera intrinsic parameter matrices K1 and K2 corresponding to the two images obtained above and the basic parameter value F, according to the following formula

[0091] E=K′2*F*K1

[0092] We can obtain the essential matrix E and perform singular value decomposition on it to obtain E = U * Λ * V′, where the third column of U is the translation vector between the two images, and U * W * V′ is the image rotation vector. Based on the translation and rotation vectors between the two images, the two images can be stitched together. Four images are taken at different angles and stitched together.

[0093] Optionally, the accessory further comprises: a concave lens group and a convex lens group;

[0094] The concave lens group is arranged between the proximal end side of the tube assembly and the liquid lens group, and the convex lens group is arranged between the concave lens group and the wedge lens group;

[0095] The concave lens group is used to eliminate chromatic aberration, and the convex lens group is used to focus light.

[0096] Specifically, in the accessories described in the embodiments of the present application, there may be certain chromatic aberrations due to the use of multiple different lenses. Therefore, the embodiments of the present application also consider designing a concave lens group between the proximal side of the tube assembly and the liquid lens group to eliminate chromatic aberrations.

[0097] The concave lens group may specifically include one or more concave lenses with a fixed focal length. The concave lens group should be arranged close to the interface of the endoscope to ensure that the light finally incident on the endoscope has its chromatic aberration effectively eliminated.

[0098] In an embodiment of the present application, in order to control the comprehensive focal length range, a convex lens group is also provided between the concave lens group and the wedge-shaped lens group. The convex lens group may contain one or more convex lenses. In addition, when there are multiple convex lenses in the concave lens group, the convex lens group does not have to be arranged together. For example, when there are two convex lenses in the convex lens group, they can be respectively arranged in the front and rear positions of the liquid lens group.

[0099] In the embodiment of the present application, the concave lens group and the convex lens group can effectively eliminate chromatic aberration and achieve focal length control, thereby helping the endoscope to achieve better imaging.

[0100] Optionally, the accessory further comprises: a first input unit and a second input unit, both of which are connected to the liquid lens assembly;

[0101] Wherein, the first input unit is used to adjust the curvature of the liquid lens assembly in response to the magnification increase input to magnify the imaging of the endoscope;

[0102] The second input unit is used to respond to a reduction magnification input and adjust the curvature of the liquid lens assembly to reduce the imaging of the endoscope.

[0103] Specifically, the first input unit and the second input unit described in the embodiment of the present application can be two input buttons. When the user presses the first input unit, it is equivalent to receiving an input to increase the magnification. When the user presses the second input unit, it is equivalent to receiving an input to decrease the magnification.

[0104] For example, in the embodiment of the present application, the focal length and magnification are set in advance according to the commonly used working distance (100mm) of minimally invasive abdominal surgery. The first of the two liquid lenses is used to adjust the focal length, and the second liquid lens is used to adjust the magnification. The voltage value of the first liquid lens in the first magnification working mode is set to 24V, the working voltage of the second liquid lens is set to 66.52V, and the magnification is 1.44. The voltage value of the second liquid lens in the second magnification working mode is set to 59.68V, the working voltage of the second liquid lens is set to 24V, and the magnification is 0.68. Two buttons are provided on the top of the integrated endoscope to facilitate quick switching between the above two magnification working modes during use.

[0105] In the embodiment of the present application, through the first input unit and the second input unit, the user can conveniently and quickly manually adjust the magnification of the endoscope, thereby quickly obtaining the magnification the user wants.

[0106] Optionally, the liquid lens assembly further includes a third liquid lens, and the driver assembly further includes a third driver;

[0107] The third driver is electrically connected to the third liquid lens;

[0108] The third liquid lens and the first liquid lens are used together for focal length adjustment.

[0109] As another optional embodiment, the liquid lens assembly described in the embodiment of the present application may also include three liquid lenses, which can have a larger magnification range and working distance based on two liquid lenses.

[0110] As an optional embodiment, the liquid lens assembly described in the embodiment of the present application may also use only one liquid lens, which can realize the automatic focusing function and solve the problem that traditional endoscopes will be out of focus when the working distance is less than 30 mm.

[0111] Figure 2 This is one of the schematic diagrams of the endoscope system structure provided in the embodiment of the present application, such as Figure 2 As shown, it includes an endoscope 5 and a tube assembly, and the inner ends of the endoscope 5 and the tube assembly are detachably connected.

[0112] Figure 3 This is the second structural diagram of the endoscope system described in the embodiment of the present application, as shown in FIG. Figure 3As shown, the endoscope 5 comprises two small-aperture liquid lenses 31, two fixed-focal-length convex lenses 33, two fixed-focal-length concave lenses 34, and two wedge-shaped mirrors 35. The endoscope 5 is composed of a CMOS and a fixed-focal-length convex lens. The first liquid lens is used for focusing, and the second liquid lens is used for switching the magnification to achieve a change in the viewing angle and synchronous focusing of the object under the endoscope. The fixed-focal-length convex lens is used to control the integrated focal length, and the concave lens is used to eliminate the chromatic aberration generated by the system. The wedge-shaped mirror is connected to a micro-motor 36 for changing the working tilt angle to expand the surgical field. The system calibrates the magnification of the two working modes to achieve imaging under magnified and reduced viewing angles, which can be quickly switched through two buttons 32 on the housing.

[0113] Figure 4 A flow chart of the control method provided in the embodiment of the present application is shown in FIG. Figure 4 As shown, including:

[0114] Step 410 , performing clarity detection on the edge area of ​​the endoscope imaging;

[0115] The endoscopic imaging described in the embodiment of the present application refers to the image currently taken by the endoscope, and edge detection is performed on the edge area of ​​the endoscopic imaging to determine the clarity of the pixels in the edge area, thereby determining whether the endoscopic imaging is blurred.

[0116] Specifically, if blur occurs during shooting, the image will be affected by the blur radius, as shown in the following formula:

[0117]

[0118] Where c is the blur radius, f0 is the current focal length, d is the depth value under the focus state corresponding to the focal length, and d f is the actual depth value, and N is the aperture value.

[0119] The edge detection Sobel operator is used to perform real-time edge detection on the region of interest in the captured image. The Sobel operator performs convolution on the x-axis and y-axis respectively. The convolution kernel is as follows:

[0120]

[0121] The operation of these two convolution kernels can obtain the partial derivatives in the horizontal and vertical directions. Finally, the clarity of each pixel is measured by the square difference between the two:

[0122]

[0123] Step 420: Comparing the contrast between the current image and the previous image and using the golden ratio segmentation method to search and approximate the driving voltage of the driver group within a preset voltage range, thereby obtaining a target driving voltage of the driver group, wherein the target driving voltage is the driving voltage corresponding to the highest definition of the pixel points in the edge area;

[0124] Since the image clarity and focal length are in a Gaussian distribution curve relationship at the same object distance, the detected edges are searched for extreme values ​​through algorithm iteration.

[0125] Since the object distance remains unchanged and the magnification is not adjusted, it is only necessary to determine the target driving voltage of the liquid lens in the liquid lens group that performs focusing.

[0126] The search approximation is performed using the golden ratio segmentation method. In each iteration, the algorithm compares the sharpness at positions 0.38 and 0.62 within the interval containing two voltage values. Depending on which of these two points has the best sharpness, the new interval is reduced to the left or right by 0.38 of the previous interval. In each iteration, the span is reduced to a known amount (0.62), so that the convergence criterion is fixed: regardless of the position of the starting point and the target focus, once the depth of field is fixed, the number of iterations is also fixed. The specific process of the algorithm is as follows: In the first iteration, the Sobel operator sharpness is calculated at the voltage values ​​of 0.618 and 0.38 between the maximum and minimum, and the larger value between the two is used as the boundary point for the next round. In the second and subsequent iterations, the voltage value of the previous step is used as the maximum, and the Sobel operator sharpness is repeatedly calculated at the voltage values ​​of 0.382 and 0.618 between the maximum and minimum until the sharpness search range is less than 0.02V, at which time the search is stopped. After n iterations, the error value is:

[0127] Err=(1-R) (n-1) *(Max-Min) (4)

[0128] Where Max is the maximum adjustable diopter of the liquid lens, Min is the minimum adjustable diopter of the liquid lens, n is the total number of iterations, and R is the error value of each iteration.

[0129] The voltage value of the last iteration is used as the target driving voltage of the liquid lens group. The liquid lens group forms a continuous focal length variation range by achieving different voltage combination values ​​of the two ends of the calibration. The combined focal length between the target driving voltages can be obtained by the following formula:

[0130]

[0131] Where A is the distance between the first liquid lens and the object, B is the distance between the two liquid lenses, and C is the distance between the second liquid lens and the imaging film. is the ratio of the image distance to the object distance of the second liquid lens, is the ratio of the image distance to the object distance of the first liquid lens.

[0132] Because data on commonly used magnifications during endoscope use is limited, we can pre-generate voltage combinations corresponding to each magnification and store them in a table. During use, the table can be looked up to obtain the corresponding voltage values ​​for the two liquid lenses for the desired combination. The entire process typically requires 8-12 images to achieve real-time, high-speed focusing of organs and instruments during surgery.

[0133] Step 430 : Adjusting the curvature of the liquid lens assembly based on the target driving voltage to form an image according to the adjusted liquid lens assembly.

[0134] Specifically, after the curvature of the liquid filter is adjusted, the endoscope can now focus better, thereby obtaining a sharper image.

[0135] In an embodiment of the present application, the observer can clearly and continuously observe the surgical field and stereoscopic superimposed images within a larger field of view in a working mode in which the magnification can be freely switched. The automatic focusing endoscope three-dimensional reconstruction system based on the liquid lens in the present invention has the advantages of a wide observation range, a clear field of view, and can perform accurate measurement and three-dimensional reconstruction according to different needs, and can also realize a lightweight and miniaturized system.

[0136] Optionally, the performing clarity detection on the edge area of ​​the endoscopic imaging includes:

[0137] When the endoscope detects a change in the object distance, performing a clarity detection on an edge area of ​​the endoscope imaging;

[0138] Or; when the magnification of the liquid lens assembly changes, the clarity of the edge area of ​​the endoscope imaging is detected.

[0139] Specifically, the object distance described in the embodiments of the present application changes, which may be that a new target object appears in the field of view of the endoscope, or that the object distance between the target object and the lens changes significantly.

[0140] Optionally, when the magnification of the liquid lens assembly changes significantly, the endoscopic imaging may become blurred and refocusing is also required.

[0141] In an embodiment of the present application, when the endoscope detects a change in object distance or a change in magnification, the edge area of ​​the endoscope imaging is re-detected for clarity to refocus and ensure that the endoscope can continue to acquire clear images.

[0142] Optionally, in an embodiment of the present application, the actual length of the target object captured during endoscope operation can also be estimated. When the curvature of the liquid lens assembly is adjusted based on the target drive voltage, the clearest point in this state is obtained. The combined focal length of this point and the liquid lens assembly is used as input. According to the triangulation principle of the optical system, it can be concluded that the ratio of the object length in the image to the time length is only the inverse of the pixel density of the camera sensor:

[0143]

[0144] where d 3D is the actual length of the object, d 2D is the length of the object in the image, α is the pixel density of the camera sensor, d0 is the depth value in the current focus mode, and f is the focal length parameter of the camera. From this relationship, the actual length of the two points in the image can be obtained.

[0145] In the embodiment of the present application, the triangulation principle of the optical system can be used to effectively help observe and understand the real information of the target object.

[0146] Optionally, a plurality of image groups at different wedge mirror angles collected during a process of searching and approximating the driving voltage of the driver group within a preset voltage range are acquired, wherein each of the image groups includes at least two images at different magnifications;

[0147] Perform SIFT feature point detection and matching on each of the images to determine the translation vector and rotation vector between the images;

[0148] Each group of the images is spliced ​​based on the translation vector and the rotation direction to obtain a target image.

[0149] The motor drives the wedge mirror to rotate and capture multiple images at different viewing angles. The focal length parameters of the liquid lens are read when each image is captured, and the corresponding camera intrinsic parameter matrix is ​​obtained as follows:

[0150]

[0151] Where f is the focal length, dx is the horizontal pixel size, dy is the vertical pixel size, (u o , v o ) is the intersection of the optical axis and the camera sensor.

[0152] By performing SIFT-based feature point detection and matching on the two images, we can obtain the homogeneous coordinates of the corresponding feature points in the two images (x1, y1, 1) and (x2, y2, 1). According to the epipolar geometry relationship, we can obtain x′Fx=0, where x′ and x correspond to the coordinates of the feature points in the two images. According to the equation, the basic parameter value F of the camera can be solved.

[0153] According to the camera intrinsic parameter matrices K1 and K2 corresponding to the two images obtained above and the basic parameter value F, according to the following formula

[0154] E=K′2*F*K1

[0155] The essential matrix E is obtained, and then subjected to singular value decomposition (SVD) to obtain E = U * Λ * V′, where the third column of U is the translation vector between the two images, and U * W * V′ is the image rotation vector. Based on the translation and rotation vectors between the two images, the two images can be stitched together. Simultaneously, based on the definition of the projection matrix, the final 3D coordinates of the feature matching point pairs can be calculated. By sequentially calculating the 3D coordinates of each image according to the above steps and stitching their 3D point clouds together, the corresponding wide-view 3D image can be obtained.

[0156] Optionally, after the step of determining the translation vector and the rotation vector between the images, the method further comprises:

[0157] constructing a first depth map based on the translation vector and the rotation vector;

[0158] Performing focus edge detection on each of the images using a Laplacian of Gaussian kernel to obtain a second depth map, wherein a corresponding depth value of each pixel in the second depth map is determined based on a layer where a norm of the Laplacian of Gaussian kernel is maximum;

[0159] performing normalization processing on the second depth map and performing depth optimization processing on the second depth map to obtain a third depth map;

[0160] Adding and averaging the first depth map and the third depth map to obtain a fourth fused depth map;

[0161] Performing splicing processing on the fourth fused depth map based on the translation vector and the rotation vector to obtain a target fused map;

[0162] A target three-dimensional reconstruction model is determined based on the target fused depth map and the three-dimensional model texture coverage information, wherein the three-dimensional model texture coverage information is determined based on imaging of the adjusted liquid lens group.

[0163] During autofocusing, the multiple images collected during the focusing process are images of the same object at different focal lengths. The degree of defocus in this sequence of images varies. The algorithm implementation in the depth estimation and 3D reconstruction module is as follows: Based on the known focal length value, the camera intrinsic parameter matrix corresponding to the two images can be obtained as follows:

[0164]

[0165] Where f is the focal length, dx is the horizontal pixel size, dy is the vertical pixel size, (u o , v o ) is the intersection of the optical axis and the camera sensor.

[0166] By performing SIFT-based feature point detection and matching on the two images, we can obtain the homogeneous coordinates of the corresponding feature points in the two images (x1, y1, 1) and (x2, y2, 1). According to the epipolar geometry relationship, we can obtain x′Fx=0, where x′ and x correspond to the coordinates of the feature points in the two images. According to the equation, the basic parameter value F of the camera can be solved.

[0167] According to the camera intrinsic parameter matrices K1 and K2 corresponding to the two images obtained above and the basic parameter value F, according to the following formula

[0168] E=K′2*F*K1

[0169] We can obtain the essential matrix E and perform singular value decomposition on it to obtain E = U * Λ * V', where the third column of U is the translation vector between the two images, and U * W * V' is the image rotation vector. Based on the translation and rotation vectors between the two images, the two images can be registered. At the same time, according to the definition of the projection matrix, the final 3D coordinates of the feature matching point pairs and the corresponding first depth map can be calculated.

[0170] The image is focused and edge detected according to the Laplace Gaussian kernel to calculate the focus degree f m The implementation formula is as follows:

[0171]

[0172] Where LOG is the Laplacian of Gaussian kernel, i(x, y) represents the image contrast, N represents the window size, and (i, j) represents the pixel currently being calculated. Based on the above calculation results, the norm is calculated. The layer with the maximum norm for each pixel (i, j) is the depth layer of that point. Normalization yields a depth map corresponding to the entire image. This depth map is then subjected to depth optimization operations such as hole filling and mean filtering to generate a second depth map.

[0173] The first depth map and the second depth map are added and averaged to obtain a target fused depth map; the fused depth map is spliced ​​according to the rotation and translation matrices between the images at different angles of the wedge mirror to obtain a fused depth map with a larger viewing angle;

[0174] After the autofocus module reaches its sharpest point, the image captured at this voltage value is considered fully focused. This image serves as the texture input for depth estimation and 3D reconstruction. 3D reconstruction is performed based on this depth map, and the fully focused image is overlaid with the texture to produce a 3D reconstructed model of the target object.

[0175] Figure 5 A schematic diagram of the control device structure provided in the embodiment of the present application is shown in FIG. Figure 5 As shown, including:

[0176] The clarity measurement module 510 is used to detect the clarity of the edge area of ​​the endoscope imaging;

[0177] The extreme value search module 520 is configured to compare the contrast between the current image and the previous frame image and use the golden ratio segmentation method to search and approximate the driving voltage of the driver group within a preset voltage range to obtain a target driving voltage of the driver group, wherein the target driving voltage is the driving voltage corresponding to the highest definition of the pixel points in the edge area;

[0178] The adjustment module 530 is configured to adjust the curvature of the liquid lens assembly based on the target driving voltage, so as to form an image according to the adjusted liquid lens assembly.

[0179] In an embodiment of the present application, the range of the surgical field can be effectively expanded by using a wedge-shaped mirror group arranged on the front side of the insertion end of the endoscope. Through the liquid lens and its corresponding driver group, rapid focusing can be achieved while freely switching the magnification, thereby ensuring clear and continuous observation of the surgical field area.

[0180] Figure 6 An example of a physical structure diagram of an electronic device is shown below. Figure 6As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communication interface 620, and the memory 630 communicate with each other via the communication bus 640. The processor 610 may call the logic instructions in the memory 630 to execute a control method, which includes: performing clarity detection on the edge area of ​​the endoscope imaging; searching and approximating the driving voltage of the driver group within a preset voltage range by comparing the contrast strength of the current image and the previous frame image and using the golden ratio segmentation method to obtain a target driving voltage of the driver group, wherein the target driving voltage is the driving voltage corresponding to the highest clarity of the pixel points in the edge area; and adjusting the curvature of the liquid lens group based on the target driving voltage to form an image according to the adjusted liquid lens group.

[0181] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0182] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the control methods provided by the above methods, the method including: when the clarity of the pixel points in the edge area is lower than a preset threshold, the driving voltage of the driver group is searched and approximated within a preset voltage range by using the golden ratio segmentation method to obtain a target driving voltage of the driver group, wherein the target driving voltage is the driving voltage corresponding to the highest clarity of the pixel points in the edge area; and adjusting the curvature of the liquid lens group based on the target driving voltage to form an image according to the adjusted liquid lens group.

[0183] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the control method provided by the above-mentioned methods, the method comprising: when the clarity of the pixel points in the edge area is lower than a preset threshold, searching and approximating the driving voltage of the driver group within a preset voltage range by a golden ratio segmentation method to obtain a target driving voltage of the driver group, wherein the target driving voltage is the driving voltage corresponding to when the clarity of the pixel points in the edge area is the highest; adjusting the curvature of the liquid lens group based on the target driving voltage to form an image according to the adjusted liquid lens group.

[0184] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0185] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, or of course, by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the existing technology can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or certain parts of the embodiments.

[0186] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. An endoscope system, characterized in that: include: Endoscopes and accessories for endoscopes, the accessories comprising: a tube assembly, a liquid lens assembly, a wedge lens assembly, and a driver assembly; The proximal end side of the tube assembly is adapted to be detachably connected to the insertion end of the endoscope; The wedge mirror group is arranged at the distal end side of the tube assembly, the liquid lens group and the wedge mirror are spaced apart, and the driver group is electrically connected to the liquid lens group; The driver group is used to adjust the curvature of the liquid lens group to adjust the focal length and magnification of the endoscope, and the wedge lens group is used to increase the imaging angle of the endoscope; The accessory is applied to an endoscope system, and the endoscope system is used for: Perform clarity detection on the edge area of ​​endoscopic imaging; The endoscope system is also used for: Acquire multiple image groups at different wedge mirror angles collected during a process of searching and approximating the driving voltage of the driver group within a preset voltage range, wherein each image group includes at least two images at different magnifications; Perform SIFT feature point detection and matching on each of the images to determine the translation vector and rotation vector between the images; Each group of the images is spliced ​​based on the translation vector and the rotation direction to obtain a target image.

2. The endoscope system according to claim 1, wherein: The liquid lens assembly includes a first liquid lens and a second liquid lens, and the driver assembly includes a first driver and a second driver; The first driver is electrically connected to the first liquid lens, and the second driver is electrically connected to the second liquid lens; The first liquid lens is used for adjusting the focal length, and the second liquid lens is used for adjusting the magnification.

3. The endoscope system according to claim 1, wherein: The accessories also include: a micro motor; The micromotor is connected to the wedge-shaped mirror group, and the micromotor is used to control the wedge-shaped mirror group to adjust the tilt angle.

4. The endoscope system according to claim 1, wherein: The accessories also include: a concave lens group and a convex lens group; The concave lens group is arranged between the proximal end side of the tube assembly and the liquid lens group, and the convex lens group is arranged between the concave lens group and the wedge lens group; The concave lens group is used to eliminate chromatic aberration, and the convex lens group is used to focus light.

5. The endoscope system according to claim 1, wherein: The accessory further comprises: a first input unit and a second input unit, both of which are connected to the liquid lens assembly; Wherein, the first input unit is used to adjust the curvature of the liquid lens assembly in response to the magnification increase input to magnify the imaging of the endoscope; The second input unit is used to respond to a reduction magnification input and adjust the curvature of the liquid lens assembly to reduce the imaging of the endoscope.

6. The endoscope system according to claim 2, wherein: The liquid lens assembly further includes a third liquid lens, and the driver assembly further includes a third driver; The third driver is electrically connected to the third liquid lens; The third liquid lens and the first liquid lens are used together for focal length adjustment.

7. A control method for an endoscope system according to any one of claims 1 to 6, characterized in that: include: When the clarity of the pixels in the edge area is lower than a preset threshold, the driving voltage of the driver group is searched and approximated within a preset voltage range by using a golden ratio segmentation method to obtain a target driving voltage of the driver group, wherein the target driving voltage is the driving voltage corresponding to the highest clarity of the pixels in the edge area; The curvature of the liquid lens assembly is adjusted based on the target driving voltage to obtain an image of the adjusted liquid lens assembly.

8. The control method according to claim 7, characterized in that: The clarity detection of the edge area of ​​the endoscopic imaging includes: When the endoscope detects a change in the object distance, performing a clarity detection on an edge area of ​​the endoscope imaging; Or; when the magnification of the liquid lens group changes, the clarity of the edge area of ​​the endoscope imaging is detected.

9. The control method according to claim 7, characterized in that: The method further comprises: constructing a first depth map based on the translation vector and the rotation vector; Performing focus edge detection on each of the images using a Laplacian of Gaussian kernel to obtain a second depth map, wherein a corresponding depth value of each pixel in the second depth map is determined based on a layer where a norm of the Laplacian of Gaussian kernel is maximum; performing normalization processing on the second depth map and performing depth optimization processing on the second depth map to obtain a third depth map; Adding and averaging the first depth map and the third depth map to obtain a fourth fused depth map; Performing splicing processing on the fourth fused depth map based on the translation vector and the rotation vector to obtain a target fused map; A target three-dimensional reconstruction model is determined based on the target fused depth map and the three-dimensional model texture coverage information, wherein the three-dimensional model texture coverage information is determined based on imaging of the adjusted liquid lens group.

Images