Depth-of-field continuation imaging device and method for endoscope and electronic equipment

Abstract

The invention relates to the technical field of endoscope imaging, and discloses a depth-of-field continuation imaging device and method for an endoscope and electronic equipment. The microlens array is composed of a plurality of microlenses with positive focal power, the microlens array is used for obtaining an original image collected by the endoscope, a light field image comprising sub-aperture images corresponding to the microlenses is obtained, depth calculation is carried out according to the parallax value between the different sub-aperture images, and the depth of the endoscope is calculated according to the parallax value between the sub-aperture images. Obtaining a pixel depth corresponding to an image pixel in the original image, refocusing each sub-aperture image on a focal plane corresponding to the pixel depth to obtain a refocused image corresponding to each sub-aperture image, and determining a definition weight corresponding to the image pixel from each refocused image; and carrying out image fusion on the pixel regions of the image pixels in the corresponding definition weights to obtain a target image. Compared with depth-of-field continuation according to multiple frames of images, the target picture with clear texture and extended depth-of-field can be obtained, and the image quality is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of endoscopic imaging, and in particular, to a depth-of-field extension imaging device, method, and electronic device for an endoscope. Background Art

[0002] Medical endoscopes are high-precision medical diagnostic and treatment devices integrating optics, electronics, and software. Due to the influence of factors such as the lens aperture, lens focal length, and the distance between the focal plane and the shooting target, the imaging device can only obtain clear images of the shooting targets within a target distance range, which is called the depth of field. In the field of medical endoscopic imaging, traditional optical systems are limited by the inherent contradiction between physical focal depth and optical parameters. There are differences in the depths of different heterogeneous tissues relative to the endoscope, and it is difficult for the depth of field of the endoscope to cover all heterogeneous tissues, resulting in the fact that a single image collected by the endoscope cannot meet the clinical needs of synchronous clear imaging of various heterogeneous tissues. Taking laparoscopy as an example, although its optical characteristics of short focal length and large field of view can penetrate the body cavity to obtain lesion information, due to the balance constraint between the aperture parameter and the object distance range, the actual depth of field can only cover a limited depth range. When facing complex anatomical structures such as the folds of the inner wall of the cavity and deep blood vessels, the operator needs to frequently mechanically focus to switch the focus, increasing the operation time and operation risk.

[0003] Even if algorithms such as multi-frame synthesis and feature matching are used for depth-of-field extension, due to the sub-pixel displacement caused by physiological tremors of the human body such as breathing and heartbeat, these algorithms are difficult to accurately align the multi-frame sequence, resulting in motion artifacts and detail loss, and the image quality of the extended depth-of-field image is low. Summary of the Invention

[0004] To provide a basic understanding of some aspects of the disclosed embodiments, a simple summary is given below. The summary is not a general review, nor is it intended to identify key important elements or delineate the protection scope of these embodiments, but rather serves as a preface to the subsequent detailed description.

[0005] In view of the above-mentioned disadvantages of the prior art, the present application provides a depth-of-field extension imaging device, method, and electronic device for an endoscope to improve the image quality of depth-of-field extension.

[0006] The present application provides a depth-of-field extension imaging device for an endoscope, comprising: an imaging module for obtaining an original image collected by the endoscope by using a microlens array to obtain a light field image, wherein the microlens array comprises a plurality of microlenses with positive optical power, and the light field image comprises sub-aperture images respectively corresponding to the microlenses; a microprocessing chip for performing depth calculation according to the parallax differences between different sub-aperture images to obtain the pixel depth corresponding to the image pixels in the original image; refocusing each sub-aperture image on the focal plane corresponding to the pixel depth to obtain refocused images respectively corresponding to the sub-aperture images, and determining the clarity weight corresponding to the image pixels from the refocused images; and performing image fusion on the pixel regions of each image pixel in the corresponding clarity weight to obtain a target image.

[0007] In an embodiment of the present application, the imaging module comprises: the microlens array, the object side of the microlens array is arranged on the eyepiece side of the endoscope; a first image sensor arranged on the image side of the microlens array, and the first image sensor is used for collecting the light field image after the original image passes through the microlens array.

[0008] In an embodiment of the present application, the imaging module further comprises: a beam splitter arranged between the eyepiece side and the microlens array, and the beam splitter is configured to decompose the original image collected by the endoscope into a first optical path and a second optical path, wherein the microlens array and the first image sensor are both arranged on the first optical path; a second image sensor arranged on the second optical path, and the second image sensor is configured to perform image acquisition on the second optical path.

[0009] In an embodiment of the present application, before performing depth calculation, the microprocessing chip is further used for: obtaining the plane coordinates of each microlens in the microlens array; and performing image extraction from the light field image according to the plane coordinates to obtain sub-aperture images respectively corresponding to the microlenses.

[0010] In an embodiment of the present application, the microprocessing chip refocuses each sub-aperture image on the focal plane corresponding to the pixel depth in the following manner: using a spatial domain digital refocusing algorithm or a frequency domain digital refocusing algorithm to refocus each sub-aperture image on the focal plane corresponding to the pixel depth.

[0011] In an embodiment of the present application, the microprocessing chip performs depth calculation in the following manner: determining a first image and a second image from any two sub-aperture images respectively, finding the minimum disparity matching cost between the first image and the second image according to the matching cost formula to determine the disparity value of any image pixel in the original image between the first image and the second image; calculating the disparity value using the depth solving formula to obtain the pixel depth corresponding to the image pixel; and performing fusion by combining the pixel depths corresponding to each image pixel to obtain a depth image.

[0012] In an embodiment of the present application, the microprocessing chip determines the clarity weight corresponding to the image pixel in the following manner: obtaining the pixel regions of the image pixel in each refocused image respectively; determining the clearest region with the highest clarity from the pixel regions of the image pixel in each refocused image, and taking the refocused image corresponding to the clearest region as the clarity weight of the image pixel.

[0013] In an embodiment of the present application, the device further includes: a user terminal for displaying a target image carrying texture information and / or a depth image carrying depth information.

[0014] The present application provides a depth-of-field extension imaging method for an endoscope, including: using a microlens array to obtain an original image collected by the endoscope to obtain a light field image, where the microlens array includes a plurality of microlenses with positive optical power, and the light field image includes sub-aperture images respectively corresponding to each microlens; performing depth calculation according to the disparity values between different sub-aperture images to obtain the pixel depth corresponding to the image pixel in the original image; refocusing each sub-aperture image on the focal plane corresponding to the pixel depth to obtain refocused images respectively corresponding to each sub-aperture image, and determining the clarity weight corresponding to the image pixel from each refocused image; and performing image fusion on the pixel regions of each image pixel in the corresponding clarity weight to obtain a target image.

[0015] In an embodiment of the present application, before performing depth calculation, the method further includes: obtaining the plane coordinates of each microlens in the microlens array respectively; and performing image extraction from the light field image according to the plane coordinates to obtain sub-aperture images respectively corresponding to each microlens.

[0016] In an embodiment of the present application, each sub-aperture image is refocused on the focal plane corresponding to the pixel depth in the following manner: using a spatial domain digital refocusing algorithm or a frequency domain digital refocusing algorithm to refocus each sub-aperture image on the focal plane corresponding to the pixel depth.

[0017] In one embodiment of the present application, depth calculation is performed in the following manner: Determine a first image and a second image from any two sub-aperture images respectively, find the minimum disparity matching cost between the first image and the second image according to the matching cost formula to determine the disparity value of any image pixel in the original image between the first image and the second image; Use the depth solving formula to calculate the disparity value to obtain the pixel depth corresponding to the image pixel; Combine the pixel depths corresponding to each image pixel for fusion to obtain a depth image.

[0018] In one embodiment of the present application, the clarity weight corresponding to an image pixel is determined in the following manner: Obtain the pixel regions of the image pixel in each refocused image respectively; Determine the clearest region with the highest clarity from the pixel regions of the image pixel in each refocused image, and use the refocused image corresponding to the clearest region as the clarity weight of the image pixel.

[0019] The present application provides an electronic device, including: a processor and a memory; The memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory so that the electronic device executes the above method.

[0020] Advantages of the present application: A microlens array is composed of multiple microlenses with positive optical power. The original image collected by the endoscope is obtained by using the microlens array to obtain a light field image including sub-aperture images respectively corresponding to each microlens. Depth calculation is performed according to the disparity values between different sub-aperture images to obtain the pixel depth corresponding to the image pixel in the original image. Each sub-aperture image is refocused on the focal plane corresponding to the pixel depth to obtain refocused images respectively corresponding to each sub-aperture image, and the clarity weight corresponding to the image pixel is determined from each refocused image, so as to perform image fusion on the pixel regions of each image pixel in the corresponding clarity weight to obtain a target image. In this way, compared with extending the depth of field according to multiple frames of images, by capturing a single frame of image through a microlens array, obtaining the pixel depth corresponding to the image pixel in the original image according to the sub-aperture images, and the refocused images formed on the focal plane corresponding to the pixel depth, and then combining the pixel depth to fuse the clear regions of the image pixel in each refocused image, a target picture with both clear texture and extended depth of field is obtained, improving the image quality. Description of the Drawings

[0021] Figure 1 It is a schematic structural diagram of a depth of field extension imaging device for an endoscope in an embodiment of the present application; Figure 2 It is a schematic structural diagram of an imaging module in an embodiment of the present application; Figure 3 It is a schematic structural diagram of a microlens array in an embodiment of the present application; Figure 4 It is a schematic structural diagram of another imaging module in an embodiment of the present application; Figure 5 It is a schematic flowchart of a depth-of-field extension imaging method for an endoscope in an embodiment of the present application; Figure 6 It is a schematic flowchart of another depth-of-field extension imaging method for an endoscope in an embodiment of the present application; Figure 7 It is a schematic structural diagram of an electronic device in an embodiment of the present invention. Detailed implementation manners

[0022] The following uses specific specific examples to illustrate the implementation manners of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific implementation manners. Various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, without conflict, the following embodiments and sub-samples in the embodiments can be combined with each other.

[0023] It should be noted that the drawings provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner. Therefore, only the components related to the present invention are shown in the drawings, rather than being drawn according to the number, shape, and size of the components in actual implementation. The types, quantities, and proportions of the components in actual implementation can be arbitrarily changed, and the component layout type may also be more complex.

[0024] In the following description, a large number of details are explored to provide a more thorough explanation of the embodiments of the present invention. However, it is obvious to those skilled in the art that the embodiments of the present invention can be implemented without these specific details. In other embodiments, well-known structures and devices are shown in the form of block diagrams rather than in detail to avoid making the embodiments of the present invention difficult to understand.

[0025] In the specification, claims, and above-mentioned drawings of the present application, the terms "first", "second", etc. are used to distinguish similar objects, and do not have to be used to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so as to implement the embodiments of the present application described herein. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion.

[0026] Unless otherwise specified, the term "plurality" means two or more.

[0027] In the present application, the character " / " means that the objects before and after are in an "or" relationship. For example, A / B means: A or B.

[0028] The term "and / or" describes the relationship between objects and indicates that three relationships can exist. For example, A and / or B means: A or B, or, A and B, these three relationships.

[0029] The following is an introduction and explanation of several terms involved in this application and the background technology: Refocusing refers to generating clear images of scenes with different depths by adjusting the projection and integration of light field data on different focal planes.

[0030] The focal plane refers to the plane perpendicular to the main optical axis of the endoscope.

[0031] The filter film is a functional film that realizes optical regulation by selectively transmitting or reflecting light of specific wavelengths. On the one hand, by blocking interfering light, the filter film can highlight specific colors. On the other hand, the band-pass filter film is used to separate specific fluorescence signals, such as accurately capturing the target wavelength in cell labeling detection to improve the diagnostic accuracy.

[0032] Combined Figure 1 As shown, this application provides a depth-of-field extension imaging device for an endoscope, including an imaging module 101 and a microprocessing chip 102.

[0033] The imaging module 101 is used to obtain the original image collected by the endoscope using a microlens array to obtain a light field image. Among them, the microlens array includes a plurality of microlenses with positive optical power, and the light field image includes sub-aperture images corresponding to each microlens respectively.

[0034] In this embodiment, since the microlens array is composed of a plurality of microlenses in the same plane, and each microlens has positive optical power, an image on the object side of the microlens forms a reduced image on the image side of the microlens. Therefore, when the original image passes through each microlens from the object side of the microlens array, a sub-aperture image can be formed on the image side of the microlens.

[0035] Combined Figure 1 As shown, the imaging module 101 is arranged on the eyepiece side of the endoscope.

[0036] Combined Figure 1 As shown, the microprocessing chip 102 is connected to a user terminal 103, and the user terminal is used to display a target image carrying texture information and / or a depth image carrying depth information.

[0037] Combined Figure 1 and Figure 2As shown in the figure, the depth-of-field extension imaging device of the present application further includes an optical lens 201, which is disposed between the eyepiece side of the endoscope and the imaging module 101. Among them, the imaging module 101 sequentially includes a microlens array 202 and a first image sensor 203 along the optical axis from the object side to the image side.

[0038] For the optical lens 201, a first lens 2011, a second lens 2012, a third lens 2013, a fourth lens 2014, a fifth lens 2015, a sixth lens 2016, and a seventh lens 2017 are sequentially arranged along the optical axis. Among them, the optical lens 201 is close to the eyepiece side of the endoscope.

[0039] The first lens 2011 is a protective lens.

[0040] The second lens 2012 has a negative optical power, the object side is concave, and the image side is convex.

[0041] An aperture 2018 is provided between the first lens 2011 and the second lens 2012.

[0042] The third lens 2013 has a positive optical power, the object side is convex, and the image side is concave.

[0043] The fourth lens 2014 has a positive optical power, and the image side is convex.

[0044] The fifth lens 2015 has a positive optical power, and the object side is convex.

[0045] The sixth lens 2016 has a positive optical power, the object side is convex, and the image side is concave.

[0046] The seventh lens 2017 has a negative optical power, the object side is convex, and the image side is concave.

[0047] The sixth lens 2016 and the seventh lens 2017 are cemented lenses.

[0048] A housing is provided outside the optical lens 201.

[0049] In this embodiment, by gluing lenses made of different materials with different refractive indices, aberration can be eliminated and the imaging quality can be improved. At the same time, gluing lenses can reduce the number of interfaces between air and glass, thereby reducing reflection loss. In addition, gluing lenses can also compensate for the difference in the radius of curvature of the glued surface, thereby reducing the requirement for processing accuracy and simplifying the processing process. Additionally, the second lens 2012 has a negative optical power to diverge the incident parallel light beam, shifting the principal plane backward. Thus, with a relatively short physical focal length, a working distance longer than the actual focal length can be obtained. The intermediate lenses all have positive optical power to focus the diverged light beam. At the same time, through the optical power distribution of the negative-positive combination, the incident angle of the off-axis light beam can be significantly reduced, achieving a larger field of view. The seventh lens 2017, which has a negative optical power, can further adjust the optical path to adapt to the image sensor and avoid deterioration of the edge image quality.

[0050] The object side of the microlens array 202 is disposed on the eyepiece side of the endoscope.

[0051] Combined Figure 3 As shown, the microlens array includes a plurality of microlenses with positive optical power, and all the microlenses are in the same plane.

[0052] The first image sensor 203 is disposed on the image side of the microlens array, and the first image sensor is configured to collect the light field image on the image side of the microlens array.

[0053] A first protective glass 204 is provided on the back of the first image sensor 203. Among them, the first protective glass 204 is provided with a filter film.

[0054] Combined Figure 4 As shown, the present application also provides an imaging device. The imaging module 101 is disposed on the image side of the optical lens 201. Among them, the imaging module 101 includes a microlens array 202, a first image sensor 203, a beam splitter 205, and a second image sensor 206.

[0055] The beam splitter 205 is disposed between the eyepiece side and the microlens array.

[0056] The beam splitter 205 is disposed between the optical lens 201 and the microlens array 202, and the beam splitter 205 is configured to decompose the original image collected by the endoscope into a first optical path and a second optical path.

[0057] The microlens array 202 and the first image sensor 203 are sequentially disposed along the first optical path along the optical axis.

[0058] The second image sensor 206 is disposed on the second optical path, and the second image sensor is configured to collect images of the second optical path.

[0059] A second protective glass 207 is provided on the back surface of the second image sensor 206, wherein the second protective glass 207 is provided with a filter film coated thereon.

[0060] In some embodiments, both the first image sensor and the second image sensor are CMOS (Complementary Metal Oxide Semiconductor) sensors.

[0061] In this way, through the dual-channel design of the beam splitting system combined with the jointly optimized cemented lens group, the optical signals in the first optical path are captured by the first image sensor and used to provide depth information, and the optical signals in the second optical path are captured by the second image sensor and used for high-definition imaging to provide high-resolution details, so as to obtain a target picture with both clear texture and extended depth of field, realize the synchronous capture of multi-focus pixel depth, eliminate mechanical focusing components, and reduce the device complexity and operation time. At the same time, by setting the beam splitter, the optical path is split into two, and only minor modifications to the original endoscope structure are required to implement this solution, which can effectively save design and manufacturing costs.

[0062] The microprocessing chip 102 is used to calculate the depth according to the parallax between different sub-aperture images, and obtain the pixel depth corresponding to the image pixels in the original image; refocus each sub-aperture image on the focal plane corresponding to the pixel depth, obtain the refocused images respectively corresponding to each sub-aperture image, and determine the clarity weight corresponding to the image pixels from the refocused images; perform image fusion on the pixel regions of each image pixel in the corresponding clarity weight to obtain the target image.

[0063] In some embodiments, the microprocessing chip 102 includes a processing chip with algorithm logic such as an MCU (Microcontroller Unit) and an FPGA (Field Programmable Gate Array). The light field image captured by the imaging module is input to the microprocessing chip, and the processing chip is controlled to perform depth extension on the original image collected by the endoscope through the light field image.

[0064] Optionally, the microprocessing chip determines the clarity weight corresponding to the image pixels in the following manner: obtain the pixel regions of the image pixels in each refocused image respectively; determine the clear region with the highest clarity from the pixel regions of the image pixels in each refocused image respectively, and use the refocused image corresponding to the clear region as the clarity weight of the image pixel.

[0065] In some embodiments, for each focal plane, there exists an image sequence containing multiple refocused images. Each refocused image in the image sequence is identified to obtain an image identifier. The clearly imaged regions in each refocused image are different. At the same time, the image pixels of the original image exist in the pixel regions of each refocused image. Therefore, the region that can be clearly imaged is searched for from the pixel regions of the image pixels in each refocused image, and the refocused image corresponding to this clear region is used as the clarity weight of the image pixel and recorded with the image identifier.

[0066] In some embodiments, a Laplacian pyramid is used to capture high-frequency details from the pixel regions of each image pixel in the corresponding clarity weight to avoid halo artifacts, and a multi-scale decomposition and fusion strategy is used to fuse the pixel regions of each image pixel in the corresponding clarity weight according to the coordinate positions of the image pixels in the original image to obtain a target image, and this target image can cover the full depth of field.

[0067] Using the depth-of-field extension imaging device for endoscopes provided by the present application, a microlens array is composed of multiple microlenses with positive optical power. The original image collected by the endoscope is obtained by using the microlens array to obtain a light field image including sub-aperture images respectively corresponding to each microlens. Depth calculation is performed according to the parallax between different sub-aperture images to obtain the pixel depth corresponding to the image pixels in the original image. Each sub-aperture image is refocused on the focal plane corresponding to the pixel depth to obtain refocused images respectively corresponding to each sub-aperture image, and the clarity weight corresponding to the image pixels is determined from each refocused image to perform image fusion on the pixel regions of each image pixel in the corresponding clarity weight to obtain a target image. In this way, compared with performing depth-of-field extension based on multiple frames of images, by capturing a single frame of image through a microlens array, obtaining the pixel depth corresponding to the image pixels in the original image according to the sub-aperture images, and the refocused images formed on the focal plane corresponding to the pixel depth, and then combining the pixel depth to fuse the clear regions of the image pixels in each refocused image to obtain a target picture with both clear texture and extended depth of field, improving the image quality.

[0068] Optionally, before performing depth calculation, the microprocessing chip is further configured to: obtain the planar coordinates of each microlens in the microlens array; perform image extraction from the light field image according to the planar coordinates to obtain sub-aperture images respectively corresponding to each microlens.

[0069] In some embodiments, a light field image of the microlens array on the image side is obtained through a first image sensor, and a set of sub-aperture images is extracted from the light field data through an extraction formula, where the set of sub-aperture images includes sub-aperture images respectively corresponding to each microlens.

[0070] In some embodiments, sub-aperture images are extracted through formula (1): Formula (1) In Formula (1), is the sub-aperture image,[[]] is the four-dimensional light field function,[[]] are the planar coordinates of the main lens in the endoscope, and the main lens is used to collect the original image,[[]] are the planar coordinates of the microlens,[[]] is the viewing angle step,[[]] is the horizontal serial number of the microlens in the microlens matrix,[[]] is the vertical serial number of the microlens in the microlens matrix.[[]]

[0071] In this way, by capturing the four-dimensional light field data at one time through the microlens array, combining the parallax analysis and the multi-focal plane refocusing algorithm, it is possible to fuse the clear regions at different depths in a single-frame image without mechanical focusing, breaking through the physical focal depth limitation of the traditional endoscope.[[]]

[0072] Optionally, the microprocessing chip refocuses each sub-aperture image on the focal plane corresponding to the pixel depth in the following manner: using the spatial domain digital refocusing algorithm or the frequency domain digital refocusing algorithm to refocus each sub-aperture image on the focal plane corresponding to the pixel depth.[[]]

[0073] In some embodiments, the sub-aperture images are translated and accumulated through the refocusing algorithm to generate the refocused images corresponding to different focal planes, wherein each refocused image corresponding to a focal plane is an image sequence.[[]]

[0074] In some embodiments, the spatial domain digital refocusing algorithm is characterized by Formula (2):[[]] Formula (2)[[]] In Formula (2),[[]] is the refocused image,[[]] is the relative depth, and[[]] ,[[]] is the distance between the plane where the microlens is located and the focal plane,[[]] is the distance between the plane where the microlens is located and the refocusing plane, wherein the image sensor is used to obtain the sub-aperture image set, and the refocusing plane is the focal plane corresponding to the refocused image.[[]]

[0075] In some embodiments, the frequency domain digital refocusing algorithm is characterized by Formulas (3) to (6):[[]] Formula (3)[[]] Formula (4)[[]] Formula (5)[[]] Formula (6)[[]] In the formula,[[]] For re - focused image, is the cropping matrix, is the inverse transform of 2 - D Fourier transform, is the 4 - D Fourier transform.

[0076] Optionally, the micro - processing chip performs depth calculation in the following way: determine the first image and the second image from any two sub - aperture images respectively, find the minimum disparity matching cost between the first image and the second image according to the matching cost formula to determine the disparity value of any image pixel in the original image between the first image and the second image; calculate the disparity value using the depth solving formula to obtain the pixel depth corresponding to the image pixel; fuse by combining the pixel depths corresponding to each image pixel to obtain a depth image.

[0077] In some embodiments, there is a disparity value for an image pixel between different images. Calculate the matching cost corresponding to the image pixel at different disparity values through the matching cost formula to find the disparity value corresponding to the minimum matching cost; calculate the disparity value using the depth solving formula to obtain the pixel depth; combine the pixel depths corresponding to each image pixel and use multi - view disparity fusion to generate a depth image.

[0078] In some embodiments, the matching cost formula is represented by formula (7): Formula (7) In formula (7), is the pixel coordinate at the disparity value of the matching cost, is the first image, is the second image, is the pixel coordinate of the corresponding window neighborhood.

[0079] In some embodiments, the depth solving formula is represented by formula (8): Formula (8) In formula (8), is the depth value corresponding to the pixel coordinate , is the disparity value corresponding to the pixel coordinate , is the calibrated focal length, is the baseline length, where the baseline length , is the distance between adjacent two microlenses.

[0080] Optionally, the apparatus further includes: a user terminal for displaying a target image carrying texture information and / or a depth image carrying depth information. Specifically, the user terminal may be an endoscope monitor, an endoscope display screen, or a device with a display device such as a computer, a mobile phone, or a tablet that is wired / wirelessly connected to the endoscope.

[0081] In some embodiments, the target image and the depth image are simultaneously displayed through the display interface of the user terminal. In this way, the user can obtain texture information and depth information from different images, providing more reference information for the user and thus improving the diagnostic accuracy.

[0082] Combined Figure 5 As shown, the present application provides a method for extended depth of field imaging for an endoscope, including: Step S501, using a microlens array to obtain the original image collected by the endoscope to obtain a light field image; Wherein, the microlens array includes a plurality of microlenses with positive optical power, and the light field image includes sub-aperture images respectively corresponding to the microlenses; Step S502, performing depth calculation according to the parallax between different sub-aperture images to obtain the pixel depth corresponding to the image pixels in the original image; Step S503, refocusing each sub-aperture image on the focal plane corresponding to the pixel depth to obtain a refocused image respectively corresponding to each sub-aperture image, and determining the clarity weight corresponding to the image pixels from each refocused image; Step S504, performing image fusion on the pixel regions of each image pixel in the corresponding clarity weight to obtain a target image.

[0083] By using the method for extended depth of field imaging for an endoscope provided by the present application, a microlens array is composed of a plurality of microlenses with positive optical power. The original image collected by the endoscope is obtained by using the microlens array to obtain a light field image including sub-aperture images respectively corresponding to the microlenses. Depth calculation is performed according to the parallax between different sub-aperture images to obtain the pixel depth corresponding to the image pixels in the original image. Each sub-aperture image is refocused on the focal plane corresponding to the pixel depth to obtain a refocused image respectively corresponding to each sub-aperture image, and the clarity weight corresponding to the image pixels is determined from each refocused image, so as to perform image fusion on the pixel regions of each image pixel in the corresponding clarity weight to obtain a target image. In this way, compared with performing extended depth of field based on multiple frames of images, a single frame of image is captured by the microlens array, the pixel depth corresponding to the image pixels in the original image is obtained from the sub-aperture images, and the refocused images formed on the focal plane corresponding to the pixel depth are combined. Then, the clear regions of the image pixels in each refocused image are fused according to the pixel depth to obtain a target picture with both clear texture and extended depth of field, improving the image quality.

[0084] In some embodiments, before performing depth calculation, the method further includes: obtaining the planar coordinates of each microlens in the microlens array; extracting an image from the light field image according to the planar coordinates to obtain a sub-aperture image corresponding to each microlens.

[0085] In some embodiments, the following method is used to refocus each sub-aperture image on the focal plane corresponding to the pixel depth: using a spatial domain digital refocusing algorithm or a frequency domain digital refocusing algorithm to refocus each sub-aperture image on the focal plane corresponding to the pixel depth.

[0086] In some embodiments, depth calculation is performed by the following method: determining a first image and a second image from any two sub-aperture images respectively, finding the minimum disparity matching cost between the first image and the second image according to the matching cost formula to determine the disparity value of any image pixel in the original image between the first image and the second image; calculating the disparity value using the depth solving formula to obtain the pixel depth corresponding to the image pixel; combining the pixel depths corresponding to each image pixel for fusion to obtain a depth image.

[0087] In some embodiments, the clarity weight corresponding to the image pixel is determined by the following method: obtaining the pixel regions of the image pixel in each refocused image respectively; determining the clearest region with the highest clarity from the pixel regions of the image pixel in each refocused image, and using the refocused image corresponding to the clearest region as the clarity weight of the image pixel.

[0088] Combined Figure 6 As shown, the present application provides a depth of field extension imaging method for an endoscope, including: Step S601, using a microlens array to obtain the original image collected by the endoscope to obtain a light field image; Step S602, extracting an image from the light field image according to the planar coordinates to obtain a sub-aperture image corresponding to each microlens, and jumping to Step S603 and Step S604; Step S603, performing depth calculation according to the disparity value between different sub-aperture images to obtain the pixel depth corresponding to the image pixel in the original image, and jumping to Step S605; Step S604, refocusing each sub-aperture image on the focal plane corresponding to the pixel depth to obtain a refocused image corresponding to each sub-aperture image, and jumping to Step S605; Step S605, determining the clarity weight corresponding to the image pixel from each refocused image; Step S606, performing image fusion on the pixel regions of each image pixel in the corresponding clarity weight to obtain a target image.

[0089] By adopting the depth-of-field extension imaging method for an endoscope provided in this application, a microlens array is composed of multiple microlenses with positive optical power. The microlens array is used to obtain the original image collected by the endoscope, and a light field image including sub-aperture images respectively corresponding to the microlenses is obtained. Depth calculation is performed based on the parallax between different sub-aperture images to obtain the pixel depth corresponding to the image pixels in the original image. Each sub-aperture image is refocused on the focal plane corresponding to the pixel depth to obtain the refocused images respectively corresponding to the sub-aperture images, and the clarity weight corresponding to the image pixels is determined from each refocused image, so as to perform image fusion on the pixel regions of each image pixel in the corresponding clarity weight to obtain a target image. In this way, compared with performing depth-of-field extension based on multiple frames of images, a single-frame image is captured through the microlens array, the pixel depth corresponding to the image pixels in the original image is obtained based on the sub-aperture images, and the refocused images formed on the focal plane corresponding to the pixel depth are combined. Then, the clear regions of the image pixels in each refocused image are fused in combination with the pixel depth to obtain a target picture with both clear texture and extended depth of field, improving the image quality.

[0090] This application also provides an electronic device, including: a processor and a memory; the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory so that the electronic device executes the above method.

[0091] Figure 7 The structural schematic diagram of a computer system of an electronic device suitable for implementing the embodiments of this application is shown. It should be noted that, Figure 7 The computer system 700 of the electronic device shown is only an example and should not bring any limitation to the functions and usage scope of the embodiments of this application.

[0092] As Figure 7 shown, the computer system 700 includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 702 or the program loaded from the storage section 708 into the random access memory (RAM) 703, such as executing the method in the above embodiments. In the RAM 703, various programs and data required for system operation are also stored. The CPU 701, ROM 702, and RAM 703 are connected to each other through a bus 704. The input / output (I / O) interface 705 is also connected to the bus 704.

[0093] The following components are connected to the I / O interface 705: an input section 706 including a keyboard, a mouse, etc.; an output section 707 including, for example, a cathode ray tube (CRT), a liquid crystal display (LCD), etc. and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is mounted on the drive 710 as needed so that a computer program read out therefrom is installed into the storage section 708 as needed.

[0094] The electronic device disclosed in this embodiment includes a processor, a memory, a transceiver, and a communication interface. The memory and the communication interface are connected to the processor and the transceiver and complete communication therebetween. The memory is used to store a computer program, the communication interface is used for communication, and the processor and the transceiver are used to run the computer program so that the electronic device executes each step of the above method.

[0095] The above description and drawings fully illustrate the embodiments of the present disclosure, enabling those skilled in the art to practice them. Other embodiments may include structural, logical, electrical, process, and other changes. Embodiments merely represent possible variations. Unless explicitly required, individual components and functions are optional, and the order of operations may vary. Parts and sub-samples of some embodiments may be included in or replace parts and sub-samples of other embodiments. Moreover, the terms used in this application are only for describing embodiments and do not limit the claims. As used in the description of embodiments and claims, unless the context clearly indicates otherwise, the singular forms "a", "an", and "the" are intended to also include the plural forms. Similarly, as used in this application, the term "and / or" refers to any and all possible combinations of one or more of the associated listed items. Additionally, when used in this application, the term "comprise" and its variants "comprises" and / or "comprising" etc. mean the presence of the stated sub-samples, wholes, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other sub-samples, wholes, steps, operations, elements, components, and / or groupings of these. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, or device comprising the element. Herein, what each embodiment focuses on may be the differences from other embodiments, and the same or similar parts among various embodiments may be referred to each other. For the methods, products, etc. disclosed in the embodiments, if they correspond to the method parts disclosed in the embodiments, the relevant parts may refer to the description of the method parts.

[0096] Those skilled in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner may depend on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods for each specific application to implement the described functions, but such implementation should not be considered to exceed the scope of this application. Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the systems, devices, and units described above can refer to the corresponding processes in the foregoing method embodiments, and will not be elaborated herein.

[0097] In the embodiments disclosed in this article, the disclosed methods, products (including but not limited to devices, equipment, etc.) can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of units can be merely a logical function division. In actual implementation, there can be other division methods. For example, multiple units or components can be combined or integrated into another system, or some sub-samples can be ignored or not executed. Additionally, the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces. The indirect couplings or communication connections of devices or units can be in electrical, mechanical, or other forms. The units described as separate components can be or can not be physically separated. The components displayed as units can be or can not be physical units, that is, they can be located in one place or can be distributed to multiple network units. Some or all of the units can be selected according to actual needs to implement this embodiment. Additionally, in this application, each functional unit can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit.

[0098] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of the systems, methods, and computer program products according to this application. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or part of the code, and the module, program segment, or part of the code contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions marked in the boxes can also occur in a different order than that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the reverse order, which can depend on the functions involved. In the descriptions corresponding to the flowcharts and block diagrams in the accompanying drawings, the operations or steps corresponding to different boxes can also occur in a different order than that disclosed in the description. Sometimes, there is no specific order between different operations or steps. For example, two consecutive operations or steps can actually be executed substantially in parallel, and they can sometimes be executed in the reverse order, which can depend on the functions involved. Each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system for performing the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

Claims

1. An imaging device for extending the depth of field of an endoscope, characterized in that, Comprising: An imaging module, configured to obtain an original image collected by an endoscope using a microlens array to obtain a light field image, wherein the microlens array includes a plurality of microlenses with positive optical power, and the light field image includes sub-aperture images respectively corresponding to the microlenses; A microprocessing chip, configured to perform depth calculation based on the parallax between different sub-aperture images to obtain the pixel depth corresponding to an image pixel in the original image; refocus each sub-aperture image on the focal plane corresponding to the pixel depth to obtain a refocused image respectively corresponding to each sub-aperture image, and determine the clarity weight corresponding to the image pixel from each refocused image; perform image fusion on the pixel regions of each image pixel in the corresponding clarity weight to obtain a target image.

2. The device according to claim 1, characterized in that, The imaging module includes: The microlens array, the object side of which is arranged on the eyepiece side of the endoscope; A first image sensor, arranged on the image side of the microlens array, and the first image sensor is configured to collect the light field image after the original image passes through the microlens array.

3. The device according to claim 2, characterized in that, The imaging module further includes: A beam splitter, arranged between the eyepiece side and the microlens array, and the beam splitter is configured to decompose the original image collected by the endoscope into a first optical path and a second optical path, wherein the microlens array and the first image sensor are both arranged on the first optical path; A second image sensor, arranged on the second optical path, and the second image sensor is configured to perform image acquisition on the second optical path.

4. The device according to claim 2, wherein Before performing depth calculation, the microprocessing chip is further configured to: Obtain the planar coordinates of each microlens in the microlens array; Extract images from the light field image according to the planar coordinates to obtain sub-aperture images respectively corresponding to the microlenses.

5. The device according to any one of claims 1 to 4, characterized in that The microprocessing chip refocuses each sub-aperture image on the focal plane corresponding to the pixel depth in the following manner: Adopt a spatial domain digital refocusing algorithm or a frequency domain digital refocusing algorithm to refocus each sub-aperture image on the focal plane corresponding to the pixel depth.

6. The device according to any one of claims 1 to 4, characterized in that The microprocessing chip performs depth calculation in the following manner: Determine a first image and a second image from any two sub-aperture images respectively, and find the minimum parallax matching cost between the first image and the second image according to the matching cost formula to determine the parallax value between any image pixel in the original image in the first image and the second image; Calculate the parallax value using the depth solving formula to obtain the pixel depth corresponding to the image pixel; Fuse the pixel depths corresponding to each image pixel to obtain a depth image.

7. The device according to any one of claims 1 to 4, characterized in that, The microprocessing chip determines the clarity weight corresponding to the image pixel in the following manner: Obtain the pixel regions of the image pixel in each refocused image respectively; Determine the clearest region with the highest clarity from the pixel regions of the image pixel in each refocused image respectively, and use the refocused image corresponding to the clearest region as the clarity weight of the image pixel.

8. The device according to any one of claims 1 to 4, characterized in that, The device further includes: A user terminal for displaying a target image carrying texture information and / or a depth image carrying depth information.

9. An imaging method for extending the depth of field of an endoscope, characterized in that, Comprising: Obtaining a light field image by using a microlens array to acquire an original image collected by an endoscope, wherein the microlens array includes a plurality of microlenses with positive optical power, and the light field image includes sub-aperture images respectively corresponding to the microlenses; Performing depth calculation based on the parallax between different sub-aperture images to obtain the pixel depth corresponding to an image pixel in the original image; Refocusing each sub-aperture image on the focal plane corresponding to the pixel depth to obtain a refocused image respectively corresponding to each sub-aperture image, and determining the clarity weight corresponding to the image pixel from each refocused image; Performing image fusion on the pixel regions of each image pixel in the corresponding clarity weight to obtain a target image.

10. An electronic device, characterized in that, Comprising: A processor and a memory; The memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the electronic device executes the depth of field extension imaging method for an endoscope as described in claim 9.

Images