De-occlusion image display device and method for endoscope, and electronic device

Abstract

The invention relates to the technical field of endoscope imaging, and provides a de-blocking image display device and method for an endoscope and electronic equipment. According to the method, the shielding image is acquired, the multi-aperture image corresponding to the shielding image is acquired by using the micro-lens array, phase distribution calculation is performed according to the multi-aperture image to obtain wavefront phase distribution corresponding to the multi-aperture image, and phase distortion correction is performed on the shielding image according to the wavefront phase distribution to obtain the corrected image. Compared with a space-time registration algorithm for recovering image occlusion through calculation of inter-frame correlation, the space-time registration algorithm has the advantages that the inter-frame correlation does not need to be calculated by collecting multiple frames of images, so that the time-space registration accuracy is improved, and the time-space registration efficiency is improved. Only the occlusion image and the multi-aperture image of the same frame need to be collected, so that the calculation complexity of an occlusion removing algorithm is reduced, and the occlusion removing efficiency of the endoscope image is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of endoscope imaging, and in particular to a de-occlusion image display device and method for an endoscope and an electronic device. BACKGROUND

[0002] As a medical device integrating optics, electronics and software technology, the core function of an endoscope relies on the visual presentation of an optical imaging system to an optical image. Traditional endoscopes use visible light band imaging, but when there is turbid occlusion medium such as blood or tissue fluid between the endoscope and the target object, the medium has strong absorption characteristics to visible light, which will cause rapid attenuation of the light signal, and then form occlusion phenomena such as speckle noise on the endoscope image, resulting in a decrease in image contrast and loss of tissue tomography information, and affecting the image quality.

[0003] At present, although image clarification can be performed by de-occlusion display methods such as speckle autocorrelation algorithm, these de-occlusion display methods rely on space-time registration, not only need to collect multiple frames of dynamic occlusion images to de-occlude by calculating the inter-frame correlation, but also the blood flow will cause the speckle pattern to change rapidly with time, and the phase difference of each frame needs to be accurately aligned, resulting in an exponential increase in the amount of calculation with the frame rate. Therefore, the existing de-occlusion display method has the defects of complex calculation process and insufficient stability, which directly leads to low de-occlusion efficiency of the endoscope image, and it is difficult to meet the requirements of image timeliness in the field of endoscope application. SUMMARY

[0004] To provide a simple overview of some aspects of the disclosed embodiments the following is presented. The summary is not an extensive overview of all of the embodiments nor is it intended to determine key / critical elements of the embodiments or to delineate the scope of the embodiments but is presented to provide a basic understanding of some aspects of the disclosed embodiments.

[0005] In view of the above-mentioned defects of the prior art, the present application provides a de-occlusion image display device and method for an endoscope and an electronic device to solve the technical problem of low de-occlusion efficiency of the endoscope image.

[0006] The application provides a kind of unoccluded image display device for endoscope, the endoscope is in the environment where target observer is in the presence of occluded medium, image acquisition is carried out to the target observer, and the occluded image is obtained, the device includes: imaging module, for obtaining occluded image, while, using microlens array obtains the multi-aperture image corresponding to the occluded image, wherein the microlens array includes a plurality of microlenses with positive focal power;Microprocessing chip is used for calculating phase distribution according to the multi-aperture image, and the wavefront phase distribution corresponding to the multi-aperture image is obtained;According to the wavefront phase distribution, the phase distortion correction of the occluded image is carried out, and the corrected image is obtained;By phase recovery to the corrected image, the medium-free image corresponding to the occluded image is obtained.

[0007] In an embodiment of the present application, the imaging module includes: a beamsplitter for dividing the light beam transmission path of the occluded image into a first optical path and a second optical path; a Hartmann sensor is arranged in the first optical path, wherein the Hartmann sensor includes the microlens array and a first image sensor, the first image sensor is arranged on the image side of the microlens array, and the first image sensor is used to acquire the multi-aperture image; a second image sensor is arranged in the second optical path, and the second image sensor is used to acquire the occluded image.

[0008] In an embodiment of the present application, the microprocessing chip calculates the phase distribution according to the multi-aperture image to obtain the wavefront phase distribution corresponding to the multi-aperture image in the following way: the multi-aperture image includes a sub-aperture image corresponding to each microlens; according to the multi-aperture image, the light spot displacement corresponding to each sub-aperture image is obtained, and the wavefront tilt angle of each sub-aperture image mapped in the multi-aperture image is calculated according to each light spot displacement; the wavefront tilt angle corresponding to each sub-aperture image is taken as a first tilt angle, the wavefront tilt angle between the first tilt angles is completed by interpolation method to obtain a second tilt angle, and the wavefront tilt distribution is generated according to the first tilt angle and the second tilt angle; the wavefront tilt distribution is converted to wavefront phase gradient by calculation according to the wavefront tilt distribution, and the wavefront phase gradient is integrated to obtain the wavefront phase distribution.

[0009] In an embodiment of the present application, the microprocessing chip obtains the wavefront tilt angle of the sub-aperture image mapped in the multi-aperture image by the following formula: ; In the formula, is the wavefront tilt angle of the first sub-aperture image mapped in the multi-aperture image, is the light spot displacement corresponding to the first sub-aperture image, The focal length of the microlens.

[0010] In an embodiment of the present application, the micro-processing chip obtains the wavefront phase gradient by the following formula: , wherein, is the wavefront phase distribution, is the gradient operator, is the wavelength of the light wave, is the wavefront tilt distribution.

[0011] In an embodiment of the present application, the micro-processing chip corrects the phase distortion of the occluded image according to the wavefront phase distribution to obtain a corrected image by the following method: converting the wavefront phase distribution to a frequency domain function by Fourier transform to obtain a frequency domain distortion distribution; calculating the frequency domain compensation factor according to the frequency domain distortion distribution, and converting the frequency domain compensation factor to a spatial domain function by inverse Fourier transform to obtain a spatial domain correction kernel; and convolving the occluded image according to the spatial domain correction kernel to obtain a corrected image with phase information.

[0012] In an embodiment of the present application, the micro-processing chip obtains the spatial domain correction kernel by the following formula: , wherein, is the Fourier transform, is the regularization coefficient, is the frequency domain compensation factor, is the inverse Fourier transform.

[0013] In an embodiment of the present application, the micro-processing chip obtains the medium-free image corresponding to the occluded image by phase recovery of the corrected image by the following method: obtaining a training sample set, wherein the training sample set includes a plurality of corrected image samples and medium-free image labels corresponding to each of the corrected image samples; performing model training on a preset neural network model according to the training sample set, and calculating the model loss result between the model output result and the medium-free image label according to a preset loss function, so as to control the number of iterations of the model training according to the model loss result, wherein the model output result is obtained by inputting the corrected image sample into the neural network model; taking the neural network model after the model training as a phase recovery model; and inputting the corrected image into the phase recovery model to obtain a medium-free image.

[0014] The application provides a method for displaying unobstructed images of an endoscope, which collects images of a target object in an environment with an obstructive medium to obtain an obstructed image, and the method comprises the following steps: acquiring the obstructed image, and simultaneously acquiring a multi-aperture image corresponding to the obstructed image by using a microlens array, wherein the microlens array comprises a plurality of microlenses with positive focal lengths; calculating a phase distribution according to the multi-aperture image to obtain a wavefront phase distribution corresponding to the multi-aperture image; correcting a phase distortion of the obstructed image according to the wavefront phase distribution to obtain a corrected image; and obtaining a medium-free image corresponding to the obstructed image by performing phase recovery on the corrected image.

[0015] In an embodiment of the application, the obstructed image and the multi-aperture image are acquired by an imaging module, wherein the imaging module comprises: a beam splitter for dividing a light beam transmission path of the obstructed image into a first optical path and a second optical path; a Hartmann sensor arranged in the first optical path, wherein the Hartmann sensor comprises the microlens array and a first image sensor, the first image sensor is arranged on an image side of the microlens array, and the first image sensor is used to acquire the multi-aperture image; and a second image sensor arranged in the second optical path, the second image sensor is used to acquire the obstructed image.

[0016] In an embodiment of the application, the wavefront phase distribution corresponding to the multi-aperture image is calculated according to the multi-aperture image, which comprises the following steps: the multi-aperture image comprises a plurality of sub-aperture images corresponding to the microlenses respectively; the light spot displacements corresponding to the sub-aperture images are obtained by measuring the multi-aperture image, and the wavefront tilt angles of the sub-aperture images respectively mapped on the multi-aperture image are obtained by calculating according to the light spot displacements respectively; the wavefront tilt angles corresponding to the sub-aperture images are taken as first tilt angles respectively, the wavefront tilt angles between the first tilt angles are completed by using an interpolation method to obtain second tilt angles, and the wavefront tilt distribution is generated according to the first tilt angles and the second tilt angles; the wavefront tilt distribution is converted to a wavefront phase gradient by calculating according to the wavefront tilt distribution, and the wavefront phase distribution is obtained by integrating the wavefront phase gradient.

[0017] In an embodiment of the application, the wavefront tilt angles of the sub-aperture images respectively mapped on the multi-aperture image are obtained by the following formula: ; wherein, is the wavefront tilt angle of the i-th sub-aperture image mapped on the multi-aperture image, is the wavefront tilt angle of the j-th sub-aperture image mapped on the multi-aperture image, is the light spot displacement corresponding to the i-th sub-aperture image, is the light spot displacement corresponding to the j-th sub-aperture image, and is the focal length of the microlens.

[0018] In an embodiment of the present application, the wavefront phase gradient is obtained by the following formula: ; wherein, is the wavefront phase distribution, is the gradient operator, is the wavelength of the light wave, is the wavefront tilt distribution.

[0019] In an embodiment of the present application, the phase distortion correction is performed on the occluded image according to the wavefront phase distribution to obtain a corrected image, including: converting the wavefront phase distribution to a frequency domain function by Fourier transform to obtain a frequency domain distortion distribution; calculating according to the frequency domain distortion distribution to obtain a frequency domain compensation factor, and converting the frequency domain compensation factor to a spatial domain function by inverse Fourier transform to obtain a spatial domain correction kernel; and performing convolution on the occluded image according to the spatial domain correction kernel to obtain a corrected image with phase information.

[0020] In an embodiment of the present application, the spatial domain correction kernel is obtained by the following formula: ; wherein, is the Fourier transform, is the regularization coefficient, is the frequency domain compensation factor, is the inverse Fourier transform.

[0021] In an embodiment of the present application, the medium-free image corresponding to the occluded image is obtained by performing phase recovery on the corrected image, including: obtaining a training sample set, wherein the training sample set includes a plurality of corrected image samples, and a medium-free image label corresponding to each of the corrected image samples; performing model training on a preset neural network model according to the training sample set, and calculating a model loss result between a model output result and the medium-free image label according to a preset loss function, so as to control the number of iterations of the model training according to the model loss result, wherein the model output result is obtained by inputting the corrected image sample into the neural network model; taking the neural network model after the model training as a phase recovery model; and inputting the corrected image into the phase recovery model to obtain a medium-free image.

[0022] The present application provides an electronic device, including: a processor and a memory; the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory to make the electronic device execute the above-mentioned method.

[0023] The present application discloses a computer readable storage medium, which stores a computer program: the computer program is executed by a processor to realize the above-mentioned method.

[0024] The beneficial effects of the present application are as follows: By acquiring the occluded image, simultaneously acquiring the multi-aperture image corresponding to the occluded image by using the microlens array, performing phase distribution calculation according to the multi-aperture image to obtain the wavefront phase distribution corresponding to the multi-aperture image, and performing phase distortion correction on the occluded image according to the wavefront phase distribution to obtain the corrected image, the phase recovery is performed on the corrected image to obtain the medium-free image. In this way, compared with the time-space registration algorithm for recovering the occluded image by calculating the inter-frame correlation, the inter-frame correlation does not need to be calculated by acquiring multiple frames of images, only the occluded image and the multi-aperture image of the same frame need to be acquired, the wavefront phase distribution is directly measured according to the multi-aperture image, the phase distortion correction is performed on the occluded image according to the wavefront phase distribution to obtain the corrected image, and then the phase recovery is performed on the corrected image to estimate the original propagation path of the light field through the phase inverse operation, to reconstruct the structural information of the occluded area, and to obtain the medium-free image corresponding to the occluded image, thereby reducing the calculation complexity of the occlusion removal algorithm and improving the occlusion removal efficiency of the endoscope image. BRIEF DESCRIPTION OF DRAWINGS

[0025] The accompanying drawings, which are incorporated into and form a part of the specification, illustrate one embodiment consistent with the present application and, together with the description, serve to explain the principles of the application. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained from these drawings without creative labor for those skilled in the art.

[0026] In the drawings: Figure 1 is a structural schematic diagram of an occlusion-removed image display device for an endoscope in an embodiment of the present application; Figure 2 is a structural schematic diagram of an optical lens in an embodiment of the present application; Figure 3 is a structural schematic diagram of a microlens array in an embodiment of the present application; Figure 4 is a structural schematic diagram of an imaging device in an embodiment of the present application; Figure 5 is a flow schematic diagram of a training sample set acquisition method in an embodiment of the present application; Figure 6 is a structural schematic diagram of a model structure of a phase recovery model in an embodiment of the present application; Figure 7 is a flow schematic diagram of a phase recovery model training method in an embodiment of the present application; Figure 8 is a flow schematic diagram of an occlusion-removed image display method for an endoscope in an embodiment of the present application; Figure 9Fig. 1 is a structural schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0027] The above and other advantages and features of the present application will become apparent from the following description of the embodiments, taken in conjunction with the accompanying drawings, which illustrate, by way of example, the principles of the application. This description is given for the sake of example and the details are not intended to limit the present application. The present application can be implemented in many different ways and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete and will fully convey the scope of the application to those skilled in the art.

[0028] It is noted that the drawings of the embodiments provided in the following description are only to illustrate the basic idea of the present application, and the drawings only show the components related to the present application, not the number, shape and size of the components when actually implemented. The actual implementation of each component can be a random change, and the component layout pattern can be more complex.

[0029] In the following description, numerous specific details are discussed so as to provide a thorough understanding of the embodiments of the present application. However, it will be apparent to one of ordinary skill in the art that the embodiments of the present application can be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form in order to avoid obscuring the embodiments of the present application.

[0030] The terms "first", "second", and the like, in the description and in the claims of the present application and the above drawings, are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion.

[0031] Unless otherwise specified, the term "a plurality of" means two or more.

[0032] In the present application, the character " / " represents an "or" relationship between the objects before and after. For example, A / B means: A or B.

[0033] The term "and / or" is a description of the relationship between the objects, which means that there can be three relationships. For example, A and / or B means: A or B, or, A and B, three relationships.

[0034] Combination Figure 1As shown, the present application provides a kind of unoccluded image display device for endoscope, including imaging module 101 and micro processing chip 102, wherein endoscope is in the environment where target observation object is in the presence of shielding medium, image acquisition is carried out to the target observation object, and shielding image is obtained.

[0035] In some embodiments, the target observation object is in the environment where shielding medium exists, wherein the shielding medium includes blood, tissue fluid, etc., especially in the blood environment, blood forms a local turbidity layer in front of the lens, and the line of sight is shielded to affect the observation effect;In addition, red blood cells and hemoglobin in blood have strong absorption characteristics to visible light, resulting in light intensity attenuation, and scattering to form a random phase screen, generating speckle noise, so that the phase distortion presents high-frequency random characteristics.

[0036] It should be noted that the above-mentioned endoscope for image acquisition of target observation object specifically refers to endoscope viewfinder, including hard viewfinder and soft viewfinder, hard viewfinder is composed of hard mirror tube which cannot be deformed and its internal objective group, eyepiece group, rod lens group, illumination optical fiber, instrument channel, water injection and suction channel, etc.;Soft viewfinder is composed of one part of tip portion, bending portion and operation portion, etc., the tip portion is provided with objective lens for imaging, illumination lens for providing light source, etc., the bending portion can be flexibly bent, the observation direction is adjusted, the inside of the bending portion is integrated with light guide beam for transmitting light source, image guide beam for transmitting image, etc., and the operation portion is used to adjust the bending direction and angle of the bending portion, control the air flow and water flow for cleaning lens or expanding body cavity, etc.

[0037] Imaging module 101 is used to obtain the above-mentioned shielding image, and at the same time, the multi-aperture image corresponding to the shielding image is obtained by using microlens array, wherein the microlens array includes a plurality of microlenses with positive focal length.

[0038] In the embodiment, since the microlens array is composed of a plurality of microlenses in the same plane, and each microlens has positive focal length, so that the image on the object side of the microlens forms a reduced image on the image side of the microlens, therefore, when the original image passes through each microlens from the object side of the microlens array, a sub-aperture image can be formed on the image side of the microlens.

[0039] Combined Figure 1 As shown, imaging module 101 is arranged on the eyepiece side of endoscope.

[0040] Combined Figure 1 As shown, the shielding image recovery device of the present application further includes optical lens 201, which is arranged between the eyepiece side of endoscope and imaging module 101.

[0041] Combined Figure 2As shown, the application also provides an optical lens, the optical lens 201 is sequentially provided with a first lens 2011, a second lens 2012, a third lens 2013, a fourth lens 2014, a fifth lens 2015, a sixth lens 2016 and a seventh lens 2017 along an optical axis, wherein the optical lens 201 is close to an ocular side of the endoscope.

[0042] The first lens 2011 is a protective lens.

[0043] The second lens 2012 has a negative focal length, and the object side is concave and the image side is convex.

[0044] In this embodiment, the second lens 2012 uses a negative focal length to produce a diverging effect on the incident parallel light beam, so that the principal plane is moved backward, thereby obtaining a longer working distance than the actual focal length in the case of a shorter physical focal length.

[0045] The first lens 2011 and the second lens 2012 are provided with a diaphragm 2018.

[0046] The third lens 2013 has a positive focal length, and the object side is convex and the image side is concave.

[0047] The fourth lens 2014 has a positive focal length, and the image side is convex.

[0048] The fifth lens 2015 has a positive focal length, and the object side is convex.

[0049] The sixth lens 2016 has a positive focal length, and the object side is convex and the image side is concave.

[0050] In this embodiment, the third lens to the sixth lens are respectively positive focal length lenses, which focus the diverging light beam, and through the negative-positive combination of focal length distribution, the incident angle of the off-axis light beam can be significantly reduced, and a larger field of view angle can be achieved.

[0051] The seventh lens 2017 has a negative focal length, and the object side is convex and the image side is concave.

[0052] In this embodiment, the seventh lens 2017 uses a negative focal length lens to further adjust the optical path, adapt to the image sensor, and avoid edge image quality degradation.

[0053] The sixth lens 2016 and the seventh lens 2017 are cemented lenses.

[0054] In this embodiment, by cementing lenses of different materials and refractive indexes, aberration can be eliminated and imaging quality can be improved; at the same time, the cemented lens can reduce the number of air and glass interface, thereby reducing the reflection loss, in addition, the cemented lens can also compensate for the difference in curvature radius of the cemented surface, thereby reducing the requirement for processing precision and simplifying the processing process.

[0055] The optical lens 201 is provided with a shell outside.

[0056] In combination Figure 3 As shown, the present application also provides a microlens array, wherein the microlens array comprises a plurality of microlenses with positive focal power, and each microlens is in the same plane.

[0057] In some embodiments, in the microlens array, the microlens focal length of each microlens is the same.

[0058] Optionally, the imaging module comprises: a beam splitter configured to split the occlusion image into a first optical path and a second optical path; a Hartmann sensor disposed in the first optical path, wherein the Hartmann sensor comprises a microlens array and a first image sensor, the first image sensor is disposed on the image side of the microlens array, and the first image sensor is configured to acquire a multi-aperture image; and a second image sensor disposed in the second optical path, the second image sensor being configured to acquire the occlusion image.

[0059] In combination Figure 4 As shown, the present application also provides an imaging device, wherein the imaging module 101 is disposed on the image side of the optical lens 201, and the imaging module 101 comprises a microlens array 202, a first image sensor 203, a beam splitter 205, and a second image sensor 206.

[0060] The beam splitter 205 is disposed between the optical lens 201 and the microlens array 202, and the beam splitter 205 is configured to split the original image collected by the endoscope into a first optical path and a second optical path.

[0061] The first optical path sequentially comprises the microlens array 202 and the first image sensor 203 along the optical axis, wherein the Hartmann sensor is a module composed of the microlens array 202 and the first image sensor 203.

[0062] The first image sensor 203 is disposed on the image side of the microlens array, and the first image sensor is configured to acquire a multi-aperture image on the image side of the microlens array.

[0063] A first protective glass 204 is disposed between the first image sensor 203 and the microlens array 202, wherein the first protective glass 204 is provided with a filter film.

[0064] The first protective glass serves as the last ring of the optical path, protecting the first image sensor from damage such as collision and scratching, and at the same time, the parallel surface design between the protective glass and the first image sensor can reduce optical path refraction distortion, ensure optical flatness, and avoid phase calculation errors.

[0065] The filter film of the first protective glass can filter ambient stray light, and ensure image accuracy. Alternatively, through different settings, the filter film can also be used to filter other special light to achieve special functions, for example: the filter film adopts an infrared band-pass filter film, which allows infrared light to pass through and blocks visible light, thereby reducing speckle noise caused by blood scattering.

[0066] The second image sensor 206 is arranged in the second optical path, and the second image sensor is configured to capture images of the second optical path.

[0067] The second protective glass 207 is arranged between the second image sensor 206 and the beam splitter 205, wherein the second protective glass 207 is provided with a filter film.

[0068] The second protective glass serves as the last ring of the optical path, and protects the second image sensor from damage such as collision and scratching.

[0069] The filter film of the second protective glass can filter ambient stray light, and ensure image accuracy. Alternatively, through different settings, the filter film can also be used to filter other special light to achieve special functions, for example: the filter film adopts an infrared band-pass filter film, which allows infrared light to pass through and blocks visible light, thereby reducing speckle noise caused by blood scattering.

[0070] In some embodiments, the first image sensor and the second image sensor are both CMOS (Complementary Metal Oxide Semiconductor) sensors.

[0071] In this way, through the double-channel design of the beam splitting system combined with the jointly optimized cemented lens group, the light signal of the first optical path is captured by the first image sensor, and the phase distribution is calculated according to the captured multi-aperture image to obtain the wavefront phase distribution corresponding to the multi-aperture image. At the same time, the light signal of the second optical path is captured by the second image sensor to obtain a blocked image. In this way, the phase distortion of the blocked image is corrected according to the wavefront phase distribution to obtain a corrected image. Through phase recovery of the corrected image, the original propagation path of the light field is estimated through phase inverse operation, the structural information of the blocked area is reconstructed, the detail information of the blocked area is recovered, and a non-medium image is obtained. At the same time, by splitting the optical path into two parts through the beam splitter, only a small modification needs to be made to the original endoscope structure to realize the present scheme, which can effectively save design and manufacturing costs.

[0072] The microprocessor chip 102 is used to calculate the phase distribution based on the multi-aperture image to obtain the wavefront phase distribution corresponding to the multi-aperture image; perform phase distortion correction on the occluded image based on the wavefront phase distribution to obtain a corrected image; and obtain a medium-free image corresponding to the occluded image by performing phase recovery on the corrected image.

[0073] In some embodiments, the microprocessor chip 102 includes a processing chip with algorithm logic, such as an MCU (Microcontroller Unit) and an FPGA (Field Programmable Gate Array). The multi-aperture image and the occlusion image captured by the imaging module are input into the microprocessor chip, and the microprocessor chip is controlled to perform phase distribution calculation based on the multi-aperture image to obtain the wavefront phase distribution corresponding to the multi-aperture image. The phase distortion of the occlusion image is corrected according to the wavefront phase distribution to obtain a corrected image. By performing phase recovery on the corrected image, a medium-free image corresponding to the occlusion image is obtained.

[0074] In some embodiments, the microprocessor chip 102 is connected to a user terminal, which is used to display the corrected image and / or the medium-free image, wherein the user terminal includes a monitor, a touch screen, a mobile display device, etc.

[0075] The de-occlusion image display device for an endoscope provided in the present application acquires an occlusion image and uses a microlens array to acquire a multi-aperture image corresponding to the occlusion image. Phase distribution calculation is performed based on the multi-aperture image to obtain a wavefront phase distribution corresponding to the multi-aperture image. Phase distortion correction is performed on the occlusion image based on the wavefront phase distribution to obtain a corrected image. Phase recovery is then performed on the corrected image to eliminate the occluding medium in the occlusion image and obtain a medium-free image. Thus, compared to spatiotemporal registration algorithms that recover image occlusion by calculating inter-frame correlation, there is no need to acquire multiple frames of images to calculate inter-frame correlation. Instead, only the occlusion image and its multi-aperture image of the same frame need to be acquired. The wavefront phase distribution is directly measured based on the multi-aperture image. Phase distortion correction is performed on the occlusion image based on the wavefront phase distribution to obtain a corrected image. Phase recovery is then performed on the corrected image to estimate the original propagation path of the light field through a phase inversion operation, reconstruct the structural information of the occluded area, and obtain a medium-free image corresponding to the occlusion image. This reduces the computational complexity of the de-occlusion algorithm and improves the de-occlusion efficiency of endoscopic images.

[0076] Optionally, the phase distribution is calculated according to the multi-aperture image to obtain a wavefront phase distribution corresponding to the multi-aperture image, including: the multi-aperture image includes sub-aperture images corresponding to the respective micro-lenses; the multi-aperture image is measured to obtain a light spot displacement corresponding to each sub-aperture image, and the wavefront tilt angle of each sub-aperture image mapped to the multi-aperture image is calculated according to the respective light spot displacements; the wavefront tilt angles corresponding to the respective sub-aperture images are taken as first tilt angles, the wavefront tilt angles between the first tilt angles are completed by interpolation to obtain second tilt angles, and the wavefront tilt distribution is generated according to the first tilt angles and the second tilt angles; the wavefront tilt distribution is converted to a wavefront phase gradient by calculation, and the wavefront phase gradient is integrated to obtain the wavefront phase distribution.

[0077] In some embodiments, the wavefront tilt angle of the sub-aperture image mapped to the multi-aperture image is obtained by formula (1): Formula (1) In formula (1), is the wavefront tilt angle of the i-th sub-aperture image mapped to the multi-aperture image, is the light spot displacement corresponding to the i-th sub-aperture image, is the focal length of the micro-lens. In some embodiments, the light spot displacement and the wavefront tilt angle are both two-dimensional parameters, including horizontal and vertical directions, wherein the wavefront tilt angle in the horizontal direction is obtained by solving according to the light spot displacement in the horizontal direction, the wavefront tilt angle in the vertical direction is obtained by solving according to the light spot displacement in the vertical direction, and the wavefront tilt angle of the sub-aperture image mapped to the multi-aperture image is obtained by combining the wavefront tilt angle in the horizontal direction and the wavefront tilt angle in the vertical direction.

[0078] In some embodiments, the image resolution of the multi-aperture image is , the micro-lens array is composed of micro-lenses; the sub-aperture image corresponding to each micro-lens can be calculated to obtain a wavefront tilt angle by the light spot displacement, therefore, the number of the first tilt angles is

[0079] ; the difference value of the second tilt angle between the adjacent two sub-aperture images is calculated by taking the first tilt angles corresponding to the adjacent two sub-aperture images as the reference, and the wavefront tilt distribution is finally formed, for example, the wavefront tilt distribution of is obtained by completing first tilt angles according to the bilinear interpolation algorithm, each pixel corresponds to a wavefront tilt angle.

[0080] ​​​In some embodiments, the wavefront phase gradient is obtained by formula (2): Formula (2) In formula (2), is the wavefront phase distribution, is the gradient operator, is the wavelength of the light wave, is the wavefront tilt distribution.

[0081] In some embodiments, the wavefront tilt is essentially the scattering or refraction of the occlusion medium to the light wave, forming the wavefront phase distortion. Based on geometric optics, the wavefront phase gradient and the wavefront tilt distribution are positively correlated, and the proportional coefficient is determined by the wavelength of the light wave ; based on the phase period being , the wavelength normalization factor is determined according to the wavelength of the light wave; based on the wavelength normalization factor, the angle of the wavefront tilt angle is converted into the phase gradient.

[0082] In some embodiments, the wavefront phase distribution is obtained by formula (3): Formula (3) In formula (3), is the two-dimensional integral.

[0083] In some embodiments, according to the basic theorem of vector calculus, the integral of the gradient of a scalar field along a path is equal to the difference of the scalar field at the endpoints of the path. Since the wavefront tilt represents the local tilt angle of the light wavefront at , the wavefront tilt distribution is equivalent to the directional derivative of the wavefront phase gradient. By regarding the wavefront phase gradient as a local slope and the wavefront phase distribution as a global cumulative height, the wavefront phase gradient is integrated to obtain the wavefront phase distribution.

[0084] Optionally, the phase distortion correction of the occlusion image is performed according to the wavefront phase distribution to obtain a corrected image, including: converting the wavefront phase distribution to a frequency domain function by Fourier transform to obtain a frequency domain distortion distribution; calculating the frequency domain compensation factor according to the frequency domain distortion distribution, and converting the frequency domain compensation factor to a spatial domain function by inverse Fourier transform to obtain a spatial domain correction kernel; and convolving the occlusion image according to the spatial domain correction kernel to obtain a corrected image with phase information.

[0085] In an embodiment, the spatial correction kernel is calculated based on the wavefront phase distribution. When performing convolution operation on the occluded image, two effects are achieved: on the one hand, the phase distortion caused by the medium is compensated based on the frequency domain deconvolution principle to achieve clear restoration of the image spatial structure; on the other hand, the spatial distribution information of the phase gradient is embedded in the corrected image in the form of convolution kernel parameters, providing spatial-phase joint features for the subsequent deep learning-based phase recovery model, thereby achieving an image de-occlusion link.

[0086] Optionally, the spatial domain correction kernel is determined by formula (4): Formula (4) In formula (4), is the Fourier transform, is the regularization coefficient, is the frequency domain compensation factor, is the inverse Fourier transform.

[0087] In some embodiments, combined with speckle imaging theory, the relationship between the distorted image and the undistorted image in the frequency domain is represented as: ,in, is the distorted image, is an undistorted image, is the image noise, is the frequency domain response of the point spread function (PSF). is the frequency domain distortion distribution ; The undistorted image spectrum and Multiplication causes the phase of the low-frequency component to slow down, blurring the image as a whole, and the phase of the high-frequency component to fuse, resulting in the loss of details such as cell texture, forming a distorted image spectrum. ; It can be seen that if the distorted image is to be Restore to undistorted image , need to be based on Spectrum of distorted image Inverse and get the undistorted image spectrum However, due to image noise The energy is high in the high frequency region. If we directly analyze the distorted image spectrum, Inverse operation will lead to noise amplification, and a regularization term needs to be introduced to balance the recovery effect and noise suppression. Therefore, formula (4) is used as a frequency domain deconvolution algorithm based on phase measurement. It is essentially used to perform an inverse operation on the known phase distortion in the frequency domain. By regularizing and balancing signal recovery and noise suppression, a spatial domain correction kernel that can be convolved in real time is finally generated.

[0088] In some embodiments, the frequency domain distortion distribution The Fourier transform of the wavefront phase distribution is used to identify the dominant frequencies that cause the image blur, including low frequency scattering, high frequency speckle, etc.

[0089] In some embodiments, the frequency domain energy spectrum of the phase distortion is used to quantify the distortion intensity of each frequency component, and the higher the energy, the stronger the compensation is needed.

[0090] In some embodiments, the regularization coefficient is used to control the correction strength corresponding to the spatial domain correction kernel, and suppress high frequency noise to avoid numerical instability caused by the denominator approaching zero.

[0091] In some embodiments, the frequency domain compensation factor is derived based on Wiener filtering, and is a compensation function with band-pass filtering characteristics, for example, if the image region is in the low frequency band of the frequency response, the frequency domain energy spectrum is large, and the frequency domain compensation factor is large accordingly, so as to enhance the attenuated low frequency signal in the image region, and if the image region is in the high frequency band of the frequency response, the frequency domain energy spectrum is small, so that the frequency domain compensation factor tends to decrease to 1, thereby suppressing the noise amplification in the image region.

[0092] In some embodiments, the frequency domain compensation factor is converted to the spatial domain by inverse Fourier transform to form a spatial domain correction kernel, and the spatial domain distribution of the spatial domain correction kernel corresponds to the spatial position and amplitude of the wavefront distortion; the spatial domain correction kernel is convolved with the occluded image, which is equivalent to linear compensation of phase distortion for each pixel in the spatial domain, and at the same time, the phase information is fused into the occluded image through the convolution operation to form a corrected image with phase information.

[0093] Optionally, the medium-free image corresponding to the occluded image is obtained by performing phase recovery on the corrected image, including: obtaining a training sample set, wherein the training sample set includes a plurality of corrected image samples and medium-free image labels corresponding to each corrected image sample; performing model training on a preset neural network model according to the training sample set, and calculating a model loss result between a model output result and the medium-free image label according to a preset loss function, so as to control the number of iterations of the model training according to the model loss result, wherein the model output result is obtained by inputting the corrected image sample into the neural network model; taking the neural network model after the model training as a phase recovery model; and inputting the corrected image into the phase recovery model to obtain the medium-free image.

[0094] In combination with Figure 5As shown, the present application also provides a training sample set acquisition method, comprising: when the target observation object is in an environment without a shielding medium, acquiring an image of the target observation object through an endoscope to obtain a medium-free image; and when the target observation object is in an environment with a shielding medium, acquiring an image of the target observation object through an endoscope to obtain a shielding image corresponding to the medium-free image, and a multi-aperture image corresponding to the shielding image; performing phase distribution calculation on the multi-aperture image to obtain a wavefront phase distribution; calculating the wavefront phase distribution to obtain a spatial domain correction kernel; convolving the shielding image corresponding to the multi-aperture image with the spatial domain correction kernel to correct phase distortion and obtain a corrected image sample; taking the medium-free image corresponding to the corrected image sample as the medium-free image label corresponding to the corrected image sample to form a sample image pair; and generating a training sample set according to a plurality of sample image pairs, wherein the training sample set , is the first corrected image sample, is the medium-free image label corresponding to the first corrected image sample, is the first corrected image sample, is the medium-free image label corresponding to the first corrected image sample, is the total amount of corrected image samples.

[0095] As shown in Figure 6 , the present application also provides a model structure of a phase recovery model, the phase recovery model adopts a U-NET (U-shaped Network) structure and is divided into an encoding unit and a decoding unit; the phase recovery model sequentially performs 4 times of downsampling and 4 times of upsampling on model input data, wherein each layer performs twice convolution and once 2x2 maximum pooling on the input data, and an activation function is processed after each convolution.

[0096] In some embodiments, the activation function adopts a linear rectifier (ReLU) function, and the linear rectifier function is represented as .

[0097] As shown in Figure 7 , the present application also provides a phase recovery model training method, comprising: inputting the corrected image sample into the phase recovery model to obtain a model output result; and iteratively training the phase recovery model according to a model loss result between the model output result and the medium-free image label by using a preset loss function, wherein the preset loss function adopts a mean squared error loss function (MSE, Mean Squared Error).

[0098] In some embodiments, the mean squared error loss function is represented by formula (5): Formula (5) In formula (5), is the model loss result, correcting image samples based on the neural network model outputted model output result.

[0099] in combination Figure 8 As shown in the drawings, the present application also provides a method for displaying a de-occlusion image of an endoscope, which comprises the following steps: In step S801, an occlusion image is acquired, and a multi-aperture image corresponding to the occlusion image is acquired by using a microlens array; The microlens array comprises a plurality of microlenses with positive focal power. The occlusion image is obtained by using the endoscope to collect an image of a target object in an environment where the target object is in the presence of an occlusion medium. In step S802, phase distribution calculation is performed according to the multi-aperture image to obtain a wavefront phase distribution corresponding to the multi-aperture image. In step S803, phase distortion correction is performed on the occlusion image according to the wavefront phase distribution to obtain a corrected image. In step S804, phase retrieval is performed on the corrected image to obtain a medium-free image corresponding to the occlusion image.

[0100] The method for displaying a de-occlusion image of an endoscope provided by the present application acquires an occlusion image, acquires a multi-aperture image corresponding to the occlusion image by using a microlens array, performs phase distribution calculation according to the multi-aperture image to obtain a wavefront phase distribution corresponding to the multi-aperture image, performs phase distortion correction on the occlusion image according to the wavefront phase distribution to obtain a corrected image, and then performs phase retrieval on the corrected image to eliminate the occlusion medium in the occlusion image and obtain a medium-free image. Thus, compared with a time-space registration algorithm for recovering image occlusion by calculating inter-frame correlation, the present application does not need to acquire multiple frames of images to calculate inter-frame correlation, but only needs to acquire the occlusion image and its multi-aperture image of the same frame, directly measures the wavefront phase distribution according to the multi-aperture image, performs phase distortion correction on the occlusion image according to the wavefront phase distribution to obtain a corrected image, and then performs phase retrieval on the corrected image to estimate the original propagation path of the light field through phase inverse operation, reconstructs the structural information of the occlusion area, and obtains a medium-free image corresponding to the occlusion image, thereby reducing the calculation complexity of the de-occlusion algorithm and improving the de-occlusion efficiency of the endoscope image.

[0101] In some embodiments, the specific implementation process of the method for displaying a de-occlusion image of an endoscope can refer to the embodiments of the device for displaying a de-occlusion image of an endoscope described above, and the method execution steps correspond to the device hardware architecture.

[0102] The application further provides an electronic device, comprising: a processor and a memory; the memory is used for storing a computer program, and the processor is used for executing the computer program stored in the memory, so that the electronic device executes the method described above.

[0103] Figure 9 A structural diagram of a computer system of an electronic device suitable for implementing the embodiments of the application is shown. It should be noted that, Figure 9 The computer system 900 of the electronic device shown is only an example, and should not impose any limitation on the functions and use range of the embodiments of the application.

[0104] As Figure 9 shown, the computer system 900 comprises a central processing unit (CPU) 901, which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 902 or programs loaded from a storage portion 908 into a random access memory (RAM) 903, such as performing the methods in the above embodiments. In the RAM 903, various programs and data required for system operation are also stored. The CPU 901, the ROM 902 and the RAM 903 are connected to each other through a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0105] The following components are connected to the I / O interface 905: an input portion 906 comprising a keyboard, a mouse, etc.; an output portion 907 comprising a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 908 comprising a hard disk, etc.; and a communication portion 909 comprising a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication portion 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to the I / O interface 905 as needed. A removable medium 911 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is mounted on the drive 910 as needed, so that a computer program read therefrom is installed into the storage portion 908 as needed.

[0106] The electronic device disclosed in the embodiment comprises a processor, a memory, a transceiver and a communication interface. The memory and the communication interface are connected with the processor and the transceiver and complete communication between each other. The memory is used for storing a computer program. The communication interface is used for communication. The processor and the transceiver are used for running the computer program, so that the electronic device executes each step of the method.

[0107] The embodiment of the present disclosure further provides a computer readable storage medium, which stores a computer program. The program is executed by a processor to implement any method in the embodiment.

[0108] The computer readable storage medium in the embodiment of the present disclosure can be understood by those skilled in the art that all or part of the steps of the above-mentioned method embodiments can be completed by the hardware of the computer program. The aforementioned computer program can be stored in a computer readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes ROM, RAM, magnetic disc or optical disc and various media that can store program codes.

[0109] The above description and drawings are illustrative of embodiments of the present disclosure and are not intended to be limiting. Other embodiments can include structural, logical, electrical, process, and other changes. Embodiments are merely representative of possible variations. Individual components and functions are optional and the order of operations can vary. Portions and sub-combinations of some embodiments can be included or replaced in or by other embodiments. Also, the phraseology and terminology used herein is for the purpose of description and should not be regarded as limiting. As used in the description of the embodiments and the claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. Similarly, the term "and / or" as used herein refers to any and all possible combinations of one or more of the associated listed items. Additionally, as used in this application, the term "comprises" and variations thereof do not intend to preclude the presence or addition of one or more other items to those stated in the compositions, integers, steps, operations, elements, and / or components. Without more limitations, an element defined by the phrase "comprises a..." does not exclude the presence of additional identical elements in the process, method, or apparatus including the element. In this document, each embodiment focuses on the differences from other embodiments, and the same or similar parts between embodiments can be referred to each other. For the method, product, etc. disclosed by the embodiments, if it corresponds to the method part disclosed by the embodiments, the relevant part can be referred to the description of the method part.

[0110] Those skilled in the art can understand that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application. Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described system, device and unit can refer to the corresponding process in the foregoing method embodiments, which will not be repeated here.

[0111] In the embodiments disclosed herein, the disclosed methods, products (including but not limited to devices, apparatuses, etc.) can be implemented in other manners. For example, the described device embodiments are merely schematic. For example, the division of the units is merely a logical function division. There can be another division manner for the actual implementation, for example, multiple units or components can be combined or integrated into another system, or some of the components can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections can be indirect couplings or communication connections through some interfaces, devices or units, and can be in electrical, mechanical or other forms. The unit illustrated as a separate component can or can not be physically separate, and can or can not be a physical component. Some or all of the units can be selected according to actual needs to implement the embodiments. In addition, the units in the application can be integrated into a processing unit, or each unit can exist physically as a separate entity, or two or more units can be integrated into a unit.

[0112] The flowcharts and block diagrams in the drawings show the architectural, functional and operational aspects of possible implementations of systems, methods and computer program products according to the present application. In this regard, each block in the flowcharts or block diagrams can represent a module, a program segment or a portion of code that contains one or more executable instructions for implementing the specified logical functions. In some alternative implementations, the functions noted in the blocks can occur in a different order than that shown in the drawings. For example, two consecutive blocks can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. In the descriptions corresponding to the flowcharts and block diagrams in the drawings, the operations or steps corresponding to different blocks can also occur in an order different from that disclosed in the description, and sometimes there is no specific order between different operations or steps. For example, two consecutive operations or steps can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. Each block in the block diagrams and / or flowcharts, and the combination of blocks in the block diagrams and / or flowcharts, can be implemented by a dedicated hardware-based system that performs the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

Claims

1. A de-obstructed image display device for an endoscope, wherein the endoscope collects an image of a target object under an environment where the target object is in the presence of an obstructing medium to obtain an obstructed image, characterized in that: The device comprises: An imaging module, configured to acquire the obstructed image and, at the same time, acquire a multi-aperture image corresponding to the obstructed image using a microlens array, wherein the microlens array includes a plurality of microlenses having positive optical power; The microprocessor chip is used to calculate the phase distribution based on the multi-aperture image to obtain the wavefront phase distribution corresponding to the multi-aperture image; perform phase distortion correction on the occluded image based on the wavefront phase distribution to obtain a corrected image; and obtain a medium-free image corresponding to the occluded image by performing phase recovery on the corrected image.

2. The device according to claim 1, characterized in that The imaging module includes: a beam splitter, configured to divide a transmission path of the light beam blocking the image into a first optical path and a second optical path; a Hartmann sensor disposed in the first optical path, wherein the Hartmann sensor comprises the microlens array and a first image sensor, the first image sensor being disposed on the image side of the microlens array and being used to capture the multi-aperture image; The second image sensor is arranged in the second optical path, and is used to collect the blocked image.

3. The device according to claim 1, characterized in that The microprocessor chip calculates the phase distribution based on the multi-aperture image in the following manner to obtain the wavefront phase distribution corresponding to the multi-aperture image: The multi-aperture image includes sub-aperture images corresponding to the microlenses respectively; Measuring according to the multi-aperture image to obtain the spot displacement corresponding to each of the sub-aperture images, and calculating according to each of the spot displacements to obtain the wavefront tilt angle of each of the sub-aperture images mapped to the multi-aperture image; taking the wavefront tilt angle corresponding to each of the sub-aperture images as a first tilt angle, interpolating the wavefront tilt angles between the first tilt angles to obtain a second tilt angle, and generating a wavefront tilt distribution based on the first tilt angles and the second tilt angles; Calculation is performed based on the wavefront tilt distribution to convert the wavefront tilt distribution into a wavefront phase gradient, and integration calculation is performed on the wavefront phase gradient to obtain a wavefront phase distribution.

4. The device according to claim 3, characterized in that The microprocessor chip obtains the wavefront tilt angle of the sub-aperture image mapped to the multi-aperture image through the following formula: Where, For the The sub-aperture images are mapped to the wavefront tilt angle of the multi-aperture image. For the The spot displacement corresponding to the sub-aperture image is is the focal length of the microlens.

5. The device according to claim 3, characterized in that The microprocessor chip obtains the wavefront phase gradient using the following formula: Where, is the wavefront phase distribution, is the gradient operator, is the wavelength of light, is the wavefront tilt distribution.

6. The device according to claim 1, characterized in that The microprocessor chip performs phase distortion correction on the blocked image according to the wavefront phase distribution in the following manner to obtain a corrected image: Converting the wavefront phase distribution into a frequency domain function by Fourier transform to obtain a frequency domain distortion distribution; Calculating according to the frequency domain distortion distribution to obtain a frequency domain compensation factor, and converting the frequency domain compensation factor into a spatial domain function through inverse Fourier transform to obtain a spatial domain correction kernel; The occluded image is convolved according to the spatial domain correction kernel to obtain a corrected image with phase information.

7. The device according to claim 6, characterized in that The microprocessor chip obtains the spatial domain correction kernel using the following formula: Where, is the Fourier transform, is the regularization coefficient, is the frequency domain compensation factor, is the inverse Fourier transform.

8. The device according to any one of claims 1 to 7, characterized in that The microprocessor chip performs phase recovery on the corrected image to obtain a medium-free image corresponding to the blocked image in the following manner: Acquire a training sample set, wherein the training sample set includes a plurality of corrected image samples and a non-medium image label corresponding to each of the corrected image samples; Performing model training on a preset neural network model based on the training sample set, and calculating a model loss result between a model output result and a label of the image without medium based on a preset loss function, so as to control the number of iterations of the model training based on the model loss result, wherein the model output result is obtained by inputting the rectified image sample into the neural network model; The trained neural network model is used as a phase recovery model; The corrected image is input into the phase recovery model to obtain a medium-free image.

9. A method for displaying an obstructed image for an endoscope, wherein the endoscope captures an image of a target object under an environment where the target object is in the presence of an obstructing medium to obtain an obstructed image, characterized in that: The method comprises: Acquire the occlusion image, and simultaneously acquire a multi-aperture image corresponding to the occlusion image using a microlens array, wherein the microlens array includes a plurality of microlenses with positive optical power; performing phase distribution calculation according to the multi-aperture image to obtain a wavefront phase distribution corresponding to the multi-aperture image; performing phase distortion correction on the blocked image according to the wavefront phase distribution to obtain a corrected image; By performing phase recovery on the corrected image, a medium-free image corresponding to the blocked image is obtained.

10. An electronic device, characterized in that: include: processor and memory; The memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the electronic device performs the method according to claim 9.

Images