Image processing method and application thereof, endoscopic optical lens and capsule endoscope

Abstract

The invention provides an image processing method and application thereof, an endoscopic optical lens and a capsule endoscope, which can realize field depth expansion by utilizing a wavefront coding mode so as to have a larger optical field depth on the premise of not sacrificing an optical resolution and a view field. The image processing method comprises the following steps: calibrating point spread functions of an endoscope module introducing a phase surface under a plurality of object distances and noise under a single object distance to obtain a noise model of the endoscope module and point spread functions of a plurality of different view fields; constructing a training data set based on the point spread functions of a plurality of different view fields under different object distances and the focusing images collected by the calibration camera under the corresponding object distances; based on the noise model of the endoscope module and the constructed training data set, training a depth-of-field extension model; and processing the original degraded image acquired by the endoscope module through the trained depth-of-field expansion model so as to expand the depth-of-field range of the endoscopic image.

Description

Technical Field

[0001] The present invention relates to the technical field of endoscopes, and in particular to an image processing method and application thereof, an optical lens for endoscopy, and a capsule endoscope. Background Art

[0002] Early cancer screening of tissues in the body, such as the gastrointestinal tract, is of great significance to the treatment of cancer, because effective early cancer screening can greatly improve the survival rate of cancer patients. At present, early cancer screening of the gastrointestinal tract mainly uses an insertable endoscope to observe the morphology of gastrointestinal tissues through the natural cavity of the human body, screen out suspicious lesions, and then take suspicious lesions for in vitro biopsy to determine the benign or malignant nature of the tumor. Because this insertable endoscopic detection method will cause great psychological and physical discomfort to the person being examined during the insertion of the endoscope, it generally needs to be used in conjunction with general anesthesia. Therefore, most people will only choose gastrointestinal endoscopy when there is obvious discomfort in the gastrointestinal tract, and this test is usually ignored in daily physical examinations, which may lead to missing the golden period of cancer treatment.

[0003] In addition, for some people with special underlying diseases such as heart disease and vascular disease, insertable endoscopy can cause serious complications in some patients, and generally cannot be used for examination, which makes the emerging capsule endoscope gradually accepted by the public. Capsule endoscope mainly uses wireless communication to transmit instructions and image data. When examining the gastrointestinal tract, it no longer needs to be connected to a long catheter, so it can greatly reduce the pain of users, and it is easy to use and can be used for daily examinations by the public.

[0004] However, due to the small size of the capsule endoscope, the length and number of lenses of its internal optical module are strictly limited, making it difficult to have a large optical depth of field without sacrificing optical resolution and field of view. Summary of the invention

[0005] One advantage of the present invention is that it provides an image processing method and its application, as well as an optical lens and capsule endoscope for endoscopy, which can achieve depth of field extension by wavefront coding so as to have a larger optical depth of field without sacrificing optical resolution and field of view.

[0006] Another advantage of the present invention is that it provides an image processing method and its application, an optical lens for endoscopy, and a capsule endoscope. In one embodiment of the present invention, the optical lens for endoscopy can introduce a third-order phase plane, so that the modulation transfer function (MTF) or point spread function (PSF) of the optical system is insensitive to defocus, ensuring that the same PSF is maintained throughout the depth of field.

[0007] Another advantage of the present invention is to provide an image processing method and its application, an optical lens for endoscopy, and a capsule endoscope. In one embodiment of the present invention, the optical lens for endoscopy can make the MTF have no zero point in the entire depth of field after introducing the third-order phase plane, thereby avoiding information loss.

[0008] Another advantage of the present invention is that it provides an image processing method and its application, an optical lens for endoscopy, and a capsule endoscope. In one embodiment of the present invention, the image processing method can restore the depth of field image by image restoration or AI reconstruction, thereby greatly expanding the depth of field range without affecting the original system parameters.

[0009] Another advantage of the present invention is to provide an image processing method and its application, an optical lens for endoscopy, and a capsule endoscope, wherein in order to achieve the above-mentioned purpose, no complicated structure and material are required in the present invention. Therefore, the present invention successfully and effectively provides a solution, not only providing a simple image processing method and its application, an optical lens for endoscopy, and a capsule endoscope, but also increasing the practicality and reliability of the image processing method and its application, and the optical lens for endoscopy, and the capsule endoscope.

[0010] In order to achieve at least one of the above advantages or other advantages and purposes of the present invention, the present invention provides an image processing method, comprising the steps of:

[0011] Calibrate the point spread function of the endoscope module introduced with the phase plane at multiple object distances and the noise at a single object distance to obtain the noise model of the endoscope module and the point spread function of multiple different fields of view;

[0012] A training data set is constructed based on point spread functions of multiple different fields of view at different object distances and in-focus images collected by a calibrated camera at corresponding object distances;

[0013] Based on the noise model of the endoscope module and the constructed training data set, training the depth of field extension model; and

[0014] The original degraded image collected by the endoscope module is processed by the trained depth of field extension model to expand the depth of field range of the endoscopic image.

[0015] According to one embodiment of the present application, the step of calibrating the point spread function of the endoscope module introduced with the phase plane at multiple object distances and the noise at a single object distance to obtain the noise model of the endoscope module and the point spread function of multiple different fields of view includes the steps of:

[0016] The endoscope module is used to photograph a calibration plate at K object distances in a bright room environment to sample original degraded images at K object distances within a depth of field, where K is a positive integer greater than 1;

[0017] Calibrate the point spread function of the original degraded image at each object distance to obtain the point spread functions of M*N different fields of view of the endoscope module at each object distance, where M and N are both positive integers greater than 1; and

[0018] The endoscope module is used to photograph a color card plate at a single object distance in a darkroom environment to calibrate the noise model of the endoscope module at the current object distance.

[0019] According to an embodiment of the present application, the calibration plate is a series of oblique checkerboards; the color card plate is a 24-color card.

[0020] According to one embodiment of the present application, the step of constructing a training data set based on point spread functions of multiple different fields of view at different object distances and in-focus images collected by a calibrated camera at corresponding object distances includes the steps of:

[0021] Using the calibration camera to shoot different scenes in the current application scenario under focus, so as to sample the in-focus images at the K object distances within the depth of field;

[0022] Perform extended depth of field fusion on the K in-focus images acquired at the K object distances to obtain a fully-focused image as label data;

[0023] The K in-focus images are convolved in blocks and calibrated at corresponding object distances to obtain M*N point spread functions of different fields of view, so as to obtain K single-layer convolution images at the K object distances;

[0024] Superimposing and fusing the K single-layer convolution images to obtain a blurred image as input data; and

[0025] Repeat the above steps in different application scenarios, and form training data pairs with the label data and input data obtained in different application scenarios to construct the training data set.

[0026] According to one embodiment of the present application, the step of repeating the above steps in different application scenarios and forming a training data pair with the label data and input data obtained in different application scenarios to construct the training data set includes the steps of:

[0027] Determine whether the number of training data pairs in the training data set meets the requirements;

[0028] If yes, complete the construction of the training data set; and

[0029] If not, construct a training data pair in the next application scenario to update the training data set.

[0030] According to one embodiment of the present application, the step of training the depth of field extension model based on the noise model of the endoscope module and the constructed training data set includes the steps of:

[0031] When obtaining a current training data pair from the training data set, randomly adding different gains to input data of the current training data pair to obtain gain input data;

[0032] Reading noises of different intensities in the noise model and adding them to the gain input data to obtain gain input data carrying noise;

[0033] Inputting the gain input data carrying noise into the depth of field extension model for training to output a network result;

[0034] Calculating a loss function based on the network structure and the label data of the current training data pair; and

[0035] The model parameters are iteratively optimized by gradient feedback to finally obtain a trained depth of field extension model.

[0036] According to one embodiment of the present application, the network structure of the depth of field extension model is Res-Unet; the Loss constraints of the depth of field extension model include an absolute value loss function and a total variational loss function.

[0037] According to an embodiment of the present application, the step of processing the original degraded image collected by the endoscope module through the trained depth of field extension model to expand the depth of field range of the endoscopic image includes the steps of:

[0038] According to the requirements of the actual application platform, the trained depth of field extension model is compressed and quantized;

[0039] Deploy according to the relevant interface documents of the actual application platform, and restore the original degraded image collected by the endoscope module to obtain a restored image; and

[0040] The restored image is further processed by an image signal processing model to output an endoscopic image with an extended depth of field.

[0041] According to another aspect of the present application, an embodiment of the present application further provides an electronic device, comprising

[0042] Processor; and

[0043] A memory, wherein at least one instruction is stored in the memory, and when the instruction is executed by the processor, the processor executes the above-mentioned image processing method.

[0044] According to another aspect of the present application, an embodiment of the present application further provides a medical device, including:

[0045] the electronic equipment described above; and

[0046] A capsule endoscope comprises: an optical lens for endoscopy having a three-dimensional phase plane, a photosensitive component arranged on the image side of the optical lens for endoscopy, a wireless communication module communicatively connected to the photosensitive component, and a capsule shell, wherein the optical lens for endoscopy, the photosensitive component and the wireless communication module are assembled inside the capsule shell, and the wireless communication module can be wirelessly connected to the electronic device.

[0047] According to another aspect of the present application, an embodiment of the present application further provides an optical lens for endoscopy, comprising a first lens with positive optical power, a second lens with optical power, a third lens with positive optical angle, and a fourth lens with negative optical power, which are coaxially arranged in sequence from the object side to the image side along the optical axis; the object side surface of the second lens is a cubic phase surface;

[0048] The endoscope optical lens satisfies the following relationship:

[0049] TTL / ImgH ≤ 2.4; and

[0050] 0.5<α<1.2;

[0051] Wherein, TTL is the axial distance between the object side surface of the first lens and the imaging surface; ImgH is the diagonal length of the effective pixel area on the imaging surface; α is the phase modulation intensity of the third-order phase plane.

[0052] According to an embodiment of the present application, the surface shape of the cubic phase surface is an extended polynomial aspheric surface, and the 2D mask function Q(x, y) of the cubic phase surface is: Q(x, y) = α(x 3 ,y 3 ), where α is the phase modulation intensity of the three-dimensional phase plane; (x, y) is the position coordinate of each point on the three-dimensional phase plane.

[0053] According to one embodiment of the present application, the object-side surface of the first lens is a concave surface, and the image-side surface of the first lens is a convex surface.

[0054] According to one embodiment of the present application, the surface shapes of the object side surface and the image side surface of the first lens are both even-order aspheric surfaces, and the surface shape equation of the even-order aspheric surface is:

[0055]

[0056] Where: Z is the height in the direction of the optical axis; c is the curvature of the surface; k is the cone coefficient; r is the aperture in the radial direction; Ai is the aspheric coefficient.

[0057] According to one embodiment of the present application, the image side surface of the second lens is a plane to cooperate with the cubic phase surface to form a cubic phase plate.

[0058] According to one embodiment of the present application, the object-side surface of the third lens is a concave surface, and the image-side surface of the third lens is a convex surface.

[0059] According to one embodiment of the present application, the object-side surface of the fourth lens is a convex surface, and the image-side surface of the fourth lens is a concave surface.

[0060] According to one embodiment of the present application, the optical lens for endoscopy further includes an aperture and a filter element, wherein the aperture is located between the first lens and the second lens, and the filter element is located on the image side of the fourth lens.

[0061] According to one embodiment of the present application, the full-depth MTF of the endoscopic optical lens is greater than 0.2.

[0062] According to one embodiment of the present application, the aperture number of the endoscopic optical lens is less than or equal to 3.

[0063] According to one embodiment of the present application, the endoscopic optical lens satisfies the relationship:

[0064] 105°≤(FOV*EFL) / ImgH≤115°;

[0065] Wherein: FOV is the field of view of the endoscopic optical lens; EFL is the effective focal length of the endoscopic optical lens; ImgH is the diagonal length of the effective pixel area on the imaging surface of the endoscopic optical lens.

[0066] According to one embodiment of the present application, the endoscopic optical lens satisfies the relationship:

[0067] 0.004 / °≤D / (ImgH*FOV)≤0.008 / °;

[0068] Wherein: D is the aperture of the first lens; ImgH is the diagonal length of the effective pixel area on the imaging surface of the endoscopic optical lens; FOV is the field of view of the endoscopic optical lens.

[0069] According to one embodiment of the present application, the endoscopic optical lens satisfies the relationship:

[0070] 0.6≤RS7 / EFL≤0.65;

[0071] Wherein: RS7 is the curvature radius of the object side surface of the fourth lens; EFL is the effective focal length of the endoscopic optical lens.

[0072] According to another aspect of the present application, an embodiment of the present application further provides a capsule endoscope, comprising:

[0073] The above-mentioned endoscopic optical lens;

[0074] A photosensitive component, arranged on the image side of the endoscopic optical lens;

[0075] A wireless communication module, communicatively connected to the photosensitive component; and

[0076] The capsule shell, the endoscopic optical lens, the photosensitive component and the wireless communication module are assembled inside the capsule shell. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] Figure 1 is a schematic structural diagram of a capsule endoscope according to an embodiment of the present application;

[0078] Figure 2 Example 1 of an optical lens for endoscopy in a capsule endoscope according to the above embodiment of the present application is shown;

[0079] FIG. 3A to FIG. 3C Schematic diagrams of MTIF curves of complex light diffraction of the endoscopic optical lens according to Example 1 of the present application when the object distances are 5 mm, 13 mm and 35 mm are shown in sequence;

[0080] Figure 4 Example 2 of the optical lens for endoscopy in the capsule endoscope according to the above embodiment of the present application is shown;

[0081] FIG. 5A to FIG. 5C Schematic diagrams of MTIF curves of complex light diffraction of the endoscopic optical lens according to Example 2 of the present application when the object distances are 5 mm, 13 mm and 35 mm are shown in sequence;

[0082] Figure 6 is a flowchart of an image processing method according to an embodiment of the present application;

[0083] Figure 7 A schematic diagram showing a flow chart of a calibration step in the image processing method according to the above embodiment of the present application is shown;

[0084] Figure 8 A schematic flow chart of a data set construction step in the image processing method according to the above embodiment of the present application is shown;

[0085] Fig. 9 A specific example of the data set construction step according to the above embodiment of the present application is shown;

[0086] Fig.10 A schematic flow chart of a model training step in the image processing method according to the above embodiment of the present application is shown;

[0087] Fig.11 A schematic diagram of the network structure of a depth of field extension model in the image processing method according to the above embodiment of the present application is shown;

[0088] Fig.12 A schematic flow chart of a model training step in the image processing method according to the above embodiment of the present application is shown;

[0089] Fig.13 is a schematic block diagram of an electronic device according to an embodiment of the present application;

[0090] Fig.14 is a schematic block diagram of a medical device according to an embodiment of the present application;

[0091] Fig.15 A schematic diagram comparing the effects of the depth of field extension solution disclosed in the present application and the traditional depth of field extension solution is shown.

[0092] Explanation of the main component symbols: 1. Capsule endoscope; 10. Optical lens for endoscopy; 11. First lens; 12. Second lens; 13. Third lens; 14. Fourth lens; 15. Aperture; 16. Filter element; 20. Photosensitive component; 30. Wireless communication module; 40. Capsule shell; 50. Electronic device; 51. Processor; 52. Memory; 53. Input device; 54. Output device.

[0093] The above description of the main component symbols is combined with the accompanying drawings and specific implementation methods to further illustrate the present invention in detail. DETAILED DESCRIPTION

[0094] The following description is used to disclose the present invention so that those skilled in the art can implement the present invention. The preferred embodiments described below are only examples, and those skilled in the art can think of other obvious variations. The basic principles of the present invention defined in the following description can be applied to other embodiments, variations, improvements, equivalents, and other technical solutions that do not deviate from the spirit and scope of the present invention.

[0095] Those skilled in the art should understand that, in the disclosure of the present invention, the terms "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicating the orientation or position relationship are based on the orientation or position relationship shown in the drawings, which are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation. Therefore, the above terms should not be understood as limiting the present invention.

[0096] In the present invention, the term "one" in the claims and the specification should be understood as "one or more", that is, in one embodiment, the number of an element can be one, while in another embodiment, the number of the element can be multiple. Unless it is clearly indicated in the disclosure of the present invention that the number of the element is only one, the term "one" cannot be understood as unique or single, and the term "one" cannot be understood as a limitation on the quantity.

[0097] In the description of the present invention, it should be understood that "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, "connected" and "connected" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through a medium. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0098] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0099] Considering the small size of existing capsule endoscopes, the length of the optical module and the number of lenses are strictly limited, and it is difficult to have a large optical depth of field without sacrificing optical resolution and field of view. The present application provides an image processing method and its application, an optical lens for endoscopy, and a capsule endoscope, which can achieve depth of field extension by wavefront coding, so as to have a large optical depth of field without sacrificing optical resolution and field of view.

[0100] Specifically, refer to the accompanying drawings of the specification of this application. Figure 1 According to one embodiment of the present application, a capsule endoscope 1 is provided, which may include an optical lens 10 for endoscopy, a photosensitive component 20 arranged on the image side of the optical lens 10 for endoscopy, a wireless communication module 30 communicatively connected to the photosensitive component 20, and a capsule shell 40. The optical lens 10 for endoscopy, the photosensitive component 20, and the wireless communication module 30 are assembled in the capsule shell 40 so as to wrap the optical lens 10 for endoscopy, the photosensitive component 20, and the wireless communication module 30 through the capsule shell 40 to facilitate endoscopic examination.

[0101] More specifically, if Figure 2 and Figure 4 As shown, the endoscope optical lens 10 may include a first lens 11 with positive optical power, a second lens 12 with optical power, a third lens 13 with positive optical power, and a fourth lens 14 with negative optical power, which are coaxially arranged in sequence from the object side to the image side along the optical axis; the object side surface of the second lens 12 is a cubic phase surface;

[0102] The endoscope optical lens 10 satisfies the following relationship:

[0103] TTL / ImgH ≤ 2.4; and

[0104] 0.5<α<1.2;

[0105] Wherein, TTL is the axial distance between the object side surface of the first lens 11 and the imaging surface; ImgH is the diagonal length of the effective pixel area on the imaging surface; and α is the phase modulation intensity of the cubic phase surface.

[0106] It is worth noting that, since wavefront coding uses a combination of optics and digital methods to expand the depth of field, the endoscopic optical lens 10 of the present application introduces a third-order phase plane in the optical system, which can not only make the modulation transfer function (MTF) or point spread function (PSF) of the optical system insensitive to defocus, so as to have the same PSF throughout the depth of field, but also make the MTF have no zero point throughout the depth of field to avoid information loss. In this way, the present application can restore the depth of field image collected by the endoscopic optical lens 10 by image restoration or AI reconstruction, thereby greatly expanding the depth of field range without affecting the original optical system parameters.

[0107] In particular, the present application not only controls the ratio between the total optical length TTL and the maximum image height ImgH within the range of the above relationship, effectively reducing the total length of the optical system, which is conducive to the miniaturization of the entire capsule endoscope 1, but also controls the phase modulation intensity α of the three-dimensional phase plane within the range of the above relationship, which can satisfy the three-dimensional phase plane with a large depth of field wavefront coding modulation capability while having good processability and recoverability. It can be understood that if the phase modulation intensity α is greater than 1.2, it is not conducive to the processing of the three-dimensional phase plane, and the overall MTF value is too low, which is not conducive to the subsequent AI algorithm recovery; if the phase modulation intensity α is less than 0.5, the phase modulation capability of the large depth of field cannot be met, affecting the imaging capability within the entire depth of field range.

[0108] In other words, the endoscopic optical lens 10 of the present application can make the MTF insensitive to the defocus position, ensure that the MTF performance of each field of view is close to the same, and the MTF curve has no zero points from high frequency to low frequency, so that the information is well preserved, so that it can be restored to a clear image with an appropriate filter function or AI algorithm later, so as to achieve the purpose of expanding the depth of field. In addition, the cubic phase plane can modulate the optical path distribution of the optical system, so that the MTF of the entire optical system is insensitive to the lens and surface eccentricity, which is convenient for achieving the effect of reducing tolerance sensitivity. In other words, the endoscopic optical lens 10 of the present application can make the capsule endoscope 1 have better full-field MTF uniformity within the entire depth of field, which helps to effectively improve the recovery ability of the subsequent AI algorithm and reduce computing power.

[0109] Optionally, the surface shape of the cubic phase surface is implemented as an extended polynomial aspheric surface, and the 2D mask function Q(x, y) of the cubic phase surface is implemented as: Q(x, y) = α(x 3 ,y 3 ), where α is the phase modulation intensity of the three-dimensional phase plane; (x, y) is the position coordinate of each point on the three-dimensional phase plane.

[0110] Optionally, within the entire depth of field, the MTF uniformity of the endoscopic optical lens 10 in the capsule endoscope 1 is better in the entire field of view; preferably, the full-depth MTF of the endoscopic optical lens 10 is greater than 0.2, so that the minimum MTF value of the endoscopic optical lens 10 at the inspection frequency in the entire field of view is greater than 0.2, which can effectively improve the recovery ability of the AI ​​algorithm and reduce the computing power. It is understandable that if the full-depth MTF is less than 0.2, the imaging quality of the system after the phase plane is added will be seriously degraded, and there will be noise in the actual imaging process. The severely degraded image is not conducive to the recovery of the subsequent algorithm, which seriously affects the imaging quality.

[0111] Alternatively, if Figure 2 and Figure 4 As shown, the object side surface of the first lens 11 is concave, and the image side surface of the first lens 11 is convex, so that the first lens 11 bears the optical focal length required by the optical system, which is beneficial to reducing the field of view angle and reducing the pupil aberration, so as to improve the imaging quality.

[0112] Alternatively, if Figure 2 As shown, the surface shapes of the object side surface and the image side surface of the first lens 11 are both implemented as even-order aspheric surfaces, and the surface shape equation of the even-order aspheric surface is implemented as:

[0113]

[0114] Where: Z is the height in the direction of the optical axis; c is the curvature of the surface; k is the cone coefficient; r is the aperture in the radial direction; Ai is the aspheric coefficient.

[0115] Alternatively, if Figure 2 and Figure 4 As shown, the image side surface of the second lens 12 is a plane so as to cooperate with the cubic phase plane to form a cubic phase plate, so that the optical focal length of the second lens 12 depends only on the cubic phase plane, which is beneficial to reduce the manufacturing difficulty of the second lens 12 while expanding the depth of field of the system.

[0116] Alternatively, if Figure 2 and Figure 4 As shown, the object side surface of the third lens 13 is concave, and the image side surface of the third lens 13 is convex, which is beneficial to reducing the coma and astigmatism of the system, and is also beneficial to compressing the exiting glue of the light at the aperture, so as to better control the distortion shape and size.

[0117] Alternatively, if Figure 2 and Figure 4 As shown, the object side surface of the fourth lens 14 is convex, and the image side surface of the fourth lens 14 is concave, which can reasonably control the contribution of spherical aberration within a reasonable level, so that the on-axis field of view obtains good imaging quality, and at the same time can make the light converge quickly to obtain better relative illumination.

[0118] It is worth noting that Figure 2 and Figure 4 As shown, the endoscopic optical lens 10 may further include an aperture 15, which is located between the first lens 11 and the second lens 12, so that the light will be immediately modulated by the third-dimensional phase plane after passing through the aperture 15, which helps to give full play to the phase modulation characteristics of the third-dimensional phase plane and facilitates the optimization of the parameters of the third-dimensional phase plane.

[0119] In addition, if Figure 2 and Figure 4 As shown, the endoscopic optical lens 10 may further include a filter element 16, which is located on the image side of the fourth lens 14, so that the light passing through the fourth lens 14 first passes through the filter element 16 to be filtered, and then is received by the photosensitive component 20 to obtain image data, so that the image data obtained in the body by the photosensitive component 20 can be wirelessly transmitted to the outside of the body through the wireless communication module 30 to be received by the host computer for image processing.

[0120] It can be understood that the photosensitive component 20 mentioned in the present application may include but is not limited to image sensors such as CCD (charge coupled device), CMOS (complementary metal-oxide semiconductor) or digital signal processing chip (DSP); the filter element 16 mentioned in the present application may be implemented as a bandpass filter but is not limited to it, as long as it can meet the filtering requirements, and the present application will not elaborate on this.

[0121] Optionally, the aperture number FNO of the endoscopic optical lens 10 is less than or equal to 3, so that the endoscopic optical lens 10 can collect more light, improve the relative illumination of the image plane, and facilitate clear imaging in the body with dim light.

[0122] Optionally, the endoscopic optical lens 10 may also satisfy the relationship:

[0123] 105°≤(FOV*EFL) / ImgH≤115°;

[0124] Wherein: FOV is the field of view of the endoscopic optical lens 10; EFL is the effective focal length of the endoscopic optical lens 10; ImgH is the diagonal length of the effective pixel area on the imaging surface of the endoscopic optical lens 10. In this way, the endoscopic optical lens 10 can achieve a large field of view while increasing the system focal length, so that the center of the endoscopic optical lens 10 obtains a larger angular resolution, ensuring that the endoscopic optical lens 10 obtains clear imaging quality within the entire depth of field. It can be understood that if (FOV*EFL) / ImgH>115°, it is not conducive to achieving a large image height; if (FOV*EFL) / ImgH<105°, it cannot be guaranteed that the optical system has a sufficient focal length, nor can it guarantee the clarity of the optical system when observing distant targets.

[0125] Optionally, the endoscopic optical lens 10 satisfies the relationship:

[0126] 0.004 / °≤D / (ImgH*FOV)≤0.008 / °;

[0127] Wherein: D is the aperture of the first lens 11; ImgH is the diagonal length of the effective pixel area on the imaging surface of the endoscopic optical lens 10; FOV is the field of view of the endoscopic optical lens 10. In this way, the aperture of the endoscopic optical lens 10 is effectively reduced, which is conducive to the miniaturization of the capsule endoscope. It can be understood that if D / (ImgH*FOV)>0.008 / °, the front port diameter of the optical system will become larger, which is not conducive to the miniaturization of the capsule endoscope system; if D / (ImgH*FOV)<0.004 / °, the MTF and relative illumination of the edge field of view of the optical system will be reduced.

[0128] Optionally, the endoscopic optical lens 10 satisfies the relationship:

[0129] 0.6≤RS7 / EFL≤0.65;

[0130] Wherein: RS7 is the radius of curvature of the object side of the fourth lens 14; EFL is the effective focal length of the endoscope optical lens 10. In this way, the endoscope optical lens 10 can effectively control the CRA (Chief Ray Angle) of the optical system, so that when the optical lens is matched with the photosensitive component, its CRA matches the CRA of the photosensitive component to avoid color cast. It can be understood that if RS7 / EFL>0.65 or RS7 / EFL<0.6, the CRA of the optical system will become larger, which is not conducive to matching with the CRA of the photosensitive component.

[0131] Some specific but non-limiting examples of the embodiments of the present application are described in more detail below with reference to the accompanying drawings. It is understood that any one of the following examples 1 and 2 is applicable to all embodiments of the present application.

[0132] For ease of description, in the following examples, OBJ represents the object surface of the endoscope optical lens 10, S1 represents the object side surface of the first lens 11, S2 represents the image side surface of the first lens 11, STO represents the surface of the aperture 15, S3 represents the object side surface of the second lens 12 (i.e., the third-order phase surface), S4 represents the image side surface of the second lens 12, S5 represents the object side surface of the third lens 13, S6 represents the image side surface of the third lens 13, S7 represents the object side surface of the fourth lens 14, S8 represents the image side surface of the fourth lens 14, S9 represents the object side surface of the filter element 16, S10 represents the image side surface of the filter element 16, and S11 represents the imaging surface of the endoscope optical lens 10. Ai represents the i-th order even-order aspheric coefficient, i=1-8; Aj represents the j-th order extended odd-order aspheric coefficient, j=4, 6, 8, 10, 12, 14, 16, 18, 20.

[0133] Example 1

[0134] like Figure 2 As shown in FIG. 1 , an example of an endoscopy optical lens 10 is described, wherein TTL / ImgH=2.4. Specifically, Figure 2 As shown, the endoscopic optical lens 10 in Example 1 includes a first lens 11, an aperture 15, a second lens 12, a third lens 13, a fourth lens 14 and a filter element 16 in order from the object side to the image side along the optical axis direction, wherein the surface types of the object side surface S1 and the image side surface S2 of the first lens 11 are both even-order aspheric surfaces; the surface types of the object side surface S3 and the image side surface S4 of the second lens 12 are respectively an extended polynomial aspheric surface and a spherical surface; the surface types of the object side surfaces S5, S7 and the image side surfaces S6, S8 of the third lens 13 and the fourth lens 14 are all extended odd-order aspheric surfaces; and the object side surface S9 and the image side surface S10 of the filter element 16 are both spherical surfaces.

[0135] More specifically, the basic optical parameters of the endoscopic optical lens 10 of Example 1 are shown in Table 1, where the units of the radius of curvature and thickness are both millimeters.

[0136] Table 1: Basic optical parameters of the endoscope optical lens of Example 1

[0137]

[0138] Table 2 shows a table of high-order coefficients of each even-order aspheric surface in the endoscopic optical lens 10 of Example 1, as shown in Table 2.

[0139] Table 2: Table of even-order aspheric coefficients of the endoscope optical lens of Example 1

[0140] Face number A1 A2 A3 A4 A5 A6 A7 A8 S1 -1.726E+00 7.863E+00 -7.575E+01 2.933E+02 0 0 0 0 S2 -7.994E-02 6.325E+00 -8.660E+01 1.097E+03 -4.968E+03 0 0 0

[0141] Table 3 shows a table of high-order coefficients of each extended odd-order aspheric surface in the endoscopic optical lens 10 of Example 1, as shown in Table 3.

[0142] Table 3: Table of extended odd-order aspheric coefficients of the endoscopy optical lens of Example 1

[0143]

[0144]

[0145] Table 4 shows the expanded polynomial coefficient table of the cubic phase plane in the endoscopic optical lens 10 of Example 1, as shown in Table 4.

[0146] Table 4: Table of cubic phase surface coefficients of the endoscopy optical lens of Example 1

[0147] Face number <![CDATA[X 3 AND 0 ]]> <![CDATA[X 0 AND 3 ]]> S3 0.7 0.7

[0148] It should be noted that the thicknesses 5 / 13 / 35 of the object plane OBJ in Table 1 represent that the object distances of the endoscopic optical lens 10 of Example 1 are 5 mm, 13 mm, and 35 mm, respectively.

[0149] After simulation test: the MTF curve of the polychromatic light diffraction of the endoscope optical lens 10 of Example 1 when the object distance is 5 mm is as follows: Figure 3A As shown; Example 1 endoscope optical lens 10 at an object distance of 13mm when the complex light diffraction MTF curve is as follows Figure 3B As shown; Example 1 of the endoscope optical lens 10 when the object distance is 35mm when the complex light diffraction MTF curve is as follows Figure 3C As shown. FIG. 3A to FIG. 3C It can be seen that the endoscopic optical lens 10 of Example 1 can achieve good imaging quality at various object distances.

[0150] Example 2

[0151] like Figure 4 As shown in FIG. 1 , an example 2 of an endoscopy optical lens 10 is described, wherein TTL / ImgH=1.7. Specifically, Figure 4As shown, the endoscopic optical lens 10 in Example 2 includes a first lens 11, an aperture 15, a second lens 12, a third lens 13, a fourth lens 14 and a filter element 16 in order from the object side to the image side along the optical axis direction, wherein the surface types of the object side surface S1 and the image side surface S2 of the first lens 11 are both extended odd-order aspheric surfaces; the surface types of the object side surface S3 and the image side surface S4 of the second lens 12 are respectively extended polynomial aspheric surfaces and spherical surfaces; the surface types of the object side surfaces S5, S7 and the image side surfaces S6, S8 of the third lens 13 and the fourth lens 14 are all extended odd-order aspheric surfaces; and the object side surface S9 and the image side surface S10 of the filter element 16 are both spherical surfaces.

[0152] More specifically, the basic optical parameters of the endoscopic optical lens 10 of Example 2 are shown in Table 5, where the units of the radius of curvature and thickness are both millimeters.

[0153] Table 5: Basic optical parameters of the endoscope optical lens of Example 2

[0154]

[0155]

[0156] Table 6 shows a table of high-order coefficients of each extended odd-order aspheric surface in the endoscopic optical lens 10 of Example 2, as shown in Table 6.

[0157] Table 6: Table of extended odd-order aspheric coefficients of the endoscopy optical lens of Example 2

[0158]

[0159] Table 7 shows the expanded polynomial coefficient table of the cubic phase plane in the endoscopic optical lens 10 of Example 2, as shown in Table 7.

[0160] Table 7: Cubic phase surface coefficient table of the endoscope optical lens of Example 2

[0161] Face number <![CDATA[X 3 AND 0 ]]> <![CDATA[X 0 AND 3 ]]> S3 1.1526 1.1526

[0162] It should be noted that the thicknesses 5 / 13 / 35 of the object plane OBJ in Table 5 represent that the object distances of the endoscopic optical lens 10 of Example 2 are 5 mm, 13 mm, and 35 mm, respectively.

[0163] After simulation test, the MTF curve of the polychromatic light diffraction of the endoscope optical lens 10 of Example 2 when the object distance is 5 mm is as follows: Figure 5A As shown; Example 1 endoscope optical lens 10 at an object distance of 13mm when the complex light diffraction MTF curve is as follows Figure 5B As shown; Example 1 of the endoscope optical lens 10 when the object distance is 35mm when the complex light diffraction MTF curve is as follows Figure 5C As shown. FIG. 5A to FIG. 5C It can be seen that the endoscopic optical lens 10 of Example 2 can achieve good imaging quality at various object distances.

[0164] In summary, the endoscopic optical lenses in Example 1 and Example 2 satisfy the relationship shown in Table 8, as shown in Table 8.

[0165] Table 8: Relationships satisfied by endoscopy optical lenses

[0166] Conditional / Example Example 1 Example 2 TTL / ImgH 2.4 1.7 FNO 2.7 2.7 (FOV*EFL) / ImgH 113° 91° D / (ImgH*FOV) 0.0074 / ° 0.0027 / ° RS7 / EFL 0.626 0.53

[0167] Schematic method

[0168] It is worth mentioning that, since the present application adds a third phase plane to the optical lens to encode the relevant information, it is necessary to perform corresponding decoding in the image processing part to reconstruct a high-quality image and achieve the purpose of extending the depth of field. According to another aspect of the present application, an embodiment of the present application further provides an image processing method, which can use a deep learning network model to complete image reconstruction and depth of field extension, so as to improve the imaging quality of the capsule endoscope at a single object distance while ensuring clear imaging within the largest possible depth of field range to meet the actual application needs.

[0169] Specifically, Figure 6 As shown, the image processing method of the present application may include the steps of:

[0170] S100: calibrating the point spread function of the endoscope module introduced with the phase plane at multiple object distances and the noise at a single object distance to obtain a noise model of the endoscope module and point spread functions of multiple different fields of view;

[0171] S200: constructing a training data set based on point spread functions of multiple different fields of view at different object distances and in-focus images collected by a calibrated camera at corresponding object distances;

[0172] S300: training a depth of field extension model based on the noise model of the endoscope module and the constructed training data set; and

[0173] S400: Processing the original degraded image collected by the endoscope module through the trained depth of field extension model to expand the depth of field range of the endoscopic image.

[0174] It is worth noting that the endoscope module mentioned in this application can be implemented as other types of lens modules in addition to the above-mentioned capsule endoscope, as long as the optical lens of these lens modules introduces a phase plane (such as the object side and / or image side of any lens in the optical lens is a three-dimensional phase plane), and this application will not go into details. In addition, the calibration camera mentioned in this application can be implemented as an existing focusable endoscope module with a larger volume, a higher lens height and better imaging quality, but it can also be implemented as multiple camera modules with different focal lengths.

[0175] More specifically, if Figure 7 As shown, step S100 of the image processing method may include the following steps:

[0176] S110: photographing a calibration plate at K object distances in a bright room environment by using the endoscope module to sample original degraded images at K object distances within a depth of field, where K is a positive integer greater than 1;

[0177] S120: performing point spread function calibration on the original degraded image at each object distance to obtain point spread functions of M*N different fields of view of the endoscope module at each object distance, where M and N are both positive integers greater than 1; and

[0178] S130: photographing a color card plate at a single object distance by using the endoscope module in a darkroom environment to calibrate a noise model of the endoscope module at a single object distance.

[0179] Optionally, the K object distances mentioned in this application can be selected within the depth of field range according to the resolution, depth of field coverage and accuracy requirements of the photosensitive components in the endoscope module. In addition, it can be understood that the bright room environment mentioned in this application refers to an environment with sufficient light; the dark room environment mentioned in this application refers to an environment with dim light.

[0180] For example, taking the resolution of 320*320 and the range of depth of field to be covered as 30 cm, the number of objective lens sampling K=10, and each object distance is calibrated to 3*3=9 fields of view, that is, M=N=3.

[0181] It is worth noting that the calibration plate can be implemented as, but not limited to, a series of small-sized oblique checkerboards; the color card plate can be implemented as, but not limited to, a small-sized 24-color card.

[0182] According to the above embodiments of the present application, Figure 8 and Fig. 9 As shown, step S200 of the image processing method may include the following steps:

[0183] S210: photographing different scenes in the current application scenario by the calibration camera under focus, so as to sample the in-focus images at the K object distances within the depth of field;

[0184] S220: performing extended depth of field fusion on the K in-focus images acquired at the K object distances to obtain a fully-focused image as label data Label;

[0185] S230: performing block convolution on the K in-focus images respectively and calibrating the point spread functions of M*N different fields of view at the corresponding object distances to obtain K single-layer convolution images at the K object distances;

[0186] S240: superimposing and fusing the K single-layer convolution images to obtain a blurred image as input data Input; and

[0187] S250: Repeat the above steps in different application scenarios, and form a training data pair with the label data Label and the input data Input obtained in different application scenarios to construct the training data set.

[0188] Alternatively, if Fig. 9 As shown, in the above step S220 of the present application: the K in-focus images can be fused with extended depth of field using, but not limited to, a mature and effective extended depth of field technology EDOF algorithm.

[0189] Alternatively, if Fig. 9 As shown, step S250 of the present application may include the following steps:

[0190] Determine whether the number of training data pairs in the training data set meets the requirements;

[0191] If yes, complete the construction of the training data set; and

[0192] If not, construct a training data pair in the next application scenario to update the training data set.

[0193] According to the above embodiments of the present application, Fig.10 As shown, step S300 of the image processing method may include the following steps:

[0194] S310: when obtaining a current training data pair from the training data set, randomly adding different gains to input data of the current training data pair to obtain gain input data;

[0195] S320: reading noises of different intensities in the noise model, and adding them to the gain input data to obtain gain input data carrying noise;

[0196] S330: inputting the gain input data carrying noise into the depth of field extension model for training to output a network result;

[0197] S340: Calculating a loss function based on the network result and the label data of the current training data pair; and

[0198] S350: Iteratively optimize the model parameters by returning the gradient to finally obtain a trained depth of field extension model.

[0199] It is worth noting that step S300 of the image processing method of the present application, before step S310, further includes the steps of packaging the training data set into an h5 file (i.e., hierarchical data format version 5 file), and writing the noise model into a readable config file (i.e., configuration file).

[0200] Optionally, the depth of field extension model mentioned in this application can adopt a network structure and loss function constraint (i.e., Loss constraint) that is relatively friendly to the image processing field in terms of effect and deployment. Fig.11 As shown, in the depth of field extension model: the network structure can be but is not limited to being implemented as Res-Unet (that is, it goes beyond the traditional image semantic segmentation algorithm, and is an improved semantic segmentation algorithm based on ResNet and U-Net, using ResNet to obtain high-level feature representation, and using U-Net's downsampling and upsampling operations for pixel-level segmentation); Loss constraints can be but are not limited to including L1 loss (that is, L1 loss function, also known as absolute value loss function) and TV loss (that is, TV loss function, total variation loss function); the number of network layers is 4; the maximum number of channels is 512; the specific network structure is shown in Figure 5.

[0201] According to the above embodiments of the present application, Fig.12 As shown, step S400 of the image processing method may include the following steps:

[0202] S410: performing model compression and quantization operations on the trained depth of field extension model according to the requirements of the actual application platform;

[0203] S420: deploying according to the relevant interface document of the actual application platform, and restoring the original degraded image collected by the endoscope module to obtain a restored image; and

[0204] S430: further processing the restored image through an image signal processing model to output an endoscopic image with an extended depth of field range.

[0205] It is worth noting that the actual application platform mentioned in the present application can be, but is not limited to, implemented as a PC (ie, computer) with GPU (ie, graphics processing unit) resources, such as a host computer.

[0206] Optionally, in step S430 of the present application, the image signal processing model may be implemented, but is not limited to, as a traditional ISP image signal processing algorithm, so as to output high-quality endoscopic images within a large depth of field.

[0207] It can be understood that the image processing method of the present application restores the image quality and expands the depth of field of the degraded image directly output by the photosensitive component in the endoscope module (i.e., the original degraded image), so that the endoscope module can produce clear images within a large depth of field.

[0208] In addition, before deploying the trained depth of field extension model on the PC: the trained depth of field extension model can be directly converted into the onnx (Open Neural Network Exchange) format, and then further optimized by the TensorRT (deep learning inference platform) tool, so as to complete the efficient processing of the original degraded image.

[0209] Schematic electronic equipment

[0210] According to another aspect of the present application, Fig.13 As shown, an embodiment of the present application further provides an electronic device 50 , which may include one or more processors 51 and a memory 52 .

[0211] The processor 51 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 50 to perform desired functions.

[0212] The memory 52 may include one or more computer program products, which may include various forms of readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, a random access memory (RAM) and / or a cache memory (cache), etc. The non-volatile memory may include, for example, a read-only memory (ROM), a hard disk, a flash memory, etc. One or more instructions may be stored on the readable storage medium, and the processor 51 may execute the instructions to implement the image processing method of each embodiment of the present invention described above and / or other desired functions.

[0213] In one example, if Fig.13As shown, the electronic device 50 may also include: an input device 53 and an output device 54, and these components are interconnected through a bus system and / or other forms of connection mechanisms (not shown). For example, the input device 53 may be, for example, a keyboard for inputting parameters or a wireless connector for receiving image data, etc. The output device 54 may output various information to the outside, including endoscopic image results, etc. The output device 54 may include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, etc.

[0214] Of course, to simplify, Fig.13 Only some of the components related to the present invention in the electronic device 50 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application conditions, the electronic device 50 may also include any other appropriate components.

[0215] According to another aspect of the present application, Fig.14 As shown, an embodiment of the present application further provides a medical device, which may include the above-mentioned electronic device 50 and a capsule endoscope 1; the capsule endoscope 1 may include: an endoscopic optical lens 10 with a three-dimensional phase plane, a photosensitive component 20 arranged on the image side of the endoscopic optical lens 10, a wireless communication module 30 communicatively connected to the photosensitive component 20, and a capsule shell 40, wherein the endoscopic optical lens 10, the photosensitive component 20 and the wireless communication module 30 are assembled in the capsule shell 40, and the wireless communication module 30 can be wirelessly connected to the electronic device 50.

[0216] It is worth noting that in other examples of the present application, the endoscopic optical lens 10 in the capsule endoscope 1 can be replaced by other optical lenses, as long as the optical lens introduces a phase plane, and this application will not elaborate on this.

[0217] Illustrative computer program product

[0218] In addition to the above-mentioned methods and devices, an embodiment of the present invention may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to perform some steps of the image processing method according to various embodiments of the present invention described in the above "illustrative method" section of this specification.

[0219] The computer program product may be written in any combination of one or more programming languages ​​to write program code for performing the operations of the embodiments of the present invention, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as C or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user computing device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0220] In addition, an embodiment of the present invention may also be a readable storage medium having at least one instruction stored thereon, which, when executed by a computing device, can be operated to execute the steps of the image processing method according to various embodiments of the present invention described in the above “illustrative method” section of this specification.

[0221] The readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0222] In summary, the present application utilizes wavefront coding (i.e., the endoscopic optical lens that introduces a cubic phase plane) and an AI restoration algorithm (i.e., the image processing method) to expand the depth of field of the optical system without increasing the total optical length of the lens group and the number of lenses, and the imaging quality can be better than most capsule endoscopes currently on the market. In a specific example, the total optical length of the endoscopic optical lens in the capsule endoscope of the present application can be 2mm, and its depth of field extension is more than five times that of the traditional solution. For example, the depth of field extension solution disclosed in the present application is compared with the traditional depth of field extension solution. Fig.15 As shown: the first row is the effect diagram of the traditional depth of field extension solution at object distances of 5mm, 13mm and 35mm respectively; the second row is the effect diagram of the depth of field extension solution of the present application at object distances of 5mm, 13mm and 35mm respectively.

[0223] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0224] The above embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention.

Claims

1. An image processing method, characterized in that: Includes steps: Calibrate the point spread function of the endoscope module introduced with the phase plane at multiple object distances and the noise at a single object distance to obtain the noise model of the endoscope module and the point spread function of multiple different fields of view; A training data set is constructed based on point spread functions of multiple different fields of view at different object distances and in-focus images collected by a calibrated camera at corresponding object distances; Based on the noise model of the endoscope module and the constructed training data set, the depth of field extension model is trained; as well as The original degraded image collected by the endoscope module is processed by the trained depth of field extension model to expand the depth of field range of the endoscopic image.

2. The image processing method according to claim 1, characterized in that: The step of calibrating the point spread function of the endoscope module introduced with the phase plane at multiple object distances and the noise at a single object distance to obtain the noise model of the endoscope module and the point spread function of multiple different fields of view comprises the steps of: The endoscope module is used to photograph a calibration plate at K object distances in a bright room environment to sample original degraded images at K object distances within a depth of field, where K is a positive integer greater than 1; The point spread function of the original degraded image at each object distance is calibrated to obtain the point spread function of M*N different fields of view of the endoscope module at each object distance, where M and N are both positive integers greater than 1; as well as The endoscope module is used to photograph a color card plate at a single object distance in a darkroom environment to calibrate the noise model of the endoscope module at the current object distance.

3. The image processing method according to claim 2, characterized in that: The calibration plate is a series of oblique checkerboards; the color card plate is a 24-color card.

4. The image processing method according to claim 2, characterized in that: The step of constructing a training data set based on point spread functions of multiple different fields of view at different object distances and in-focus images collected by a calibrated camera at corresponding object distances comprises the steps of: Using the calibration camera to shoot different scenes in the current application scenario under focus, so as to sample the in-focus images at the K object distances within the depth of field; Perform extended depth of field fusion on the K in-focus images acquired at the K object distances to obtain a fully-focused image as label data; The K in-focus images are convolved in blocks and calibrated at corresponding object distances to obtain M*N point spread functions of different fields of view, so as to obtain K single-layer convolution images at the K object distances; The K single-layer convolution images are superimposed and fused to obtain a blurred image as input data; as well as Repeat the above steps in different application scenarios, and form training data pairs with the label data and input data obtained in different application scenarios to construct the training data set.

5. The image processing method according to claim 4, characterized in that: The step of repeating the above steps in different application scenarios and forming training data pairs with the label data and input data obtained in different application scenarios to construct the training data set includes the following steps: Determine whether the number of training data pairs in the training data set meets the requirements; If yes, complete the construction of the training data set; as well as If not, construct a training data pair in the next application scenario to update the training data set.

6. The image processing method according to claim 1, characterized in that: The step of training the depth of field extension model based on the noise model of the endoscope module and the constructed training data set comprises the steps of: When obtaining a current training data pair from the training data set, randomly adding different gains to input data of the current training data pair to obtain gain input data; Reading noises of different intensities in the noise model and adding them to the gain input data to obtain gain input data carrying noise; Inputting the gain input data carrying noise into the depth of field extension model for training to output a network result; Calculate the loss function based on the network structure and the label data of the current training data pair; as well as The model parameters are iteratively optimized by gradient feedback to finally obtain a trained depth of field extension model.

7. The image processing method according to claim 1, characterized in that: The network structure of the depth of field extension model is Res-Unet; the Loss constraints of the depth of field extension model include absolute value loss function and total variation loss function.

8. The image processing method according to any one of claims 1 to 7, characterized in that: The step of processing the original degraded image collected by the endoscope module through the trained depth of field extension model to expand the depth of field range of the endoscopic image includes the steps of: According to the requirements of the actual application platform, the trained depth of field extension model is compressed and quantized; Deploy according to the relevant interface documents of the actual application platform, and restore the original degraded image collected by the endoscope module to obtain a restored image; as well as The restored image is further processed by an image signal processing model to output an endoscopic image with an extended depth of field.

9. An electronic device, characterized in that include Processor; and A memory, wherein at least one instruction is stored in the memory, and when the instruction is executed by the processor, the processor executes the image processing method according to any one of claims 1 to 8.

10. A medical device, characterized in that include: The electronic device as claimed in claim 9; and A capsule endoscope comprises: an optical lens for endoscopy having a three-dimensional phase plane, a photosensitive component arranged on the image side of the optical lens for endoscopy, a wireless communication module communicatively connected to the photosensitive component, and a capsule shell, wherein the optical lens for endoscopy, the photosensitive component and the wireless communication module are assembled inside the capsule shell, and the wireless communication module can be wirelessly connected to the electronic device.

11. An optical lens for endoscopy, characterized in that: The optical lens comprises a first lens with positive optical power, a second lens with optical power, a third lens with positive optical angle and a fourth lens with negative optical power, which are coaxially arranged in sequence from the object side to the image side along the optical axis; the object side surface of the second lens is a cubic phase surface; The endoscope optical lens satisfies the following relationship: TTL / ImgH ≤ 2.4; and 0.5＜α＜1.2； Wherein, TTL is the axial distance between the object side surface of the first lens and the imaging surface; ImgH is the diagonal length of the effective pixel area on the imaging surface; α is the phase modulation intensity of the third-order phase plane.

12. The endoscopic optical lens according to claim 11, characterized in that: The surface shape of the cubic phase surface is an extended polynomial aspheric surface, and the 2D mask function Q(x, y) of the cubic phase surface is: Q(x, y) = α(x 3 ,y 3 ), where α is the phase modulation intensity of the three-dimensional phase plane; (x, y) is the position coordinate of each point on the three-dimensional phase plane.

13. The endoscopic optical lens according to claim 11, characterized in that: The object side surface of the first lens is a concave surface, and the image side surface of the first lens is a convex surface; the image side surface of the second lens is a plane to cooperate with the third-order phase surface to form a third-order phase plate; the object side surface of the third lens is a concave surface, and the image side surface of the third lens is a convex surface; the object side surface of the fourth lens is a convex surface, and the image side surface of the fourth lens is a concave surface; the endoscopic optical lens also includes an aperture and a filter element, the aperture is located between the first lens and the second lens, and the filter element is located on the image side of the fourth lens.

14. The endoscopic optical lens according to claim 11, characterized in that: The surface shapes of the object side surface and the image side surface of the first lens are both even-order aspheric surfaces, and the surface shape equation of the even-order aspheric surface is: Where: Z is the height in the direction of the optical axis; c is the curvature of the surface; k is the cone coefficient; r is the aperture in the radial direction; Ai is the aspheric coefficient.

15. The endoscopic optical lens according to any one of claims 11 to 14, characterized in that: The full-depth MTF of the endoscopic optical lens is greater than 0.

2.

16. The endoscopic optical lens according to any one of claims 11 to 14, characterized in that: The aperture number of the endoscopic optical lens is less than or equal to 3.

17. The endoscopic optical lens according to any one of claims 11 to 14, characterized in that: The endoscope optical lens satisfies the relationship: 105°≤(FOV*EFL) / ImgH≤115°; Wherein: FOV is the field of view of the endoscopic optical lens; EFL is the effective focal length of the endoscopic optical lens; ImgH is the diagonal length of the effective pixel area on the imaging surface of the endoscopic optical lens.

18. The endoscopic optical lens according to any one of claims 11 to 14, characterized in that: The endoscope optical lens satisfies the relationship: 0.004 / °≤D / (ImgH*FOV)≤0.008 / °; Wherein: D is the aperture of the first lens; ImgH is the diagonal length of the effective pixel area on the imaging surface of the endoscopic optical lens; FOV is the field of view of the endoscopic optical lens.

19. The endoscopic optical lens according to any one of claims 11 to 14, characterized in that: The endoscope optical lens satisfies the relationship: 0.6≤RS7 / EFL≤0.65; Wherein: RS7 is the radius of curvature of the object side of the fourth lens; EFL is the effective focal length of the endoscopic optical lens.

20. A capsule endoscope, characterized in that: include: The endoscopic optical lens according to any one of claims 11 to 19; A photosensitive component, arranged on the image side of the endoscopic optical lens; A wireless communication module, communicatively connected to the photosensitive component; as well as The capsule shell, the endoscopic optical lens, the photosensitive component and the wireless communication module are assembled inside the capsule shell.

Images