Endoscopic imaging method and related products

Abstract

The application provides an endoscope imaging method and related products. The method comprises: acquiring first phase difference information corresponding to imaging of an imaging area by an image acquisition system at an initial focal length, the first phase difference information comprising phase difference information output by multiple pairs of phase detection pixels during imaging; determining at least one target focal length adjustment amount according to the first phase difference information; when the number of target focal length adjustment amounts is greater than 1: performing a focusing operation according to the target focal length adjustment amounts and acquiring a multi-depth-of-field image sequence; and performing image fusion on endoscope images in the multi-depth-of-field image sequence to obtain a depth-of-field expansion image of the imaging area. This scheme comprehensively determines the number of in-focus planes of the imaging area and the target focal length adjustment amount corresponding to each in-focus plane according to phase difference information at multiple positions in the imaging area, and performs a focusing operation and image fusion in a targeted manner, so that the clarity requirement and real-time requirement of the endoscope image can be simultaneously met.

Description

Technical Field

[0001] This invention relates to the field of endoscopic imaging technology, and more specifically to an endoscopic imaging method, an endoscopic image processing device, an endoscopic system, a storage medium, and a computer program product. Background Technology

[0002] Endoscopic imaging systems are commonly used equipment in modern medical devices for examining patients' bodies. The image quality of the endoscopic image directly affects the examination results, and this image quality is affected by various factors, such as the size of the patient's body cavity, organ peristalsis, and endoscope vibration. If the object being observed exceeds the effective depth of field of the endoscopic imaging system, the endoscopic image will become out of focus and blurry, resulting in a sharp decline in image quality and significantly impacting the accuracy of the doctor's diagnosis.

[0003] Based on the design principles of endoscopic imaging systems, there is an irreconcilable contradiction between the effective depth of field, aperture size, and imaging resolution. Reducing the lens aperture can increase the depth of field, but this simultaneously leads to a decrease in the intensity of reflected light received by the camera module and an expansion of the Airy disk during imaging, thus reducing image sharpness and resulting in lower image quality. In related technologies, some endoscopic systems employ a step-by-step focusing method to acquire multi-depth-field image sequences and then perform image fusion to obtain a depth-extended image. However, this focusing method is equivalent to adjusting the focus step by step across the entire adjustable focusing range, resulting in extremely low focusing efficiency, high imaging delay, and hindering the doctor's observation and operation. Summary of the Invention

[0004] This invention addresses the aforementioned problems. It provides an endoscopic imaging method, an endoscopic image processing device, an endoscopic system, a storage medium, and a computer program product. This approach integrates phase difference information from multiple locations within the imaging area to determine the number of focal planes in the imaging area and the target focal length adjustment amount corresponding to each focal plane. It then performs targeted focusing operations and image fusion, improving the efficiency and flexibility of acquiring depth-of-field extended images while simultaneously meeting the requirements for both clarity and real-time performance in endoscopic images.

[0005] According to one aspect of the present invention, an endoscopic imaging method is provided for use in an endoscopic system. The endoscopic system includes an image acquisition system, which includes an adjustable focus lens group and a phase difference camera module. The phase difference camera module includes multiple pairs of phase detection pixels, each pair of phase detection pixels being used to detect phase difference information at its imaging position. The endoscopic imaging method includes: acquiring first phase difference information corresponding to the imaging region imaged by the image acquisition system at an initial focal length, the first phase difference information including phase difference information output by each pair of phase detection pixels during imaging; and determining at least one target focal length adjustment amount based on the first phase difference information, wherein... At least one target focal length adjustment corresponds one-to-one with at least one set of target sub-regions in the imaging region. For each target focal length adjustment, when the image acquisition system performs a focusing operation and images the imaging region according to the target focal length adjustment, the target sub-region corresponding to the target focal length adjustment is in focus. Wherein, when the number of target focal length adjustments is greater than 1: a multi-depth-of-field image sequence is acquired, which includes endoscopic images acquired by the image acquisition system when performing a focusing operation and images according to each target focal length adjustment; image fusion is performed on the endoscopic images in the multi-depth-of-field image sequence to obtain a depth-of-field extended image of the imaging region.

[0006] Optionally, determining at least one target focal length adjustment amount based on the first phase difference information includes: determining a phase correlation distribution matrix based on the first phase difference information; statistically analyzing the peak values ​​of the phase correlation distribution matrix; clustering the peak values ​​of the phase correlation distribution matrix to determine a first group of peak centers; and determining at least one target focal length adjustment amount based on at least some of the peak centers in the first group of peak centers.

[0007] Optionally, determining at least one target focal length adjustment based on at least some of the peak centers in the first group of peak centers includes: filtering the peak centers in the first group of peak centers according to a threshold for the number of target focal length adjustments to obtain a second group of peak centers; and determining at least one target focal length adjustment based on each peak center in the second group of peak centers.

[0008] Optionally, determining at least one target focal length adjustment amount based on at least some of the peak centers in the first group of peak centers includes: determining at least one focusing focal length adjustment amount based on at least some of the peak centers in the first group of peak centers, wherein the at least one focusing focal length adjustment amount corresponds one-to-one with at least one group of initial sub-regions in the imaging area; for each focusing focal length adjustment amount, when the image acquisition system performs a focusing operation and images the imaging area according to the focusing focal length adjustment amount, the initial sub-region corresponding to the focusing focal length adjustment amount is in a focused state; adjusting at least two focusing focal lengths that meet the second preset requirement from the at least one focusing focal length adjustment amount... The integer optimization is used as the first target focal length adjustment amount, wherein at least one of the focusing focal length adjustment amounts that does not meet the second preset requirement is used as the second target focal length adjustment amount, and the at least one target focal length adjustment amount includes the first target focal length adjustment amount and the second target focal length adjustment amount; wherein the second preset requirement includes: the effective depth of field ranges after the image acquisition system performs focusing operation based on at least two focusing focal length adjustment amounts have an intersection, and the target sub-region corresponding to each first target focal length adjustment amount is obtained by merging the initial sub-regions corresponding to the focusing focal length adjustment amounts that participated in the optimization of the first target focal length adjustment amount.

[0009] Optionally, determining at least one target focal length adjustment amount based on the first phase difference information includes: determining at least one focusing focal length adjustment amount based on the first phase difference information, wherein the at least one focusing focal length adjustment amount corresponds one-to-one with at least one set of initial sub-regions in the imaging area; for each focusing focal length adjustment amount, when the image acquisition system performs a focusing operation and images the imaging area according to the focusing focal length adjustment amount, the initial sub-region corresponding to the focusing focal length adjustment amount is in a focused state; optimizing at least two focusing focal length adjustment amounts that satisfy the second preset requirement from the at least one focusing focal length adjustment amount into a first target focal length adjustment amount. The focal length adjustment amount is defined as follows: at least one focal length adjustment amount that does not meet the second preset requirement is used as the second target focal length adjustment amount; the at least one target focal length adjustment amount includes a first target focal length adjustment amount and a second target focal length adjustment amount; wherein the second preset requirement includes: the effective depth of field ranges after the image acquisition system performs focusing operation based on at least two focal length adjustment amounts have an intersection; and the target sub-region corresponding to each first target focal length adjustment amount is obtained by merging the initial sub-regions corresponding to the focal length adjustment amounts that participated in the optimization of the first target focal length adjustment amount.

[0010] Optionally, image fusion is performed on the endoscopic images in the multi-depth-field image sequence to obtain a depth-extended image of the imaging region, including: aligning the endoscopic images in the multi-depth-field image sequence; determining the region to be fused in the aligned endoscopic images, wherein the region to be fused is a region whose clarity meets a first preset requirement; and fusing the region to be fused to obtain a depth-extended image.

[0011] Optionally, determining the region to be fused from the aligned endoscopic images includes: for each endoscopic image in a multi-depth-of-field image sequence, acquiring second phase difference information corresponding to the image acquisition system when acquiring the endoscopic image, the second phase difference information including phase difference information output by each pair of phase detection pixels when acquiring the endoscopic image; evaluating the local sharpness of the endoscopic image based on the second phase difference information, and obtaining a specific region of the endoscopic image based on the evaluation result of the local sharpness, the specific region being a region whose sharpness meets a first preset requirement; and using the corresponding region obtained after the alignment operation of the specific region as the region to be fused from the aligned endoscopic image.

[0012] Optionally, before fusing the regions to be fused to obtain a depth-extended image of the imaging region, the method further includes smoothing the edges of the regions to be fused.

[0013] Optionally, when the number of target focal length adjustments is 1, the endoscopic imaging method further includes: determining whether the target focal length adjustment is greater than a preset adjustment threshold; if so, controlling the image acquisition system to perform a focusing operation according to the target focal length adjustment to acquire and display the endoscopic image acquired by the image acquisition system at the focused focal length; if not, displaying the endoscopic image acquired by the image acquisition system at the initial focal length; or, determining whether the imaging area is in a defocused state according to the first phase difference information; if so, controlling the image acquisition system to perform a focusing operation according to the target focal length adjustment to acquire and display the endoscopic image acquired by the image acquisition system at the focused focal length; if not, displaying the endoscopic image acquired by the image acquisition system at the initial focal length.

[0014] According to another aspect of the present invention, an endoscopic image processing apparatus is also provided, comprising: a processor and a memory, wherein the memory stores computer program instructions, which are executed by the processor to perform the endoscopic imaging method described above.

[0015] According to another aspect of the present invention, an endoscope system is also provided, comprising: an image acquisition system, the image acquisition system including an adjustable focus lens group and a phase difference camera module, the phase difference camera module including multiple pairs of phase detection pixels, each pair of phase detection pixels being used to detect phase difference information at its imaging position, and an endoscope image processing device as described above, the endoscope processing device being communicatively connected to the adjustable focus lens group and the phase difference camera module respectively.

[0016] Optionally, the focusing lens group includes a deformable lens.

[0017] Optionally, the phase contrast camera module includes an image sensor, on which multiple pairs of phase detection pixels and multiple imaging pixels are arranged on the photosensitive surface. The multiple imaging pixels are used to receive light from the imaging area to generate an endoscopic image of the imaging area.

[0018] According to another aspect of the present invention, a storage medium is also provided, on which program instructions are stored, which are used to execute the above-described endoscopic imaging method when running.

[0019] According to another aspect of the present invention, a computer program product is also provided, including computer program instructions, which, when executed, are used to perform the above-described endoscopic imaging method.

[0020] The aforementioned technical solution can adaptively obtain the target focal length adjustment amount required for clear imaging of each part of the current imaging area based on the phase difference information at multiple locations within the imaging area. It can accurately adjust the initial focal length to the target focal length corresponding to each focal plane of the imaging area. Furthermore, this targeted focusing method has high focusing efficiency and can meet the real-time requirements of endoscopic images. On the other hand, the focal length after focusing corresponds to an effective depth of field range. In particular, after multiple focusing operations, the effective depth of field ranges of each endoscopic image in a multi-depth-of-field image sequence may differ, and the union of the effective depth of field ranges corresponding to each endoscopic image is relatively large. By performing image fusion on the endoscopic images acquired after focusing according to the target focal length adjustment by the image acquisition system to obtain a depth-of-field extended image, the effective depth of field range of the fused depth-of-field extended image can be increased, thereby enriching the image content of the depth-of-field extended image. In addition, by performing image fusion on endoscopic images, it is possible to ensure that the subject within the imaging area can be adaptively and clearly imaged within the effective depth of field range as much as possible.

[0021] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and in order to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0022] The above and other objects, features, and advantages of the present invention will become more apparent from the more detailed description of the embodiments of the invention in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same parts or steps.

[0023] Figure 1 A schematic diagram of the frame of an endoscope system according to an embodiment of the present invention is shown;

[0024] Figure 2 A flowchart illustrating an endoscopic imaging method according to an embodiment of the present invention is shown.

[0025] Figure 3 A schematic diagram of the algorithm flow of an endoscope system according to an embodiment of the present invention is shown;

[0026] Figure 4 This diagram illustrates the timing sequence of control signals for an endoscope system according to an embodiment of the present invention.

[0027] Figure 5 A schematic block diagram of an electronic device according to an embodiment of the present invention is shown. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of the present invention more apparent, exemplary embodiments according to the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely a part of the embodiments of the present invention, and not all of the embodiments of the present invention. It should be understood that the present invention is not limited to the exemplary embodiments described herein. Based on the embodiments of the present invention described herein, all other embodiments obtained by those skilled in the art without inventive effort should fall within the protection scope of the present invention.

[0029] As mentioned above, in related technologies, there is an irreconcilable contradiction between the effective depth of field range, aperture diameter, and imaging resolution. Increasing the depth of field range of the imaging system leads to a decrease in image quality. This invention provides an endoscopic imaging method. This method integrates phase difference information from multiple locations within the imaging area to determine the number of focal planes in the imaging area and the target focal length adjustment amount corresponding to each focal plane. It then performs targeted focusing operations and image fusion, improving the efficiency and flexibility of acquiring depth-of-field extended images while simultaneously meeting the requirements for both clarity and real-time performance of endoscopic images.

[0030] The endoscopic imaging method of this invention can be used in an endoscope system, wherein the endoscope system may include an image acquisition system.

[0031] Please see Figure 1 The diagram shown is a schematic frame of an endoscope system according to an embodiment of the present invention. The endoscope system may include an image acquisition system and an image processing system. (Refer to...) Figure 1 The image acquisition system may include an optical imaging module 1, an imaging control module 2, an optical illumination module 3, and a phase difference camera module 4.

[0032] For example, the optical imaging module 1 may include a plurality of lens groups, which include a base lens group and a focusable lens group. Figure 1The number and position of the base lens group and adjustable focus lens group shown are for illustrative purposes only. The number of base lens groups and adjustable focus lens groups can be set to any suitable number as needed. The positions of the base lens groups and adjustable focus lens groups can be set as needed. For example, an adjustable focus lens group can be placed between two base lens groups. Each base lens group may include one or more non-deformable lenses. Each adjustable focus lens group may include one or more deformable lenses. The optical properties (e.g., surface curvature, refractive index, etc.) of the deformable lens are adjustable. For example, the surface curvature of the deformable lens can be adjusted by means of an electric field or external force, thereby achieving adjustment of the system focal length and focusing position of the optical imaging module. Deformable lenses may include, but are not limited to, liquid lenses, liquid crystal lenses, birefringence crystal lenses, electric field-driven deformable lenses, piezoelectrically driven mechanical deformable lenses, etc. It can be understood that non-deformable lenses are the opposite of deformable lenses, that is, their optical properties are fixed. It is also understandable that using deformable lenses as adjustable focus lens groups can usually achieve a higher response speed and better meet the real-time requirements of endoscopic images. In other embodiments, each adjustable focus lens group may also include one or more movable lenses, and focusing is achieved by adjusting the position of the movable lens on the optical axis.

[0033] The imaging control module 2 may include a lens driving submodule for driving and controlling the adjustable focus lens group, an exposure control submodule for controlling exposure parameters (e.g., exposure time, gain, trigger mode), and a joint control submodule for sending commands to the lens driving submodule and the exposure control submodule. The joint control submodule can send commands to the lens driving submodule to reset the optical properties (shape, curvature, etc.) of the lenses in the adjustable focus lens group. In one specific embodiment, a communication interface such as a serial port is provided between the exposure control submodule and the camera control submodule, and between the lens driving submodule and the adjustable focus lens group, which can transmit specific information such as voltage, current, and control words. The lens driving submodule can send drive signals to the adjustable focus lens group, including but not limited to voltage, current, and / or control words, which can control and adjust the optical properties of the deformable lenses or the movement distance of the movable lenses in the adjustable focus lens group.

[0034] The optical illumination module 3 may include an illumination source and a light source control submodule for controlling the on and off of the illumination source. The light source control submodule may send commands to the illumination source to provide illumination light to the imaging area, which may include visible light, infrared light, etc.

[0035] The phase-contrast camera module 4 includes an image sensor and a camera control submodule for controlling the image sensor. In some embodiments, the photosensitive surface of the image sensor of the phase-contrast camera module 4 may be provided with imaging pixels and several pairs of phase detection pixels. The imaging pixels can be used to image the imaging area to output an endoscope image, and the phase detection pixels can output phase difference information at the current focal length when imaging the imaging area. In other embodiments, all pixels of some image sensors in the phase-contrast camera module 4 may be imaging pixels, and all pixels of other image sensors may be phase detection pixels. In this case, the optical imaging module 1 may also include a light path splitting submodule, which can split the incident light into multiple paths, each of which can be incident on one image sensor of the phase-contrast camera module 4, and each of the split incident light paths can be incident on each image sensor of the phase-contrast camera module 4. This approach increases the size of the optical imaging module 1, but achieves higher phase detection accuracy and image clarity. The camera control submodule is connected to the exposure control submodule. Based on the exposure parameters and other basic imaging parameters sent by the exposure control submodule, it can control the image sensor to receive illumination light to image the imaging area, and transmit the acquired image of the imaging area (which can be one frame or several frames) to the image processor.

[0036] The image processing system may include an image processor for processing images acquired by the image acquisition system, and a display terminal for outputting the processed images in real time. The image processor can perform image processing on the images transmitted by the image sensor. Image processing may include image denoising, image synthesis, image smoothing, etc., and the processed image is then transmitted to the display terminal. The image processor can communicate with the joint control submodule. The image processor can control the operation of the joint control submodule according to a preset algorithm.

[0037] It should be noted that Figure 1 The structure of the endoscope system shown is merely an example and not a limitation of the invention. For example, the display terminal in the image processing system may be omitted, and the image processed by the image processor may be stored or transmitted to other devices communicatively connected to the image processing system.

[0038] Please see Figure 2 As shown, it is a flowchart of an endoscopic imaging method according to an embodiment of the present invention. The endoscopic imaging method of the present invention may include, but is not limited to, steps S110, S120, S130 and S140.

[0039] Step S110: Obtain the first phase difference information corresponding to the imaging area formed by the image acquisition system at the initial focal length. The first phase difference information includes the phase difference information output by each pair of phase detection pixels during imaging.

[0040] Exemplarily, the focal length of the image acquisition system can be changed by adjusting the adjustable focus lens group of the optical imaging module 1 in the image acquisition system. Exemplarily, the following description will use the example of changing the focal length of the image acquisition system by adjusting the lens shape in the adjustable focus lens group as an example. The initial focal length can be any size, for example, it can be the system focal length of the optical imaging module when the lens shape in the adjustable focus lens group is in the reset state, or it can be the system focal length of the optical imaging module when the lens shape of the adjustable focus lens group is a preset shape. This preset shape can be defined by the user, and this embodiment of the invention does not limit it. In this embodiment of the invention, the imaging area can be the entire area of ​​the two-dimensional mapping of the target object under a certain imaging viewpoint, or it can be the region of interest of the two-dimensional mapping of the target object under a certain imaging viewpoint. Those skilled in the art will understand that the region of interest can be defined according to actual needs, and this embodiment of the invention does not specifically limit it. The image sensor can be equipped with multiple pixels, wherein at least some pixels are regularly symmetrical phase detection pixels (also known as "phase difference pixels," hereinafter referred to as PD pixels) used for phase detection. For each pair of PD pixels, light rays returning from the object surface within the imaging region are projected onto the pair of PD pixels via a lens in the optical imaging module, forming phase difference data. This phase difference data can be quantized using grayscale or brightness and output as phase difference information. In a specific embodiment, every two symmetrically arranged PD pixels can be defined as a pair of PD pixels, and each pair of PD pixels can include a left PD pixel and a right PD pixel. For each pair of PD pixels, the energy value of the pair of PD pixels can be calculated. Specifically, the signal values ​​of the left PD pixel and the right PD pixel of the pair of PD pixels can be extracted separately, and the difference between their signal values ​​can be calculated. This signal difference can be used as the phase difference data on the pair of PD pixels. This phase difference data can be normalized to grayscale or brightness, and the normalized phase difference data can be used as the phase difference information output by the pair of PD pixels.

[0041] Step S120: Based on the first phase difference information, determine at least one target focal length adjustment amount, wherein at least one target focal length adjustment amount corresponds one-to-one with at least one set of target sub-regions in the imaging area. For each target focal length adjustment amount, when the image acquisition system performs a focusing operation and images the imaging area according to the target focal length adjustment amount, the target sub-region corresponding to the target focal length adjustment is in focus.

[0042] For example, the phase difference can reflect the object distance offset, which can be understood as the degree of deviation between the current imaging object distance and the target imaging object distance. The target imaging object distance represents the distance between the position of the current imaging object when it can be clearly imaged and the position of the image sensor. The object distance offset corresponds to the focal length offset; similarly, the focal length offset can be understood as the degree of deviation between the current focal length and the target focal length, which represents the focal length at which the current imaging object can be clearly imaged. It can be understood that, based on the phase difference, the focal length offset can be obtained, and the focal length offset can be used as the adjustment amount for the focusing focal length during focusing operations.

[0043] In this embodiment, the phase difference camera module of the image acquisition system includes multiple pairs of phase detection pixels. Each pair of phase detection pixels is used to detect the phase difference information at its imaging position. Therefore, when the image acquisition system images the imaging area at the initial focal length, it can simultaneously acquire the phase difference information output by each pair of phase detection pixels. This phase difference information can form the first phase difference information. The first phase difference information can be used to determine the target focal length adjustment amount that makes the object in the imaging area clear. The target focal length adjustment amount can be one or more. That is, the first phase difference information can correspond to only one target focal length adjustment amount, or it can correspond to multiple target focal length adjustment amounts. Those skilled in the art will understand that the first phase difference information can include several sets of phase difference data, and each set of phase difference data corresponds to one focal length adjustment amount. Exemplarily and not limitingly, each focal length adjustment amount can be clustered according to the sharpness requirements of the image to be acquired to determine at least one target focal length adjustment amount, or all the acquired focal length adjustment amounts can be used as the target focal length adjustment amount. The lens shape of the adjustable focusing lens group can be adjusted according to the determined target focal length adjustment, allowing the image acquisition system to adjust the current focal length based on the target focal length adjustment. Specifically, any region being in focus indicates that the region can be imaged clearly. More specifically, after the image acquisition system performs a focusing operation according to the target focal length adjustment, the imaging plane of at least a portion of the imaging region A is located on the same plane as the image sensor. It can be considered that the focal point corresponding to the adjusted focal length falls on at least a portion of the imaging region A. This portion of the region serves as a group of target sub-regions, each of which can be a single connected region or multiple independent regions. Alternatively, for each target focal length adjustment, after the image acquisition system performs a focusing operation according to the target focal length adjustment, the spatial position of at least a portion of the imaging region A is within the effective depth of field of the image sensor. It can also be considered that the distance between the imaging plane of at least a portion of the imaging region A and the image sensor is within the depth of focus range corresponding to the target focal length, where the depth of focus represents the range of sharpness on both sides of the imaging plane. The target sub-regions corresponding to each target focal length adjustment can collectively cover or approximately cover the imaging region A. That is, for example, the cross-border-union ratio (CLU) between the region formed by the superposition of the above-mentioned at least one set of target sub-regions and the imaging region can be greater than or equal to a preset overlap threshold. The preset overlap threshold can be set to any suitable value as needed, and it can be in the range of [70%, 100%], such as equal to 80%, 90%, 100%, etc. When the preset overlap threshold is equal to 100%, the CLU ratio between the region formed by the superposition of the at least one set of target sub-regions and the imaging region is equal to 100%, and at this time the region formed by the superposition of the at least one set of target sub-regions can exactly cover the imaging region.When the preset overlap threshold is less than 100%, the cross-union ratio between the region formed by at least one set of target sub-regions superimposed and the imaging region can be equal to 100% or less than 100%. If the cross-union ratio between the region formed by at least one set of target sub-regions superimposed and the imaging region is less than 100% but greater than or equal to the preset overlap threshold, it can be considered that the region formed by at least one set of target sub-regions superimposed roughly covers the imaging region.

[0044] When the number of target focal length adjustments is greater than 1, execute steps S130 and S140.

[0045] Step S130: Acquire a multi-depth-of-field image sequence, which includes endoscopic images acquired by the image acquisition system when performing focusing operations and imaging based on the focal length adjustment of each target.

[0046] For example, a multi-depth-of-field image sequence may include multiple endoscopic images, wherein each endoscopic image may correspond to a target focal length adjustment. When acquiring each endoscopic image, the subject located within at least a portion of the imaging region (i.e., a "set of target sub-regions" herein) is within the effective depth of field during image acquisition. Therefore, at least a portion of each endoscopic image meets the sharpness requirement. The sharpness-compliant region corresponding to each endoscopic image may or may not include a portion of the imaging region at the initial focal length, and the sharpness-compliant region corresponding to each endoscopic image may substantially constitute the imaging region.

[0047] Step S140: Perform image fusion on the endoscopic images in the multi-depth image sequence to obtain a depth-extended image of the imaging area, which is used for display on a display device.

[0048] For example, the endoscopic images involved in the fusion can be endoscopic images from a multi-depth-field image sequence. Image fusion can employ pixel-level fusion methods, directly fusing the image data acquired by the image acquisition system using spatial domain and transform domain algorithms. The resulting depth-expanded image area corresponds to the imaging area captured by the image acquisition system at the initial focal length. The depth-expanded image can be output as the final image captured for that imaging area and displayed on the display device's interface, thus presenting the user with an endoscopic image where all areas are clearly imaged. Meanwhile, the individual endoscopic images from the multi-depth-field image sequence can serve as intermediate images and are not output to the display device's interface.

[0049] The aforementioned technical solution can adaptively obtain the target focal length adjustment amount required for clear imaging of each part of the current imaging area based on the phase difference information at multiple locations within the imaging area. It can accurately adjust the initial focal length to the target focal length corresponding to each focal plane of the imaging area, and this targeted focusing method has high focusing efficiency. On the other hand, the focal length after focusing corresponds to an effective depth of field range. In particular, after multiple focusing operations, the effective depth of field ranges of each endoscopic image in a multi-depth-of-field image sequence may differ, and the union of the effective depth of field ranges corresponding to each endoscopic image is relatively large. By performing image fusion on the endoscopic images acquired after focusing according to the target focal length adjustment amount by the image acquisition system to obtain a depth-of-field extended image, the effective depth of field range of the fused depth-of-field extended image can be increased, thereby enriching the image content of the depth-of-field extended image. Furthermore, by performing image fusion on the endoscopic images, it is possible to ensure that the subject within the imaging area can be adaptively and clearly imaged within the effective depth of field range as much as possible.

[0050] Optionally, in step S120, determining at least one target focal length adjustment amount based on the first phase difference information includes:

[0051] Determine the phase correlation distribution matrix based on the first phase difference information;

[0052] Peak values ​​of the statistical phase correlation distribution matrix;

[0053] Cluster the peaks of the phase correlation distribution matrix to determine the center of the first group of peaks;

[0054] At least one target focal length adjustment is determined based on at least some of the peak centers in the first group of peak centers.

[0055] For example, based on the first phase difference information, a matrix can be obtained that reflects the focal length adjustment amount by which the image acquisition system can achieve clear imaging at different positions in the imaging area. Normalizing this matrix yields the phase correlation distribution matrix. For example, the normalization method can be, for instance, a cross-correlation function, a phase consistency index, etc. In the phase correlation distribution matrix, each element can represent the object distance offset at the corresponding position in the imaging area. After obtaining the phase correlation distribution matrix, for example, but not limited to, a peak detection algorithm can be applied to find the maximum points in the matrix. By obtaining the position of the peak (corresponding to the coordinates of the peak in the phase correlation distribution matrix) and the corresponding phase correlation value, detection algorithms such as curve fitting, higher-order moments, principal component analysis, and clustering algorithms can be used to analyze the distribution characteristics of the peaks, and a peak threshold can be set to filter significant peaks. The concentrated distribution of a significant peak can represent a focal plane corresponding to at least a portion of the imaging area. For example, each significant peak can be defined as the center of the first set of peaks. The values ​​of each peak center in the first set of peaks have a mapping relationship with the object distance offset. This mapping relationship can be obtained through methods such as pre-calibration and interpolation. Based on the object distance offset, the focal length offset, i.e., the focusing focal length adjustment, can be determined. The determined partial or complete focusing focal length adjustment can be used as the target focal length adjustment. For example, the mapping relationship between the object distance offset and the focal length offset can also be used to convert the phase correlation distribution matrix into a focus offset distribution matrix. The focus offset distribution matrix can reflect the defocusing degree of each sub-region of the imaging area. The target focal length adjustment can also be obtained by analyzing the distribution characteristics of the peaks in the focus offset distribution matrix. The process of obtaining the target focal length adjustment based on the focus offset distribution matrix is ​​similar to that based on the phase correlation distribution matrix, and will not be elaborated here.

[0056] The above technical solution uses relevant detection algorithms to analyze the peak distribution characteristics of the phase correlation distribution matrix, which can obtain the focus shift degree of each sub-region in the imaging area at the current initial focal length. The focus shift degree can be used as a reference parameter to adjust the focal length to obtain one or more target focal length adjustment amounts for clear imaging of the imaging area, which helps to achieve adaptive division of different focal planes required for the imaging area.

[0057] Optionally, determining at least one target focal length adjustment based on at least a portion of the peak centers in the first set of peak centers includes:

[0058] The peak centers in the first group of peak centers are filtered according to the threshold of the number of target focal length adjustments to obtain the second group of peak centers;

[0059] At least one target focal length adjustment is determined based on each peak center in the second group of peak centers.

[0060] For example, when there are too many significant peaks in the acquired phase correlation distribution matrix, a threshold for the number of target focal length adjustments can be set to filter the significant peaks. For instance, by performing peak statistical analysis on the acquired phase correlation distribution matrix using a detection algorithm, 13 significant peaks are obtained, each corresponding to a different focal length adjustment. A threshold for the number of target focal length adjustments can be set, for example, 5. The focal length adjustments corresponding to the top 5 significant peaks ranked from largest to smallest absolute value among these 13 significant peaks are determined as the target focal length adjustments. It should be noted that the above method for filtering significant peaks is only an exemplary and optional implementation method. The specific filtering conditions for significant peaks are not limited in this embodiment of the invention and can be defined according to the element categories in the phase correlation distribution matrix.

[0061] The above technical solution, through parameter control of the detection algorithm, helps to ensure and balance the sensitivity and robustness of peak distribution detection of the phase correlation distribution matrix. In practical applications, a reasonable threshold can be set according to requirements to limit the maximum number of focusing attempts, while also taking into account the real-time and clarity requirements of endoscopic imaging.

[0062] Optionally, the phase-detection camera module includes at least two pixels, and the at least two pixels include multiple pairs of phase-detection pixels.

[0063] Before calculating the peak value of the phase correlation distribution matrix, at least one target focal length adjustment is determined based on the first phase difference information, and the method further includes:

[0064] Whether to perform interpolation on the phase correlation distribution matrix is ​​determined based on the comparison between the number of pixels of at least two pixels and the number of elements of the phase correlation distribution matrix. If the number of pixels of at least two pixels is not equal to the number of elements of the phase correlation distribution matrix, the phase correlation distribution matrix is ​​interpolated according to a preset interpolation method so that the number of elements of the phase correlation distribution matrix is ​​the same as the number of pixels of at least two pixels.

[0065] For example, an image sensor can be configured with imaging pixels and several pairs of phase-detection (PD) pixels. Imaging pixels are conventional pixels used solely for imaging. Unlike PD pixels, imaging pixels cannot acquire phase difference information corresponding to the imaging area. Therefore, when the number of pixels is not equal to the number of elements in the phase correlation distribution matrix, the phase correlation distribution matrix can be interpolated using a preset interpolation method to complete the phase difference information at the location of the imaging pixels. Each element of the interpolated phase correlation distribution matrix can correspond to the phase difference information of one pixel.

[0066] The above technical solution interpolates the phase correlation distribution matrix by comparing the number of pixels in the image sensor with the number of elements in the phase correlation distribution matrix. This helps to complete the phase difference information of imaging pixels other than PD pixels, making the phase difference information of each region more complete when the imaging area is imaged, which is conducive to obtaining a more complete and accurate target focal length adjustment amount.

[0067] Optionally, determining at least one target focal length based on at least a portion of the peak centers in the first set of peak centers includes:

[0068] At least one focus adjustment amount is determined based on at least some of the peak centers in the first group of peak centers. The at least one focus adjustment amount corresponds one-to-one with at least one group of initial sub-regions in the imaging area. For each focus adjustment amount, when the image acquisition system performs a focusing operation and images the imaging area based on the focus adjustment amount, the initial sub-region corresponding to the focus adjustment amount is in a focused state.

[0069] At least two of the at least one focal length adjustment amounts that meet the second preset requirement are optimized into the first target focal length adjustment amount. The focal length adjustment amount that does not meet the second preset requirement is used as the second target focal length adjustment amount. The at least one target focal length adjustment amount includes the first target focal length adjustment amount and the second target focal length adjustment amount.

[0070] The second preset requirement includes: the effective depth of field ranges of the image acquisition system after performing focusing operations based on at least two focusing focal length adjustment amounts have an overlap, and the target sub-region corresponding to each first target focal length adjustment amount is obtained by merging the initial sub-regions corresponding to the optimized focusing focal length adjustment amounts that participate in the first target focal length adjustment amount.

[0071] For example, the first set of peak centers can serve as the first phase difference information. Referring to the foregoing embodiment, for each peak center in the first set of peak centers, a focusing focal length adjustment amount can be obtained based on that peak center. It is understood that for each focusing focal length adjustment amount, if a focusing operation is performed and imaging is completed according to that focusing focal length adjustment amount, a set of initial sub-regions in the imaging region can be in a focused state. Each set of initial sub-regions can be a connected domain in the imaging region, or it can be multiple mutually independent connected domains in the imaging region. There can be overlapping parts between the sets of initial sub-regions corresponding to each focusing focal length adjustment amount, or the sets of initial sub-regions can exist independently of each other. For example, the second preset requirement may include that the effective depth of field ranges corresponding to the image acquisition system performing a focusing operation according to the focusing focal length adjustment amount have an intersection. By way of example, but not limitation, the second preset requirement may further include that after the image acquisition system performs a focusing operation based on the focal length adjustment amount, the corresponding effective depth-of-field ranges have an intersection, and the ratio of the first range corresponding to the intersection to the second range corresponding to the union of all effective depth-of-field ranges satisfies greater than or equal to a certain preset ratio, which can be defined according to actual imaging needs, for example, 0.8. For example, the focal length adjustment amount satisfying the second preset requirement can be optimized into a first target focal length adjustment amount, and the focal length adjustment amount other than the focal length adjustment amount participating in the optimization can be used as the second target focal length adjustment amount. Both the first target focal length adjustment amount and the second target focal length adjustment amount belong to the target focal length adjustment amount. It can be understood that the target sub-region corresponding to the first target focal length adjustment amount is the sub-region obtained by merging the initial sub-regions corresponding to the focal length adjustment amounts participating in the optimization of the first target focal length adjustment amount.

[0072] The above technical solution utilizes the phase difference distribution between different pixels to adaptively obtain the focal length adjustment amount required for clear imaging of different areas of the current imaging area. Based on the intersection relationship of the effective depth of field range corresponding to the focusing operation performed by the image acquisition system according to the focal length adjustment amount, the number of focal length adjustment amounts is optimized. This can minimize system latency to improve the working efficiency of the endoscope system, while ensuring that the subject in the current imaging area can be adaptively and clearly imaged within the effective depth of field range.

[0073] Optionally, in step S120, determining at least one target focal length adjustment amount based on the first phase difference information includes:

[0074] Based on the first phase difference information, at least one focusing focal length adjustment amount is determined. The at least one focusing focal length adjustment amount corresponds one-to-one with at least one set of initial sub-regions in the imaging area. For each focusing focal length adjustment amount, when the image acquisition system performs focusing operation and images the imaging area according to the focusing focal length adjustment amount, the initial sub-region corresponding to the focusing focal length adjustment amount is in a focused state.

[0075] At least two of the at least one focal length adjustment amounts that meet the second preset requirement are optimized into the first target focal length adjustment amount. The focal length adjustment amount that does not meet the second preset requirement is used as the second target focal length adjustment amount. The at least one target focal length adjustment amount includes the first target focal length adjustment amount and the second target focal length adjustment amount.

[0076] The second preset requirement includes: the effective depth of field ranges of the image acquisition system after performing focusing operations based on at least two focusing focal length adjustment amounts have an overlap, and the target sub-region corresponding to each first target focal length adjustment amount is obtained by merging the initial sub-regions corresponding to the optimized focusing focal length adjustment amounts that participate in the first target focal length adjustment amount.

[0077] For example, the focal length adjustment amount can be directly determined based on the acquired first phase difference information. Similar to the aforementioned embodiments, at least two focal length adjustment amounts that satisfy the second preset requirement can be optimized into a first target focal length adjustment amount. In a specific embodiment, if there is a focal length adjustment amount of 0.7mm, and the effective depth of field range corresponding to the image acquisition system performing a focusing operation based on this focal length adjustment amount is 1.5mm to 6.5mm, and another focal length adjustment amount of 0.5mm, and the effective depth of field range corresponding to the image acquisition system performing a focusing operation based on this focal length adjustment amount is 2.5mm to 7.5mm, then these two focal length adjustment amounts can be optimized into a target focal length adjustment amount. The optimized target focal length adjustment amount can be any value between 0.5mm and 0.7mm. Those skilled in the art will understand that in this embodiment, the target focal length adjustment amount can be a specific value within the focal length adjustment range, and this focal length adjustment range can be determined based on a combination of the focal length adjustment amount and the imaging area. For example, when the imaging area is part of the imaging field of view, the optimized target focal length adjustment amount does not require the image acquisition system to perform a focusing operation based on the target focal length adjustment amount. After that, all objects in the imaging field of view fall within the effective depth of field range corresponding to the target focal length, and only the subject in the imaging area is within the effective depth of field range. If the effective depth of field range corresponding to the focus adjustment operation performed by the image acquisition system based on a certain focal length adjustment amount does not overlap with the effective depth of field range corresponding to the focus adjustment operation performed based on other focal length adjustment amounts, then the focal length adjustment amount can be directly used as the target focal length adjustment amount. Alternatively, if the effective depth of field range corresponding to the focus adjustment operation performed by the image acquisition system based on a certain focal length adjustment amount has a small overlap with the effective depth of field range corresponding to the focus adjustment operation performed based on other focal length adjustment amounts, and after optimizing the focal length adjustment amount and other focal length adjustment amounts, the effective depth of field range corresponding to the focal length adjustment amount cannot ensure that the entire subject corresponding to the imaging area is within the effective depth of field range, then the focal length adjustment amount can be directly used as the target focal length adjustment amount.

[0078] The above technical solution uses phase difference information to determine the focal length adjustment amount, and optimizes the focal length based on the intersection relationship of the effective depth of field range corresponding to the focal length adjustment operation performed by the image acquisition system to determine the target focal length adjustment amount. This can effectively reduce the number of required focal lengths and help to shorten the focal length adjustment cycle to achieve rapid imaging.

[0079] Optionally, when the number of target focal length adjustments is 1, after step S120, the endoscopic imaging method further includes:

[0080] Determine whether the target focal length adjustment amount is greater than the preset adjustment amount threshold. If so, control the image acquisition system to perform a focusing operation according to the target focal length adjustment amount, so as to obtain and display the endoscope image acquired by the image acquisition system at the focused focal length. If not, display the endoscope image acquired by the image acquisition system at the initial focal length.

[0081] or,

[0082] Based on the first phase difference information, determine whether the imaging area is in a defocused state. If so, control the image acquisition system to perform a focusing operation according to the target focal length adjustment amount, so as to acquire and display the endoscope image acquired by the image acquisition system at the focused focal length. If not, display the endoscope image acquired by the image acquisition system at the initial focal length.

[0083] For example, the number of target focal length adjustments may be one. When the number of target focal length adjustments is one, the comparison between the target focal length adjustment and a preset adjustment threshold can be used to determine whether to perform a focusing operation. If the target focal length adjustment is less than or equal to the preset adjustment threshold, it can be considered that imaging the imaging area at the current initial focal length can bring the imaging area into focus, and no focusing is required. If the target focal length adjustment is greater than the preset adjustment threshold, it can be considered that there is only one focal plane in the imaging area, but imaging the imaging area at the current initial focal length cannot bring the imaging area into focus. In this case, the image acquisition system needs to be controlled to perform a focusing operation according to the target focal length adjustment in order to acquire and display the endoscopic image acquired by the image acquisition system at the focused focal length.

[0084] Alternatively, when only one target focal length adjustment is available, it is possible to directly determine whether the imaging area is out of focus based on the first phase difference information. For example, a phase difference threshold can be set. If the maximum phase difference of the imaging area in the first phase difference information is less than the phase difference threshold, it can be determined that the imaging area is in focus; otherwise, it is out of focus.

[0085] The above technical solution can determine whether the current actual initial focal length is equal to or close to the target focal length. If so, it can be considered that the current initial focal length can meet the requirement of clear imaging of the entire imaging area without the need to adjust the focal length of the image acquisition system to obtain the depth-of-field extended image, which is conducive to the rapid imaging of endoscopic images and saves time.

[0086] Optionally, in step S130, a multi-depth image sequence is acquired, including:

[0087] The image acquisition system is controlled to perform focusing operations and image the imaging area sequentially according to the preset adjustment order of the at least one target focal length adjustment amount;

[0088] The image acquisition system acquires endoscopic images of the imaging area after each focusing operation to obtain a multi-depth-of-field image sequence.

[0089] For example, the target focal lengths corresponding to each target focal length adjustment amount can be arranged in ascending or descending order. The image acquisition system can then perform focusing operations sequentially according to the order of the target focal length adjustment amounts. After each focusing operation, an image can be acquired from the imaging area to obtain an endoscope image. The endoscope images acquired after each focusing operation can then be combined to form a multi-depth-of-field image sequence.

[0090] The above technical solution can quickly acquire endoscopic images at various target focal lengths, effectively shortening the time to adjust the current focal length to the target focal length, which is beneficial to shortening the focal length adjustment cycle.

[0091] Optionally, in step S140, image fusion is performed on the endoscopic images in the multi-depth-field image sequence to obtain a depth-extended image of the imaging region, including:

[0092] Align endoscopic images in a multi-depth-of-field image sequence;

[0093] The region to be fused in the aligned endoscope image is determined. The region to be fused is the region whose clarity meets the first preset requirement.

[0094] The areas to be blended are blended to obtain a depth-of-field extended image.

[0095] For example, image alignment (also known as image registration) can be achieved using any existing or future image alignment method, such as feature-based image alignment. For example, image alignment can be performed as follows: select points of interest in each endoscopic image, match and associate these points, and select a reference image from the endoscopic images in a multi-depth-of-field image sequence according to actual imaging requirements. Use the other endoscopic images besides the reference image as floating images, and transform the floating images according to the matching and association results to achieve image alignment. After obtaining the aligned endoscopic images, for each aligned endoscopic image, it can be divided into multiple regions, and the sharpness of each region can be evaluated. Sharpness evaluation can be achieved using any existing or future sharpness evaluation method, such as image histogram method, energy variance method, squared gradient energy method, etc. Optionally, the sharpness of each region of the image can also be evaluated based on the correlation principle using the phase difference information corresponding to the image. The regions to be fused in each endoscope image can be determined based on the sharpness evaluation results of each region. For example, the sharpness of each region in any endoscope image can be represented by a sharpness score; a higher score indicates a sharper region. A first preset requirement may be, for example, that the sharpness score is greater than or equal to a preset sharpness threshold. For instance, if the sharpness score obtained after normalizing the image data of a certain region input into the sharpness function is greater than or equal to the preset sharpness threshold, then that region can be determined as the region to be fused. Those skilled in the art will understand that the preset sharpness threshold can be set according to the sharpness requirements of the depth-of-field extended image to be acquired, and this embodiment of the invention does not impose specific limitations. The method for fusing the regions to be fused can be, for example, stitching the regions to be fused together and then using a multi-band blending method, or a feature point matching method. The specific fusion method can be selected according to actual needs, as long as it satisfies the relative positional relationship between the coordinates of each pixel in the acquired depth-of-field extended image and the relative positional relationship between the position points of each pixel in the imaging region in the spatial coordinate system.

[0096] The above technical solution can fuse the clear areas in the endoscopic images obtained based on each target focal length, and expand the effective depth of field range by superimposing and fusing the clear areas, so that the corresponding imaging area in the obtained depth-of-field extended image meets the overall clarity requirements.

[0097] Optionally, aligning endoscopic images in a multi-depth-of-field image sequence includes:

[0098] Determine the pixel coordinates of each pixel in the endoscopic image in a unified spatial coordinate system in a multi-depth-of-field image sequence;

[0099] The pixel coordinates of each pixel in the endoscopic images in the multi-depth-of-field image sequence are matched to align the images in the multi-depth-of-field image sequence.

[0100] For example, a spatial coordinate system can be established to uniformly calibrate the coordinates of each pixel in each endoscopic image. The coordinates of each pixel can represent the spatial location information of the actual imaging area corresponding to that pixel. Based on the pixel coordinates of each pixel in each endoscopic image, pixels with the same spatial location information in each endoscopic image are matched to achieve image alignment of multiple images. Image alignment algorithms can be, for example, feature matching algorithms, template matching methods, and phase correlation methods. Specifically, the pixel coordinates of each pixel in each endoscopic image can be partially matched or fully matched. When the pixel coordinates of each pixel in each endoscopic image are partially matched, image alignment can be performed on each endoscopic image based on the pixels with matched pixel coordinates. Exemplarily, but not limitingly, before performing image alignment on each endoscopic image, image preprocessing can be performed on each endoscopic image. Image preprocessing methods can be, for example, image denoising, image enhancement, and image brightness compensation; the specific process of image preprocessing methods is not elaborated here.

[0101] The above technical solution, by matching each pixel of each endoscope image, can achieve the matching of the actual spatial information corresponding to the image regions of different endoscope images. This helps to make the position information corresponding to each pixel of the acquired depth-of-field extended image closer to the position information of the actual spatial location of the pixel, thereby making the information contained in the acquired depth-of-field extended image more reliable.

[0102] Optionally, determining the region to be fused from the aligned endoscopic images includes:

[0103] For each endoscopic image in a multi-depth-of-field image sequence, the second phase difference information corresponding to the acquisition system when acquiring the endoscopic image is obtained. The second phase difference information includes the phase difference information output by each pair of phase detection pixels when acquiring the endoscopic image.

[0104] The local sharpness of the endoscope image is evaluated based on the second phase difference information, and a specific region of the endoscope image is obtained based on the evaluation result of the local sharpness. The specific region is the region whose sharpness meets the first preset requirement.

[0105] The corresponding region obtained after the alignment operation of a specific region is used as the region to be fused in the aligned endoscopic image.

[0106] For example, the image acquisition system can perform an adjustment operation based on the focal length adjustment of each target image before imaging. During imaging, corresponding phase difference information can be acquired. Each endoscopic image in the multi-depth-of-field image sequence can correspond to a set of phase difference information, which is the second phase difference information. Similar to the first phase difference information, the second phase difference information can be output by the PD pixels. For example, each set of second phase difference information can include several second phase difference data points. Based on each second phase difference data point, the local sharpness of the corresponding position in the endoscopic image can be evaluated. Specifically, the closer the phase difference is to 0, the closer it is to a focused state. A certain phase difference threshold (which can be called a preset phase difference threshold) can be preset, and the area in the endoscopic image that is less than or equal to the preset phase difference threshold is taken as a specific area of ​​the endoscopic image. The preset phase difference threshold can be set to any suitable value as needed, such as 0.05. It is understood that the endoscopic image may undergo affine transformation processing during image alignment, and the position, shape, etc., of a certain area in the endoscopic image will change before and after alignment. For any region in any endoscopic image, the corresponding region in the aligned endoscopic image can be determined according to the image alignment method used. In a specific embodiment, the multi-depth image sequence includes a first endoscopic image, a second endoscopic image, and a third endoscopic image. A specific region 'a' of the first endoscopic image is acquired, specific regions 'b' and 'c' of the second endoscopic image are acquired, and a specific region 'd' of the third endoscopic image is acquired. If specific regions 'b' and 'c' can cover the entire area of ​​specific region 'd', and the clarity of specific regions 'b' and 'c' is higher than that of specific region 'd', then only the corresponding regions of specific regions 'a', 'b', and 'c' in their respective aligned endoscopic images can be used as the regions to be fused. In another specific embodiment, specific regions a, b, c, and d each contain distinct, non-overlapping sub-regions. For each specific region, the non-overlapping sub-regions belong to at least a portion of the imaging region. Therefore, the corresponding regions of specific regions a, b, c, and d in their respective aligned endoscopic images can be used as regions to be fused. More specifically, if overlapping regions exist among the regions to be fused, the region with the highest clarity among those containing the overlapping region can be selected, allowing the overlapping region to participate in the image overlay and fusion.

[0107] The above technical solution directly evaluates the local sharpness of each endoscopic image using phase difference information to quickly identify sharp areas. For example, areas with a phase difference closer to zero are considered closer to being in focus. Regions with a phase difference less than or equal to a preset phase difference threshold are considered the closest to being in focus (i.e., their sharpness meets the first preset requirement). Compared to conventional image sharpness evaluation methods such as Laplacian filtering or feature operators, this sharpness evaluation method has a simple algorithm and high computational efficiency.

[0108] Optionally, before fusing the regions to be fused to obtain a depth-extended image, the method further includes:

[0109] Smooth the edges of the areas to be merged.

[0110] For example, before fusing the regions to be merged, the edges of the regions to be merged can be smoothed to reduce noise in each region. This embodiment of the invention does not limit the specific method of smoothing; optional smoothing methods may include, for example, mean filtering, median filtering, Gaussian filtering, etc., and the specific filtering process will not be described in detail here.

[0111] The above technical solution utilizes the characteristic that endoscopic images usually have depth continuity. By smoothing the edges of the areas to be fused in each endoscopic image, the noise in each area to be fused can be reduced, which is beneficial to improving the image effect of the final depth-extended image.

[0112] Please see Figure 3 The diagram shown is a schematic flowchart of an algorithm for an endoscope system according to an embodiment of the present invention. The endoscopic imaging method used in the endoscope system applied in this embodiment is based on... Figure 2 The algorithm flow shown is implemented, and the structure of the endoscope system used in this embodiment can be referred to. Figure 1In this embodiment, when the endoscope system starts working, that is, when it begins imaging the imaging area, the deformable lens is driven to reset via the lens driving submodule. In the reset state, the optical properties of the deformable lens (e.g., shape, curvature, etc.) allow the focal length of the optical imaging module to be adjusted to the default focal length (i.e., the initial focal length in step S110 above). This default focal length can be set manually before imaging the imaging area according to actual needs, which will not be elaborated here. When the focal length of the optical imaging module is adjusted to the default focal length, the control device sends a light source control signal to the optical illumination module and the imaging control module to adjust the light source parameters and camera control parameters (e.g., exposure parameters, including exposure time, gain, etc.) to the specified parameter values, and sends an image acquisition drive signal to the camera control submodule to trigger image acquisition. When the camera control submodule is triggered to acquire an image, the image sensor can detect the phase correlation distribution of each pair of PD pixels in a row / column / block manner to obtain the phase difference information of each pair of PD pixels, forming the first phase difference information and the corresponding phase correlation distribution matrix (referring to the aforementioned embodiment, this phase correlation distribution matrix can also be converted into a focus offset distribution matrix). The peak distribution of this phase correlation distribution matrix (or focus offset distribution matrix) is statistically analyzed using statistical methods (e.g., curve fitting, higher-order moments, principal component analysis, clustering algorithms, etc.) to obtain significant peaks. The concentrated distribution of each significant peak can represent a focal plane corresponding to at least a portion of the imaging area, thereby achieving intelligent adaptive division of the focal plane of the imaging area. Based on each significant peak, the number of target focal length adjustments required for clear imaging can be determined. Each target focal length adjustment corresponds to a target focal length. When the number of target focal length adjustments is only a single value, it can be determined whether the target focal length adjustment is greater than a preset adjustment threshold, or whether the imaging area is in a defocused state based on the first phase difference information, to determine whether the current initial focal length can achieve clear imaging. If the target focal length adjustment amount is greater than the preset adjustment amount threshold, or if the imaging area is determined to be in a defocused state based on the first phase difference information, the lens driving submodule sends a lens driving control signal to drive the focal length of the deformable lens to adjust according to the target focal length adjustment amount. The camera control submodule can trigger image acquisition to obtain an endoscope image for display when the focal length of the deformable lens is adjusted according to the target focal length adjustment amount. If the absolute value of the difference between the default focal length and the target focal length is less than or equal to the preset difference threshold, or if the imaging area is determined to be in a focused state based on the first phase difference information, the endoscope image for display can be directly output at the current default focal length.When there are multiple target focal length adjustment amounts, the lens driving submodule generates and sends corresponding periodic lens driving control signals to achieve rapid focusing. The camera control submodule can trigger image acquisition when the focal length of the deformable lens is adjusted according to the target focal length adjustment amount. Specifically, each time the current focal length of the deformable lens is adjusted to a target focal length N according to the target focal length adjustment amount, the camera control submodule triggers image acquisition accordingly. The joint control submodule judges the driving cycle. The end condition of a driving cycle is to perform the adjustment operation according to all target focal length adjustment amounts and perform image acquisition on the imaging area. If it is determined that the driving cycle has not ended, the operation of adjusting the current focal length of the deformable lens according to the target focal length adjustment amount and the operation of performing image acquisition on the imaging area after performing the focusing operation according to the target focal length adjustment amount are repeated until the driving cycle ends. After obtaining multiple depth-of-field image sequences corresponding to multiple target focal length adjustment amounts, the endoscopic images in the multiple depth-of-field image sequences are aligned. The local sharpness of each endoscopic image in the multiple depth-of-field image sequences is evaluated based on the second phase difference information corresponding to the acquisition of each endoscopic image in the multiple depth-of-field image sequences. Based on the evaluation results of local sharpness, specific regions of each endoscopic image that meet the preset sharpness requirements are obtained. Based on the position of the specific regions of each endoscopic image, the regions to be fused of the aligned endoscopic images corresponding to each endoscopic image are determined. The edges of each region to be fused are smoothed and optimized. The smoothed and optimized regions to be fused are then fused to obtain a depth-of-field extended image.

[0113] Please see Figure 4 As shown, it is a timing diagram of control signals for an endoscope system according to an embodiment of the present invention. Figure 4 The structure of the endoscope system used in the embodiments can be referred to Figure 1 . Figure 4Three sets of signal timing curves are shown. From top to bottom, they represent the timing of the lens drive control signal sent from the lens control submodule to the adjustable focus lens group, the depth-of-field change timing corresponding to the lens drive control signal based on the response characteristics of the adjustable focus lens group, and the camera exposure control signal timing sent from the joint control submodule to the exposure control submodule. When the lens drive control signal begins to change, the depth of field of the optical imaging module gradually begins to change because the adjustable focus lens group typically needs a certain amount of time to stabilize at the target focal length. When the focal length of the deformable lens has stabilized under the control of the lens drive control signal, the depth of field of the optical imaging module no longer changes with the stabilization of the focal length of the deformable lens. At this point, the depth-of-field change timing curve changes from a gradual state to a flat state. When the depth-of-field change control signal begins to flatten, that is, at the initial moment when the depth of field of the acquired image no longer changes, the camera exposure control signal changes at that initial moment. This avoids blurring in certain areas of the endoscopic image caused by the focusing process.

[0114] Please see Figure 5 The diagram shown is a schematic block diagram of an endoscopic image processing apparatus according to an embodiment of the present invention. According to another aspect of the present application, an electronic device endoscopic image processing apparatus 500 is also provided, comprising: a processor 510 and a memory 520, wherein the memory 520 stores computer program instructions, which, when executed by the processor 610, are used to perform the endoscopic imaging method in any of the above embodiments.

[0115] According to another aspect of the present invention, an endoscope system is also provided. The endoscope system includes:

[0116] An image acquisition system, comprising an adjustable focus lens group and a phase difference camera module, wherein the phase difference camera module includes multiple pairs of phase detection pixels, each pair of phase detection pixels being used to detect phase difference information at its imaging position.

[0117] as well as,

[0118] As described above, the endoscopic image processing device is communicatively connected to both the adjustable focus lens group and the phase contrast camera module.

[0119] For example, the components of an image acquisition system can be referred to Figure 1In the provided embodiments, the image sensor may be, for example, a CMOS image sensor with autofocus, which may include imaging pixels and PD pixels. The imaging pixels are configured to detect color, and the PD pixels are configured to detect phase difference. The imaging pixels include multiple color filters covering photodiodes. The color filters are arranged in staggered positions and configured to filter out a specific range of radiation spectrum. After filtering by the color filters, the photodiodes detect the intensity of radiation received by the imaging pixels. The PD pixels are arranged in pairs. A pair of PD pixels may have opposite sides open to incident radiation. In one specific embodiment, when the right half of one PD pixel in a PD pixel pair is blocked by an opaque structure, the left half is transparent to at least some of the incident radiation. When the left half of the other phase-detecting pixel in a PD pixel pair is blocked by an opaque structure, the right half is transparent to at least some of the incident radiation. Therefore, the radiation received by the PD pixel pair has a phase difference, which can be used to determine the focusing of light incident at that location. Exemplarily, the optical imaging module may include several lens groups, for example, each basic lens group may be a non-deformable lens group, and each adjustable lens group may be a deformable lens group. The optical properties (e.g., shape, curvature) of the adjustable focus lens group are adjustable. The imaging control module can communicate with the adjustable focus lens group and control the shape of the deformable lens in the adjustable focus lens group to change the focal length for imaging the imaging area.

[0120] Optionally, the focusing lens group includes a deformable lens.

[0121] For example, when the shape of the deformable lens in the adjustable focus lens group changes, the path of light entering the deformable lens also changes accordingly. More specifically, the deformable lens can be a convex lens. When the curvature of the convex lens increases, its refractive index also increases. After the light reflected from the imaging region enters the deformable lens, the focal point of the light is closer to the deformable lens than the focal point before the shape of the deformable lens changes. Therefore, the path of light can change with the shape of the deformable lens in the adjustable focus lens group.

[0122] Optionally, the phase contrast camera module includes an image sensor, and the photosensitive surface of the image sensor is provided with multiple pairs of phase detection pixels and multiple imaging pixels. The multiple imaging pixels are used to receive light from the imaging area to generate an endoscopic image of the imaging area.

[0123] For example, the descriptions of the PD pixels and imaging pixels of the image sensor can be found in the foregoing embodiments, and will not be repeated here.

[0124] According to another aspect of the embodiments of this application, a storage medium is also provided. This storage medium stores a computer program / instructions, which, when executed by a processor, implement the endoscopic imaging method described in any of the above embodiments. The storage medium may, for example, include a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disc read-only memory (CD-ROM), a USB memory, or any combination of the above storage media. A computer-readable storage medium may be any combination of one or more computer-readable storage media.

[0125] According to another aspect of the embodiments of this application, a computer program product is also provided, including computer program instructions, which, when run, are used to perform the endoscopic imaging method described in any of the above embodiments.

[0126] Those skilled in the art can easily understand the implementation structure, working principle, and beneficial effects of the endoscopic system by reading the above-described endoscopic imaging methods. For the sake of brevity, further details will not be elaborated here.

[0127] Although exemplary embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above exemplary embodiments are merely illustrative and are not intended to limit the scope of the invention. Various changes and modifications can be made therein by those skilled in the art without departing from the scope and spirit of the invention. All such changes and modifications are intended to be included within the scope of the invention as claimed in the appended claims.

[0128] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0129] In the several embodiments provided by this invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed.

[0130] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0131] Similarly, it should be understood that, in order to streamline the invention and aid in understanding one or more of the various aspects of the invention, features of the invention are sometimes grouped together in a single embodiment, figure, or description thereof in the description of exemplary embodiments of the invention. However, this approach should not be construed as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim. Rather, as reflected in the corresponding claims, its inventive point lies in solving the corresponding technical problem with fewer features than all of those in a single disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into that detailed description, wherein each claim itself is a separate embodiment of the invention.

[0132] Those skilled in the art will understand that, apart from the mutual exclusion of features, all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or units of any method or apparatus so disclosed can be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature that serves the same, equivalent, or similar purpose.

[0133] Furthermore, those skilled in the art will understand that although some embodiments described herein include certain features but not others included in other embodiments, combinations of features from different embodiments are intended to be within the scope of the invention and form different embodiments. For example, in the claims, any of the claimed embodiments can be used in any combination.

[0134] The various component embodiments of the present invention can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art will understand that microprocessors or digital signal processors (DSPs) can be used in practice to implement some or all of the functions of some modules of the endoscope system according to embodiments of the present invention. The present invention can also be implemented as an apparatus program (e.g., a computer program and computer program product) for performing some or all of the methods described herein. Such programs implementing the present invention can be stored on a computer-readable medium or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

[0135] It should be noted that the above embodiments are illustrative of the invention and not restrictive, and that those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

[0136] The above description is merely a specific embodiment of the present invention or an explanation of that embodiment. The scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. The scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. An endoscopic imaging method, characterized in that, An endoscope system is used, the endoscope system including an image acquisition system, the image acquisition system including an adjustable focusing lens group and a phase difference camera module, the phase difference camera module including multiple pairs of phase detection pixels, each pair of phase detection pixels being used to detect phase difference information at its imaging position, the endoscopic imaging method including: The first phase difference information corresponding to the imaging of the imaging area by the image acquisition system at the initial focal length is obtained. The first phase difference information includes the phase difference information output by each pair of phase detection pixels during imaging. Based on the first phase difference information, at least one target focal length adjustment amount is determined, wherein the at least one target focal length adjustment amount corresponds one-to-one with at least one set of target sub-regions in the imaging area. For each target focal length adjustment amount, when the image acquisition system performs a focusing operation and images the imaging area based on the target focal length adjustment amount, the target sub-region corresponding to the target focal length adjustment amount is in focus. Wherein, when the number of target focal length adjustments is greater than 1: Acquire a multi-depth-of-field image sequence, the multi-depth-of-field image sequence including endoscopic images acquired by the image acquisition system when performing focusing operation and imaging according to each target focal length adjustment amount; Image fusion is performed on the endoscopic images in the multi-depth-field image sequence to obtain a depth-of-field extended image of the imaging region.

2. The method according to claim 1, characterized in that, The step of determining at least one target focal length adjustment amount based on the first phase difference information includes: Determine the phase correlation distribution matrix based on the first phase difference information; Statistically analyze the peak value of the phase correlation distribution matrix; Cluster the peaks of the phase correlation distribution matrix to determine the first group of peak centers; The at least one target focal length adjustment is determined based on at least a portion of the peak centers in the first group of peak centers.

3. The method according to claim 2, characterized in that, Determining the at least one target focal length adjustment based on at least a portion of the peak centers in the first group of peak centers includes: The peak centers in the first group of peak centers are filtered according to the threshold of the number of target focal length adjustments to obtain the second group of peak centers; The at least one target focal length adjustment amount is determined based on each peak center in the second group of peak centers.

4. The method according to claim 2, characterized in that, Determining the at least one target focal length adjustment based on at least a portion of the peak centers in the first group of peak centers includes: At least one focus adjustment amount is determined based on at least some of the peak centers in the first group of peak centers. The at least one focus adjustment amount corresponds one-to-one with at least one group of initial sub-regions in the imaging area. For each focus adjustment amount, when the image acquisition system performs a focusing operation and images the imaging area based on the focus adjustment amount, the initial sub-region corresponding to the focus adjustment amount is in focus. At least two of the at least one focal length adjustment amounts that meet the second preset requirement are optimized into a first target focal length adjustment amount, wherein the at least one focal length adjustment amount that does not meet the second preset requirement is used as a second target focal length adjustment amount, and the at least one target focal length adjustment amount includes the first target focal length adjustment amount and the second target focal length adjustment amount. The second preset requirement includes: the effective depth of field ranges of the image acquisition system after performing focusing operations based on the at least two focusing focal length adjustment amounts have an overlap, and the target sub-region corresponding to each first target focal length adjustment amount is obtained by merging the initial sub-regions corresponding to the focusing focal length adjustment amounts that participated in the optimization of the first target focal length adjustment amount.

5. The method according to claim 1, characterized in that, The step of determining at least one target focal length adjustment amount based on the first phase difference information includes: Based on the first phase difference information, at least one focus adjustment amount is determined. The at least one focus adjustment amount corresponds one-to-one with at least one set of initial sub-regions in the imaging area. For each focus adjustment amount, when the image acquisition system performs a focusing operation and images the imaging area according to the focus adjustment amount, the initial sub-region corresponding to the focus adjustment amount is in focus. At least two of the at least one focal length adjustment amounts that meet the second preset requirement are optimized into a first target focal length adjustment amount, wherein the at least one focal length adjustment amount that does not meet the second preset requirement is used as a second target focal length adjustment amount, and the at least one target focal length adjustment amount includes the first target focal length adjustment amount and the second target focal length adjustment amount. The second preset requirement includes: the effective depth of field ranges of the image acquisition system after performing focusing operations based on the at least two focusing focal length adjustment amounts have an overlap, and the target sub-region corresponding to each first target focal length adjustment amount is obtained by merging the initial sub-regions corresponding to the focusing focal length adjustment amounts that participated in the optimization of the first target focal length adjustment amount.

6. The method according to claim 1, characterized in that, The step of image fusion of endoscopic images in the multi-depth-of-field image sequence to obtain a depth-of-field extended image of the imaging region includes: Align the endoscopic images in the multi-depth-of-field image sequence; The region to be fused in the aligned endoscopic image is determined, wherein the region to be fused is a region whose clarity meets a first preset requirement; The regions to be merged are then merged to obtain the depth-of-field extended image.

7. The method according to claim 6, characterized in that, The process of determining the region to be fused in the aligned endoscopic image includes: For each endoscopic image in the multi-depth-of-field image sequence The second phase difference information corresponding to the image acquisition system when acquiring the endoscope image is obtained. The second phase difference information includes the phase difference information output by each pair of phase detection pixels when acquiring the endoscope image. The local sharpness of the endoscope image is evaluated based on the second phase difference information, and a specific region of the endoscope image is obtained based on the evaluation result of the local sharpness. The specific region is the region whose sharpness meets the first preset requirement. The corresponding region obtained after the alignment operation of the specific region is used as the region to be fused in the aligned endoscopic image.

8. The method according to claim 6, characterized in that, Before fusing the regions to be fused to obtain a depth-extended image of the imaging region, the method further includes: The edges of the region to be fused are smoothed.

9. The method according to any one of claims 1-8, characterized in that, When the number of target focal length adjustments is 1, the endoscopic imaging method further includes: If the target focal length adjustment amount is greater than a preset adjustment amount threshold, the image acquisition system is controlled to perform a focusing operation according to the target focal length adjustment amount to acquire and display the endoscope image acquired by the image acquisition system at the focused focal length; otherwise, the endoscope image acquired by the image acquisition system at the initial focal length is displayed. or, Based on the first phase difference information, it is determined whether the imaging area is in a defocused state. If so, the image acquisition system is controlled to perform a focusing operation according to the target focal length adjustment amount to acquire and display the endoscope image acquired by the image acquisition system at the focused focal length. If not, the endoscope image acquired by the image acquisition system at the initial focal length is displayed.

10. An endoscopic image processing device, comprising: A processor and a memory, wherein the memory stores computer program instructions, characterized in that the computer program instructions, when executed by the processor, are used to perform the endoscopic imaging method as described in any one of claims 1 to 9.

11. An endoscope system, characterized in that, include: An image acquisition system, comprising an adjustable focus lens group and a phase difference camera module, wherein the phase difference camera module includes multiple pairs of phase detection pixels, each pair of phase detection pixels being used to detect phase difference information at its imaging position. as well as, The endoscopic image processing device as described in claim 10, wherein the endoscopic processing device is communicatively connected to the adjustable focus lens group and the phase difference camera module, respectively.

12. The endoscope system according to claim 11, characterized in that, The adjustable focus lens group includes deformable lenses.

13. The endoscopic system according to claim 11 or 12, characterized in that, The phase difference camera module includes an image sensor, and the photosensitive surface of the image sensor is provided with multiple pairs of phase detection pixels and multiple imaging pixels. The multiple imaging pixels are used to receive light from the imaging area to generate an endoscopic image of the imaging area.

14. A storage medium on which program instructions are stored, characterized in that, The program instructions are used to execute the endoscopic imaging method as described in any one of claims 1 to 9 when the program is run.

15. A computer program product comprising computer program instructions, characterized in that, The computer program instructions, when executed, are used to perform the endoscopic imaging method as described in any one of claims 1 to 9.

Images