Endoscope color crosstalk suppression method and endoscope

Abstract

The application belongs to the technical field of endoscope imaging, and provides an endoscope color crosstalk suppression method and an endoscope. The method comprises the following steps: acquiring an original endoscope image collected by an image sensor; identifying target object parameters in the original endoscope image, wherein the target object parameters comprise a target color channel corresponding to a target object and a target size parameter; determining a target blur convolution kernel corresponding to the target object parameters according to a mapping relationship between calibration target object parameters and a calibration blur convolution kernel, wherein the target blur convolution kernel represents a combination of an optical diffusion parameter corresponding to the target size parameter of the target object in the target color channel and a sensor crosstalk parameter in a target channel; and performing deconvolution processing on the original endoscope image based on the target blur convolution kernel to obtain an endoscope imaging image. The method can solve the problems of tissue boundary blurring and tissue color distortion caused by optical diffusion and image sensor crosstalk during narrowband imaging, thereby more accurately displaying the color change of the tissue.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical endoscope imaging, and in particular to an endoscope color crosstalk suppression method and an endoscope. BACKGROUND

[0002] Endoscope narrow-band light imaging combines the absorption characteristics of hemoglobin and the penetration ability of light in tissue, selects a narrow-band as an illumination light source for imaging, improves the contrast between blood vessels and tissue, and improves image visibility. CMOS image sensors have the advantages of low power consumption, small heat generation, high integration and miniaturization, so endoscope imaging systems based on CMOS image sensors have become mainstream. However, the photosensitive unit of the CMOS image sensor can accept a wide range of light spectrum, which causes a single wavelength light signal to be detected in multiple color channels (R\G\B) at the same time. Therefore, when using a CMOS image sensor for multi-spectral narrow-band light imaging, multiple narrow-band lights are simultaneously irradiated, and the reflected light causes unexpected interference between sensor pixels, resulting in unexpected signal overlap in each channel image. In narrow-band light images, color diffusion occurs at high-contrast edges, and tissue boundaries are blurred. In actual endoscope imaging, edge blurring is caused by two factors: optical diffusion and CMOS image sensor crosstalk. Tissue boundary blurring and tissue color distortion caused by optical diffusion and image sensor crosstalk reduce the tissue discrimination of narrow-band light images and make it difficult to accurately display tissue color changes.

[0003] Therefore, how to improve the tissue discrimination of narrow-band light images based on COMS image sensors and accurately display tissue color changes is a problem to be solved. SUMMARY

[0004] Embodiments of the present application provide an endoscope color crosstalk suppression method and an endoscope to solve the technical problem of how to improve the tissue discrimination of narrow-band light images based on COMS image sensors in the prior art.

[0005] According to a first aspect, the present application provides an endoscope color crosstalk suppression method, comprising:

[0006] Obtaining an original endoscope image collected by an image sensor;

[0007] Identifying a target object parameter in the original endoscope image, the target object parameter including a target color channel corresponding to a target object and a target size parameter;

[0008] determine a target blur convolution kernel corresponding to the target object parameter according to a mapping relationship between a pre-calibrated target object parameter and a calibration blur convolution kernel, wherein the target blur convolution kernel represents a combination of an optical diffusion parameter corresponding to the target size parameter of the target object under the target color channel and a sensor cross-talk parameter under the target color channel;

[0009] perform deconvolution processing on the original endoscope image based on the target blur convolution kernel to obtain an endoscope imaging image.

[0010] In an embodiment, the performing cross-talk suppression processing on the original endoscope image based on the cross-talk parameter to obtain an endoscope imaging image includes:

[0011] determining a target region where the target object is located in the original endoscope image;

[0012] performing deconvolution operation on the target region by color channel using the target blur convolution kernel to obtain a cross-talk suppression image;

[0013] fusing the cross-talk suppression image and the original endoscope image to obtain the endoscope imaging image.

[0014] In an embodiment, the calibration method of the mapping relationship includes:

[0015] obtaining a cross-talk calibration image having a calibration target object in the cross-talk calibration image;

[0016] constructing a line spread function of the calibration target object under each calibration color channel of the cross-talk calibration image;

[0017] fitting the calibration target object under each calibration color channel of the cross-talk calibration image using a Gaussian distribution model to obtain an optical diffusion parameter corresponding to different calibration size parameters of the calibration target object under different calibration color channels and a sensor cross-talk parameter under different calibration color channels;

[0018] combining the optical diffusion parameter and the sensor cross-talk parameter to construct a mapping relationship between the calibration target object parameter and the blur convolution kernel.

[0019] In an embodiment, the cross-talk calibration image includes an optical cross-talk calibration image, and the optical cross-talk calibration image is a biomimetic target image of multiple calibration color channels acquired by a calibration image sensor under illumination of a single narrowband light source corresponding to each calibration color channel;

[0020] constructing a line spread function of the calibration target object under each calibration color channel of the cross-talk calibration image includes:

[0021] The first sampling points of the calibration target objects with different calibration size parameters under each calibration color channel are obtained by sampling the calibration target objects with different calibration size parameters in the optical crosstalk image of each calibration color channel, wherein the number of the first sampling points is inversely related to the value of the calibration size parameter;

[0022] The first point spread function of the calibration target object corresponding to each calibration size parameter under each calibration color channel is extracted based on the first sampling points corresponding to each calibration size parameter;

[0023] The first line spread function is constructed based on the first point spread function.

[0024] In an embodiment, the crosstalk calibration image includes a plurality of sensor crosstalk calibration images under a plurality of calibration color channels;

[0025] The line spread function of the calibration target object under each calibration color channel in the crosstalk calibration image is constructed, including:

[0026] The second sampling points of the calibration target object under each calibration color channel are obtained by sampling the calibration target object in the sensor crosstalk calibration image under each calibration color channel;

[0027] The second point spread function of the calibration target object under each calibration color channel is extracted for the second sampling points;

[0028] The second line spread function is constructed based on the second point spread function.

[0029] In an embodiment, the fitting of the calibration target object under each calibration color channel in the crosstalk calibration image using a Gaussian distribution model includes:

[0030] The first line spread function is fitted using a first Gaussian distribution model to obtain the optical diffusion parameters corresponding to different size parameters under the calibration color channel;

[0031] A first mapping table of the optical diffusion parameters related to the calibration color channel and the calibration size parameter is established;

[0032] The second line spread function is fitted using a second Gaussian distribution model to obtain the sensor crosstalk parameters corresponding to the calibration color channel;

[0033] A second mapping table of the sensor crosstalk parameters related to the calibration color channel is established.

[0034] In an embodiment, the acquisition method of the sensor crosstalk calibration image includes:

[0035] acquire a mixed calibration color channel biomimetic target image of the calibration image sensor set under simultaneous irradiation of all the narrowband light sources corresponding to all the calibration color channels;

[0036] merge all the optical crosstalk calibration images into a composite image;

[0037] subtract the mixed calibration color channel biomimetic target image from the composite image to obtain a mixed calibration color channel sensor crosstalk image;

[0038] perform channel separation on the mixed calibration color channel sensor crosstalk image to obtain the sensor crosstalk calibration images under multiple calibration color channels.

[0039] In an embodiment, the deconvolution method includes but is not limited to one of direct inverse filtering, Wiener filtering, iterative optimization, regularization, and blind deconvolution.

[0040] In an embodiment, the identifying a target object parameter in the original endoscope image includes:

[0041] identifying a target object in an original endoscope image through spectral absorption differences between a target biological tissue corresponding to the target object and surrounding tissues.

[0042] According to a second aspect, an endoscope color crosstalk suppression device is provided, including:

[0043] an acquisition module configured to acquire an original endoscope image collected by an image sensor;

[0044] an identification module configured to identify a target object parameter in the original endoscope image, the target object parameter including a target color channel corresponding to a target object and a target size parameter;

[0045] a determination module configured to determine a target blur kernel corresponding to the target object parameter according to a mapping relationship between a calibration target object parameter and a calibration blur kernel, wherein the target blur kernel represents a combination of an optical diffusion parameter corresponding to the target size parameter of the target object under the target color channel and a sensor crosstalk parameter under the target color channel;

[0046] a crosstalk suppression module configured to perform deconvolution processing on the original endoscope image based on the target blur kernel to obtain an endoscope imaging image.

[0047] According to a third aspect, a computer readable storage medium is provided, which stores a computer program, and the computer program is executed by a processor to implement the endoscope color crosstalk suppression method according to any one of the first aspect.

[0048] According to a fourth aspect, the present application provides an endoscope, comprising an image sensor, an illumination light source and a controller, wherein the controller comprises one or more processors, one or more memories, and one or more computer program instructions, which, when executed by the processor, implement the endoscope color crosstalk suppression method of any one of the first aspect.

[0049] The present application provides an endoscope color crosstalk suppression method, which comprises the following steps: acquiring an original endoscope image collected by an image sensor; identifying a target object parameter in the original endoscope image, wherein the target object parameter comprises a target color channel corresponding to a target object and a target size parameter; determining a target blur kernel corresponding to the target object parameter according to a mapping relationship between a calibration target object parameter and a calibration blur kernel, wherein the target blur kernel represents a combination of an optical diffusion parameter corresponding to the target size parameter of the target object in the target color channel and a sensor crosstalk parameter in the target color channel; and performing deconvolution processing on the original endoscope image based on the target blur kernel to obtain an endoscope imaging image. After the original endoscope image is acquired, the target object and the target object parameter are identified in the original endoscope image, the target blur kernel used to represent the combination of the optical diffusion parameter corresponding to the target size parameter of the target object in the target color channel and the sensor crosstalk parameter in the target channel is determined for the target object according to the mapping relationship between the calibration target object parameter and the calibration blur kernel, and the deconvolution operation is performed on the original endoscope image by using the target blur kernel, so that the image is processed to eliminate the crosstalk in different regions. The problems of tissue boundary blurring and tissue color distortion caused by optical diffusion and image sensor crosstalk during narrowband imaging are solved, so that the color change of the tissue can be more accurately displayed. BRIEF DESCRIPTION OF DRAWINGS

[0050] The accompanying drawings, which are included to provide a further understanding of the present application and constitute a part of this application, illustrate certain illustrative embodiments of the present application and together with the description serve to explain the present application. In the drawings:

[0051] Figure 1 A flowchart of an endoscope color crosstalk suppression method provided for some embodiments of the present application is shown in the figure;

[0052] Figure 2 A mapping relationship calibration method provided for some embodiments of the present application is shown in the figure;

[0053] Figure 3 A blood vessel sampling method provided for some embodiments of the present application is shown in the figure;

[0054] Figure 4Structure diagram of an endoscope color crosstalk suppression device provided for some embodiments of the present application;

[0055] Figure 5 Controller diagram in an endoscope provided for some embodiments of the present application. DETAILED DESCRIPTION

[0056] In order to make the purposes, technical solutions and advantages of the present application clearer, the technical solutions of the present application will be described clearly and completely below in combination with specific embodiments of the present application and corresponding drawings. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without any creative work fall within the protection scope of the present application.

[0057] Term explanation:

[0058] Sensor crosstalk: The band crosstalk of a CMOS image sensor refers to the phenomenon that light signals of different color channels (such as red, green and blue) are mixed, details are blurred or signal-to-noise ratio is reduced due to the overlapping of light transmission ranges of filters, charge leakage of adjacent pixels or penetration of oblique incident light through adjacent pixels, which can be suppressed by optimizing filter design, increasing microlenses or improving pixel isolation process.

[0059] Endoscope: An endoscope is an optical instrument composed of a cold light source lens, a fiber light guide, an image transmission system, a screen display system and the like. It usually enters the human body through natural orifices or small incisions. When used, the doctor directs the endoscope into the position to be examined by external manual operation, directly probes the changes of the relevant parts, and cooperates with surgical instruments to perform necessary diagnosis and treatment. The imaging quality of the endoscope directly affects the use experience, probing accuracy and efficiency of the doctor, and also marks the development level of the endoscope technology.

[0060] Spectral characteristics: Biological tissues have different absorption and reflection characteristics, i.e. spectral characteristics, under different wavebands. The optical characteristics (absorption, scattering and reflection) of biological tissues are used in medicine for imaging, mainly through the following ways: techniques based on absorption differences (such as near-infrared spectral imaging for monitoring blood oxygen and photoacoustic imaging for detecting tumors); techniques based on scattering / reflection (such as OCT for micron-level structure imaging and confocal microscopy for observing cell morphology); and multi-modal fusion (such as photoacoustic-ultrasound combined imaging). These methods analyze the characteristic responses of tissues at specific wavelengths (such as hemoglobin absorption peaks and collagen scattering signals), realize disease diagnosis (cancer detection and blood vessel imaging), surgical navigation and physiological monitoring from macro to micro, and have both functional and structural information.

[0061] Narrow band imaging: Narrow band imaging is an optical imaging technology that enhances the visibility of mucosal blood vessels, lesions, inflammation and other parts by using one or several narrow-band spectrum light sources as illumination. Common narrow-band imaging modes include NBI (Narrow Band Imaging), BLI (Blue Laser Imaging), LCI (Linked Color Imaging) and the like. Narrow-band imaging will cause a significant difference in color compared with white light imaging because it only covers part of the visible spectrum band or changes the proportion of white light spectrum.

[0062] Point spread function: Point spread function (PSF) describes the response of an imaging system to a point source (object). The general term of PSF is system response, and PSF is the impulse response of a focusing optical system. In most cases, PSF can be considered as an extended block in the image that can represent an unresolved object. In terms of function, PSF is the spatial domain representation of the transfer function of the imaging system. PSF is an important concept, and Fourier optics, astronomical imaging, medical imaging, electron microscopy and other imaging technologies such as three-dimensional microscopic imaging and fluorescence microscopic imaging all have it. The degree of diffusion (blurring) of a point object is a measure of the quality of an imaging system.

[0063] The present application provides an endoscope color crosstalk suppression method, which is applied to an endoscope, the endoscope comprising a CMOS image sensor for imaging; an illumination light source, which can be composed of 5 LEDs of blue-violet light source, green light source, amber light source, red light source and the like, and the 5 LEDs can synthesize white light or other arbitrary combination of multi-spectrum mixed light. The light source illumination combination adopted in the present application is narrow-band blue-violet light source, narrow-band blue light source, narrow-band green light source, narrow-band amber light source and narrow-band red light source. Specifically, the central wavelength of the narrow-band blue-violet light source can be 415±10nm, the central wavelength of the narrow-band blue light source can be 460±10nm, the central wavelength of the narrow-band green light source can be 540±15nm, the central wavelength of the narrow-band amber light source can be 600±15nm, and the central wavelength of the narrow-band red light source can be 630±10nm. A controller is used to execute the endoscope color crosstalk suppression method. Specifically, see Figure 1 The method comprises the following steps:

[0064] S10. Obtain the original endoscope image collected by the image sensor. Under the irradiation of the illumination light source, the image sensor of the endoscope collects the image of the human tissue containing the target object as the original endoscope image. The target object can be a blood vessel, a mucosa, a tumor or the like. In the following embodiments, the blood vessel is taken as an example for illustration. The image sensor adopts a CMOS image sensor. The sensory unit in the CMOS image sensor can accept a relatively wide range of light spectrum, so that a single wavelength light signal can be detected in multiple color channels at the same time. Therefore, when a CMOS image sensor camera is used for multi-spectral narrowband light imaging, multiple narrowband lights are irradiated at the same time, and the reflected light causes unexpected interference between the sensor pixels, resulting in unexpected signal overlap in the images of the channels. There is image sensor crosstalk in the original endoscope image. At the same time, there is also optical crosstalk caused by optical diffusion in the original endoscope image.

[0065] S20. Identify the target object parameters in the original endoscope image, including the target color channel corresponding to the target object and the target size parameter. In this embodiment, after the original endoscope image is collected, the target object in the original endoscope image is identified first. Exemplarily, the target object can be identified in the original endoscope image by the spectral absorption difference between the target biological tissue corresponding to the target object and the surrounding tissue.

[0066] Taking the blood vessel as the target object, the blood vessel can be identified based on the hemoglobin content. Exemplarily, hemoglobin is an important protein in human red blood cells, and the blood vessel region is rich in hemoglobin, so the hemoglobin can be used as a reference factor for identifying the blood vessel.

[0067] The hemoglobin has a strong absorption peak near 415 nm and 540 nm, and the absorption difference is greater than that in the mucosa tissue, i.e. Therefore, the above information can be used to mark the position of the blood vessel in the image. When the illumination light source has the above two light sources, the above two wavelength main channel images can be used for processing, respectively denoted as 、 The preset blood vessel marking parameter is:

[0068] ;

[0069] wherein, and are adjustment parameters. By adjusting the values of and , when , the pixel value in the preset blood vessel marking parameter is greater than the set threshold , it is considered that the region is a blood vessel part.

[0070] In an embodiment, the target object in the original endoscope image can also be identified by morphology, for example, edge detection, region growing, thresholding, fuzzy clustering, mathematical morphology-based methods, etc.

[0071] In an embodiment, the target object in the original endoscope image can also be identified based on a deep learning model. In this embodiment, taking blood vessels as an example, first, define the image content attributes of the target endoscope application scenario, such as bronchial mucosa, mucus, superficial blood vessels, deep blood vessels, bleeding points, inflammation, etc. Then, perform region segmentation on the image, and each region represents similar image attributes.

[0072] The image region segmentation can use traditional image processing methods or deep learning methods. The traditional image processing methods include but are not limited to threshold-based segmentation, edge detection-based segmentation, genetic algorithm-based segmentation, etc. The deep learning processing methods include but are not limited to feature encoding-based segmentation methods (such as VGGNet, ResNet, etc.), region selection-based methods (such as Fast RCNN), U-Net or DeepLab series network-based image segmentation methods, etc.

[0073] S30. Determine the target blur convolution kernel corresponding to the target object parameter according to the mapping relationship between the calibrated target object parameter and the calibrated blur convolution kernel, wherein the target blur convolution kernel represents the combination of the optical diffusion parameter corresponding to the target size parameter of the target object in the target color channel and the sensor crosstalk parameter in the target channel. In an embodiment, the mapping relationship is constructed based on a first mapping relationship between the optical diffusion coefficient of the calibration target object with different size parameters in the calibration color channel and a second mapping relationship between the different calibration color channels of the calibration target object and the sensor crosstalk coefficient. For example, the first mapping relationship and the second mapping relationship can be a first mapping table of the optical diffusion parameter related to the blood vessel diameter of the calibration color channel of the calibration target object and a second mapping table of the sensor crosstalk parameter related to the calibration color channel of the calibration target object.

[0074] The first mapping table can include a red channel and a blood vessel diameter related optical diffusion parameter mapping table, a green channel and a blood vessel diameter related optical diffusion parameter mapping table, and a blue channel and a blood vessel diameter related optical diffusion parameter mapping table. The second mapping table can include a red channel related sensor crosstalk parameter mapping table, a green channel and a blood vessel diameter related sensor crosstalk parameter mapping table, and a blue channel related sensor crosstalk parameter mapping table.

[0075] The color diffusion at high-contrast edges and the blurring of tissue boundaries in the original endoscope image are caused by optical diffusion and sensor crosstalk. The total crosstalk is a combination of optical diffusion and sensor crosstalk. Therefore, in the embodiment, the first mapping table and the second mapping table are combined into an overall blur kernel. In the embodiment, the optical diffusion is related to the color channel and the vessel diameter, while the sensor crosstalk is only related to the color channel. Therefore, a two-dimensional look-up table of the blur kernel can be established based on the first mapping table and the second mapping table, that is, the total crosstalk of the current target object in a certain color channel can be obtained by the color channel and the size parameter of the target object.

[0076] In another embodiment, the mapping relationship can also be a mapping function obtained by fitting, for example, a univariate polynomial can be used to fit the mapping relationship.

[0077] S40. The original endoscope image is deconvolved based on the target blur kernel to obtain an endoscope imaging image. The target blur kernel corresponding to the color channel and the size parameter of the target object in the original endoscope image is determined by the mapping relationship obtained by pre-calibration, that is, the total crosstalk of the target object in the current endoscope original endoscope image. After obtaining the target blur kernel, the corresponding deconvolution operation is performed on the image in the blood vessel region, so as to restore the image with clear blood vessel boundaries.

[0078] In the embodiment, the deconvolution method includes but is not limited to direct inverse filtering, Wiener filtering, iterative optimization, regularization, blind deconvolution, etc.

[0079] For example, the original endoscope image is divided into channels and Fourier transform is performed to obtain the frequency spectrum of each color channel. The frequency spectrum is divided by the frequency spectrum of the point spread function corresponding to the vessel diameter, that is, the frequency spectrum obtained by Fourier transform of the convolution kernel of each color channel, and then inverse Fourier transform is performed to obtain the restored image.

[0080] ;

[0081] Wherein, F is Fourier transform, F -1 is inverse Fourier transform, is a red channel image of the original endoscope image, is a green channel image of the original endoscope image, is a blue channel image of the original endoscope image, delta (R, d) is the convolution kernel of the red channel corresponding to the vessel diameter d, delta (G, d) is the convolution kernel of the green channel corresponding to the vessel diameter d, delta (B, d) is the convolution kernel of the blue channel corresponding to the vessel diameter d, is a red channel recovered image, is a green channel recovered image, is a blue channel recovered image.

[0082] Finally, in order to avoid boundary mutation, the processed image is fused with the original endoscope image, and the fused image is:

[0083] I final (x,y) = ωI recovered (x,y) + (1-ω)I blurred (x,y);

[0084] wherein, I final (x,y) is a fused image, I recovered (x,y) is a recovered image, I blurred (x,y) is an original endoscope image, and ω is a fusion coefficient.

[0085] In the present application, after obtaining an original endoscope image, a target object and target object parameters are identified in the original endoscope image, a mapping relationship between the target object parameters and a calibration blur kernel is determined by pre-calibration, the target object is used to determine a target blur kernel corresponding to a combination of an optical diffusion parameter of the target size parameter of the target object in the target color channel and a sensor crosstalk parameter in the target channel, and the target blur kernel is used for deconvolution operation on the original endoscope image to perform crosstalk elimination on the image in an organization and region. The problems of tissue boundary blur, tissue color distortion and the like caused by optical diffusion and image sensor crosstalk in narrowband imaging are solved, so that the color change of the tissue is more accurately displayed.

[0086] In an embodiment, for the calibration of the mapping relationship, after the crosstalk calibration image is obtained by using the image sensor to collect the bionic target, the optical diffusion parameters corresponding to different calibration color channels and different size parameters in the calibration crosstalk image and the sensor crosstalk parameters corresponding to different calibration color channels are determined respectively, and the mapping relationship is obtained.

[0087] Specifically, as Figure 2 shown, the calibration method of the mapping relationship includes the following steps:

[0088] S100. Obtain a crosstalk calibration image, and the crosstalk calibration image has the calibration target object. In the present embodiment, the crosstalk calibration image can include an optical crosstalk calibration image in a single calibration color channel and sensor crosstalk calibration images in multiple calibration color channels.

[0089] The optical crosstalk image refers to multiple biomimetic target images of calibrated color channels acquired by a calibration image sensor under illumination by a single narrowband light source corresponding to each calibrated color channel. For example, the single narrowband light source sequentially illuminates the biomimetic target individually, and images are acquired separately using a calibration camera, resulting in multiple narrowband images. These are denoted as follows: , , , , The biomimetic target includes calibration targets with various size parameters, such as vascular structures ranging from 50 to 500 μm.

[0090] The main spectral response color channel image of each narrowband image is extracted as the optical crosstalk calibration image. For example, when only a narrowband light source with a main spectrum of 415 nm is used as the illumination source, the main response is on the B channel; therefore, the main response image of the 415 nm wavelength light in the B channel is extracted and denoted as... Similarly, the main response image of light with a wavelength of 460nm on channel B can be obtained. The main response image of light with a wavelength of 540nm on the G channel. Master response images at wavelengths of 600nm and 620nm on the R channel. and .

[0091] The main response images of 415nm and 460nm light wavelengths are combined into a blue channel optical crosstalk calibration image using the following formula. The master response image of 540nm wavelength light was used as the green channel optical crosstalk calibration image. The master response images at wavelengths of 600 nm and 620 nm are combined to form the red channel optical crosstalk calibration image. This will give you the optical crosstalk calibration image for each calibrated color channel:

[0092]

[0093] The acquisition of sensor crosstalk calibration images under multiple calibration color channels can be determined by the optical crosstalk calibration image of a single calibration color channel in the above embodiments.

[0094] The optical crosstalk calibration images of each individual calibrated color channel are merged into an RGB image to obtain the composite image. . It is an image containing only optical diffusion. Acquire a hybrid calibration color channel biomimetic target image of the calibration image sensor set under simultaneous illumination by all narrowband light sources corresponding to all calibration color channels. ; Hybrid calibration color channel biomimetic target image An image containing both optical diffusion crosstalk and sensor crosstalk.

[0095] The mixed calibration color channel biomimetic target image is constructed with the synthetic image The difference is obtained to obtain a mixed calibration color channel sensor crosstalk image I cmos , see formula: .

[0096] The mixed calibration color channel sensor crosstalk image is channel separated to obtain the sensor crosstalk calibration image under multiple calibration color channels. The sensor crosstalk calibration images are respectively denoted as a red channel sensor crosstalk calibration image , a green channel sensor crosstalk calibration image , and a blue channel sensor crosstalk calibration image . Sensor crosstalk is independent of blood vessel size and only related to the characteristics of the CMOS image sensor itself, so when calibrating the CMOS image sensor crosstalk, it is not necessary to distinguish the blood vessel diameter.

[0097] Since the optical diffusion crosstalk is related to the color channel and the size parameter of the target, and the sensor crosstalk is only related to the color channel, in this embodiment, two different width Gaussian kernels are used to quantize the optical diffusion parameter and the sensor crosstalk parameter respectively, and finally combined to complete the calibration of the total crosstalk characteristics. Specifically, see the description of steps S200 to S400.

[0098] S200. Construct the line spread function of the calibration target in each calibration color channel of the crosstalk calibration image. For the optical crosstalk calibration image, construct the line spread function of the optical crosstalk image under each calibration color channel. Specifically, sample the calibration target of different calibration size parameters in the optical crosstalk image of each calibration color channel to obtain the first sampling point of the calibration target of different calibration size parameters under each calibration color channel, wherein the number of the first sampling point is inversely related to the value of the calibration size parameter. Extract the first point spread function of the calibration target corresponding to each calibration size parameter under each calibration color channel based on the first sampling point corresponding to each calibration size parameter; and construct a first line spread function based on the first point spread function.

[0099] For example, taking the red channel as an example, see Figure 3 , for the red channel optical crosstalk calibration image , for example, in Different diameter blood vessels are marked, and normals are generated perpendicular to the vessel centerline, extending beyond the diffusion boundary and covering the entire diffusion transition region (e.g., the diffusion boundary is ±3 times the vessel diameter). Because small blood vessels are close to the optical diffraction limit, dense sampling is required to capture subtle optical diffusion. Therefore, for small blood vessels (50-100 μm), the normal spacing is... The number of sampling points (number of normals) is Medium-sized blood vessels (100-300 μm), with normal septa of... The number of sampling points is Large blood vessels (300-500μm), with normal septa of... The number of sampling points is The following relationship must be satisfied: < < , > > .

[0100] The first point spread function (PSF) is performed using the first sampling point. d (x) is fitted as follows:

[0101]

[0102] Where d is the diameter of the blood vessel, and N is the number of normals at that diameter (x, y) i () represents the coordinates of the first sampling point.

[0103] The first-point diffusion function is fitted to the first-line diffusion function LSF(x) according to the following formula:

[0104] .

[0105] For the sensor crosstalk parameters, the calibration target in the sensor crosstalk calibration image under each calibration color channel is sampled to obtain the second sampling point of the calibration target under each calibration color channel; the second point diffusion function of the calibration target under each calibration color channel is extracted based on the second sampling point; and the second line diffusion function is constructed based on the second point diffusion function.

[0106] Taking the red channel as an example, this paper addresses the crosstalk calibration image of the red channel sensor. ,right The process involves marking M large, medium, and small blood vessels, with N second sampling points for each, and a normal interval of [missing information]. The second point spread function PSF(x) corresponding to the red channel sensor crosstalk calibration image is obtained by performing superposition averaging:

[0107] .

[0108] The second point spread function is fitted into a second line spread function LSF(x) according to the following formula:

[0109] .

[0110] S300. The calibration target in each calibration color channel of the crosstalk calibration image is fitted using a Gaussian distribution model to obtain the optical diffusion parameters corresponding to different calibration size parameters of the calibration target in different calibration color channels and the sensor crosstalk parameters in different calibration color channels.

[0111] Specifically, the first line spread function is fitted using a first Gaussian distribution model to obtain the optical diffusion parameters corresponding to different size parameters in the calibration color channel. A first mapping table of the optical diffusion parameters related to the calibration size parameters in the calibration color channel is established. The first mapping table includes a red channel optical diffusion parameter mapping table related to the calibration size parameters , a green channel optical diffusion parameter mapping table related to the calibration size parameters , and a blue channel optical diffusion parameter mapping table related to the calibration size parameters .

[0112] The second line spread function is fitted using a second Gaussian distribution model to obtain the sensor crosstalk parameters corresponding to the calibration color channel. A second mapping table of the sensor crosstalk parameters related to the calibration color channel is established, and the second mapping table includes a red channel sensor crosstalk parameter mapping table related to the calibration color channel , a green channel sensor crosstalk parameter mapping table related to the calibration color channel , and a blue channel sensor crosstalk parameter mapping table related to the calibration color channel . Wherein the Gaussian kernel width of the first Gaussian distribution model and the second Gaussian distribution model is different.

[0113] S400. The optical diffusion parameters and the sensor crosstalk parameters are combined to construct the mapping relationship between the calibration target parameters and the blur convolution kernel. Since the optical crosstalk is related to the color channel and the blood vessel diameter, and the sensor crosstalk is only related to the color channel, a two-dimensional lookup table of the blur convolution kernel is established , for the color channel, and the blood vessel diameter. Wherein the two-dimensional lookup table of the blur convolution kernel δ(channal,d)=δ channal *δ d , δ channal is the combination of the optical diffusion parameters related to the color channel in the first mapping table and the sensor crosstalk parameters related to the color channel in the second mapping table, The size parameter related optical diffusion parameter in the first mapping table. Using two different width Gaussian kernels, respectively quantifying the edge color diffusion caused by lens blur and sensor crosstalk, completing multi-dimensional color diffusion calibration, and then performing sub-organization and sub-regional color diffusion elimination on the image. Solve the problems of tissue boundary blur, tissue color distortion and other problems caused by color crosstalk when narrowband imaging, so as to more accurately display the color change of the tissue.

[0114] As Figure 4 shown, the embodiment of the present application also provides an endoscope color crosstalk suppression device, comprising:

[0115] The acquisition module 401 is configured to acquire an original endoscope image collected by an image sensor.

[0116] The identification module 402 is configured to identify target object parameters in the original endoscope image, wherein the target object parameters include a target color channel corresponding to a target object and a target size parameter.

[0117] The determination module 403 is configured to determine a target blur convolution kernel corresponding to the target object parameters according to a mapping relationship between a calibration target object parameter and a calibration blur convolution kernel, wherein the target blur convolution kernel represents a combination of an optical diffusion parameter corresponding to the target size parameter of the target object in the target color channel and a sensor crosstalk parameter in the target channel.

[0118] The crosstalk suppression module 404 is configured to perform deconvolution processing on the original endoscope image based on the target blur convolution kernel to obtain an endoscope imaging image.

[0119] The embodiment of the present application also provides an endoscope, comprising an image sensor, an illumination light source and a controller, wherein, as Figure 5 shown in the structure block diagram of the controller, the controller comprises a processor 501, a communication interface 502, a memory 503 and a communication bus 504, wherein the processor 501, the communication interface 502 and the memory 503 complete mutual communication through the communication bus 504, wherein,

[0120] The memory 503 is configured to store a computer program.

[0121] The processor 501 is configured to execute the computer program stored in the memory 503 to realize the following steps:

[0122] Acquire an original endoscope image collected by an image sensor.

[0123] Identify target object parameters in the original endoscope image, wherein the target object parameters include a target color channel corresponding to a target object and a target size parameter.

[0124] According to a mapping relationship between a pre-calibrated target object parameter and a calibration blur kernel, a target blur kernel corresponding to the target object parameter is determined, wherein the target blur kernel represents a combination of an optical diffusion parameter corresponding to the target size parameter of the target object in the target color channel and a sensor cross-talk parameter in the target channel.

[0125] Based on the target blur kernel, the original endoscope image is deconvoluted to obtain an endoscope imaging image.

[0126] Optionally, in the embodiment, the communication bus can be a PCI (Peripheral Component Interconnect) bus, an EISA (Extended Industry Standard Architecture) bus, or the like. The communication bus can be divided into an address bus, a data bus, a control bus, and the like. For ease of representation, Figure 5 In the figure, only one thick line is used to represent the bus, but it does not mean that there is only one bus or only one type of bus.

[0127] The communication interface is used for communication between the controller and other devices.

[0128] The memory can include a RAM and can also include a non-volatile memory, for example, at least one disk memory. Optionally, the memory can also be at least one storage device located away from the aforementioned processor.

[0129] The processor can be a general-purpose processor, which can include but is not limited to a CPU (Central Processing Unit), an NP (Network Processor), and the like; and can also be a DSP (Digital Signal Processing), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0130] Optionally, specific examples in the embodiment can refer to the examples described in the above-described embodiments, and the embodiment will not be described here again.

[0131] Those skilled in the art can understand that all or part of the steps in the above-mentioned embodiments can be completed by instructing the terminal device related hardware through programs, and the programs can be stored in a computer readable storage medium, which can include a flash disk, a ROM, a RAM, a magnetic disk or an optical disk, etc.

[0132] According to another aspect of the embodiments of the present application, a storage medium is further provided. Optionally, in the embodiments, the storage medium can be used to store program codes for executing the endoscope color crosstalk suppression method.

[0133] Optionally, in the embodiments, the storage medium can be located on at least one of the network devices in the network shown in the embodiments.

[0134] Optionally, in the embodiments, the storage medium is configured to store program codes for executing the following steps:

[0135] Obtaining an original endoscope image collected by an image sensor;

[0136] Identifying a target object parameter in the original endoscope image, the target object parameter including a target color channel corresponding to a target object and a target size parameter;

[0137] Determining a target blur convolution kernel corresponding to the target object parameter according to a mapping relationship between a calibration target object parameter and a calibration blur convolution kernel, wherein the target blur convolution kernel represents a combination of an optical diffusion parameter corresponding to the target size parameter of the target object in the target color channel and a sensor crosstalk parameter in the target channel;

[0138] Performing deconvolution processing on the original endoscope image based on the target blur convolution kernel to obtain an endoscope imaging image.

[0139] Optionally, specific examples in the embodiments can refer to the examples described in the above-mentioned embodiments, and the embodiments will not be described here.

[0140] Optionally, in the embodiments, the storage medium can include but is not limited to a U disk, a ROM, a RAM, a mobile hard disk, a magnetic disk or an optical disk, and various media that can store program codes.

[0141] The serial numbers of the embodiments of the present application are only for description, and do not represent the advantages or disadvantages of the embodiments.

[0142] The integrated units in the above embodiments, if implemented in the form of software function units and sold or used as independent products, can be stored in the above computer-readable storage medium. Based on such understanding, the technical solutions of the present application, essentially or in other words, the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes a number of instructions to make one or more computer devices (which can be personal computers, servers or network devices, etc.) execute all or part of the steps of the methods described in various embodiments of the present application.

[0143] In the above embodiments of the present application, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the relevant description of other embodiments.

[0144] In the several embodiments provided in the present application, it should be understood that the disclosed client can be implemented by other means. Among them, the above-mentioned device embodiments are only schematic, for example, the division of the units is only a logical function division, and in actual implementation, there can be another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the shown or discussed mutual ones can be indirect coupling or communication connection through some interfaces, units or modules, and can be electrical or other forms.

[0145] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, they can be located in one place or distributed on multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the scheme provided in the embodiments.

[0146] In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The above integrated unit can be realized in the form of hardware or in the form of software function unit.

[0147] The places not mentioned in the present application can be realized by using or referring to the existing technology.

[0148] Each embodiment in the specification is described in a progressive manner, and the same or similar parts between each embodiment can be referred to each other. Each embodiment focuses on the different parts from other embodiments.

[0149] The above merely provides an example of the present application, and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present application should be included in the scope of claims of the present application.

Claims

1. A method for suppressing color crosstalk in endoscopes, characterized in that, Acquire raw endoscopic images captured by the image sensor; Identify target parameters in the original endoscopic image, the target parameters including the target color channel and target size parameters corresponding to the target object; The target fuzzy convolution kernel corresponding to the target object parameters is determined according to the pre-calibrated mapping relationship between the calibration target object parameters and the calibration fuzzy convolution kernel. The target fuzzy convolution kernel represents the combination of the optical diffusion parameter corresponding to the target size parameter of the target object in the target color channel and the sensor crosstalk parameter in the target color channel. The original endoscope image is deconvolved based on the target blur convolution kernel to obtain an endoscope imaging image. The method for determining the mapping relationship includes: Acquire a crosstalk calibration image, wherein the crosstalk calibration image contains a calibration target object; Construct the line spread function of the calibrated target object for each calibrated color channel in the crosstalk calibration image; The calibration target object in each calibration color channel of the crosstalk calibration image is fitted using a Gaussian distribution model to obtain the optical diffusion parameters corresponding to different calibration size parameters of the calibration target object in different calibration color channels and the sensor crosstalk parameters in different calibration color channels. By combining the optical diffusion parameters and the sensor crosstalk parameters, a mapping relationship between the calibration target parameters and the fuzzy convolution kernel is constructed; The crosstalk calibration image includes an optical crosstalk calibration image, which is a biomimetic target image of multiple calibration color channels acquired by a calibration image sensor under illumination by a single narrowband light source corresponding to the calibration color channel. Constructing the line spread function of the calibrated target object in each calibrated color channel of the crosstalk calibration image includes: For each calibration color channel, the calibration target objects with different calibration size parameters in the optical crosstalk image are sampled to obtain the first sampling points of the calibration target objects with different calibration size parameters under each calibration color channel, wherein the number of the first sampling points is inversely correlated with the value of the calibration size parameter; Based on the first sampling point corresponding to each calibration size parameter, the first point diffusion function of the calibration target object corresponding to each calibration size parameter under each calibration color channel is extracted; Construct a first line diffusion function based on the first point diffusion function; The crosstalk calibration image includes sensor crosstalk calibration images under multiple calibration color channels; Constructing the line spread function of the calibrated target object in each calibrated color channel of the crosstalk calibration image includes: The calibration target object in the sensor crosstalk calibration image under each calibration color channel is sampled to obtain the second sampling point of the calibration target object under each calibration color channel; For the second sampling point, extract the second point diffusion function of the calibrated target object under each calibrated color channel; Construct a second line diffusion function based on the second point diffusion function; The step of fitting the calibration target object in each calibration color channel of the crosstalk calibration image using a Gaussian distribution model includes: The first line diffusion function is fitted using the first Gaussian distribution model to obtain the optical diffusion parameters corresponding to different size parameters under the calibrated color channel. Establish a first mapping table of optical diffusion parameters related to the calibration color channels and calibration size parameters; The second line diffusion function is fitted using the second Gaussian distribution model to obtain the sensor crosstalk parameters corresponding to the calibrated color channel; Establish a second mapping table for calibrating sensor crosstalk parameters related to color channels.

2. The endoscopic color crosstalk suppression method as described in claim 1, characterized in that, The step of performing deconvolution processing on the original endoscopic image based on the target blur convolution kernel to obtain the endoscopic imaging image includes: Determine the target region where the target object is located in the original endoscopic image; The target region is deconvolved by the target blur convolution kernel according to its color channels to obtain a crosstalk suppression image; The crosstalk-suppressed image is fused with the original endoscopic image to obtain the endoscopic imaging image.

3. The endoscopic color crosstalk suppression method as described in claim 1, characterized in that, The method for acquiring the sensor crosstalk calibration image includes: Acquire a mixed calibration color channel biomimetic target image captured by the calibration image sensor under simultaneous illumination by all narrowband light sources corresponding to all calibration color channels; All the optical crosstalk calibration images are merged into a composite image; The difference between the hybrid calibrated color channel bionic target image and the synthesized image is used to obtain the hybrid calibrated color channel sensor crosstalk image; Channel separation is performed on the mixed calibration color channel sensor crosstalk image to obtain the sensor crosstalk calibration image under multiple calibration color channels.

4. The endoscopic color crosstalk suppression method as described in claim 2, characterized in that, The deconvolution methods include, but are not limited to, one of the following: direct inverse filtering, Wiener filtering, iterative optimization, regularization, and blind deconvolution.

5. The endoscopic color crosstalk suppression method as described in claim 1, characterized in that, The parameters for identifying the target object in the original endoscopic image include: The target object is identified in the original endoscopic image by the difference in spectral absorption between the target biological tissue and the surrounding tissue.

6. An endoscope, characterized in that, The device includes an image sensor, an illumination source, and a controller, wherein the controller includes one or more processors, one or more memories, and one or more computer program instructions, which, when executed by the processor, implement the endoscopic color crosstalk suppression method as described in any one of claims 1 to 5.

Images