Endoscopic Imaging Method, Device, Computer Equipment and Storage Medium

Abstract

The present invention relates to the field of medical endoscope imaging technology, and discloses an endoscope imaging method, device, computer device, and storage medium; the endoscope imaging method includes: acquiring a first reference image and a second reference image collected by an endoscope; obtaining a differential pixel image of the first reference image relative to the second reference image based on the pixel attribute differences of each pixel in the first reference image and the second reference image; determining a gain correction coefficient of the differential pixel image based on the first reference image and the second reference image; performing gain correction on the differential pixel image based on the gain correction coefficient to obtain an estimated image corresponding to a third color channel in an RGB image; fusing the estimated image, the first reference image, and the second reference image to obtain an endoscope imaging image; the finally fused endoscope imaging image can present a color expression closer to white light on the whole while retaining the prominent detail information.

Description

Technical Field

[0001] The present invention relates to the technical field of medical endoscope imaging, and particularly relates to an endoscope imaging method, device, computer device and storage medium. Background Art

[0002] With the booming development of endoscope applications, doctors' demand for high-quality endoscope imaging has also been continuously increasing. In the endoscope application scenarios, especially during the examination, diagnosis and treatment of digestive endoscopes (gastrointestinal endoscopes), doctors usually use the white light imaging mode that conforms to the human eye vision for routine operations. When further exploring lesions is needed, the narrow-band light imaging mode will be switched to.

[0003] Due to the strong absorption and reflection characteristics of the narrow-band wavelength band for specific biological tissues, narrow-band light images can effectively highlight detailed lesions. However, due to the incomplete light wavelength band, there are obvious differences in tone between narrow-band light images and white light. Taking the commonly used Olympus NBI mode as an example, the central wavelength bands of its illumination light source are composed of 415nm and 540nm. In order to highlight blood vessels and lesions that are not available in white light imaging, the red channel is blocked, making the overall image tone present cyan and sepia, with a huge difference from the white light tone.

[0004] Therefore, how to obtain a color expression similar to white light imaging while retaining the detail enhancement brought by changing the proportion of the narrow-band light source has become an urgent problem to be solved. Summary of the Invention

[0005] In view of this, the present invention provides an endoscope imaging method, device, computer device and storage medium to solve the problem that the narrow-band light image has obvious differences in tone from white light due to the incomplete light wavelength band.

[0006] In a first aspect, the present invention provides an endoscope imaging method. The endoscope includes an illumination unit, an imaging unit, and an image processing unit. The illumination unit includes at least one first light source in at least one first band corresponding to a first color channel in the RGB image collected by the imaging unit, and at least one second light source in at least one second band corresponding to a second color channel. The imaging unit is capable of collecting an image corresponding to the first color channel as a first reference image and an image corresponding to the second color channel as a second reference image. The endoscope imaging method includes: obtaining the first reference image and the second reference image collected by the endoscope; obtaining a difference pixel image of the first reference image relative to the second reference image based on the pixel attribute differences of the pixels in the first reference image and the second reference image; wherein the difference pixel image is used to represent an image of the difference set information composed of reference pixels in the first reference image with different pixel attributes from the pixels in the second reference image corresponding to the same position, as presented in the first color channel; determining a gain correction coefficient of the difference pixel image based on the first reference image and the second reference image; performing gain correction on the difference pixel image based on the gain correction coefficient to obtain an estimated image corresponding to a third color channel in the RGB image; and fusing the estimated image, the first reference image, and the second reference image to obtain an endoscope imaging image.

[0007] As an exemplary embodiment, the obtaining a difference pixel image of the first reference image relative to the second reference image based on the pixel attribute differences of the pixels in the first reference image and the second reference image includes: performing pixel attribute marking on the pixels in the first reference image and the second reference image to obtain pixel attribute marking results of each of the reference images; extracting difference set information from the first reference image based on the pixel attribute marking results; wherein the difference set information includes a plurality of reference pixels in the first reference image, and the pixel attribute marking results of each of the reference pixels are different from the pixel attribute marking results of the pixels in the second reference image corresponding to the same position; determining an illuminance estimation map of the difference set information based on the first reference image; and fusing the illuminance estimation map and the difference set information to obtain the difference pixel image.

[0008] As an exemplary embodiment, the pixel attribute marking of each pixel in the first reference image and the second reference image to obtain the pixel attribute marking results of the respective reference images includes: fusing the first reference image and the second reference image to obtain a fused image; classifying each pixel in the fused image to obtain a pixel classification result; and respectively marking the pixels at the corresponding positions in the first reference image and the second reference image based on the pixel classification result and a preset attribute relationship to obtain a first pixel attribute corresponding to the first reference image and a second pixel attribute corresponding to the second reference image.

[0009] As an exemplary embodiment, the determining of the gain correction coefficient of the differential pixel image based on the first reference image and the second reference image includes: determining union information based on the pixels of the first reference image and the second reference image; wherein the union information is obtained by weighted fusion of the first reference image and the second reference image; determining the gain correction coefficient of the differential pixel image based on the union information and the difference set information; wherein the gain correction coefficient is positively correlated with the union information and negatively correlated with the difference set information.

[0010] As an exemplary embodiment, the determining of the union information based on the pixels of the first reference image and the second reference image includes: determining the average brightness of the differential pixel image based on the overall average brightness of the first reference image and the second reference image; fusing, based on a preset fusion rule, the average brightness as a first fusion parameter of a third color channel, the pixels in the first reference image as a second fusion parameter of a first color channel, and the pixels in the second reference image as a third fusion parameter of a second color channel to obtain the union information; wherein the preset fusion rule is the conversion rule of the Y channel when converting from the RGB domain to the YUV domain.

[0011] As an exemplary embodiment, the determining of the gain correction coefficient of the differential pixel image based on the union information and the difference set information includes:

[0012]

[0013] wherein, represents the gain correction coefficient, represents the union information, represents the difference set information, represents an adjustable parameter.

[0014] As an exemplary embodiment, the endoscopic imaging method further includes: after normalizing the gain correction coefficient, performing neighborhood smoothing and neighborhood low-pass filtering to obtain a modulated gain correction coefficient.

[0015] In a second aspect, the present invention provides an endoscopic imaging device, which includes: an acquisition module for acquiring a first reference image and a second reference image collected by the endoscope; a differential pixel image determination module for obtaining a differential pixel image of the first reference image relative to the second reference image based on the pixel attribute differences of the pixels in the first reference image and the second reference image; wherein the differential pixel image is used to represent an image presented under a first color channel of difference set information composed of reference pixels in the first reference image with different pixel attributes from those of the second reference image; a gain correction coefficient determination module for determining a gain correction coefficient of the differential pixel image based on the first reference image and the second reference image; a gain correction module for performing gain correction on the differential pixel image based on the gain correction coefficient to obtain an estimated image corresponding to a third color channel in an RGB image; and a fusion module for fusing the estimated image, the first reference image, and the second reference image to obtain an endoscopic imaging image.

[0016] In a third aspect, the present invention provides a computer device, including: a memory and a processor, which are communicatively connected to each other. The memory stores computer instructions, and the processor executes the computer instructions to execute the method according to the first aspect or any corresponding embodiment thereof.

[0017] In a fourth aspect, the present invention provides a computer-readable storage medium, on which computer instructions are stored. The computer instructions are used to cause a computer to execute the method according to the first aspect or any corresponding embodiment thereof.

[0018] The present invention provides an endoscope imaging method, and the endoscope imaging method includes: acquiring the first reference image and the second reference image collected by the endoscope; obtaining a difference pixel image of the first reference image relative to the second reference image based on the pixel attribute differences of the pixels in the first reference image and the second reference image; wherein, the difference pixel image is used to represent an image presented under the first color channel of the difference set information composed of reference pixels with different pixel attributes from those of the second reference image in the first reference image; determining a gain correction coefficient of the difference pixel image based on the first reference image and the second reference image; performing gain correction on the difference pixel image based on the gain correction coefficient to obtain an estimated image corresponding to the third color channel in the RGB image; fusing the estimated image, the first reference image, and the second reference image to obtain an endoscope imaging image; in the above endoscope imaging method of the present invention, on the one hand, the estimated image is determined by performing gain correction on the difference pixel image. The difference pixel image is obtained from the difference set information composed of reference pixels with different pixel attributes between the first reference image and the second reference image. The gain correction coefficient during gain correction is determined by the first reference image and the second reference image. Therefore, the estimated image achieves an expression close to the red channel of the endoscope under white light imaging by considering the relationship between the pixels in the first reference image and the second reference image; on the other hand, the first reference image and the second reference image contain detailed information such as prominent detailed lesions; therefore, the finally fused endoscope imaging image can present a color expression closer to white light on the basis of retaining the prominent detailed information. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the drawings required for use in the description of the specific embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0020] Figure 1 is a schematic flowchart of an endoscope imaging method according to an embodiment of the present invention;

[0021] Figure 2 is a structural block diagram of an endoscope imaging device according to an embodiment of the present invention;

[0022] Figure 3 is a schematic hardware structure diagram of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0023] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Apparently, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0024] According to an embodiment of the present invention, an embodiment of an endoscope imaging method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. And although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than here.

[0025] In one embodiment, the endoscope imaging method is applied to an endoscope.

[0026] In one embodiment, the endoscope includes an illumination unit, an imaging unit, and an image processing unit. Among them, the illumination unit includes at least one first light source of at least one first band corresponding to the first color channel in the RGB image collected by the imaging unit, and at least one second light source of at least one second band corresponding to the second color channel; the imaging unit can collect the image corresponding to the first color channel as a first reference image and the image corresponding to the second color channel as a second reference image.

[0027] In one embodiment, the first color channel is the green channel, the first light source includes at least a narrow-band green light source, the second color channel is the blue channel, and the second light source includes at least a narrow-band blue-violet light source and a narrow-band blue light source.

[0028] In another embodiment, the first color channel is the blue channel, the first light source includes at least a narrow-band blue-violet light source and a narrow-band blue light source; the second color channel is the green channel, and the second light source includes at least a narrow-band green light source.

[0029] As a possible implementation, the central wavelength corresponding to the narrow-band blue-violet light source can be 415±10nm, the central wavelength corresponding to the narrow-band blue light source can be 457±10nm, and the central wavelength corresponding to the narrow-band green light source can be 540±15nm.

[0030] Exemplarily, taking the first color channel as the green channel, the first light source including at least a narrow-band green light source, the second color channel as the blue channel, and the second light source including at least a narrow-band blue-violet light source and a narrow-band blue light source as an example, the technical solution of the present application will be described.

[0031] In this embodiment, an endoscopic color tone correction method is provided, which can be used for the above-mentioned endoscope or image processing unit. Figure 1 It is a flowchart of the endoscopic color tone correction method according to an embodiment of the present invention. As Figure 1 shown, the process includes the following steps:

[0032] Step S101, obtain the first reference image and the second reference image collected by the endoscope.

[0033] The reflected light after the emitted light of the illumination unit irradiates the object to be photographed passes through the imaging unit and the image processing unit to form an original raw domain image, where the blue channel is denoted as B, the green channel is denoted as G, and the red channel is denoted as R.

[0034] Among them, the green channel has a strong correlation with the narrow-band green light source, and the image corresponding to the green channel G is used as the first reference image I1. The blue channel B has a strong correlation with the narrow-band blue-violet light source and the narrow-band blue light source, and the image corresponding to the blue channel B is used as the second reference image I2.

[0035] Step S102, obtain a difference pixel image of the first reference image relative to the second reference image based on the pixel attribute differences of each pixel in the first reference image and the second reference image; wherein, the difference pixel image is used to represent an image of the difference set information formed by reference pixels with different pixel attributes from the second reference image in the first reference image in the first color channel.

[0036] Due to the strong absorption and reflection characteristics of the narrow-band light image for specific biological tissues in the narrow-band wavelength band, it can effectively highlight the detailed lesions. However, due to the incomplete light wavelength band, its color tone is significantly different from that of white light; in this embodiment, the estimated image of the red channel when the narrow-band light image is synthesized into a white light image is characterized by the difference pixel image of the first reference image relative to the second reference image; among them, the first reference image and the second reference image are used to highlight specific tissues different from white light imaging. Therefore, the difference pixels with different pixel attributes in the first reference image and the second reference image can characterize the pixels that the red channel corresponding to the first reference image should represent when imaging with white light.

[0037] In one embodiment, the difference pixel image is obtained by fusing a reflection map composed of reference pixels and an illuminance estimation map obtained by estimating the illuminance of the difference pixel image.

[0038] In one embodiment, since the reference pixels are pixels in the first reference image, the illuminance estimation map can be obtained by performing low-pass filtering on the first reference image at at least three different scales.

[0039] Exemplarily, the illuminance estimation map is obtained by performing low-pass filtering on the first reference image using Equation (1):

[0040] (1)

[0041] In Equation (1), denotes the illuminance estimation map, and I gua1 , I gua2 , I gua3 denote the low-pass filtered images obtained by performing low-pass filtering on the first reference image at different scales.

[0042] Step S103: Determine the gain correction coefficient of the differential pixel image based on the first reference image and the second reference image.

[0043] In step S102, the differential pixel image is used to represent the image of the difference set information formed by the reference pixels in the first reference image with different pixel attributes from those in the second reference image under the first color channel. However, the brightness and contrast of the reference pixels under the first color channel are different from the brightness and contrast characterized in the image when presented in the red channel of the corresponding white light imaging. Therefore, it is necessary to further perform gain correction on the differential pixel image so that it can match the brightness and contrast characterized in the red channel.

[0044] Based on this, in this embodiment, after obtaining the differential pixel image, the gain correction coefficient of the differential pixel image is determined based on the first reference image and the second reference image.

[0045] In one embodiment, the gain correction coefficient is determined based on the correlation between the pixels with the same and different pixel attributes in the first reference image and the second reference image.

[0046] Exemplarily, the gain correction coefficient can be determined based on the union information characterizing the correlation between the same and different pixel attributes of the pixels in the first reference image and the second reference image and the difference set information characterizing the correlation information between the different attributes of the pixels; wherein, the gain correction coefficient is positively correlated with the union information and negatively correlated with the difference set information.

[0047] Exemplarily, the union information can be obtained by fusing the first reference image and the second reference image.

[0048] As a possible implementation, the union information is obtained by fusing the corresponding brightness and pixels of the first reference image and the second reference image according to the conversion rule of the Y channel when converting the RGB domain to the YUV domain.

[0049] Furthermore, after obtaining the union information and the difference set information, calculate the correction gain coefficient according to the union information and the difference set information.

[0050] Among them, the calibration gain coefficient can be calculated by a preset formula such as formula (2):

[0051] (2)

[0052] In formula (2), represents the gain calibration coefficient, represents the union information, represents the difference set information, represents the adjustable parameter.

[0053] Step S104 performs gain calibration on the difference pixel image based on the gain calibration coefficient to obtain an estimated image corresponding to the third color channel in the RGB image.

[0054] In this embodiment, the difference pixel image can be gain-calibrated by using the gain calibration coefficient as in formula (3):

[0055] (3)

[0056] In formula (3), represents the pixel value of the pixel point (i, j) in the estimated image, represents the adjustable parameter, represents the pixel value of the pixel point (i, j) in the difference pixel image, represents the gain calibration coefficient.

[0057] By performing gain calibration on the difference pixel image, the brightness and contrast characterized by the reference pixel under the red channel of white light imaging can be restored according to the first reference image and the second reference image, and a single-channel image corresponding to the red channel when approximating white light imaging can be obtained.

[0058] Step S105 fuses the estimated image, the first reference image, and the second reference image to obtain an endoscopic imaging image.

[0059] Exemplarily, after obtaining the estimated image approximately representing the single-channel image corresponding to the red channel of white light imaging, the estimated image corresponding to the red channel of the RGB image, the first reference image corresponding to the green channel, and the second reference image corresponding to the blue channel are fused to obtain an endoscopic imaging image.

[0060] In the above fusion method, on the one hand, the estimated image is determined by performing gain correction on the differential pixel image, and the differential pixel image is obtained from the difference set information composed of reference pixels with different pixel attributes between the first reference image and the second reference image. The gain correction coefficient during gain correction is determined by the first reference image and the second reference image. Therefore, the estimated image achieves an expression close to the red channel of the endoscope under white light imaging by considering the relationship between the pixels in the first reference image and the second reference image. On the other hand, the first reference image and the second reference image contain detailed information such as prominent detailed lesions. Therefore, the finally fused endoscope imaging image can present a color expression closer to white light on the whole while retaining the prominent detailed information.

[0061] In the above endoscope imaging method of this embodiment, the endoscope includes an illumination unit, an imaging unit, and an image processing unit. The illumination unit includes at least one first light source in at least one first band corresponding to the first color channel and at least one second light source in at least one second band corresponding to the second color channel in the RGB image collected by the imaging unit. The imaging unit can collect the image corresponding to the first color channel as the first reference image and the image corresponding to the second color channel as the second reference image. The endoscope imaging method includes: obtaining the first reference image and the second reference image collected by the endoscope; obtaining the differential pixel image of the first reference image relative to the second reference image based on the pixel attribute differences of the pixels in the first reference image and the second reference image; wherein, the differential pixel image is used to represent the image shown by the difference set information composed of reference pixels with different pixel attributes between the first reference image and the second reference image in the first color channel; determining the gain correction coefficient of the differential pixel image based on the first reference image and the second reference image; performing gain correction on the differential pixel image based on the gain correction coefficient to obtain an estimated image corresponding to the third color channel in the RGB image; fusing the estimated image, the first reference image, and the second reference image to obtain an endoscope imaging image. In the above endoscope imaging method of the present invention, on the one hand, the estimated image is determined by performing gain correction on the differential pixel image, and the differential pixel image is obtained from the difference set information composed of reference pixels with different pixel attributes between the first reference image and the second reference image. The gain correction coefficient during gain correction is determined by the first reference image and the second reference image. Therefore, the estimated image achieves an expression close to the red channel of the endoscope under white light imaging by considering the relationship between the pixels in the first reference image and the second reference image. On the other hand, the first reference image and the second reference image contain detailed information such as prominent detailed lesions. Therefore, the finally fused endoscope imaging image can present a color expression closer to white light on the whole while retaining the prominent detailed information.

[0062] As an exemplary embodiment, obtaining the difference pixel image of the first reference image relative to the second reference image based on the pixel attribute differences of each pixel in the first reference image and the second reference image includes: performing pixel attribute marking on each pixel in the first reference image and the second reference image to obtain the pixel attribute marking results of each reference image; extracting difference set information in the first reference image based on the pixel attribute marking results; wherein the difference set information includes a plurality of reference pixels in the first reference image, and the pixel attribute marking results of each reference pixel are different from the pixel attribute marking results of the pixels in the second reference image corresponding to the position; determining an illuminance estimation map of the difference set information based on the first reference image; and fusing the illuminance estimation map and the difference set information to obtain the difference pixel image.

[0063] In one embodiment, the pixel attribute marking results of each reference image can be obtained by performing pixel attribute marking on the first reference image and the second reference image respectively.

[0064] Exemplarily, the pixel attribute marking of the first reference image and the second reference image respectively can be implemented by methods including traditional image processing, machine learning, and deep learning; the specific methods include but are not limited to methods such as threshold segmentation, region growing, and edge detection based on traditional image processing, k-means clustering, SVM, etc. based on machine learning, and U-Net architecture, DeepLab model, etc. based on deep learning.

[0065] As a possible implementation manner, the first preset reference image corresponding to the first color channel and the second preset reference image corresponding to the second color channel can be collected in advance based on an endoscope, and pixel attribute marking is performed on each pixel of the first preset reference image and the second preset reference image in advance to obtain a first data set composed of the first preset reference image and its corresponding pixel attribute marking result and a second data set composed of the second preset reference image and its corresponding pixel attribute marking result respectively, and a pixel attribute marking model is trained based on the first data set and the second data set respectively. Further, the first reference image and the second reference image are respectively input into the pre-trained pixel attribute marking model to obtain the pixel attribute marking results of each reference image.

[0066] In one embodiment, a fused image can be obtained by fusing the first reference image and the second reference image, further pixel attribute marking is performed on the fused image, and then the pixel attribute marking results of the first reference image and the second reference image are obtained through the preset attribute relationship between the pixels in the fused image and the first reference image and the second reference image.

[0067] Exemplarily, after obtaining the pixel attribute marking results of each of the reference images, further extract difference set information from the first reference image based on the pixel attribute marking results; wherein, the difference set information includes a plurality of reference pixels in the first reference image, and the pixel attribute marking results of each of the reference pixels are different from the pixel attribute marking results of the pixels in the second reference image corresponding to the position.

[0068] Exemplarily, after obtaining the difference set information composed of reference pixels, since the reference pixels are pixels in the first reference image, an illuminance estimation map of the difference set information can be obtained by applying Equation (1) to the first reference image.

[0069] Further, fuse the reflection map and the illuminance estimation map composed of reference pixels to obtain a differential pixel image.

[0070] As an exemplary embodiment, the step of performing pixel attribute marking on each pixel in the first reference image and the second reference image to obtain the pixel attribute marking results of each of the reference images includes: fusing the first reference image and the second reference image to obtain a fused image; classifying each pixel in the fused image to obtain a pixel classification result; and respectively marking the pixels at the corresponding positions in the first reference image and the second reference image based on the pixel classification result and a preset attribute relationship to obtain a first pixel attribute corresponding to the first reference image and a second pixel attribute corresponding to the second reference image.

[0071] In this embodiment, first fuse the first and second reference images into a fused image; exemplarily, in the fused image, different tissue structure features such as superficial blood vessels, middle and deep blood vessels, and background tissue can be represented by different colors.

[0072] Further, according to the imaging characteristics of the fused image, after mapping, the superficial blood vessels appear brown, the blood appears dark brown, the background tissue appears light brown, the middle layer blood vessels appear cyan brown, and the deep layer blood vessels appear cyan. The pixel attribute categories in the fused image include but are not limited to superficial blood vessels, middle layer blood vessels, deep layer blood vessels, background tissue, blood, etc., and each pixel in the fused image is classified by means of image processing to obtain a pixel classification result.

[0073] Among them, the means of image processing can include traditional image processing, machine learning, and deep learning methods to achieve the classification of the fused image; the specific means include methods such as threshold segmentation, region growing, and edge detection based on traditional image processing, k-means clustering, SVM, etc. based on machine learning, and U-Net architecture, DeepLab model, etc. based on deep learning.

[0074] As a possible implementation, the first preset reference image corresponding to the first color channel and the second preset reference image corresponding to the second color channel can be collected in advance based on the endoscope, the first preset reference image and the second preset reference image are fused to obtain a preset fused image, and pixel attribute marking is performed on each pixel of the preset fused image to obtain a third data set composed of the preset fused image and its corresponding pixel attribute marking result. The pixel attribute marking model is trained based on the third data set. Further, the fused image is input into the model to obtain the pixel attributes of each pixel point in the fused image.

[0075] Further, after obtaining the pixel attributes of each pixel point in the fused image, the pixels at the corresponding positions in the first reference image and the second reference image are marked respectively based on the pixel classification result and the preset attribute relationship to obtain the first pixel attribute corresponding to the first reference image and the second pixel attribute corresponding to the second reference image.

[0076] As a possible implementation, the preset attribute relationship can be set with reference to the following method: if the pixel attribute of a pixel point in the fused image is of the deep blood vessel category, the first pixel attribute is of the deep blood vessel category and the second pixel attribute is of the background tissue category;

[0077] if the pixel attribute of a pixel point in the fused image is of the middle blood vessel category, the first pixel attribute is of the middle blood vessel category and the second pixel attribute is of the middle blood vessel category;

[0078] if the pixel attribute of a pixel point in the fused image is of the shallow blood vessel category, the first pixel attribute is of the background tissue category and the second pixel attribute is of the shallow blood vessel category;

[0079] if the pixel attribute of a pixel point in the fused image is of the background tissue category, the first pixel attribute is of the background tissue category and the second pixel attribute is of the background tissue category.

[0080] As an exemplary embodiment, determining the gain correction coefficient of the difference pixel image based on the first reference image and the second reference image includes: determining union information based on each pixel of the first reference image and the second reference image; wherein, the union information is obtained by weighted fusion of the first reference image and the second reference image; determining the gain correction coefficient of the difference pixel image based on the union information and the difference set information; wherein, the gain correction coefficient is positively correlated with the union information and negatively correlated with the difference set information.

[0081] In this embodiment, the difference pixel image is subjected to gain correction based on the union information and the difference set information so that the difference pixel image can match the brightness and contrast characterized in the red channel.

[0082] Exemplarily, the union information is obtained by weighted fusion of the first reference image and the second reference image; wherein, as an exemplary embodiment, determining the union information based on each pixel of the first reference image and the second reference image includes: determining the average brightness of the differential pixel image based on the overall average brightness of the first reference image and the second reference image; based on a preset fusion rule, fusing the average brightness as the first fusion parameter of the third color channel, the pixels in the first reference image as the second fusion parameter of the first color channel, and the pixels in the second reference image as the third fusion parameter of the second color channel to obtain the union information; wherein, the preset fusion rule is the conversion rule of the Y channel when converting the RGB domain to the YUV domain.

[0083] Specifically, the union information can be obtained through equations (4) and (5):

[0084] (4)

[0085] In equation (4), represents the union information, represents the estimated overall average brightness of the differential pixel image, represents the pixel value in the first reference image, represents the pixel value in the second reference image.

[0086] Wherein, can be obtained through equation (5):

[0087] (5)

[0088] In equation (5), represents the estimated overall average brightness of the differential pixel image, is the average brightness of the first reference image, is the average brightness of the second reference image.

[0089] Furthermore, determine the gain correction coefficient of the differential pixel image based on the union information and the difference set information; wherein, the correction gain coefficient can be obtained through a preset formula such as equation (2).

[0090] The above method for determining the gain correction coefficient, on the one hand, determines the union information by weighted fusion of the first reference image and the second reference image, and simultaneously considers the pixel attributes and brightness characteristics of the pixels in the first reference image and the second reference image; at the same time, the gain correction coefficient is determined by the difference between the union information that includes both brightness characteristics and pixel attributes and the difference set information composed of only the reference pixels in the first reference image that have different pixel attributes from the pixels in the second reference image; the estimated image obtained through gain correction can represent the characteristics of the red channel under approximate white light imaging, and the finally fused endoscopic imaging image can present a color expression closer to white light on the basis of retaining prominent detail information.

[0091] After obtaining the gain correction coefficient, the gain correction coefficient can be further normalized, processed by neighborhood smoothing or low-pass filtering modulation to obtain the modulated gain correction coefficient.

[0092] Exemplarily, the gain correction coefficient can be normalized by Equation (6):

[0093] (6)

[0094] In Equation (6), represents the normalized gain correction coefficient, represents the maximum value of the gain correction coefficient, represents the minimum value of the gain correction coefficient.

[0095] Furthermore, the normalized gain correction coefficient is subjected to neighborhood smoothing and neighborhood low-pass filtering to obtain the modulated gain correction coefficient.

[0096] This embodiment provides an endoscopic imaging device, as Figure 2 shown, including:

[0097] An acquisition module 201, configured to acquire a first reference image and a second reference image collected by the endoscope;

[0098] A differential pixel image determination module 202, configured to obtain a differential pixel image of the first reference image relative to the second reference image based on the pixel attribute differences of the pixels in the first reference image and the second reference image; wherein, the differential pixel image is used to represent an image of the difference set information composed of the reference pixels in the first reference image that have different pixel attributes from the pixels in the second reference image under the first color channel;

[0099] A gain correction coefficient determination module 203, configured to determine a gain correction coefficient of the differential pixel image based on the first reference image and the second reference image;

[0100] The gain correction module 204 is configured to perform gain correction on the differential pixel image based on the gain correction coefficient to obtain an estimated image corresponding to the third color channel in the RGB image;

[0101] The fusion module 205 is configured to fuse the estimated image, the first reference image, and the second reference image to obtain an endoscopic imaging image.

[0102] It should be noted here that the examples and application scenarios implemented by the above modules and the corresponding steps are the same, but are not limited to the content disclosed in the above embodiments.

[0103] It should be noted that the above modules, as part of the device, can be implemented by software or by hardware, where the hardware environment includes a network environment.

[0104] An embodiment of the present invention further provides a computer device, including a processor, a communication interface, a memory, and a communication bus. Among them, the processor, the communication interface, and the memory complete mutual communication through the communication bus. The memory is used to store a computer program; the processor is configured to execute the method in the above-mentioned embodiments of any one by running the computer program stored on the memory.

[0105] Figure 3 is a structural block diagram of an optional computer device according to an embodiment of the present application. As Figure 3 shown, it includes a processor 10, a communication interface 20, a memory 30, and a communication bus 40. Among them, the processor 10, the communication interface 20, and the memory 30 complete mutual communication through the communication bus 40. Among them,

[0106] The memory 30 is used to store a computer program;

[0107] When the processor 10 is configured to execute the computer program stored on the memory 30, it implements the method of any of the above embodiments.

[0108] Optionally, in this embodiment, the above communication bus may be a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. This communication bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of representation, Figure 3 only a thick line is used to represent it in the figure, but it does not mean that there is only one bus or one type of bus.

[0109] The communication interface is used for communication between the above computer device and other devices.

[0110] The memory may include RAM and may also include non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located far from the aforementioned processor.

[0111] The aforementioned processor may be a general-purpose processor, including but not limited to: CPU (Central Processing Unit), NP (Network Processor), etc.; it may also be a DSP (Digital Signal Processing), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0112] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments, and will not be elaborated here.

[0113] Those of ordinary skill in the art can understand that Figure 3 The structure shown is only schematic. The device for implementing the method of any one of the above embodiments may be a terminal device, which may be a smart phone (such as an Android phone, an IOS phone, etc.), a tablet computer, a palm computer, and a mobile Internet device (MID), a PAD and other terminal devices. Figure 3 It does not limit the structure of the above electronic device. For example, the terminal device may further include more or fewer components (such as a network interface, a display device, etc.) than those shown in Figure 3 or have a different configuration from that shown in Figure 3 shown.

[0114] Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the relevant hardware of the terminal device through a program, and the program can be stored in a computer-readable storage medium. The storage medium may include: a flash drive, a ROM, a RAM, a magnetic disk or an optical disc, etc.

[0115] As an exemplary embodiment, the present application further provides a computer-readable storage medium, in which a computer program is stored, and the computer program is configured to execute the method steps of any one of the embodiments when running.

[0116] Optionally, in this embodiment, the above storage medium may be used to execute the program code of the method steps of the embodiments of the present application.

[0117] Optionally, in this embodiment, the above storage medium may be located on at least one of the multiple network devices in the network shown in the above embodiment.

[0118] Optionally, in this embodiment, the storage medium is configured to store the method for executing the above embodiment.

[0119] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments, and details are not described herein again.

[0120] Optionally, in this embodiment, the above storage medium may include but is not limited to: various media that can store program code such as USB flash drives, ROMs, RAMs, mobile hard disks, magnetic disks, or optical discs.

[0121] The serial numbers of the embodiments of the present application above are only for description and do not represent the advantages or disadvantages of the embodiments.

[0122] If the integrated unit in the above embodiment is implemented in the form of a software functional unit and sold or used as an independent product, it may be stored in the above computer-readable storage medium. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, may be embodied in the form of a software product. The computer software product is stored in the storage medium and includes several instructions for causing one or more computer devices (which may be personal computers, servers, or network devices, etc.) to execute all or part of the steps of the method in the above embodiment.

[0123] In several embodiments provided by the present application, it should be understood that the disclosed client can be implemented in other ways. Among them, the device embodiments described above are only illustrative. For example, the division of units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection may be through some interfaces. The indirect coupling or communication connection of units or modules can be in an electrical or other form.

[0124] The units described as separate components may or may not be physically separated. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution provided in this embodiment.

[0125] In addition, each functional unit in various embodiments of the present application may be integrated into one processing unit, may exist separately as individual physical units, or two or more units may be integrated into one unit. The above-mentioned integrated units may be implemented in the form of hardware or in the form of software functional units.

[0126] In the above embodiments of the present application, the descriptions of the various embodiments have their own emphases. For the parts not detailed in a certain embodiment, reference may be made to the relevant descriptions of other embodiments.

[0127] The above are only the preferred embodiments of the present application. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present application, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the present application.

Claims

1. An endoscopic imaging method, characterized in that: The endoscope comprises an illumination unit, an imaging unit and an image processing unit, wherein the illumination unit comprises at least one first light source of at least one first band corresponding to a first color channel and at least one second light source of at least one second band corresponding to a second color channel in an RGB image acquired by the imaging unit; the imaging unit is capable of acquiring an image corresponding to the first color channel as a first reference image and an image corresponding to the second color channel as a second reference image, and the endoscope imaging method comprises: Acquire the first reference image and the second reference image acquired by the endoscope; A difference pixel image of the first reference image relative to the second reference image is obtained based on the difference in pixel attributes of each pixel in the first reference image and the second reference image; wherein the difference pixel image is used to represent an image represented in the first color channel by difference set information composed of reference pixels in the first reference image having different pixel attributes from those in the second reference image; and the pixel attributes include at least one of a superficial blood vessel category, a middle blood vessel category, a deep blood vessel category, and a background tissue category; determining a gain correction coefficient of the difference pixel image based on the first reference image and the second reference image; Performing gain correction on the difference pixel image based on the gain correction coefficient to obtain an estimated image corresponding to the third color channel in the RGB image; The estimated image, the first reference image and the second reference image are fused to obtain an endoscopic imaging image.

2. The endoscopic imaging method according to claim 1, characterized in that: The step of obtaining a difference pixel image of the first reference image relative to the second reference image based on a difference in pixel attributes of each pixel in the first reference image and the second reference image includes: Performing pixel attribute marking on each pixel in the first reference image and the second reference image to obtain pixel attribute marking results of each of the reference images; Extracting difference information in the first reference image based on the pixel attribute labeling result; wherein the difference information includes a plurality of reference pixels in the first reference image, and the pixel attribute labeling result of each of the reference pixels is different from the pixel attribute labeling result of the pixel in the second reference image corresponding to the position; Determine an illumination estimation map of the difference set information based on the first reference image; wherein the illumination estimation map is obtained by performing low-pass filtering of three different scales on the first reference image; The illumination estimation map and the difference information are fused to obtain the difference pixel image.

3. The endoscopic imaging method according to claim 2, characterized in that: The step of marking pixel attributes of each pixel in the first reference image and the second reference image to obtain pixel attribute marking results of each reference image includes: fusing the first reference image and the second reference image to obtain a fused image; Classifying each pixel in the fused image to obtain a pixel classification result; Based on the pixel classification result and the preset attribute relationship, pixels at corresponding positions in the first reference image and the second reference image are marked respectively to obtain a first pixel attribute corresponding to the first reference image and a second pixel attribute corresponding to the second reference image.

4. The endoscopic imaging method according to claim 1, wherein: The step of determining a gain correction coefficient of the difference pixel image based on the first reference image and the second reference image comprises: Determining union information based on each pixel of the first reference image and the second reference image; wherein the union information is obtained by weighted fusion of the first reference image and the second reference image; A gain correction coefficient of the difference pixel image is determined based on the union information and the difference information; wherein the gain correction coefficient is positively correlated with the union information and negatively correlated with the difference information.

5. The endoscopic imaging method according to claim 4, characterized in that: The determining union information based on each pixel of the first reference image and the second reference image includes: Determine an average brightness of the difference pixel image based on full-image average brightness of the first reference image and the second reference image; Based on a preset fusion rule, the average brightness is used as a first fusion parameter of the third color channel, the pixels in the first reference image are used as second fusion parameters of the first color channel, and the pixels in the second reference image are used as third fusion parameters of the second color channel to obtain the union information; wherein the preset fusion rule is a conversion rule for the Y channel when converting the RGB domain to the YUV domain.

6. The endoscopic imaging method according to any one of claims 4 or 5, characterized in that: The step of determining a gain correction coefficient of the difference pixel image based on the union information and the difference information includes: ， In the formula, represents the gain correction factor, represents the union information, Indicates the difference information. Indicates an adjustable parameter.

7. The endoscopic imaging method according to claim 6, characterized in that: The endoscopic imaging method further comprises: After the gain correction coefficient is normalized, neighborhood smoothing and neighborhood low-pass filtering are performed to obtain the modulation gain correction coefficient.

8. An endoscopic imaging device, characterized in that: The endoscopic imaging device comprises: An acquisition module, used for acquiring a first reference image and a second reference image acquired by the endoscope; A difference pixel image determination module is used to obtain a difference pixel image of the first reference image relative to the second reference image based on the difference in pixel attributes of each pixel in the first reference image and the second reference image; wherein the difference pixel image is used to represent an image represented in a first color channel by difference set information composed of reference pixels in the first reference image having different pixel attributes from those in the second reference image; the pixel attributes include at least one of a superficial blood vessel category, a middle blood vessel category, a deep blood vessel category, and a background tissue category. a gain correction coefficient determination module, configured to determine a gain correction coefficient of the difference pixel image based on the first reference image and the second reference image; A gain correction module, configured to perform gain correction on the difference pixel image based on the gain correction coefficient to obtain an estimated image corresponding to the third color channel in the RGB image; A fusion module is used to fuse the estimated image, the first reference image and the second reference image to obtain an endoscopic imaging image.

9. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the endoscopic imaging method according to any one of claims 1 to 7 by executing the computer instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the endoscopic imaging method according to any one of claims 1 to 7.

Images