Endoscope image color temperature correction method and endoscope

Abstract

The present application relates to the technical field of image processing, and discloses an endoscope image color temperature correction method and an endoscope, the endoscope comprises at least two preset light sources, each preset light source emits light with a single preset color temperature, and the endoscope image color temperature correction method comprises the following steps: acquiring a mixed color temperature acquisition image of the endoscope under the at least two preset light sources; inputting the mixed color temperature acquisition image into a pre-trained endoscope image color temperature correction model, and obtaining a color temperature correction image based on the endoscope image color temperature correction model; and eliminating color cast caused by the formation of dynamic superimposed light fields in the transition area due to the mixing of preset light sources with different color temperatures.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an endoscopic image color temperature correction method and an endoscope. Background Art

[0002] With the vigorous development of endoscope applications, doctors' demand for high-quality endoscope imaging is also increasing.

[0003] Endoscopes have the application scenario of "mirror within a mirror"; in this scenario, an endoscope with a relatively large clamp opening is used as a mother mirror to observe the target area alone. At this time, the light emitted by the light source at the mirror head end is a single color temperature K1, and the image parameters of the mother mirror are also set according to the K1 color temperature; in order to observe more detailed areas in the target area or other areas, one or several extremely fine endoscopes are required as sub-mirrors, which enter the working channel of the mother mirror and extend from the clamp opening of the mother mirror to continue in-depth exploration; the light emitted by the light source at the mirror head end of each sub-mirror is a single color temperature K2, and the image parameters of the sub-mirrors are also set according to the K2 color temperature.

[0004] Since the light sources of the daughter mirror and the main mirror are different, their color temperatures K1 and K2 often differ to a certain extent. As a result, when the daughter mirror is extended from the clamping port of the main mirror, the light sources of two or more mirrors fill in the light at the same time, resulting in a short period of dynamic mixed color temperature, which also affects the imaging color of each mirror. In addition, due to the lack of standard white reference objects in the human body cavity, the traditional white balance algorithm fails, which increases the difficulty of solving the problem.

[0005] In view of this, in the endoscopy scenario, when the mother and child mirrors work synchronously, the color cast caused by the dynamic superposition of light fields formed by light sources of different color temperatures in the transition area is a pain point problem that needs to be solved urgently. Summary of the Invention

[0006] In view of this, the present invention provides an endoscopic image color temperature correction method and an endoscope to solve the problem of color cast caused by different color temperature light sources forming a dynamic superposition light field in the transition area when the mother and child mirrors work synchronously.

[0007] In a first aspect, the present invention provides a method for correcting the color temperature of an endoscopic image, wherein the endoscope includes at least two preset light sources, each of the preset light sources emits light of a single preset color temperature, and the method for correcting the color temperature of an endoscopic image includes: obtaining a mixed color temperature acquisition image of the endoscope under at least two of the preset light sources; inputting the mixed color temperature acquisition image into a pre-trained endoscopic image color temperature correction model, and obtaining a color temperature corrected image based on the endoscopic image color temperature correction model; wherein the endoscopic image color temperature correction model is based on a first historical acquisition image acquired by the endoscope under a single preset light source and a second historical acquisition image acquired by the endoscope under at least two of the preset light sources in the same scene as training data for model training, and during the model training process, the mapping relationship when mapping the second historical acquisition image to the first historical acquisition image is obtained based on the common biological features of the first historical acquisition image and the second historical acquisition image.

[0008] As an exemplary embodiment, the learning is based on the common biometric features of the first historical acquisition image and the second historical acquisition image, and the mapping relationship when mapping the second historical acquisition image to the first historical acquisition image includes: performing target segmentation on the first historical acquisition image and the second historical acquisition image based on the spectral reflectance of a preset target object to obtain the common biometric features; obtaining the first color features of the first historical acquisition image and the second color features of the second historical acquisition image; based on the common biometric features, continuously adjusting the model parameters of the model to be trained, so that the model to be trained learns the mapping relationship when mapping the second color features to the first color features until the model to be trained converges.

[0009] As an exemplary embodiment, obtaining the first color feature of the first historical acquisition image and the second color feature of the second historical acquisition image includes: obtaining a first light source parameter corresponding to the first historical acquisition image and a second light source parameter corresponding to the second historical acquisition image; determining the first color feature based on the first light source parameter; and determining the second color feature based on the first light source parameter, the second light source parameter and the second historical acquisition image.

[0010] As an exemplary embodiment, the color feature includes a color temperature distribution map, and the determining of the second color feature based on the first light source parameter, the second light source parameter and the second historical acquisition image includes: determining the relative distance between at least two of the preset light sources based on the first light source parameter, the second light source parameter and the second historical acquisition image; calculating the spot coverage parameter based on the relative distance; determining the distance attenuation parameter based on the first light source parameter and the second light source parameter; determining the fusion weight when fusing the color temperature distribution of at least two of the preset light sources to obtain the color temperature distribution map included in the second color feature based on the spot coverage parameter and the distance attenuation parameter; obtaining the color temperature distribution map of at least two of the preset light sources corresponding to the second historical acquisition image; and fusing the color temperature distribution map based on the fusion weight to obtain a second color temperature distribution map as the second color feature.

[0011] As an exemplary embodiment, the relative distance between at least two preset light sources is determined based on the first light source parameters, the second light source parameters and the second historical acquisition image, including: obtaining the focal length and pixel size of the first preset light source and the outer diameter of the second preset light source in the first light source parameters and the second light source parameters; obtaining the number of target pixels in the second historical acquisition image; wherein the target pixels are used to characterize the outer diameter of the head end of the second preset light source; and calculating the relative distance based on the focal length, the pixel size, the outer diameter and the number of pixels.

[0012] As an exemplary embodiment, determining the distance attenuation parameter based on the first light source parameter and the second light source parameter includes: obtaining a preset maximum relative distance in the light source parameters; wherein the preset maximum relative distance can be obtained by the relative distance between the preset light sources when the preset light source corresponding to the first historical acquisition image reaches the field of view limit of the preset light source corresponding to the second historical acquisition image; and calculating the light spot coverage parameter based on the relative distance and the maximum relative distance.

[0013] As an exemplary embodiment, determining the distance attenuation parameter based on the first light source parameter and the second light source parameter includes: obtaining the first reference light intensity and the first propagation distance contained in the first light source parameter, and the second reference light intensity and the second propagation distance contained in the second light source parameter, from the first light source parameter and the second light source parameter; and determining the distance attenuation parameter based on the first reference light intensity, the first propagation distance, the first reference light intensity, and the second propagation distance.

[0014] As an exemplary embodiment, the color feature also includes a light intensity distribution diagram, and determining the second color feature based on the first light source parameter, the second light source parameter and the second historical acquisition image also includes: determining the relative distance between at least two of the preset light sources based on the first light source parameter, the second light source parameter and the second historical acquisition image; obtaining the reference light intensity and light path attenuation coefficient of each of the preset light sources in the second light source parameters; and determining a second light intensity distribution diagram as the second color feature based on the reference light intensity, the relative distance and the light path attenuation coefficient.

[0015] As an exemplary embodiment, the common biometric feature is used as a benchmark to continuously adjust the training parameters of the preset model so that the preset model learns the mapping relationship when mapping the second color feature to the first color feature, including: constructing a target loss function based on the pixel difference information between the first historical acquisition image and the second historical acquisition image and the color feature difference between the second color features; training the model to be trained with the loss function as a constraint; during the model training process, based on the loss function, the model to be trained learns to extract features from the second historical acquisition image data to obtain a first feature extraction relationship when image features are obtained; based on the loss function, the model to be trained learns to extract features from the second historical color temperature distribution map, the second historical light intensity distribution map and the biometric feature to obtain a second feature extraction relationship when environmental features are obtained; based on the loss function, the model to be trained learns to fuse the image features and the environmental features to obtain a fused feature relationship when the image features and the environmental features are obtained; based on the loss function, the model to be trained learns to output an image output relationship when a color temperature corrected image is output based on the feature fusion relationship.

[0016] In a second aspect, the present invention provides an endoscope, comprising at least two preset light sources, at least two imaging units corresponding to the preset light sources, and at least two image processing units; each of the preset light sources emits light of a single preset color temperature, and the imaging unit is capable of capturing a single color temperature captured image obtained by the light of a single preset color temperature or a mixed color temperature captured image under at least two of the preset light sources; the image processing unit is capable of acquiring the captured image captured by the imaging unit, and the image processing unit comprises a memory and a processor, the memory and the processor being communicatively connected to each other, computer instructions stored in the memory, and the processor executing the computer instructions to execute the endoscopic image color temperature correction method described in any one of the above embodiments.

[0017] The present invention provides an endoscopic image color temperature correction method and an endoscope, wherein the endoscope includes at least two preset light sources, each of the preset light sources emits light of a single preset color temperature, and the endoscopic image color temperature correction method includes: obtaining a mixed color temperature acquisition image of the endoscope under at least two of the preset light sources; inputting the mixed color temperature acquisition image into a pre-trained endoscopic image color temperature correction model, and obtaining a color temperature corrected image based on the endoscopic image color temperature correction model; wherein the endoscopic image color temperature correction model is based on a first historical acquisition image acquired by the endoscope under a single preset light source and a second historical acquisition image acquired by the endoscope under at least two of the preset light sources in the same scene as training data for model training, and in the process of model training, learning is based on the common biological features of the first historical acquisition image and the second historical acquisition image, and the second historical acquisition image is used as a benchmark. The mapping relationship when mapped to the first historical acquisition image is obtained; in the above method, the endoscopic image color temperature correction model takes the common biological features of the first historical acquisition image representing a single color temperature and the second historical acquisition image representing a mixed color temperature under historical circumstances as a benchmark, and learns the mapping relationship of mapping the second historical acquisition image representing a mixed color temperature under historical circumstances to the first historical acquisition image representing a single color temperature; based on this, when the mixed color temperature acquisition image is input into the pre-trained endoscopic image color temperature correction model, the endoscopic color temperature correction model can map the mixed color temperature acquisition image to a color temperature correction image representing a single color temperature according to the common biological features of the mixed color temperature acquisition image, the first historical acquisition image and the second historical acquisition image, by mapping the second historical acquisition image to the first historical acquisition image, so as to eliminate the color cast caused by the dynamic superposition light field formed in the transition area by mixing preset light sources of different color temperatures. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0019] Figure 1 is a flow chart of a method for color temperature correction of an endoscopic image according to an embodiment of the present invention;

[0020] Figure 2 Schematic diagram of the hardware structure of the endoscope according to an embodiment of the present invention. DETAILED DESCRIPTION

[0021] To make the purpose, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative efforts shall fall within the scope of protection of the present invention.

[0022] According to an embodiment of the present invention, an embodiment of an endoscopic image imaging method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0023] In one embodiment, the method for color temperature correction of an endoscopic image is applied to an endoscope.

[0024] In one embodiment, the endoscope includes at least two preset light sources, at least two imaging units corresponding to the preset light sources, and at least two image processing units; each of the preset light sources emits light of a single preset color temperature, and the imaging unit is capable of capturing a single color temperature capture image obtained by the light of a single preset color temperature or a mixed color temperature capture image under at least two of the preset light sources.

[0025] In one embodiment, the image processing unit is configured to execute an endoscopic image color temperature correction method.

[0026] In one embodiment, the endoscope includes a first preset light source and a second preset light source, the first preset light source emits light of a first preset color temperature, and the second preset light source emits light of a second preset color temperature.

[0027] In one embodiment, the endoscope includes a first lens and a second lens, the first lens is provided with a first preset light source, and the second lens is provided with a second preset light source, the first preset light source emits light of a first preset color temperature, and the second preset light source emits light of a second preset color temperature, the endoscope simultaneously detects the target object through the first lens and the second lens, and obtains a mixed color temperature captured image through each or a common image processing unit and an imaging unit.

[0028] In one embodiment, the endoscope includes a first lens and a second lens, the first lens accommodates the second lens through the clamp opening, a first preset light source is provided on the first lens, and a second preset light source is provided on the second lens, the first preset light source emits light of a first preset color temperature, and the second preset light source emits light of a second preset color temperature, the endoscope simultaneously detects the target object through the first lens and the second lens, and obtains a mixed color temperature captured image through the respective image processing units and imaging units.

[0029] Exemplarily, the technical solution of the present application is described by taking an endoscope including a first preset light source, a second preset light source, an imaging unit corresponding to the first preset light source and the second preset light source as an example; wherein, the first preset light source can provide light with a color temperature of K1, and the second preset light source can provide light with a color temperature of K2, the first preset light source accommodates the second preset light source through the clamp channel, the first preset light source and its corresponding imaging unit and image processing unit can acquire a capture image of the target tissue, the second preset light source and its corresponding imaging unit and image processing unit can extend from the working channel of the first preset light source and its corresponding imaging unit and image processing unit to acquire a capture image of the target tissue; the image processing unit can acquire a single color temperature capture image obtained by the single preset color temperature light provided by the first preset light source or the second preset light source or a mixed color temperature capture image provided by the first preset light source and the second preset light source through the imaging unit, and the image processing unit is used to execute the endoscopic image color temperature correction method.

[0030] According to an embodiment of the present invention, an embodiment of a method for color temperature correction of an endoscopic image is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0031] In this embodiment, a method for color temperature correction of an endoscope image is provided. Figure 1 FIG. 1 is a flow chart of a method for correcting color temperature of an endoscopic image according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:

[0032] Step S101 : acquiring a mixed color temperature image of the endoscope under at least two of the preset light sources.

[0033] Exemplarily, the mixed color temperature captured image is captured when at least two preset light sources are enabled.

[0034] Exemplarily, the mixed color temperature captured image can be acquired by communicating between the image processing unit and the imaging units corresponding to the preset light sources when at least two preset light sources are enabled.

[0035] Step S102: input the mixed color temperature acquisition image into a pre-trained endoscopic image color temperature correction model, and obtain a color temperature corrected image based on the endoscopic image color temperature correction model; wherein, the endoscopic image color temperature correction model is based on a first historical acquisition image acquired by the endoscope under a single preset light source and a second historical acquisition image acquired by the endoscope under at least two preset light sources in the same scene as training data for model training. During the model training process, the mapping relationship when mapping the second historical acquisition image to the first historical acquisition image is obtained by learning based on the common biological features of the first historical acquisition image and the second historical acquisition image.

[0036] In the related art, corresponding image parameters are usually set separately for the fixed color temperature of a single light source; for scenes where at least two preset light sources provide illumination, the light sources influence each other, resulting in a dynamic mixed color temperature, which also affects the color of the captured image corresponding to each light source.

[0037] To solve this problem, in this embodiment, the mixed color temperature acquired image is input into a pre-trained endoscopic image color temperature correction model, and a color temperature corrected image is obtained based on the endoscopic image color temperature correction model.

[0038] Exemplarily, the endoscopic image color temperature correction model is obtained by model training based on a first historical acquisition image acquired by the endoscope under a single preset light source in the same scene and a second historical acquisition image acquired by the endoscope under at least two preset light sources as training data; during the model training process, the mapping relationship when mapping the second historical acquisition image to the first historical acquisition image is learned based on the common biological features of the first historical acquisition image and the second historical acquisition image.

[0039] In the above embodiment, for a mixed color temperature captured image affected by a dynamic mixed color temperature, when corresponding to a color temperature corrected image not affected by other color temperatures, the mixed color temperature captured image and the color temperature corrected image are in the same scene and capture the same target object, and therefore both have the same common biometric features. The endoscopic image color temperature correction model uses the common biometric features of a first historical captured image representing a single color temperature and a second historical captured image representing a mixed color temperature as a benchmark, and learns a mapping relationship for mapping the second historical captured image representing a mixed color temperature under historical circumstances to the first historical captured image representing a single color temperature. Based on this, when the mixed color temperature captured image is input into a pre-trained endoscopic image color temperature correction model, the endoscopic color temperature correction model can map the mixed color temperature captured image to a color temperature corrected image representing a single color temperature based on the common biometric features of the mixed color temperature captured image, the first historical captured image, and the second historical captured image, by mapping the second historical captured image to the first historical captured image, thereby eliminating color cast caused by a dynamic superposition light field formed in a transition area by mixing preset light sources of different color temperatures.

[0040] The present embodiment provides an endoscopic image color temperature correction method, wherein the endoscope includes at least two preset light sources, each of the preset light sources emits light of a single preset color temperature, and the endoscopic image color temperature correction method includes: obtaining a mixed color temperature acquisition image of the endoscope under at least two of the preset light sources; inputting the mixed color temperature acquisition image into a pre-trained endoscopic image color temperature correction model, and obtaining a color temperature corrected image based on the endoscopic image color temperature correction model; wherein the endoscopic image color temperature correction model is based on a first historical acquisition image acquired by the endoscope under a single preset light source and a second historical acquisition image acquired by the endoscope under at least two of the preset light sources in the same scene as training data for model training, and during the model training process, learning is based on the common biological features of the first historical acquisition image and the second historical acquisition image, and mapping the second historical acquisition image to the target image. The endoscopic image color temperature correction model is obtained by the mapping relationship when the first historical acquisition image is acquired; in the above method, the endoscopic image color temperature correction model takes the common biological features of the first historical acquisition image representing a single color temperature and the second historical acquisition image representing a mixed color temperature under historical circumstances as a benchmark, and learns to map the second historical acquisition image representing a mixed color temperature under historical circumstances to the first historical acquisition image representing a single color temperature; based on this, when the mixed color temperature acquisition image is input into the pre-trained endoscopic image color temperature correction model, the endoscopic color temperature correction model can map the mixed color temperature acquisition image to a color temperature correction image representing a single color temperature according to the common biological features of the mixed color temperature acquisition image, the first historical acquisition image and the second historical acquisition image, by mapping the second historical acquisition image to the first historical acquisition image, so as to eliminate the color cast caused by the dynamic superposition light field formed in the transition area by mixing preset light sources of different color temperatures.

[0041] As an exemplary embodiment, the learning is based on the common biometric features of the first historical acquisition image and the second historical acquisition image, and the mapping relationship when mapping the second historical acquisition image to the first historical acquisition image includes: performing target segmentation on the first historical acquisition image and the second historical acquisition image based on the spectral reflectance of a preset target object to obtain the common biometric features; obtaining the first color features of the first historical acquisition image and the second color features of the second historical acquisition image; based on the common biometric features, continuously adjusting the model parameters of the model to be trained, so that the model to be trained learns the mapping relationship when mapping the second color features to the first color features until the model to be trained converges.

[0042] Since the acquisition scenes of the first historical acquisition image and the second historical acquisition image are the same, the corresponding images contain the same target object; therefore, for the first historical acquisition image and the second historical acquisition image, although their lighting conditions are affected by different preset light sources, the same target object they contain remains unchanged, and the corresponding common biometric features determined by the target object remain unchanged.

[0043] At the same time, spectral reflectance is an important physical property that is independent of lighting and camera equipment. It represents the reflection ratio of the object surface to light of different wavelengths. By measuring the reflectance of the target object at each wavelength, the true color representation of the target object under standard observer conditions can be calculated; therefore, for the first historical acquisition image and the second historical acquisition image, although their lighting conditions are affected by different preset light sources, the same target object contained therein remains unchanged, and the corresponding spectral reflectance distribution map determined by the target object remains unchanged; when training the model, the changed color features are mapped based on the unchanged spectral reflectance distribution map. During the training process, the mapping relationship when mapping the different features of the first historical acquisition image and the second historical acquisition image can be learned by taking the common features of the first historical acquisition image and the second historical acquisition image as the benchmark.

[0044] Based on this, in this embodiment, target segmentation is performed on the first historically collected image and the second historically collected image respectively based on the spectral reflectance of a preset target object to obtain the common biometric feature.

[0045] When performing target segmentation, it can be achieved through image processing methods; specifically, first define the image content attributes of the target endoscopy application scenario, such as bronchial mucosa, mucus, superficial blood vessels, deep blood vessels, bleeding points, inflammation, etc., and then perform regional segmentation on the image, where each region represents similar image attributes.

[0046] Among them, image region segmentation can adopt both traditional image processing methods and deep learning methods; traditional image processing methods include but are not limited to threshold segmentation method, edge detection segmentation method, genetic algorithm segmentation method, etc.; deep learning processing methods may include, for example, feature coding-based segmentation methods such as deep residual neural network (Residual Neural Network, ResNet), VGGNet, region selection-based methods such as fast region convolutional network (Fast Region-based Convolutional Network, Fast RCNN), image segmentation methods based on recurrent neural network (Recurrent Neural Network, RNN), etc.

[0047] Since the spectral reflectance of the target object is independent of the lighting and imaging characteristics, in this embodiment, the spectral reflectance is pre-calibrated. For example, the process of offline calibration of the spectral reflectance is as follows:

[0048] Step 1: Obtain a spectral reflectance curve: Set up an experimental scene and obtain a sample of a preset target object. The reflectance of the target object at different wavelengths within the visible light wavelength range (approximately 400nm to 700nm) can be measured using a spectrometer. Discrete sampling can be performed at a sampling interval of N nm and fitted into a continuous spectral reflectance curve.

[0049] Step 2: Multiply the spectral reflectance by the corresponding standard observer function; the International Commission on Illumination (CIE) defined two standard observer modes, CIE 1931 and CIE 1964, which represent the average sensitivity of human vision to different wavelengths as a way to represent the color perception of standardized objects.

[0050] Step 3: Multiply the result of step 2 by the spectral power distribution weight at the corresponding wavelength and integrate it over the entire wavelength to obtain the XYZ tristimulus values, where the Y value represents brightness and the X and Z values ​​represent chromaticity.

[0051] Step 4: Since XYZ is a device-independent color space, it is converted to the corresponding sRGB color space based on the current camera system settings and the corresponding RGB calibration values ​​are obtained.

[0052] Finally, the spectral reflectance of the preset target object is obtained.

[0053] Furthermore, the difference feature is reflected in this embodiment by the first color feature of the first historical acquisition image and the second color feature of the second historical acquisition image; wherein, the color feature can be determined by the image feature of the acquisition image and / or the light source parameters of the preset light source.

[0054] In one embodiment, the color feature includes a color temperature distribution map for representing the color temperature distribution of the image.

[0055] In one embodiment, the color feature includes a color temperature distribution graph for representing the color temperature distribution of the image and a light intensity distribution graph for representing the light intensity distribution of the image.

[0056] Furthermore, after obtaining the common biometric features and color features, the model parameters of the model to be trained are continuously adjusted based on the common biometric features, so that the model to be trained learns the mapping relationship when mapping the second color feature to the first color feature until the model to be trained converges.

[0057] As an exemplary embodiment, obtaining the first color feature of the first historical acquisition image and the second color feature of the second historical acquisition image includes: obtaining a first light source parameter corresponding to the first historical acquisition image and a second light source parameter corresponding to the second historical acquisition image; determining the first color feature based on the first light source parameter; and determining the second color feature based on the first light source parameter, the second light source parameter and the second historical acquisition image.

[0058] In this embodiment, since the first color feature corresponds to the first historically captured image under a single preset light source and is not affected by other light sources, the acquisition of the first color feature can be determined solely based on the first light source parameters of the preset light source corresponding to the first historically captured image.

[0059] Since the second color feature corresponds to the second historical acquisition image captured under at least two of the preset light sources, which is affected by other light sources, the acquisition of the second color feature can be determined based on the first light source parameters, the second light source parameters and the second historical acquisition image; based on this, as an exemplary embodiment, the color feature includes a color temperature distribution map, and the determination of the second color feature based on the first light source parameters, the second light source parameters and the second historical acquisition image includes: determining the relative distance between at least two of the preset light sources based on the first light source parameters, the second light source parameters and the second historical acquisition image; calculating the spot coverage parameter based on the relative distance; determining the distance attenuation parameter based on the first light source parameters and the second light source parameters; determining the fusion weight when fusing the color temperature distribution of at least two of the preset light sources to obtain the color temperature distribution map contained in the second color feature based on the spot coverage parameter and the distance attenuation parameter; obtaining the color temperature distribution map of at least two of the preset light sources corresponding to the second historical acquisition image; fusing the color temperature distribution map based on the fusion weight to obtain the second color temperature distribution map as the second color feature.

[0060] In this embodiment, the relative distance between at least two preset light sources is considered to determine their respective contributions to the imaging of the second historical acquisition image, and the second color characteristics of the second historical acquisition image are further determined based on the light source parameter characteristics of the at least two preset light sources and the second historical acquisition image.

[0061] Specifically, in this embodiment, the relative distance between at least two preset light sources is first determined based on the first light source parameters, the second light source parameters and the second historical acquisition image, and the spot coverage parameter is further calculated based on the relative distance; and at the same time, the distance attenuation parameter is determined based on the first light source parameters and the second light source parameters.

[0062] In one embodiment, since the second preset light source and its corresponding imaging unit and image processing unit can extend from the working channel of the first preset light source and its corresponding imaging unit and image processing unit to acquire a captured image of the target tissue, the relative positions of the first preset light source and the second preset light source are fixed, and the relative distance can be determined by calibrating in advance the correspondence between the relative positions of the first preset light source and the second preset light source and the image features of the captured image; the spot coverage parameter is further determined based on the relative distance; wherein, illustratively, the spot coverage parameter is inversely correlated with the relative distance.

[0063] Moreover, since the relative positions of the first preset light source and the second preset light source are fixed, the distance attenuation parameters can be determined by the first light source parameters and the second light source parameters; specifically, the first light source parameters and the second light source parameters may include the light intensity of the first preset light source and the second preset light source at a reference distance, and the propagation distance from the first preset light source and the second preset light source to the target actual tissue, and the distance attenuation parameters are further determined based on the light intensity at the reference distance and the propagation distance.

[0064] Furthermore, a fusion weight for obtaining a color temperature distribution graph included in the second color feature by fusing the color temperature distributions of at least two preset light sources is determined based on the light spot coverage parameter and the distance attenuation parameter.

[0065] In one embodiment, the color temperature of each pixel of the second historically collected image may be determined using formula (1), and the color temperature distribution map may be further determined:

[0066] (1)

[0067] In formula (1), is the color temperature value of the pixel with coordinates (x, y) in the second color temperature distribution diagram, K1 represents the color temperature value of the pixel with coordinates (x, y) under the first preset light source, and K2 represents the color temperature value of the pixel with coordinates (x, y) under the second preset light source. represents the fusion weight.

[0068] in, Based on the spot coverage parameter and the distance attenuation parameter, it can be calculated using formula (2):

[0069] (2)

[0070] In formula (2), ω represents the fusion weight, ω space represents the spot coverage parameter, ω dist represents the distance attenuation parameter, α Indicates the modulation parameters used in calculating the fusion weights, which are preset through offline experimental debugging.

[0071] The spot coverage is calculated based on the relative distance. Specifically, it is calculated using formula (3):

[0072] (3)

[0073] In formula (3), ω space represents the spot coverage parameter, L Indicates relative distance, L max Indicates the preset maximum relative distance. L max The distance between the first preset light source and the second preset light source can be obtained by calibrating the relative distance when the mutual influence between the first preset light source and the second preset light source is zero.

[0074] As an exemplary embodiment, the relative distance between at least two preset light sources is determined based on the first light source parameters, the second light source parameters and the second historical acquisition image, including: obtaining the focal length and pixel size of the first preset light source and the outer diameter of the second preset light source in the first light source parameters and the second light source parameters; obtaining the number of target pixels in the second historical acquisition image; wherein the target pixels are used to characterize the outer diameter of the head end of the second preset light source; and calculating the relative distance based on the focal length, the pixel size, the outer diameter and the number of pixels.

[0075] The focal length and pixel size of the first preset light source and the outer diameter of the second preset light source can be obtained through the device specification and stored offline in the image processing unit.

[0076] For example, the relative distance can be calculated using formula (4):

[0077] (4)

[0078] In formula (4), L Represents the calculated relative distance, D represents the outer diameter of the second preset light source, f represents the focal length of the first preset light source, s represents the pixel size of the first preset light source, a Indicates the number of target pixels corresponding to the outer diameter of the head end of the second preset light source.

[0079] In one embodiment, D =2.8mm, f =3mm, s =1.2μm.

[0080] As an exemplary embodiment, determining the distance attenuation parameter based on the first light source parameter and the second light source parameter includes: obtaining a preset maximum relative distance in the light source parameters; wherein the preset maximum relative distance can be obtained by the relative distance between the preset light sources when the preset light source corresponding to the first historical acquisition image reaches the field of view limit of the preset light source corresponding to the second historical acquisition image; and calculating the light spot coverage parameter based on the relative distance and the maximum relative distance.

[0081] In one embodiment, the preset maximum relative distance can be obtained by calibrating the relative distance between the preset light sources when the preset light source corresponding to the first historical acquisition image reaches the field of view limit of the preset light source corresponding to the second historical acquisition image.

[0082] As an exemplary embodiment, determining the distance attenuation parameter based on the first light source parameter and the second light source parameter includes: obtaining the first reference light intensity and the first propagation distance contained in the first light source parameter, and the second reference light intensity and the second propagation distance contained in the second light source parameter, from the first light source parameter and the second light source parameter; and determining the distance attenuation parameter based on the first reference light intensity, the first propagation distance, the first reference light intensity, and the second propagation distance.

[0083] In one embodiment, the first light source parameters and the second light source parameters include a first reference light intensity and a first propagation distance of a first preset light source, and a second reference light intensity and a second propagation distance of the second preset light source; and can be obtained through factory calibration data of the light source.

[0084] Further, the distance attenuation parameter is determined based on the first reference light intensity, the first propagation distance, the first reference light intensity, and the second propagation distance.

[0085] For example, the distance attenuation parameter can be calculated using formula (5):

[0086] (5)

[0087] In formula (5), ω dist represents the distance attenuation parameter, I A Indicates the first reference light intensity, I B Indicates the second reference light intensity; d A represents the first propagation distance, d B Represents the second propagation distance.

[0088] When at least two preset light sources are used for filling light at the same time, the superposition of the light sources will also affect the light intensity of the image, and ultimately affect the respective imaging; based on this, in the present invention, the light intensity distribution of the image is further corrected to restore the light intensity representation of the image.

[0089] Based on this, as an exemplary embodiment, the color feature also includes a light intensity distribution diagram, and the determination of the second color feature based on the first light source parameter, the second light source parameter and the second historical acquisition image also includes: determining the relative distance between at least two of the preset light sources based on the first light source parameter, the second light source parameter and the second historical acquisition image; obtaining the reference light intensity and light path attenuation coefficient of each of the preset light sources in the second light source parameter; and determining the second light intensity distribution diagram as the second color feature based on the reference light intensity, the relative distance and the light path attenuation coefficient.

[0090] In this embodiment, the relative distance between the at least two preset light sources can be determined with reference to the implementation of the above embodiment.

[0091] In this embodiment, the reference light intensity of the preset light sources can be obtained by acquiring factory calibration data of each preset light source.

[0092] In this embodiment, the optical path attenuation coefficient can be obtained through an off-axis integral calibration experiment.

[0093] Furthermore, the second light intensity distribution diagram can be determined based on the reference light intensity, the relative distance and the light path attenuation coefficient using formula (6):

[0094] (6)

[0095] In formula (6), represents the light intensity value of the pixel at the coordinate (x, y) in the second historical acquisition image, IA represents the reference light intensity of the first preset light source, and IB represents the reference light intensity of the second preset light source, which are obtained through the factory calibration data of the light source; L represents the relative distance, β represents the optical path attenuation coefficient of the first preset light source, and γ represents the optical path attenuation coefficient of the second preset light source. β and γ can be obtained through off-axis integral calibration experiments.

[0096] As an exemplary embodiment, the common biometric feature is used as a benchmark to continuously adjust the training parameters of the preset model so that the preset model learns the mapping relationship when mapping the second color feature to the first color feature, including: constructing a target loss function based on the pixel difference information between the first historical acquisition image and the second historical acquisition image and the color feature difference between the second color features; training the model to be trained with the loss function as a constraint; during the model training process, based on the loss function, the model to be trained learns to extract features from the second historical acquisition image data to obtain a first feature extraction relationship when image features are obtained; based on the loss function, the model to be trained learns to extract features from the second historical color temperature distribution map, the second historical light intensity distribution map and the biometric feature to obtain a second feature extraction relationship when environmental features are obtained; based on the loss function, the model to be trained learns to fuse the image features and the environmental features to obtain a fused feature relationship when the image features and the environmental features are obtained; based on the loss function, the model to be trained learns to output an image output relationship when a color temperature corrected image is output based on the feature fusion relationship.

[0097] In the present invention, the endoscopic image color temperature correction model uses the first historical image collected by the endoscope under a single preset light source and the second historical image collected by the endoscope under at least two preset light sources as training data for model training.

[0098] For example, the first historical image captured by the endoscope under the single preset light source is obtained as the Ground Truth data; specifically, the Ground Truth data is obtained by directly acquiring the image RGB by the endoscope under the default color temperature of the single preset light source. GT , and obtain the color temperature distribution map K under this scene GT , spectral reflectance distribution diagram R , light intensity distribution diagram I GT get.

[0099] Exemplarily, a second historical image captured by the endoscope under at least two of the preset light sources is obtained as Distort data; specifically, the Distort data is obtained by adding at least one other preset light source fill light under the same scene by the endoscope to obtain the image RGB dis , and obtain the color temperature distribution map K of the scene at the same time dis , spectral reflectance distribution diagram R , light intensity distribution diagram I dis get.

[0100] Furthermore, during model training, nonlinear compensation is applied to the Distort data so that the model can learn the mapping relationship when mapping the Distort data to the Ground Truth data through Equation (7): f :

[0101] (7)

[0102] in, R pred 、G pred 、B pred Represents the RGB value output by the model, R dis 、 G dis 、 B dis Represents image RGB dis RGB value, K dis Represents image RGB dis Corresponding color temperature distribution diagram, R Represents image RGB dis The corresponding spectral reflectance distribution diagram, I dis Represents image RGB dis Corresponding light intensity distribution diagram.

[0103] In one embodiment, the preset model is a neural network model; the neural network model may include a coding layer, a feature fusion layer, a color decoding layer and an output layer; the coding layer includes a first encoder and a second encoder, the first encoder is used to input an RGB image, and the second encoder is used to input a color temperature distribution map, a spectral reflectance distribution map and a light intensity distribution map; after the first encoder and the second encoder encode the features, they are sent to the feature fusion layer for feature extraction and feature fusion. After obtaining the fused features, the feature fusion layer sends the fused features to the color decoder, and the color decoder decodes the fused features and sends them to the output layer. The output layer outputs an RGB image according to the input of the color decoder.

[0104] In one embodiment, the second feature encoder may be configured using formula (8):

[0105] (8)

[0106] Among them, V env represents the features encoded by the second feature encoder, K dis Represents image RGB dis Corresponding color temperature distribution diagram, R Represents image RGB dis The corresponding spectral reflectance distribution diagram, Idis Represents image RGB dis The corresponding light intensity distribution diagram, , .

[0107] In one embodiment, the neural network is l The layer residual convolution structure realizes feature extraction. Specifically, the feature extraction performed by the neural network can be realized by using formula (9):

[0108] (9)

[0109] In formula (9), Indicates that through l -1 layer of residual network extracted features, Indicates passing l Features extracted by the layer residual network.

[0110] In one embodiment, the neural network performs feature fusion by channel attention weighting. Specifically, the feature fusion performed by the neural network can be implemented using formula (10):

[0111] (10)

[0112] In formula (10), represents the fusion feature after feature fusion, Indicates passing l The features extracted by the layer residual network, represents the channel attention weighted operation, V env Represents the features encoded by the second feature encoder.

[0113] In one embodiment, the constraint function of the neural network is constructed based on pixel difference information between the first historically collected image and the second historically collected image and color feature differences between the second color features.

[0114] Specifically, the constraint function of the neural network is constructed using formula (11):

[0115] (11)

[0116] In formula (11), represents the constraint function, The term is the data fidelity term; ω1 represents the weight ratio parameter of the data fidelity term; RGB pred Represents the RGB value output by the neural network; RGB GT represents the RGB value of the first historically acquired image; ||…||2 represents the L2 norm (Euclidean norm), which measures the difference between the predicted image and the true image using the weighted Euclidean distance using the L2 norm; The term is the color temperature constraint term, ω2 represents the weight ratio parameter of the color temperature constraint term, and ΔE represents the predicted color temperature K pred With the real color temperature K GT difference.

[0117] An embodiment of the present invention also provides an endoscope, which includes at least two preset light sources, at least two imaging units corresponding to the preset light sources, and at least two image processing units; each of the preset light sources emits light of a single preset color temperature, and the imaging unit can capture a single color temperature capture image obtained by the light of a single preset color temperature or a mixed color temperature capture image under at least two of the preset light sources.

[0118] The image processing unit is capable of acquiring the captured image captured by the imaging unit, and includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus, the memory is used to store computer programs; the processor is used to execute the method in any of the above embodiments by running the computer program stored in the memory.

[0119] Figure 2 is a structural block diagram of an optional endoscope according to an embodiment of the present application, such as Figure 2 As shown, it includes a processor 10, a communication interface 20, a memory 30 and a communication bus 40, wherein the processor 10, the communication interface 20 and the memory 30 communicate with each other through the communication bus 40, wherein,

[0120] Memory 30, for storing computer programs;

[0121] The processor 10 is configured to implement the endoscopic image color temperature correction method according to any of the above embodiments when executing the computer program stored in the memory 30 .

[0122] Optionally, in this embodiment, the communication bus may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The communication bus may be divided into an address bus, a data bus, a control bus, and the like. For ease of representation, Figure 2 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0123] The communication interface is used for communication between the endoscope and other devices.

[0124] The memory may include RAM, or may include non-volatile memory, such as at least one disk memory. Alternatively, the memory may also be at least one storage device located away from the aforementioned processor.

[0125] The above-mentioned processor can be a general-purpose processor, which can include but is not limited to: CPU (Central Processing Unit), NP (Network Processor), etc.; it can also be DSP (Digital Signal Processing), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0126] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments, and this embodiment will not be described in detail here.

[0127] It can be understood by those skilled in the art that Figure 2 The structure shown is for illustration only. The device for implementing any one of the methods in the above embodiments may be a terminal device, which may be a smart phone (such as an Android phone, an IOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (MID), a PAD, or other terminal devices. Figure 2 It does not limit the structure of the above electronic device. For example, the terminal device may also include Figure 2 More or fewer components (such as network interfaces, display devices, etc.) shown in, or with Figure 2 Different configurations shown.

[0128] A person skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, which can include: a flash drive, ROM, RAM, a magnetic disk or an optical disk, etc.

[0129] As an exemplary embodiment, the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute any one of the method steps of the present embodiment when run.

[0130] Optionally, in this embodiment, the above-mentioned storage medium can be used to execute the program code of the method steps of the embodiment of the present application.

[0131] Optionally, in this embodiment, the above-mentioned storage medium may be located on at least one network device among the multiple network devices in the network shown in the above-mentioned embodiment.

[0132] Optionally, in this embodiment, the storage medium is configured to store data for executing the method in the above embodiment.

[0133] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments, which will not be described in detail in this embodiment.

[0134] Optionally, in this embodiment, the storage medium may include, but is not limited to, various media that can store program codes, such as a USB flash drive, a ROM, a RAM, a mobile hard disk, a magnetic disk, or an optical disk.

[0135] The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0136] If the integrated units in the above embodiments are implemented in the form of software functional units and sold or used as independent products, they can be stored in the above-mentioned computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for causing one or more endoscopes (which can be personal computers, servers, or network devices, etc.) to execute all or part of the steps of the method in the above embodiments.

[0137] In the several embodiments provided in this application, it should be understood that the disclosed client can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is merely a logical functional division. In actual implementation, there may be other division methods, such as combining or integrating multiple units or components into another system, or ignoring or not implementing some features. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interface, indirect coupling or communication connection of units or modules, and may be electrical or other forms.

[0138] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected based on actual needs to achieve the purpose of the solution provided in this embodiment.

[0139] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0140] In the above embodiments of the present application, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0141] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for color temperature correction of an endoscopic image, characterized in that: The endoscope includes at least two preset light sources, each of which emits light of a single preset color temperature. The endoscope image color temperature correction method includes: Acquire a mixed color temperature captured image of the endoscope under at least two of the preset light sources; The mixed color temperature acquisition image is input into a pre-trained endoscopic image color temperature correction model, and a color temperature corrected image is obtained based on the endoscopic image color temperature correction model; wherein the endoscopic image color temperature correction model is based on a first historical acquisition image acquired by the endoscope under a single preset light source and a second historical acquisition image acquired by the endoscope under at least two preset light sources in the same scene as training data for model training. During the model training process, a mapping relationship is learned when the second historical acquisition image is mapped to the first historical acquisition image based on the common biometric features of the first historical acquisition image and the second historical acquisition image; The model training process includes: Performing target segmentation on the first historically collected image and the second historically collected image based on the spectral reflectance of a preset target object to obtain the shared biometric feature; Acquiring a first color feature of the first historically acquired image and a second color feature of the second historically acquired image, wherein the color feature includes a color temperature distribution diagram, specifically by: acquiring a first light source parameter corresponding to the first historically acquired image and a second light source parameter corresponding to the second historically acquired image; determining the first color feature based on the first light source parameter; and determining the second color feature based on the first light source parameter, the second light source parameter, and the second historically acquired image; specifically by: determining a relative distance between at least two of the preset light sources based on the first light source parameter, the second light source parameter, and the second historically collected image; Calculating a spot coverage parameter based on the relative distance; determining a distance attenuation parameter based on the first light source parameter and the second light source parameter; determining, based on the light spot coverage parameter and the distance attenuation parameter, a fusion weight when fusing the color temperature distributions of at least two preset light sources to obtain a color temperature distribution map included in the second color feature; Obtaining color temperature distribution diagrams of at least two of the preset light sources corresponding to the second historical acquisition image; fusing the color temperature distribution maps based on the fusion weight to obtain a second color temperature distribution map as the second color feature; Based on the shared biometric feature, the model parameters of the model to be trained are continuously adjusted so that the model to be trained learns the mapping relationship when mapping the second color feature to the first color feature until the model to be trained converges.

2. The method for color temperature correction of an endoscopic image according to claim 1, wherein: The determining the relative distance between at least two preset light sources based on the first light source parameter, the second light source parameter, and the second historically collected image includes: From the first light source parameters and the second light source parameters, obtain the focal length and pixel size of the first preset light source, and obtain the outer diameter of the second preset light source; Acquire the number of target pixels in the second historical acquisition image; wherein the target pixels are used to represent the outer diameter of the head end of the second preset light source; The relative distance is calculated based on the focal length, the pixel size, the outer diameter, and the number of pixels.

3. The method for color temperature correction of an endoscopic image according to claim 1, wherein: The determining of the distance attenuation parameter based on the first light source parameter and the second light source parameter includes: In the light source parameters, a preset maximum relative distance is obtained; wherein the preset maximum relative distance can be obtained by the relative distance between the preset light sources when the preset light source corresponding to the first historical acquisition image reaches the field of view limit of the preset light source corresponding to the second historical acquisition image; The light spot coverage parameter is calculated based on the relative distance and the maximum relative distance.

4. The method for color temperature correction of an endoscopic image according to claim 1, wherein: The determining of the distance attenuation parameter based on the first light source parameter and the second light source parameter includes: From the first light source parameters and the second light source parameters, obtain a first reference light intensity and a first propagation distance included in the first light source parameters, and a second reference light intensity and a second propagation distance included in the second light source parameters; The distance attenuation parameter is determined based on the first reference light intensity, the first propagation distance, the first reference light intensity, and the second propagation distance.

5. The method for color temperature correction of an endoscopic image according to any one of claims 1 to 4, wherein: The color feature further includes a light intensity distribution diagram, and the determining the second color feature based on the first light source parameter, the second light source parameter, and the second historically collected image further includes: determining a relative distance between at least two of the preset light sources based on the first light source parameter, the second light source parameter, and the second historically collected image; Among the second light source parameters, obtain the reference light intensity and light path attenuation coefficient of each of the preset light sources; A second light intensity distribution diagram is determined as the second color feature based on the reference light intensity, the relative distance, and the light path attenuation coefficient.

6. The method for color temperature correction of an endoscopic image according to claim 1, wherein: The step of continuously adjusting the training parameters of the preset model based on the shared biometric feature so that the preset model learns the mapping relationship when mapping the second color feature to the first color feature includes: constructing a target loss function based on pixel difference information between the first historically collected image and the second historically collected image and color feature differences between the second color features; The model to be trained is trained using the loss function as a constraint; during the model training process, based on the loss function, the model to be trained learns to extract features from the second historically collected image data to obtain a first feature extraction relationship when image features are obtained; Based on the loss function, the to-be-trained model learns to extract features from the second historical color temperature distribution graph, the second historical light intensity distribution graph, and the biological features to obtain a second feature extraction relationship when the environmental features are obtained; Based on the loss function, the model to be trained learns a feature fusion relationship when fusing the image features and the environment features to obtain a fusion feature; Based on the loss function, the model to be trained learns the image output relationship when outputting the color temperature corrected image based on the feature fusion relationship.

7. An endoscope, characterized in that: The endoscope includes at least two preset light sources, at least two imaging units corresponding to the preset light sources, and at least two image processing units; Each of the preset light sources emits light of a single preset color temperature, and the imaging unit is capable of capturing a single color temperature captured image obtained by the light of the single preset color temperature or a mixed color temperature captured image under at least two of the preset light sources; The image processing unit is capable of acquiring the acquired image acquired by the imaging unit. The image processing unit includes a memory and a processor. The memory and the processor are communicatively connected to each other. Computer instructions are stored in the memory. The processor executes the computer instructions to perform the endoscopic image color temperature correction method according to any one of claims 1 to 6.

Images