Endoscopic image brightness correction method, device, equipment and storage medium

Abstract

The present invention relates to the field of medical endoscopic imaging technology, and in particular to an endoscopic image brightness correction method, device, equipment and storage medium. The method comprises: obtaining a brightness image to be corrected collected by an endoscope; performing image decomposition on the brightness image to be corrected to obtain a reflection map and an illumination map; performing image segmentation on the reflection map according to image content attributes of an endoscope application scenario to obtain target areas with different image content attributes; determining the spectral reflectance of the target area based on a pre-calibrated spectral reflectance calibration result of the image content attributes; correcting the reflection map using the spectral reflectance to obtain a corrected reflection map; fusing the corrected reflection map and the illumination map to obtain the brightness corrected image; the present invention uses the image content attributes of the target area to correct the original reflection map to adapt to the application scenario, obtains a more accurate corrected reflection map, and further obtains a brightness corrected image to adapt to the application scenario.

Description

Technical Field

[0001] The present invention relates to the field of medical endoscopic imaging technology, and in particular to an endoscopic image brightness correction method, device, equipment and storage medium. Background Art

[0002] With the vigorous development of endoscopic applications, doctors' demand for high-quality endoscopic imaging is also increasing. In endoscopic application scenarios, especially digestive endoscopy (gastroenteroscopy) and bronchoscopy, the non-uniform distribution of image brightness has always been a pain point for doctors' observation and diagnosis.

[0003] Due to the limitation of the width of human cavity, the size of endoscope lens has strict requirements, resulting in serious non-uniformity of light source and lens. At the same time, human cavity is narrow and soft, resulting in extremely uneven light distribution after the endoscope enters the human body. The difference between bright and dark areas of the image is huge, and medical scenes also have extremely high requirements for the authenticity of the image. Therefore, how to correct the non-uniform brightness of the endoscopic image from the system and restore the image with uniform brightness and true color is an urgent problem to be solved.

[0004] In the related art, in order to solve the problem of huge difference between bright and dark areas and uneven brightness distribution in the above-mentioned image, one method is to perform brightness correction considering the influence of the optical characteristics of the lens in the endoscope camera system on the uniformity of image brightness; in this method, the endoscopic image is usually corrected according to the brightness distribution with bright middle and dark edges, but the illumination distribution of the endoscope in the actual endoscopic application scenario is more complex and changeable, and the brightness correction effect is not good when the above-mentioned general correction method is used without considering the illumination distribution of the endoscope.

[0005] Another method usually decomposes the original image into an illumination map and an original reflection map according to the Retinex theory, and adds the auxiliary means of CLAHE local histogram enhancement. However, this method still focuses on the processing of general images, and the auxiliary means of CLAHE is only a general post-processing contrast enhancement algorithm, which does not consider the characteristics of the endoscope camera system and application scenarios from a system perspective.

[0006] Therefore, there is an urgent need to propose an endoscopic image brightness correction method to solve the technical problem that the endoscopic image brightness correction method in the related art only considers the optical properties of the lens in the endoscopic camera system, but does not consider the application scenario characteristics of the endoscope, resulting in poor brightness correction effect. Summary of the invention

[0007] In view of this, the present invention provides an endoscopic image brightness correction method, device, equipment and storage medium to solve the problem that the endoscopic image brightness correction method in the related art does not take into account the application scenario characteristics of the endoscope, resulting in poor brightness correction effect.

[0008] In a first aspect, the present invention provides an endoscopic image brightness correction method, which includes: obtaining a brightness image to be corrected collected by an endoscope; performing image decomposition on the brightness image to be corrected to obtain a reflection map and an illumination map; performing image segmentation on the reflection map according to image content attributes of an endoscope application scenario to obtain a target area with different image content attributes; determining the spectral reflectance of the target area based on a pre-calibrated spectral reflectance calibration result of the image content attributes; correcting the reflection map using the spectral reflectance to obtain a corrected reflection map; and fusing the corrected reflection map and the illumination map to obtain a brightness corrected image.

[0009] As an exemplary embodiment, the reflectance map is corrected using the spectral reflectance to obtain a corrected reflectance map, including: obtaining a spectral reflectance preset weight corresponding to a spectral reflectance calibration result; correcting the reflectance map based on the spectral reflectance preset weight and the spectral reflectance calibration result to obtain a first corrected reflectance map; and using the first corrected reflectance map as the corrected reflectance map.

[0010] As an exemplary embodiment, the reflection map is corrected by using the spectral reflectance to obtain a corrected reflection map, and the method also includes: obtaining the segmentation confidence and the reflection map noise interference when performing image segmentation on each target area; determining the segmentation confidence influence weight and / or the noise influence weight for correcting the reflection map based on the segmentation confidence and the reflection map noise interference; wherein the segmentation confidence influence weight is proportional to the segmentation confidence, and the noise influence weight is proportional to the reflection map noise interference; correcting the reflection map based on the segmentation confidence influence weight and / or the noise influence weight and the spectral reflectance calibration result to obtain a second corrected reflection map; and fusing the first corrected reflection map and the second corrected reflection map to obtain a corrected reflection map.

[0011] As an exemplary embodiment, the reflection map is corrected by using the spectral reflectance to obtain a corrected reflection map, and the method also includes: obtaining the regional boundary pixels of each target area and the spectral reflectance calibrated smoothing value corresponding to the boundary area; determining the boundary area pixel influence weight for correcting the reflection map based on the regional boundary pixels; correcting the reflection map based on the boundary area pixel influence weight and the spectral reflectance calibrated smoothing value to obtain a third corrected reflection map; and fusing the first corrected reflection map, the second corrected reflection map and the third corrected reflection map to obtain a corrected reflection map.

[0012] As an exemplary embodiment, the endoscopic image brightness correction method also includes: obtaining a spectral reflectance curve of the target object corresponding to each image content attribute; wherein the spectral reflectance curve is obtained by pre-spectral measurement of the target object; determining the spectral power distribution weight corresponding to each wavelength based on the spectral reflectance curve; obtaining three stimulus values ​​based on the spectral reflectance curve, the spectral power distribution weight and a preset standard observer function; converting the three stimulus values ​​to the sRGB color space, and obtaining the correspondence between the image content attributes and the RGB calibration values ​​as the reflectance calibration result.

[0013] As an exemplary embodiment, obtaining an image to be corrected for brightness captured by an endoscope includes: obtaining preset lens correction parameters, preset illumination correction parameters and a captured image of the endoscope; correcting the captured image based on the preset lens correction parameters to obtain a lens correction image; correcting the lens correction image based on the preset illumination correction parameters to obtain an illumination correction image; and using the illumination correction image as the image to be corrected for brightness.

[0014] As an exemplary embodiment, before fusing the corrected reflection map and the illuminance map to obtain a brightness-corrected image, it also includes: obtaining a preset brightness correction process; when brightness correction is performed before the preset brightness correction process is gamma, obtaining a preset gamma correction curve, and correcting the illuminance map based on the preset gamma correction curve.

[0015] In a second aspect, an endoscope image brightness correction device is provided, the device comprising: an acquisition module for acquiring a brightness image to be corrected collected by the endoscope; wherein the brightness image to be corrected is obtained by collecting the image after the endoscope performs lens correction and illumination correction in sequence; an image decomposition module for performing image decomposition on the brightness image to be corrected to obtain a reflection map and an illumination map; an image segmentation module for performing image segmentation on the reflection map according to image content attributes of the endoscope application scenario to obtain target areas with different image content attributes; a spectral reflectance determination module for determining the spectral reflectance of the target area based on a pre-calibrated spectral reflectance calibration result of the image content attributes; a correction module for correcting the reflection map using the spectral reflectance to obtain a corrected reflection map; and a fusion module for fusing the corrected reflection map and the illumination map to obtain a brightness corrected image.

[0016] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the endoscopic image brightness correction method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0017] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the endoscopic image brightness correction method of the first aspect or any corresponding embodiment thereof.

[0018] The present invention provides an endoscope image brightness correction method, device, equipment and storage medium. The endoscope image brightness correction method includes: obtaining a brightness image to be corrected collected by an endoscope; performing image decomposition on the brightness image to be corrected to obtain a reflection map and an illumination map; performing image segmentation on the reflection map according to the image content attributes of the endoscope application scene to obtain a target area with different image content attributes; determining the spectral reflectance of the target area based on a pre-calibrated spectral reflectance calibration result of the image content attribute; correcting the reflection map using the spectral reflectance to obtain a corrected reflection map; fusing the corrected reflection map and the illumination map to obtain a brightness correction image. The above method corrects the reflection map by using the relatively accurate spectral reflectance of the target area determined by the pre-calibrated spectral reflectance calibration result to obtain an accurate corrected reflection map that meets the current application scene of the endoscope and the relative spatial position of the lens, and further adopts the accurate corrected reflection map to fuse with the illumination map to obtain a brightness correction image; the reflectance calibration is combined with the reflection decomposition map to finally obtain an endoscopic imaging with uniform brightness and comfortable viewing. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 is a flow chart of an endoscope image brightness correction method according to an embodiment of the present invention;

[0021] Figure 2 is a structural block diagram of an endoscope image brightness correction device according to an embodiment of the present invention;

[0022] Figure 3 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0024] According to an embodiment of the present invention, an embodiment of an endoscopic image brightness correction method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0025] In this embodiment, a method for calibrating the brightness of an endoscope image is provided. Figure 1 FIG. 1 is a flow chart of a method for calibrating brightness of an endoscope image according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:

[0026] Step S101, obtaining a brightness image to be corrected collected by an endoscope; wherein the brightness image to be corrected is obtained by collecting images after the endoscope performs lens correction and illumination correction in sequence.

[0027] An endoscope is an optical instrument, which is composed of a cold light source lens, a fiber optic conductor, an image transmission system, a screen display system, etc. Among them, the image to be corrected for brightness is the image obtained after the cold light source lens performs image acquisition.

[0028] In the related art, there are strict requirements on the size of the endoscope lens, which leads to serious non-uniformity of the light source and the lens; therefore, in this embodiment, the endoscope is pre-calibrated with lens and illumination, and the brightness image to be corrected is obtained by collecting the image after the endoscope is pre-calibrated with lens and illumination.

[0029] Step S102, performing image decomposition on the image to be corrected for brightness to obtain a reflectance map and an illumination map.

[0030] Based on the Retinex theory, the image signal collected by the endoscope is composed of an illumination map and a reflection map. The illumination map is used to represent the illumination component of the ambient light, and the reflection map is used to represent the reflection component of the target object that carries the image detail information. It is necessary to perform image decomposition on the image to be corrected for brightness in advance to obtain the reflection map and the illumination map.

[0031] In one embodiment, a traditional method based on Retinex theory may be used to decompose the image to be corrected for brightness to obtain a reflectance map and an illumination map; specifically, the traditional method based on Retinex theory may include single-scale Retinex, multi-scale Retinex, and multi-scale adaptive gain Retinex.

[0032] In one embodiment, a deep learning method may be used to decompose the image to be corrected for brightness to obtain a reflectance map and an illumination map; specifically, the deep learning method may include RetinexNet, KinD++, etc.

[0033] Step S103 , performing image segmentation on the reflection image according to the image content attributes of the endoscope application scenario to obtain target areas with different image content attributes.

[0034] In order to adaptively modify the reflection image according to the attributes of the target object later, in this embodiment, the reflection image is segmented in advance according to the image content attributes of the endoscope application scenario to obtain target areas with different image content attributes.

[0035] Among them, application scenarios may include digestive endoscopy (gastroenteroscopy) and bronchoscopy, and image content attributes may include surface background, shallow blood vessels, deep blood vessels, bleeding points, inflammation, etc.

[0036] In one embodiment, traditional image processing means are used to implement image segmentation of the reflectance image; specifically, traditional image processing methods include but are not limited to threshold-based segmentation methods, edge detection segmentation methods, genetic algorithm segmentation methods, and the like.

[0037] In another embodiment, a deep learning method is used to implement image segmentation of the reflectance map; specifically, the deep learning processing method may include, for example, a feature coding-based segmentation method such as a deep residual neural network (Residual Neural Network, ResNet), VGGNet, a region selection-based method such as a fast region convolutional network (Fast Region-based Convolutional Network, Fast RCNN), an image segmentation method based on a recurrent neural network (Recurrent Neural Network, RNN), etc.

[0038] In a specific implementation, image segmentation of the reflection map can be achieved by configuring an image segmentation model based on deep learning offline; specifically, an online end and an offline end of the communication connection can be pre-configured for the endoscope system, the online end obtains the image to be corrected for brightness as a real-time image, the real-time image is input into a pre-configured offline end, image segmentation is performed through the offline end, and the offline end outputs the image segmentation result and returns it to the online end; wherein the offline end is provided with an image segmentation model; exemplarily, when training the image segmentation model, an image data set including reflection maps corresponding to different image labels composed of different image content attributes is pre-constructed, a network structure for identifying different image content attributes is constructed according to the data features of the image data set, and a loss function is determined according to the data features in the image data set, and hyperparameters are trained at the same time, and the network structure, loss function and trained hyperparameters are used as model parameters; after the model parameters are determined, the image data set is input into the model for model training until the model converges.

[0039] Step S104: determining the spectral reflectance of the target area based on the pre-calibrated spectral reflectance calibration result of the image content attribute.

[0040] When light source shines on the surface of an object, the object will selectively reflect electromagnetic waves of different wavelengths. Spectral reflectivity refers to the ratio of the luminous flux reflected by the object in a certain band to the luminous flux incident on the object, which is the essential property of the object's surface. Spectral reflectivity is the representation of the color of the object itself. It not only comprehensively records the color information of the object, but also is a way of representing the material of the object's surface. Spectral reflectivity is an important physical property that is independent of lighting and camera equipment. It represents the reflection ratio of the object surface to light of different wavelengths. Therefore, the true color representation of the target object under standard observer conditions can be calculated by measuring the reflectivity of the target object corresponding to each image content attribute at each wavelength.

[0041] In one embodiment, the spectral reflectance of target objects with different image content attributes may be calibrated in advance to obtain spectral reflectance calibration results, and then the spectral reflectance of the target area may be determined based on the pre-calibrated spectral reflectance calibration results of the image content attributes.

[0042] Step S105, correcting the reflection map using the spectral reflectance to obtain a corrected reflection map.

[0043] Changes in the application scenario of the endoscope and the relative spatial position of the lens will cause changes in the reflection component represented by the target object on the reflection map. Without considering the above changes, the traditional correction method of performing brightness correction on the endoscope by only considering the lens optical parameters of the endoscope will result in inaccurate reflection components represented by the reflection map and poor brightness correction effect. Therefore, in the present invention, the reflection map of the reflection component of the target object representing the image detail information is corrected to consider the differences in the reflection components of the target object caused by different application scenarios from the application scenario of the endoscope. Specifically, in the present invention, the reflection map is corrected by using the relatively accurate spectral reflectance of the target area determined by a pre-calibrated spectral reflectance calibration result to obtain a corrected reflection map of the target object corresponding to different image content attributes in the current application scenario of the endoscope and the relative spatial position of the lens.

[0044] In one embodiment, the influence of the spectral reflectance on the reflection map when correcting the reflection map can be pre-set according to different image content attributes to determine the preset weight of the reflectance calibration rate, and then the reflection map is corrected according to the preset weight of the reflectance calibration, so as to combine the reflectance calibration with the reflection map through the preset weight to obtain a more accurate corrected reflection map.

[0045] In one embodiment, the reflection map can be corrected according to the image segmentation error caused by different image content attributes when performing image segmentation, so as to combine the reflectance calibration with the reflection decomposition map through the image segmentation error to obtain a more accurate corrected reflection map; specifically, as a possible implementation method, the above-mentioned image segmentation error can be considered through the overall impact when performing image segmentation; the overall impact may include the segmentation confidence during image segmentation and the noise interference of the reflection map during segmentation; further, the corresponding correction weight is configured according to the segmentation confidence and the noise interference of the reflection map during segmentation, and then the reflection map is corrected according to the correction weight combined with the reflectance calibration value, so as to combine the reflectance calibration with the reflection decomposition map through the preset weight to obtain a more accurate corrected reflection map.

[0046] As another possible implementation method, the error of the above-mentioned image segmentation can be considered by considering the influence of local aspects when performing image segmentation; the influence of such local aspects may include region boundary pixels, wherein the region boundary pixels are pixels located at the junction of regions in the reflection map, and when being corrected, it is necessary to consider the smoothness with the surrounding pixels, otherwise the image will be prone to fracture after correction; therefore, different preset weights can be set according to the region boundary pixels, and the reflection map can be corrected in combination with the corresponding reflectance calibration value, so that the reflectance calibration can be combined with the reflection decomposition map through the preset weights to obtain a more accurate corrected reflection map.

[0047] As another possible implementation method, a solution combining the above-mentioned multiple implementation methods is adopted to combine the reflectance calibration with the reflection decomposition map through preset weights to obtain a more accurate reflection map; wherein, when considering the error of image segmentation in terms of the overall influence and local influence during image segmentation, different confidence influence weights are set for the influencing factors in the consideration of the overall influence and the local influence, respectively, and the reflection map is further corrected by the confidence influence weight, the preset weight and the reflectance calibration value, so as to combine the reflectance calibration with the reflection decomposition map through the preset weights to obtain a more accurate corrected reflection map.

[0048] The above method corrects the reflection map by using the relatively accurate spectral reflectance of the target area determined by the pre-calibrated spectral reflectance calibration result, and obtains a corrected reflection map that conforms to the target object in the current application scenario of the endoscope and the relative spatial position of the lens.

[0049] Step S106, fusing the corrected reflection image and the illumination image to obtain a brightness corrected image.

[0050] In this embodiment, the corrected reflection map is obtained by correcting the reflection map using the spectral reflectance corresponding to the pre-calibrated spectral reflectance calibration result of the image content attribute. Therefore, the brightness-corrected image conforms to the performance of the target object in the current application scenario of the endoscope and the relative spatial position of the lens. Combining the reflectance calibration with the reflection decomposition map can ultimately obtain endoscopic imaging with uniform brightness and comfortable viewing.

[0051] The endoscopic image brightness correction method of the present embodiment obtains a brightness image to be corrected collected by the endoscope; wherein the brightness image to be corrected is obtained by collecting the image after the endoscope performs lens correction and illumination correction in sequence; the brightness image to be corrected is decomposed to obtain a reflection map and an illumination map; the reflection map is segmented according to the image content attributes of the endoscope application scene to obtain a target area with different image content attributes; the spectral reflectance of the target area is determined based on the pre-calibrated spectral reflectance calibration result of the image content attribute; the reflection map is corrected by the spectral reflectance to obtain a corrected reflection map; the corrected reflection map and the illumination map are fused to obtain a brightness corrected image; the above method corrects the reflection map by the relatively accurate spectral reflectance of the target area determined by the pre-calibrated spectral reflectance calibration result, so as to obtain an accurate corrected reflection map that meets the current application scene of the endoscope and the relative spatial position of the lens, and further the accurate corrected reflection map is fused with the illumination map to obtain a brightness corrected image; the reflectance calibration is combined with the reflection decomposition map to finally obtain endoscopic imaging with uniform brightness and comfortable viewing.

[0052] As an exemplary embodiment, the reflectance map is corrected using the spectral reflectance to obtain a corrected reflectance map, including: obtaining a spectral reflectance preset weight corresponding to a spectral reflectance calibration result; correcting the reflectance map based on the spectral reflectance preset weight and the spectral reflectance calibration result to obtain a first corrected reflectance map; and using the first corrected reflectance map as the corrected reflectance map.

[0053] In this embodiment, the preset spectral reflectance weight corresponding to the spectral reflectance calibration result can be calibrated in advance; specifically, in the experimental scenario, for the spectral reflectance calibration result corresponding to each image content attribute, different spectral reflectance calibration weights from 0 to 1 can be used to correct the reflectance map in turn to obtain the correction result of the reflectance map, and then the spectral reflectance calibration weight of the reflectance calibration result that meets the preset requirements is selected as the preset spectral reflectance weight, and the corresponding relationship of the reflectance calibration weight corresponding to each image content attribute is generated. Among them, the preset requirement can be that when the reflectance map is corrected by using the spectral reflectance calibration weight, the error between the actual spectral reflectance corresponding to the corrected reflectance map and the preset spectral reflectance corresponding to the target object corresponding to the image content attribute of the reflectance map is less than the preset error; further, in actual use, the preset spectral reflectance weight corresponding to the spectral reflectance calibration result is determined by the spectral reflectance calibration result and the above-mentioned corresponding relationship.

[0054] In one embodiment, the reflection map can be corrected using formula (1) to obtain a corrected reflection map:

[0055] R out (x,y)=ω i M+R i (x,y) (1)

[0056] In formula (1), R out (x,y) represents the corrected reflection map, ω i represents the preset weight of the spectral reflectance of the i-th target area, M represents the calibration result of the spectral reflectance of the i-th target area, and R i (x,y) represents the original reflectance map of the i-th target area.

[0057] When the spectral reflectance is used to correct the reflectance map, since the spectral reflectance is obtained from the original reflectance map after image segmentation, the segmentation error generated during image segmentation will also affect the correction of the reflectance map. In the present invention, the influence may include the overall influence of the segmentation confidence during image segmentation and the noise interference of the reflectance map during segmentation and the local influence of the regional boundary pixels. Based on this, in order to consider the overall influence of the segmentation confidence during image segmentation and the noise interference of the reflectance map during segmentation, as an exemplary embodiment, the reflectance map is corrected using the spectral reflectance to obtain a corrected image. The regular reflection map also includes: obtaining the segmentation confidence and reflection map noise interference when performing image segmentation on each target area; determining the segmentation confidence influence weight and / or noise influence weight for correcting the reflection map based on the segmentation confidence and the reflection map noise interference; wherein the segmentation confidence influence weight is proportional to the segmentation confidence, and the noise influence weight is proportional to the reflection map noise interference; correcting the reflection map based on the segmentation confidence influence weight and / or noise influence weight and the spectral reflectance calibration result to obtain a second corrected reflection map; fusing the first corrected reflection map and the second corrected reflection map to obtain a corrected reflection map.

[0058] In one embodiment, the influence of segmentation confidence on the correction of the reflection map can be considered separately; specifically, the segmentation confidence and the reflection map noise interference when performing image segmentation on each target area are obtained; based on the segmentation confidence, the segmentation confidence influence weight for correcting the reflection map is determined; wherein the segmentation confidence influence weight is proportional to the segmentation confidence; based on the segmentation confidence influence weight and the spectral reflectance calibration result, the reflection map is corrected to obtain a second corrected reflection map; the first corrected reflection map and the second corrected reflection map are fused to obtain a corrected reflection map.

[0059] In one embodiment, the influence of the reflection map noise interference on the reflection map correction can be considered separately; specifically, the reflection map noise interference when each target area is segmented is obtained; based on the reflection map noise interference, the noise influence weights for correcting the reflection map are determined respectively; wherein the noise influence weight is proportional to the reflection map noise interference; based on the noise influence weight and the spectral reflectance calibration result, the reflection map is corrected to obtain a second corrected reflection map; and the first corrected reflection map and the second corrected reflection map are fused to obtain a corrected reflection map.

[0060] In another embodiment, the influence of segmentation confidence and reflection map noise interference on reflection map correction can be considered simultaneously; specifically, the segmentation confidence and reflection map noise interference of each target area during image segmentation are obtained; based on the segmentation confidence and reflection map noise interference, the segmentation confidence influence weight and noise influence weight for correcting the reflection map are determined respectively; wherein the segmentation confidence influence weight is proportional to the segmentation confidence, and the noise influence weight is proportional to the reflection map noise interference; based on the segmentation confidence influence weight and the noise influence weight and the spectral reflectance calibration result, the reflection map is corrected to obtain a second corrected reflection map; the first corrected reflection map and the second corrected reflection map are fused to obtain a corrected reflection map.

[0061] As a possible implementation method, in the experimental scenario, for each segmentation confidence and reflectance map noise interference of the target area corresponding to each image content attribute, different segmentation confidence influence weights and noise influence weights from 0 to 1 are used to correct the reflectance map separately or together to obtain the reflectance map correction result, and then the segmentation confidence and reflectance map noise interference of the reflectance calibration result that meet the preset requirements are selected as the spectral reflectance preset weight, and the corresponding relationship between the segmentation confidence and the segmentation confidence influence weight and the corresponding relationship between the reflectance map noise interference and the noise influence weight, as well as the corresponding relationship between the segmentation confidence, the reflectance map noise and the segmentation confidence influence weight, and the noise influence weight are generated. The preset requirement can be that when the reflectance map is corrected by using the segmentation confidence influence weight and / or the noise influence weight, the error between the actual spectral reflectance corresponding to the corrected reflectance map and the preset spectral reflectance corresponding to the target object corresponding to the image content attribute of the reflectance map is less than the preset error; further, in actual use, the spectral reflectance preset weight corresponding to the spectral reflectance calibration result is determined by the segmentation confidence influence weight and / or the noise influence weight and the above-mentioned corresponding relationship.

[0062] In one embodiment, the reflection map can be corrected using formula (2) to obtain a corrected reflection map:

[0063] Rout(x,y)=αi*βi*M+(1-αi*βi) *Ri(x,y) (2)

[0064] In formula (2), R out (x, y) represents the corrected reflection map, α i represents the segmentation confidence influence weight of the i-th target region, β i represents the noise impact weight of the i-th target area, M represents the spectral reflectance calibration result, R i (x,y) represents the original reflectance map of the i-th target area.

[0065] As a possible implementation method, when obtaining the second corrected reflection map, the first corrected reflection map and the second corrected reflection map are fused to obtain the corrected reflection map; when fusing, pre-configured fusion weights can be used for fusion; wherein the sum of the fusion weight of the first corrected reflection map and the fusion weight of the second corrected reflection map is 1, and the fusion weight of the second corrected reflection map is inversely proportional to each segmentation confidence and the reflection map noise interference.

[0066] As an exemplary embodiment, the reflection map is corrected by using the spectral reflectance to obtain a corrected reflection map, and the method also includes: obtaining the regional boundary pixels of each target area and the spectral reflectance calibrated smoothing value corresponding to the boundary area; determining the boundary area pixel influence weight for correcting the reflection map based on the regional boundary pixels; correcting the reflection map based on the boundary area pixel influence weight and the spectral reflectance calibrated smoothing value to obtain a third corrected reflection map; and fusing the first corrected reflection map, the second corrected reflection map and the third corrected reflection map to obtain a corrected reflection map.

[0067] In this embodiment, the regional boundary pixels are located at the junction of the regions in the reflection map. When being corrected, it is necessary to consider the smoothing with the surrounding pixels, otherwise the image is prone to discontinuity after correction; at the same time, the boundary pixels are located on the transition zone, and the reflectivity calibration value of a certain area must also be smoothly interpolated with the reflectivity calibration value of the neighborhood; therefore, when considering the influence of the regional boundary pixels on the reflection map, the regional boundary pixels of each target area and the spectral reflectivity calibration smoothing value corresponding to the boundary area are obtained, and the spectral reflectivity calibration smoothing value is obtained by smoothing the reflectivity calibration value and the reflectivity calibration value of the neighborhood; the reflection map is further corrected based on the boundary area pixel influence weight and the spectral reflectivity calibration smoothing value to obtain a third corrected reflection map.

[0068] In one embodiment, after obtaining the third corrected reflection map, the first corrected reflection map, the second corrected reflection map and the third corrected reflection map are fused to obtain a corrected reflection map; wherein the sum of the fusion weight of the first corrected reflection map, the fusion weight of the second corrected reflection map and the fusion weight of the third corrected reflection map is 1, and the fusion weight of the second corrected reflection map is inversely proportional to each segmentation confidence and the reflection map noise interference.

[0069] In one embodiment, the reflection map can be corrected using formula (3) to obtain a corrected reflection map:

[0070] R out (x,y)=α i *β i *(ψ i *M smooth +(1-ψ i ) *M +(1-α i *β i ) *R i(x,y)(3)

[0071] In formula (3), R out (x, y) represents the third corrected reflection map, α i represents the segmentation confidence influence weight of the i-th target region, β i represents the noise impact weight of the i-th target area, ψ i represents the pixel influence weight of the boundary area of ​​the i-th target area, M smooth represents the spectral reflectance calibration smoothing value of the i-th target area, M represents the spectral reflectance calibration result, R i (x,y) represents the original reflection image.

[0072] As an exemplary embodiment, the endoscopic image brightness correction method also includes: obtaining a spectral reflectance curve of the target object corresponding to each image content attribute; wherein the spectral reflectance curve is obtained by pre-spectral measurement of the target object; determining the spectral power distribution weight corresponding to each wavelength based on the spectral reflectance curve; obtaining three stimulus values ​​based on the spectral reflectance curve, the spectral power distribution weight and a preset standard observer function; converting the three stimulus values ​​to the sRGB color space, and obtaining the correspondence between the image content attributes and the RGB calibration values ​​as the reflectance calibration result.

[0073] In this embodiment, the spectral reflectance curve can be obtained by setting up an experimental scene and performing spectral measurement on the target object; specifically, the experimental scene is set up, and a sample of the target object is obtained. The reflectance of the target object at different wavelengths in the visible light wavelength range (approximately 400nm to 700nm) can be measured by a spectrometer, and discrete sampling can be performed at a sampling interval of N nm, and the samples can be fitted into a continuous spectral reflectance curve.

[0074] Further, after obtaining the spectral reflectance curve, the spectral power distribution weight corresponding to each wavelength is determined based on the spectral reflectance curve;

[0075] Furthermore, the spectral reflectance is multiplied by the corresponding standard observer function; wherein, the standard observer function can select two standard observer modes, CIE 1931 and CIE 1964, defined by the International Commission on Illumination, which represent the average sensitivity of human vision to different wavelengths as a way of expressing the perceived color of standardized objects.

[0076] Furthermore, the result of multiplying the spectral reflectance by the corresponding standard observer function is multiplied by the spectral power distribution weight at the corresponding wavelength, and the whole wavelength is integrated to obtain the XYZ tristimulus value, where the Y value represents the brightness, and X and Z represent the chromaticity.

[0077] Furthermore, since XYZ is a device-independent color space, it is converted into the corresponding sRGB color space according to the setting of the current camera system, and the corresponding RGB calibration value is obtained.

[0078] As an exemplary embodiment, preset lens correction parameters, preset illumination correction parameters and a captured image of the endoscope are obtained; the captured image is corrected based on the preset lens correction parameters to obtain a lens correction image; the lens correction image is corrected based on the preset illumination correction parameters to obtain a illumination correction image; and the illumination correction image is used as the image to be brightness corrected.

[0079] In one embodiment, the preset lens correction parameters are obtained by lens non-uniformity correction; specifically, the processing flow of lens body non-uniformity correction is as follows:

[0080] Step 1: Turn off the endoscope illumination source, and in an external environment with uniform illumination, aim the lens at a uniform grayscale panel, fix the shutter and gain parameters of the camera system, and obtain the endoscope image to be corrected. The external environment with uniform illumination can be a light box.

[0081] Step 2: According to the brightness information of the image to be corrected, a set of lens correction parameters is determined, and the set of parameters represents the gain amplitude of the corresponding area or pixel. The specific method for determining the calibration parameters includes at least one of a pixel-by-pixel method, a grid interpolation method, and a concentric circle method.

[0082] As a possible implementation manner, after obtaining the group of lens correction parameters, the lens correction parameters are integrated into the ISP Pipeline of the camera system as preset lens correction parameters, and lens correction processing is performed on the real-time target image.

[0083] In one embodiment, the preset illumination correction parameters are obtained by illumination non-uniformity correction; specifically, the processing flow of illumination non-uniformity correction is as follows:

[0084] Step 1: Turn on the endoscope illumination light source, fix the light source illumination, and in a dark external environment, aim the lens at a uniform grayscale panel to obtain an endoscope image corrected based on lens non-uniformity. The dark external environment may be a black box.

[0085] Step 2: According to the brightness information of the image to be corrected, determine a set of illumination correction parameters, which represent the gain amplitude of the corresponding area or pixel. The specific method for determining the calibration parameters includes but is not limited to the pixel-by-pixel method, the grid interpolation method, the concentric circle method, etc.

[0086] Step 3: Adjust the illumination of the light source, and repeat the above operation until all preset light source illumination groups are traversed to obtain multiple groups of illumination correction parameters.

[0087] As a possible implementation, after obtaining the illumination correction parameters, the illumination correction parameters are integrated into the ISP Pipeline of the camera system as preset lens illumination correction parameters, and lens correction processing is performed on the real-time target image.

[0088] The solution in the present invention, when fusing the corrected reflection map and the illuminance map, can support both linear RGB domain processing and nonlinear RGB domain processing, that is, it supports brightness correction before gamma and also supports brightness correction after gamma; therefore, as an exemplary embodiment, before fusing the corrected reflection map and the illuminance map to obtain the brightness corrected image, it also includes: obtaining a preset brightness correction process; when the preset brightness correction process is to perform brightness correction before gamma, obtaining a preset gamma correction curve, and correcting the illuminance map based on the preset gamma correction curve.

[0089] In one embodiment, the following formula may be used to implement brightness correction before gamma:

[0090] I out (x,y)=(R out (x,y)﹡L(x,y)) γ (4)

[0091] In formula (4), I out (x, y) represents the brightness correction image, R out (x,y) represents the corrected reflection map, L(x,y) represents the illumination map, and γ is the pre-configured gamma correction curve parameter.

[0092] As an exemplary embodiment, the following formula may be used to implement gamma and then perform brightness correction:

[0093] I out (x,y)=R out (x,y)﹡L(x,y) (5)

[0094] In formula (5), I out (x, y) represents the brightness correction image, R out (x,y) represents the corrected reflection map, and L(x,y) represents the illumination map.

[0095] This embodiment provides an endoscope image brightness correction device, such as Figure 2 As shown, including:

[0096] The acquisition module 201 is used to acquire the image to be corrected for brightness acquired by the endoscope; wherein the image to be corrected for brightness is acquired by the endoscope after performing lens correction and illumination correction in sequence;

[0097] An image decomposition module 202 is used to decompose the image to be corrected for brightness to obtain a reflectance map and an illumination map;

[0098] An image segmentation module 203 is used to segment the reflection image according to the image content attributes of the endoscope application scenario to obtain target areas with different image content attributes;

[0099] A spectral reflectance determination module 204, configured to determine the spectral reflectance of the target area based on a pre-calibrated spectral reflectance calibration result of an image content attribute;

[0100] A correction module 205 is used to correct the reflection map using the spectral reflectance to obtain a corrected reflection map;

[0101] The fusion module 206 is used to fuse the corrected reflection image and the illumination image to obtain a brightness-corrected image.

[0102] It should be noted here that the examples and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the contents disclosed in the above embodiments.

[0103] It should be noted that the above modules as part of the device can be implemented by software or hardware, wherein the hardware environment includes a network environment.

[0104] An embodiment of the present invention also provides a computer device, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus, the memory is used to store a computer program; the processor is used to execute a method in any of the above embodiments by running the computer program stored in the memory.

[0105] Figure 3 is a structural block diagram of an optional computer device according to an embodiment of the present application, such as Figure 3 As shown, it includes a processor 10, a communication interface 20, a memory 30 and a communication bus 40, wherein the processor 10, the communication interface 20 and the memory 30 communicate with each other through the communication bus 40, wherein:

[0106] A memory 30, for storing computer programs;

[0107] The processor 10 is used to implement the method for constructing a disease database as described in any of the above embodiments when executing the computer program stored in the memory 30.

[0108] Optionally, in this embodiment, the communication bus may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The communication bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0109] The communication interface is used for communication between the above-mentioned computer device and other devices.

[0110] The memory may include RAM, or may include non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0111] The above-mentioned processor can be a general-purpose processor, which can include but not be limited to: CPU (Central Processing Unit), NP (Network Processor), etc.; it can also be DSP (Digital Signal Processing), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0112] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments, and this embodiment will not be described in detail here.

[0113] It can be understood by those skilled in the art that Figure 3 The structure shown is for illustration only. The device for implementing any one of the methods in the above embodiments may be a terminal device, which may be a smart phone (such as an Android phone, an IOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (Mobile Internet Devices, MID), a PAD, or other terminal devices. Figure 3 It does not limit the structure of the above electronic device. For example, the terminal device may also include Figure 3 More or fewer components (such as network interfaces, display devices, etc.) shown in, or with Figure 3Different configurations shown.

[0114] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, which can include: a flash drive, ROM, RAM, a magnetic disk or an optical disk, etc.

[0115] As an exemplary embodiment, the present application also provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute any one of the method steps of the present embodiment when running.

[0116] Optionally, in this embodiment, the above-mentioned storage medium can be used to execute the program code of the method steps of the embodiment of the present application.

[0117] Optionally, in this embodiment, the storage medium may be located on at least one network device among a plurality of network devices in the network shown in the above embodiment.

[0118] Optionally, in this embodiment, the storage medium is configured to store data for executing the method in the above embodiment.

[0119] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments, which will not be described in detail in this embodiment.

[0120] Optionally, in this embodiment, the storage medium may include, but is not limited to, various media that can store program codes, such as a USB flash drive, a ROM, a RAM, a mobile hard disk, a magnetic disk, or an optical disk.

[0121] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0122] If the integrated units in the above embodiments are implemented in the form of software functional units and sold or used as independent products, they can be stored in the above computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling one or more computer devices (which may be personal computers, servers or network devices, etc.) to execute all or part of the steps of the method in the above embodiments.

[0123] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0124] In the several embodiments provided in the present application, it should be understood that the disclosed client can be implemented in other ways. Among them, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0125] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution provided in this embodiment.

[0126] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0127] The above are only preferred implementations of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

[0128] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0129] The above are only preferred implementations of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for calibrating endoscopic image brightness, characterized in that: The endoscopic image brightness correction method comprises: Acquire the image to be corrected for brightness collected by the endoscope; Decomposing the image to be corrected for brightness to obtain a reflection map and an illumination map; Performing image segmentation on the reflection image according to the image content attributes of the endoscope application scenario to obtain target areas with different image content attributes; Determining the spectral reflectance of the target area based on a pre-calibrated spectral reflectance calibration result of the image content attribute; Correcting the reflection map using the spectral reflectance to obtain a corrected reflection map; The method of correcting the reflection map by using the spectral reflectance to obtain a corrected reflection map includes: Obtaining a preset spectral reflectance weight corresponding to the spectral reflectance calibration result; wherein the preset spectral reflectance weight can be obtained through calibration; the calibration process is: for the spectral reflectance calibration result corresponding to each of the image content attributes, sequentially using different spectral reflectance calibration weights from 0 to 1 to correct the reflectance map to obtain a corrected reflectance map result; based on the corrected reflectance map result, selecting a spectral reflectance calibration weight that meets a preset requirement as the preset spectral reflectance weight, and generating a corresponding relationship between the reflectance calibration weights corresponding to each of the image content attributes; wherein the preset requirement is that when the spectral reflectance calibration weight is used to correct the reflectance map, the error between the actual spectral reflectance corresponding to the corrected reflectance map and the preset spectral reflectance corresponding to the target object corresponding to the image content attribute of the reflectance map is less than a preset error; determining the preset spectral reflectance weight corresponding to the spectral reflectance calibration result based on the spectral reflectance calibration result and the corresponding relationship; Correcting the reflection map based on the preset spectral reflectance weight and the spectral reflectance calibration result to obtain a first corrected reflection map; using the first corrected reflection image as the corrected reflection image; The corrected reflection map and the illumination map are fused to obtain a brightness-corrected image.

2. The endoscopic image brightness correction method according to claim 1, characterized in that: The method of correcting the reflection map by using the spectral reflectance to obtain a corrected reflection map further includes: Obtaining the segmentation confidence and reflection image noise interference when performing image segmentation on each of the target areas; Based on the segmentation confidence and the reflection image noise interference, respectively determine the segmentation confidence influence weight and / or the noise influence weight for correcting the reflection image; wherein the segmentation confidence influence weight is proportional to the segmentation confidence, and the noise influence weight is proportional to the reflection image noise interference; Correcting the reflection map based on the segmentation confidence influence weight and / or the noise influence weight and the spectral reflectance calibration result to obtain a second corrected reflection map; The first corrected reflection map and the second corrected reflection map are fused to obtain the corrected reflection map.

3. The endoscopic image brightness correction method according to claim 2, characterized in that: The method of correcting the reflection map by using the spectral reflectance to obtain a corrected reflection map further includes: Obtaining the region boundary pixels of each target region and the spectral reflectance calibration smoothing value corresponding to the boundary region; Determining the influence weight of the boundary area pixels for correcting the reflection image based on the area boundary pixels; Correcting the reflection map based on the pixel influence weight of the boundary area and the spectral reflectance calibration smoothing value to obtain a third corrected reflection map; The first corrected reflection map, the second corrected reflection map and the third corrected reflection map are fused to obtain the corrected reflection map.

4. The endoscopic image brightness correction method according to claim 1, characterized in that: The endoscope image brightness correction method further comprises: Acquire a spectral reflectance curve of the target object corresponding to each of the image content attributes; wherein the spectral reflectance curve is obtained by pre-spectrally measuring the target object; Determine the spectral power distribution weight corresponding to each wavelength based on the spectral reflectance curve; Obtaining tristimulus values ​​based on the spectral reflectance curve, the spectral power distribution weight and a preset standard observer function; The tristimulus values ​​are converted into the sRGB color space, and the corresponding relationship between the image content attribute and the RGB calibration value is obtained as the reflectance calibration result.

5. The endoscopic image brightness correction method according to claim 1, characterized in that: The step of obtaining the image to be corrected for brightness collected by the endoscope comprises: Acquiring preset lens correction parameters, preset illumination correction parameters and a captured image of the endoscope; Correcting the captured image based on the preset lens correction parameters to obtain a lens correction image; Correcting the lens correction image based on the preset illumination correction parameters to obtain an illumination correction image; The illumination-corrected image is used as the image to be corrected for brightness.

6. The endoscopic image brightness correction method according to claim 1, characterized in that: Before fusing the corrected reflection map and the illumination map to obtain the brightness-corrected image, the method further includes: Get the preset brightness correction process; When the preset brightness correction process is to perform brightness correction before gamma, a preset gamma correction curve is obtained, and the illumination map is corrected based on the preset gamma correction curve.

7. An endoscope image brightness correction device, characterized in that: The device comprises: An acquisition module is used to acquire a brightness image to be corrected acquired by an endoscope; wherein the brightness image to be corrected is acquired by acquiring the image after the endoscope performs lens correction and illumination correction in sequence; An image decomposition module, used for performing image decomposition on the image to be corrected for brightness to obtain a reflectance map and an illumination map; An image segmentation module, used to perform image segmentation on the reflection image according to the image content attributes of the endoscope application scenario to obtain target areas with different image content attributes; a spectral reflectance determination module, configured to determine the spectral reflectance of the target area based on a pre-calibrated spectral reflectance calibration result of the image content attribute; A correction module, used to correct the reflection map by using the spectral reflectivity to obtain a corrected reflection map; wherein, the correction of the reflection map by using the spectral reflectivity to obtain a corrected reflection map includes: obtaining a spectral reflectivity preset weight corresponding to the spectral reflectivity calibration result; wherein, the spectral reflectivity preset weight can be obtained through calibration; the calibration process is: for the spectral reflectivity calibration result corresponding to each of the image content attributes, the reflection map is corrected using different spectral reflectivity calibration weights from 0 to 1 in sequence to obtain a reflection map correction result; based on the reflection map correction result, a spectral reflectivity calibration weight that meets the preset requirements is selected as the spectral reflectivity preset weight , and generate a corresponding relationship between the reflectivity calibration weights corresponding to each of the image content attributes; wherein the preset requirement is that when the reflectivity map is corrected by using the spectral reflectivity calibration weights, the error between the actual spectral reflectivity corresponding to the corrected reflectivity map and the preset spectral reflectivity corresponding to the target object corresponding to the image content attribute of the reflectivity map is less than a preset error; based on the spectral reflectivity calibration result and the corresponding relationship, the spectral reflectivity preset weight corresponding to the spectral reflectivity calibration result is determined; based on the spectral reflectivity preset weight and the spectral reflectivity calibration result, the reflectivity map is corrected to obtain a first corrected reflectivity map; the first corrected reflectivity map is used as the corrected reflectivity map A fusion module is used to fuse the corrected reflection image and the illumination image to obtain the brightness correction image.

8. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the endoscopic image brightness correction method according to any one of claims 1 to 6 by executing the computer instructions.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the endoscopic image brightness correction method according to any one of claims 1 to 6.

Images