Endoscopic image brightness correction method and apparatus, device, and storage medium

Abstract

The present invention relates to the technical field of medical endoscopic imaging, and specifically relates to an endoscopic image brightness correction method and apparatus, a device, and a storage medium. The method comprises: acquiring an image collected by an endoscope and having brightness to be corrected; performing image decomposition on said image to obtain a reflection image and an illuminance image; performing image segmentation on the reflection image on the basis of image content attributes of an endoscope application scenario to obtain target regions having different image content attributes; determining spectral reflectance of the target regions on the basis of pre-calibrated spectral reflectance calibration results of the image content attributes; using the spectral reflectance to correct the reflection image to obtain a corrected reflection image; and fusing the corrected reflection image and the illuminance image to obtain a brightness-corrected image. In the present invention, the image content attributes of the target regions are used to correct the original reflection map to adapt to the application scenario, so as to obtain a more accurate corrected reflection map, thereby obtaining a brightness-corrected image adapted to the application scenario.

Description

Endoscope image brightness correction method, device, equipment and storage medium TECHNICAL FIELD

[0001] The present application relates to the medical endoscope imaging technical field, in particular to an endoscope image brightness correction method, device, equipment and storage medium. BACKGROUND

[0002] With the vigorous development of endoscope application, doctors' demand for high-quality imaging of endoscopes is also increasing. In endoscope application scenarios, especially in digestive endoscopy (gastrointestinal endoscopy) and bronchoscopy, the non-uniform distribution of image brightness is always a pain point for doctors to observe and diagnose.

[0003] Due to the limitation of the width of the human body cavity, the size of the endoscope lens has strict requirements, resulting in serious non-uniformity of the light source and the lens. At the same time, the human body cavity is long and soft, so that after the endoscope enters the human body, the light distribution is extremely uneven, and the bright and dark areas of the image have a huge difference. In addition, the medical scene also has a high requirement for the authenticity of the image. Therefore, how to correct the non-uniform brightness of the endoscope image from the system and restore the image with uniform brightness and true color is a problem to be solved.

[0004] In related technologies, in view of the above problems of large difference between bright and dark areas of the image and uneven brightness distribution, one method is to consider the influence of the optical characteristics of the lens in the endoscope camera system on the uniformity of the image brightness for brightness correction. In this method, the endoscope image is usually corrected according to the brightness distribution of the middle bright and edge dark, while the illumination distribution of the endoscope in the actual application scenario is more complex and variable. If the illumination distribution of the endoscope is not considered, the brightness correction effect is not good by using the above general correction method.

[0005] Another method usually decomposes the original image into an illumination map and an original reflection map according to the Retinex theory, and increases the auxiliary means of CLAHE local histogram enhancement, but this method still focuses on the processing of general images, and the auxiliary means of CLAHE is only a general post-processing contrast enhancement algorithm, and does not consider the characteristics of the endoscope camera system and the application scenario from the system perspective.

[0006] Therefore, it is urgent to propose an endoscope image brightness correction method to solve the technical problem of poor brightness correction effect caused by only considering the optical characteristics of the lens in the endoscope camera system and not considering the characteristics of the application scenario of the endoscope in the related art.

[0007] SUMMARY

[0008] Therefore, the present application provides an endoscope image brightness correction method, device, equipment and storage medium to solve the problem of poor brightness correction effect caused by the fact that the related art endoscope image brightness correction method does not consider the application scene characteristics of the endoscope.

[0009] In a first aspect, the present application provides an endoscope image brightness correction method, which includes: obtaining a brightness to-be-corrected image collected by an endoscope; performing image decomposition on the brightness to-be-corrected image to obtain a reflection map and an illumination map; performing image segmentation on the reflection map according to the image content attribute of the application scene of the endoscope to obtain a target region with different image content attributes; determining the spectral reflectance of the target region based on the pre-labeled spectral reflectance labeling result of the image content attribute; correcting the reflection map using the spectral reflectance to obtain a corrected reflection map; and fusing the corrected reflection map and the illumination map to obtain a brightness corrected image.

[0010] As an exemplary embodiment, the correction of the reflection map using the spectral reflectance to obtain a corrected reflection map includes: obtaining a spectral reflectance preset weight corresponding to the spectral reflectance labeling result; correcting the reflection map based on the spectral reflectance preset weight and the spectral reflectance labeling result to obtain a first corrected reflection map; and taking the first corrected reflection map as the corrected reflection map.

[0011] As an exemplary embodiment, the correction of the reflection map using the spectral reflectance to obtain a corrected reflection map further includes: obtaining a segmentation confidence when each target region is segmented; determining a segmentation confidence influence weight and / or a noise influence weight for correcting the reflection map based on the segmentation confidence and the noise interference of the reflection map, respectively; wherein the segmentation confidence influence weight is proportional to the segmentation confidence, and the noise influence weight is proportional to the noise interference of the reflection map; correcting the reflection map based on the segmentation confidence influence weight and / or the noise influence weight and the spectral reflectance labeling result to obtain a second corrected reflection map; and fusing the first corrected reflection map and the second corrected reflection map to obtain the corrected reflection map.

[0012] As an exemplary embodiment, the correction of the reflection map using the spectral reflectance to obtain a corrected reflection map further includes: obtaining a spectral reflectance labeling smoothing value corresponding to the region boundary pixels and the boundary region of each target region; determining a boundary region pixel influence weight for correcting the reflection map based on the region boundary pixels; correcting the reflection map based on the boundary region pixel influence weight and the spectral reflectance labeling smoothing value to obtain a third corrected reflection map; and fusing the first corrected reflection map, the second corrected reflection map and the third corrected reflection map to obtain the corrected reflection map.

[0013] As an exemplary embodiment, the endoscope image brightness correction method further comprises: obtaining a spectral reflectance curve of a target object corresponding to each image content attribute; wherein the spectral reflectance curve is obtained by pre-spectrometric measurement on the target object; determining a spectral power distribution weight corresponding to each wavelength based on the spectral reflectance curve; obtaining tristimulus values based on the spectral reflectance curve, the spectral power distribution weight, and a preset standard observer function; and converting the tristimulus values to an sRGB color space to obtain a corresponding relationship between the image content attribute and an RGB calibration value as a reflectance calibration result.

[0014] As an exemplary embodiment, the endoscope image brightness correction method further comprises: obtaining a spectral reflectance curve of a target object corresponding to each image content attribute; wherein the spectral reflectance curve is obtained by pre-spectrometric measurement on the target object; determining a spectral power distribution weight corresponding to each wavelength based on the spectral reflectance curve; obtaining tristimulus values based on the spectral reflectance curve, the spectral power distribution weight, and a preset standard observer function; and converting the tristimulus values to an sRGB color space to obtain a corresponding relationship between the image content attribute and an RGB calibration value as a reflectance calibration result.

[0015] As an exemplary embodiment, the endoscope image brightness correction method further comprises: obtaining a spectral reflectance curve of a target object corresponding to each image content attribute; wherein the spectral reflectance curve is obtained by pre-spectrometric measurement on the target object; determining a spectral power distribution weight corresponding to each wavelength based on the spectral reflectance curve; obtaining tristimulus values based on the spectral reflectance curve, the spectral power distribution weight, and a preset standard observer function; and converting the tristimulus values to an sRGB color space to obtain a corresponding relationship between the image content attribute and an RGB calibration value as a reflectance calibration result.

[0016] In a second aspect, an endoscope image brightness correction device is provided, which comprises: an acquisition module configured to acquire a brightness image to be corrected collected by an endoscope; wherein the brightness image to be corrected is obtained by image collection after lens correction and illumination correction of the endoscope; an image decomposition module configured to perform image decomposition on the brightness image to be corrected to obtain a reflectance image and an illumination image; an image segmentation module configured to perform image segmentation on the reflectance image according to image content attributes of an application scene of the endoscope to obtain target regions with different image content attributes; a spectral reflectance determination module configured to determine spectral reflectance of the target regions based on pre-calibrated spectral reflectance calibration results of the image content attributes; a correction module configured to correct the reflectance image by using the spectral reflectance to obtain a corrected reflectance image; and a fusion module configured to fuse the corrected reflectance image and the illumination image to obtain a brightness correction image.

[0017] In a third aspect, the present application provides a computer device, which comprises a memory and a processor, the memory and the processor are communicatively connected with each other, the memory stores computer instructions, and the processor executes the computer instructions to perform the endoscope image brightness correction method of the first aspect or any of the corresponding embodiments thereof.

[0018] In a fourth aspect, the present application provides a computer readable storage medium, which stores computer instructions for causing a computer to execute the endoscope image brightness correction method of the first aspect or any of the corresponding embodiments thereof.

[0019] The present application provides an endoscope image brightness correction method, device, equipment and storage medium. The endoscope image brightness correction method comprises: acquiring a brightness image to be corrected collected by an endoscope; performing image decomposition on the brightness image to be corrected to obtain a reflection image and an illumination image; performing image segmentation on the reflection image according to image content attributes of an application scene of the endoscope to obtain a target region with different image content attributes; determining spectral reflectance of the target region based on pre-labeled spectral reflectance labeling results of the image content attributes; correcting the reflection image by using the spectral reflectance to obtain a corrected reflection image; and fusing the corrected reflection image and the illumination image to obtain a brightness correction image. The above method corrects the reflection image by using the relatively accurate spectral reflectance of the target region determined by the pre-labeled spectral reflectance labeling results, to obtain an accurate corrected reflection image of the target object under the current application scene of the endoscope and the relative spatial position of the lens, and further fuses the accurate corrected reflection image and the illumination image to obtain the brightness correction image. The combination of the reflectance labeling and the reflection decomposition image can finally obtain an endoscope imaging with uniform brightness and comfortable visual perception. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the specific embodiments of the present application or the technical solutions in the prior art, the following will briefly introduce the drawings needed to be used in the specific embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0021] Fig. 1 is a flow diagram of an endoscope image brightness correction method according to an embodiment of the present application;

[0022] Fig. 2 is a structural block diagram of an endoscope image brightness correction device according to an embodiment of the present application;

[0023] Fig. 3 is a hardware structure diagram of a computer device according to an embodiment of the present application. DETAILED DESCRIPTION

[0024] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative efforts belong to the protection scope of the present application.

[0025] According to the embodiments of the present application, an endoscope image brightness correction method embodiment is provided. It should be noted that the steps shown in the flowchart of the drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that here.

[0026] In the present embodiment, an endoscope image brightness correction method is provided. FIG. 1 is a flowchart of an endoscope image brightness correction method according to an embodiment of the present application. As shown in FIG. 1, the flow includes the following steps:

[0027] In step S101, a brightness to-be-corrected image collected by an endoscope is obtained. The brightness to-be-corrected image is obtained by image collection after lens correction and illumination correction of the endoscope.

[0028] The endoscope is an optical instrument composed of a cold light source lens, a fiber optic line, an image transmission system, a screen display system, etc. The brightness to-be-corrected image is the image obtained after image collection by the cold light source lens.

[0029] In the related art, the size of the endoscope lens has strict requirements, resulting in serious non-uniformity of the light source and the lens. Therefore, in the present embodiment, the endoscope is pre-corrected in sequence for lens correction and illumination correction, and the brightness to-be-corrected image obtained by image collection is obtained by image collection after lens correction and illumination correction of the endoscope.

[0030] In step S102, the brightness to-be-corrected image is decomposed to obtain a reflection map and an illumination map.

[0031] Based on the Retinex theory, the image signal collected by the endoscope is composed of an illumination map and a reflection map. The illumination map is used to represent the illumination component of the ambient light, and the reflection map is used to represent the reflection component of the target object carrying image detail information. The brightness to-be-corrected image needs to be pre-decomposed to obtain the reflection map and the illumination map.

[0032] In an embodiment, a traditional method based on Retinex theory can be used to perform image decomposition on the brightness image to be corrected to obtain a reflectance map and an illumination map; specifically, the traditional method based on Retinex theory can include single-scale Retinex, multi-scale Retinex, and multi-scale adaptive gain Retinex, etc.

[0033] In an embodiment, a deep learning method can be used to perform image decomposition on the brightness image to be corrected to obtain a reflectance map and an illumination map; specifically, the deep learning method can include RetinexNet, KinD++, etc.

[0034] Step S103: performing image segmentation on the reflectance map according to image content attributes of the endoscope application scenario to obtain target regions with different image content attributes.

[0035] In order to subsequently adaptively correct the reflectance map according to the attributes of the target object, in the embodiment, the reflectance map is pre- segmented according to image content attributes of the endoscope application scenario to obtain target regions with different image content attributes.

[0036] The application scenario can include a digestive endoscope (gastrointestinal endoscope) and a bronchoscope, etc., and the image content attributes can include a superficial background, a shallow blood vessel, a deep blood vessel, a bleeding point, inflammation, etc.

[0037] In an embodiment, a traditional image processing method is used to perform image segmentation on the reflectance map; specifically, the traditional image processing method can include but is not limited to a threshold segmentation method, an edge detection segmentation method, a genetic algorithm segmentation method, etc.

[0038] In another embodiment, a deep learning method is used to perform image segmentation on the reflectance map; specifically, the deep learning processing method can include a feature coding-based segmentation method such as a deep residual network (Residual Neural Network, ResNet) and VGGNet, a region selection-based method such as a fast region-based convolutional network (Fast Region-based Convolutional Network, Fast RCNN), an image segmentation method based on a recurrent neural network (Recurrent Neural Network, RNN), etc.

[0039] In a specific embodiment, the image segmentation of the reflectogram can be achieved by an offline configuration manner of the deep learning-based image segmentation model; specifically, an online end and an offline end of the endoscope system can be configured in advance, the online end obtains the brightness to be corrected image as a real-time image, the real-time image is input into the offline end configured in advance, the image segmentation is performed by the offline end, and the image segmentation result is output; when the image segmentation model is trained, an image dataset composed of reflectograms corresponding to different image label attributes is constructed in advance, a network structure for identifying different image content attributes is built according to the data characteristics of the image dataset, a loss function is determined according to the data characteristics in the image dataset, and the network structure, the loss function and the trained hyperparameters are used as model parameters; after the model parameters are determined, the image dataset is input into the model for model training until the model converges.

[0040] In step S104, the spectral reflectance of the target region is determined based on the pre-labeled spectral reflectance result of the image content attribute.

[0041] When the light source irradiates the object surface, the object selectively reflects electromagnetic waves of different wavelengths. Spectral reflectance refers to the ratio of the light flux reflected by the object to the light flux incident on the object in a certain waveband, and is an essential attribute of the object surface. Spectral reflectance is a representation of the color of the object itself, which not only records the color information of the object comprehensively, but also represents the material of the object surface. Spectral reflectance is an important physical property independent of lighting and camera equipment, which represents the reflection ratio of the object surface to light of different wavelengths. Therefore, the true color representation of the target object under standard observer conditions can be calculated by measuring the reflectivity of the target object at each wavelength corresponding to each image content attribute.

[0042] In an embodiment, the spectral reflectance of the target object with different image content attributes can be calibrated in advance to obtain a spectral reflectance calibration result, and then the spectral reflectance of the target region is determined based on the pre-labeled spectral reflectance calibration result of the image content attribute.

[0043] In step S105, the reflectogram is corrected using the spectral reflectance to obtain a corrected reflectogram.

[0044] The change of the application scene of the endoscope and the relative spatial position of the lens can cause the change of the reflection component of the target object represented on the reflection map. Without considering the above change, the traditional correction method of only considering the lens optical parameters of the endoscope for brightness correction of the endoscope is not accurate in the reflection component represented by the reflection map, and the brightness correction effect is not good. Therefore, in the present application, the reflection map representing the reflection component of the target object representing the image detail information is corrected to consider the difference of the reflection component of the target object caused by different application scenes. Specifically, in the present application, the reflection map is corrected by the relatively accurate spectral reflectance of the target area determined by the pre-calibration spectral reflectance calibration result, and the corrected reflection map of the target object corresponding to different image content attributes under the current application scene and the relative spatial position of the lens of the endoscope is obtained.

[0045] In one embodiment, the influence of the spectral reflectance on the reflection map when correcting the reflection map is determined to determine the reflection calibration preset weight, and then the reflection map is corrected according to the reflection rate calibration preset weight, so that the reflection rate calibration is combined with the reflection decomposition map through the preset weight to obtain a more accurate reflection map.

[0046] In one embodiment, the reflection map is corrected according to the image segmentation error caused by different image content attributes when image segmentation, so that the reflection rate calibration is combined with the reflection decomposition map through the preset weight to obtain a more accurate reflection map. Specifically, as one possible implementation, the error of the above image segmentation can be considered by the overall aspect when image segmentation. The overall aspect can include the segmentation confidence when image segmentation and the noise interference of the reflection map when segmentation. Further, the corresponding correction weight is configured according to the segmentation confidence and the noise interference of the reflection map when segmentation, and then the reflection map is corrected according to the correction weight combined with the reflection rate calibration value, so that the reflection rate calibration is combined with the reflection decomposition map through the preset weight to obtain a more accurate reflection map.

[0047] As another possible implementation, the error of the above image segmentation can be considered by the local aspect when image segmentation. The local aspect can include the region boundary pixel, wherein the region boundary pixel is a pixel located at the region boundary in the reflection map. When corrected, the smoothing with the surrounding pixels needs to be considered, otherwise the image correction is easy to appear fault. Therefore, different preset weights can be set according to the region boundary pixel, and the reflection map is corrected by combining the corresponding reflection rate calibration value, so that the reflection rate calibration is combined with the reflection decomposition map through the preset weight to obtain a more accurate reflection map.

[0048] It should be understood that the reflectance calibration can be combined with the reflectance decomposition map through the preset weight to obtain a more accurate reflectance map.

[0049] The above method corrects the reflectance map by the relatively accurate spectral reflectance of the target region determined by the pre-labeled spectral reflectance calibration result, to obtain a corrected reflectance map conforming to the target object in the current application scene of the endoscope and the relative spatial position of the lens.

[0050] In step S106, the corrected reflectance map and the illumination map are fused to obtain a brightness corrected image.

[0051] In the embodiment, the corrected reflectance map is obtained by correcting the reflectance map by the spectral reflectance corresponding to the pre-labeled spectral reflectance calibration result of the image content attribute, and therefore the brightness corrected image conforms to the performance of the target object in the current application scene of the endoscope and the relative spatial position of the lens. The combination of the reflectance calibration and the reflectance decomposition map can finally obtain endoscopic imaging with uniform brightness and comfortable viewing.

[0052] The endoscopic image brightness correction method of the embodiment acquires a brightness to-be-corrected image collected by an endoscope; the brightness to-be-corrected image is obtained by image collection after lens correction and illumination correction of the endoscope in sequence; the brightness to-be-corrected image is decomposed to obtain a reflectance map and an illumination map; the reflectance map is segmented according to the image content attribute of the application scene of the endoscope to obtain target regions with different image content attributes; the spectral reflectance of the target regions is determined based on the pre-labeled spectral reflectance calibration result of the image content attribute; the reflectance map is corrected by the spectral reflectance to obtain a corrected reflectance map; the corrected reflectance map and the illumination map are fused to obtain a brightness corrected image; the above method corrects the reflectance map by the relatively accurate spectral reflectance of the target region determined by the pre-labeled spectral reflectance calibration result, to obtain a corrected reflectance map conforming to the target object in the current application scene of the endoscope and the relative spatial position of the lens. Further, the accurate corrected reflectance map is fused with the illumination map to obtain the brightness corrected image. The combination of the reflectance calibration and the reflectance decomposition map can finally obtain endoscopic imaging with uniform brightness and comfortable viewing.

[0053] As an exemplary embodiment, the reflectance map is corrected by the spectral reflectance to obtain a corrected reflectance map, including: acquiring a spectral reflectance preset weight corresponding to a spectral reflectance calibration result; correcting the reflectance map based on the spectral reflectance preset weight and the spectral reflectance calibration result to obtain a first corrected reflectance map; and taking the first corrected reflectance map as the corrected reflectance map.

[0054] In the embodiment, the spectral reflectance preset weight corresponding to the spectral reflectance calibration result can be calibrated in advance; specifically, in an experimental scene, for the spectral reflectance calibration result corresponding to each image content attribute, the reflectogram is corrected by using different spectral reflectance calibration rate weights from 0 to 1 in turn to obtain a reflectogram correction result, and then the spectral reflectance calibration rate weight of the reflectance calibration result meeting the preset requirement is selected as the spectral reflectance preset weight, and a corresponding relationship of the reflectance calibration weight corresponding to each image content attribute is generated. The preset requirement can be that when the reflectogram is corrected by using the spectral reflectance calibration weight, the error between the actual spectral reflectance corresponding to the corrected reflectogram and the preset spectral reflectance corresponding to the target object corresponding to the image content attribute of the reflectogram is less than a preset error; further, in actual use, the spectral reflectance preset weight corresponding to the spectral reflectance calibration result is determined by the spectral reflectance calibration result and the above corresponding relationship.

[0055] In an embodiment, the reflectogram can be corrected by using formula (1) to obtain a corrected reflectogram: R out (x,y)=ω i M+R i (x,y) (1)

[0056] In formula (1), R out (x,y) represents the corrected reflectogram, ω i represents the spectral reflectance preset weight of the ith target region, M represents the spectral reflectance calibration result of the ith target region, and R i (x,y) represents the original reflectogram of the ith target region.

[0057] In the correction of the reflectogram by the spectral reflectance, since the spectral reflectance is obtained from the original reflectogram segmented by the image segmentation, the segmentation error in the image segmentation also has an impact on the correction of the reflectogram; in the present application, the impact can include the overall aspect impact of the segmentation confidence in the image segmentation and the noise disturbance of the reflectogram in the segmentation and the local aspect impact of the regional boundary pixels; based on this, in order to consider the overall aspect impact of the segmentation confidence in the image segmentation and the noise disturbance of the reflectogram in the segmentation, as an exemplary embodiment, the correction of the reflectogram by the spectral reflectance to obtain the corrected reflectogram further includes: obtaining the segmentation confidence in the image segmentation of each target region; determining the segmentation confidence impact weight and / or the noise impact weight for the correction of the reflectogram based on the segmentation confidence and the reflectogram noise disturbance respectively; wherein the segmentation confidence impact weight is proportional to the segmentation confidence, and the noise impact weight is proportional to the reflectogram noise disturbance; correcting the reflectogram based on the segmentation confidence impact weight and / or the noise impact weight and the spectral reflectance calibration result to obtain a second corrected reflectogram; and fusing the first corrected reflectogram and the second corrected reflectogram to obtain the corrected reflectogram.

[0058] In an embodiment, the impact of the segmentation confidence on the correction of the reflectogram can be considered separately; specifically, the segmentation confidence in the image segmentation of each target region is obtained; the segmentation confidence impact weight for the correction of the reflectogram is determined based on the segmentation confidence; wherein the segmentation confidence impact weight is proportional to the segmentation confidence; the reflectogram is corrected based on the segmentation confidence impact weight and the spectral reflectance calibration result to obtain a second corrected reflectogram; and the first corrected reflectogram and the second corrected reflectogram are fused to obtain the corrected reflectogram.

[0059] In an embodiment, the impact of the reflectogram noise disturbance on the correction of the reflectogram can be considered separately; specifically, the reflectogram noise disturbance in the image segmentation of each target region is obtained; the noise impact weight for the correction of the reflectogram is determined based on the reflectogram noise disturbance; wherein the noise impact weight is proportional to the reflectogram noise disturbance; the reflectogram is corrected based on the noise impact weight and the spectral reflectance calibration result to obtain a second corrected reflectogram; and the first corrected reflectogram and the second corrected reflectogram are fused to obtain the corrected reflectogram.

[0060] In another embodiment, the influence of segmentation confidence and reflectance map noise disturbance on reflectance map correction can be considered simultaneously; specifically, the segmentation confidence when segmenting each target region is obtained; the segmentation confidence influence weight and the noise influence weight for correcting the reflectance map are determined based on the segmentation confidence and the reflectance map noise disturbance respectively; wherein the segmentation confidence influence weight is proportional to the segmentation confidence, and the noise influence weight is proportional to the reflectance map noise disturbance; the reflectance map is corrected based on the segmentation confidence influence weight and the noise influence weight and the spectral reflectance calibration result to obtain a second corrected reflectance map; and the first corrected reflectance map and the second corrected reflectance map are fused to obtain a corrected reflectance map.

[0061] Wherein, as one possible implementation, in an experimental scenario, for each segmentation confidence and reflectance map noise disturbance of the target region corresponding to each image content attribute, different segmentation confidence influence weights and noise influence weights between 0 and 1 are sequentially used to correct the reflectance map alone or jointly to obtain reflectance map correction results, and then the segmentation confidence and reflectance map noise disturbance of the reflectance calibration result meeting the preset requirement are selected as the spectral reflectance preset weight, and the corresponding relationship between the segmentation confidence and the segmentation confidence influence weight, the corresponding relationship between the reflectance map noise disturbance and the noise influence weight, and the corresponding relationship between the segmentation confidence, the reflectance map noise, the segmentation confidence influence weight and the noise influence weight are generated. Wherein, the preset requirement can be that when the reflectance map is corrected using the segmentation confidence influence weight and / or the noise influence weight, the error between the actual spectral reflectance corresponding to the corrected reflectance map and the preset spectral reflectance corresponding to the target object corresponding to the image content attribute of the reflectance map is less than a preset error; further, in actual use, the spectral reflectance preset weight corresponding to the spectral reflectance calibration result is determined by the segmentation confidence influence weight and / or the noise influence weight and the above corresponding relationship.

[0062] In one embodiment, the reflectance map can be corrected using formula (2) to obtain a second corrected reflectance map: out (x,y)=α i *β i *M+(1-α i *β i ) *R i (x,y) (2)

[0063] In formula (2), R out (x,y) represents the second corrected reflectance map, α i represents the segmentation confidence influence weight of the i-th target region, β i represents the noise influence weight of the i-th target region, M represents the spectral reflectance calibration result, and R i (x,y) represents the original reflectance map of the i-th target region.

[0064] As a possible implementation, after obtaining the second corrected reflectogram, the first corrected reflectogram and the second corrected reflectogram are fused to obtain the corrected reflectogram; when fusing, a pre-configured fusion weight can be used for fusion; wherein the sum of the fusion weight of the first corrected reflectogram and the fusion weight of the second corrected reflectogram is 1, and the fusion weight of the second corrected reflectogram is inversely proportional to each segmentation confidence and reflectogram noise interference.

[0065] As an exemplary embodiment, the reflectogram is corrected by using the spectral reflectance to obtain the corrected reflectogram, and further comprising: obtaining the region boundary pixels of each target region and the spectral reflectance calibration smoothing value corresponding to the boundary region; determining the boundary region pixel influence weight for correcting the reflectogram based on the region boundary pixels; correcting the reflectogram based on the boundary region pixel influence weight and the spectral reflectance calibration smoothing value to obtain the third corrected reflectogram; and fusing the first corrected reflectogram, the second corrected reflectogram and the third corrected reflectogram to obtain the corrected reflectogram.

[0066] In the embodiment, the region boundary pixels are the pixels at the region junction in the reflectogram, and when being corrected, the smoothing with the surrounding pixels needs to be considered, otherwise the image correction is easy to appear fault; meanwhile, the boundary pixels are located on the transition zone, and the reflectance calibration value of a certain region also needs to be smoothed and interpolated with the reflectance calibration value of the neighborhood; therefore, when considering the influence of the region boundary pixels on the reflectogram, the region boundary pixels of each target region and the spectral reflectance calibration smoothing value corresponding to the boundary region are obtained, and the spectral reflectance calibration smoothing value is obtained by smoothing and interpolating the reflectance calibration value with the reflectance calibration value of the neighborhood; further, the reflectogram is corrected based on the boundary region pixel influence weight and the spectral reflectance calibration smoothing value to obtain the third corrected reflectogram.

[0067] In an embodiment, after obtaining the third corrected reflectogram, the first corrected reflectogram and the third corrected reflectogram are fused to obtain the corrected reflectogram; wherein the sum of the fusion weight of the first corrected reflectogram, the fusion weight of the second corrected reflectogram and the fusion weight of the third corrected reflectogram is 1, and the fusion weight of the second corrected reflectogram is inversely proportional to each segmentation confidence and reflectogram noise interference.

[0068] In an embodiment, the reflectogram can be corrected by using formula (3) to obtain the second corrected reflectogram: out (x,y)=α i *β i *(ψ i *M smooth +(1-ψ i ) *M +(1-α i *β i ) *R i (x,y) (3)

[0069] In formula (3), R out (x,y) represents the second corrected reflectance map, a i represents the segmentation confidence influence weight of the i-th target region, β i represents the noise influence weight of the i-th target region, ψ i represents the boundary region pixel influence weight of the i-th target region, M smooth represents the spectral reflectance calibration smoothing value of the i-th target region, M represents the spectral reflectance calibration result, R i (x,y) represents the original reflectance map.

[0070] As an exemplary embodiment, the endoscope image brightness correction method further comprises: obtaining a spectral reflectance curve of a target object corresponding to each image content attribute; wherein the spectral reflectance curve is obtained by pre-spectrally measuring the target object; determining a spectral power distribution weight corresponding to each wavelength based on the spectral reflectance curve; obtaining tristimulus values based on the spectral reflectance curve, the spectral power distribution weight, and a preset standard observer function; and converting the tristimulus values to an sRGB color space to obtain a corresponding relationship between the image content attribute and the RGB calibration value as a reflectance calibration result.

[0071] In this embodiment, the spectral reflectance curve can be obtained by building an experimental scene and spectrally measuring the target object; specifically, an experimental scene is built, a sample of the target object is obtained, and the reflectance of the target object at different wavelengths of visible light (about 400 nm to 700 nm) can be measured by a spectrometer, can be discretely sampled at a sampling interval of N nm, and can be fitted into a continuous spectral reflectance curve.

[0072] Further, after obtaining the spectral reflectance curve, a spectral power distribution weight corresponding to each wavelength is determined based on the spectral reflectance curve;

[0073] Further, the spectral reflectance is multiplied by a corresponding standard observer function; wherein the standard observer function can select two standard observer modes of CIE 1931 and CIE 1964 defined by the International Commission on Illumination, which represent the average sensitivity of human vision to different wavelengths as a standardized way of perceiving colors of objects.

[0074] Further, the result of multiplying the spectral reflectance by the corresponding standard observer function is multiplied by the spectral power distribution weight at the corresponding wavelength, and the integral is performed on the full wavelength to obtain XYZ tristimulus values, wherein the Y value represents brightness, and X and Z represent chroma.

[0075] Further, since XYZ is a device independent color space, it is converted to a corresponding sRGB color space according to the current camera system settings, and a corresponding RGB calibration value is obtained.

[0076] As an exemplary embodiment, preset lens correction parameters of an endoscope, preset illumination correction parameters, and a captured image of the endoscope are obtained; the preset lens correction parameters are obtained by lens calibration of the endoscope in a closed self-provided light source and an external environment with uniform illumination; the captured image is corrected based on the preset lens correction parameters to obtain a lens corrected image; the lens corrected image is corrected based on the preset illumination correction parameters to obtain an illumination corrected image; and the illumination corrected image is taken as a brightness image to be corrected.

[0077] In an embodiment, the preset lens correction parameters are obtained by lens non-uniformity correction; specifically, the processing flow of the lens non-uniformity correction is as follows:

[0078] Step one: close the illumination light source of the endoscope, and in an external environment with uniform illumination, align the lens to a uniform gray panel, fix the shutter and gain parameters of the camera system, and obtain an image to be corrected of the endoscope. The external environment with uniform illumination can be a light box.

[0079] Step two: determine a set of lens correction parameters according to the brightness information of the image to be corrected, the set of parameters representing the gain amplitude of the corresponding region or pixel. The specific method of determining the calibration parameters includes at least one of the pixel-by-pixel method, the grid interpolation method, and the concentric circle method.

[0080] As a possible implementation, after obtaining the set of lens correction parameters, the lens correction parameters are integrated into the ISP Pipeline of the camera system as preset lens correction parameters to perform lens correction processing on a real-time target image.

[0081] In an embodiment, the preset illumination correction parameters are obtained by illumination non-uniformity correction; specifically, the processing flow of the illumination non-uniformity correction is as follows:

[0082] Step one: turn on the illumination light source of the endoscope, fix the light source illumination, and in an external environment without illumination, align the lens to a uniform gray panel to obtain an endoscope image based on lens non-uniformity correction. The external environment without illumination can be a black box.

[0083] Step two: determine a set of illumination correction parameters according to the brightness information of the image to be corrected, the set of parameters representing the gain amplitude of the corresponding region or pixel. The specific method of determining the calibration parameters includes but is not limited to the pixel-by-pixel method, the grid interpolation method, the concentric circle method, etc.

[0084] Step three: adjusting the light source illuminance, repeating the above operation until the preset light source illuminance group number is traversed, and a plurality of groups of light correction parameters are obtained.

[0085] As a possible implementation, after obtaining the light correction parameter, the light correction parameter is integrated into the ISP Pipeline of the camera system as a preset lens light correction parameter set to perform lens correction processing on the real-time target image.

[0086] The scheme in the present application can support linear RGB domain processing and non-linear RGB domain processing when fusing the corrected reflectance map and the illumination map, that is, supporting brightness correction before gamma and supporting brightness correction after gamma. Therefore, as an exemplary embodiment, before fusing the corrected reflectance map and the illumination map to obtain the brightness correction image, the method further comprises: obtaining a preset brightness correction process; when the preset brightness correction process is brightness correction before gamma, obtaining a preset gamma correction curve, and correcting the illumination map based on the preset gamma correction curve.

[0087] In an embodiment, the brightness correction before gamma can be realized by the following formula: out (x,y)=(R out (x,y)*L(x,y)) γ (4)

[0088] In formula (4), I out (x,y) represents the brightness correction image, R out (x,y) represents the corrected reflectance map, L(x,y) represents the illumination map, and γ is a pre-configured gamma correction curve parameter.

[0089] As an exemplary embodiment, the brightness correction after gamma can be realized by the following formula: out (x,y)=R out (x,y)*L(x,y) (5)

[0090] In formula (5), I out (x,y) represents the brightness correction image, R out (x,y) represents the corrected reflectance map, and L(x,y) represents the illumination map.

[0091] The present embodiment provides an endoscope image brightness correction device, as shown in FIG. 2, comprising:

[0092] The acquisition module 201 is configured to acquire a brightness image to be corrected collected by an endoscope; wherein the brightness image to be corrected is obtained by sequentially performing lens correction and light correction on the image collected by the endoscope;

[0093] an image decomposition module 202, configured to perform image decomposition on the image to be corrected to obtain a reflectance image and an illumination image;

[0094] an image segmentation module 203, configured to perform image segmentation on the reflectance image according to image content attributes of the endoscope application scenario to obtain target regions with different image content attributes;

[0095] a spectral reflectance determination module 204, configured to determine spectral reflectance of the target regions based on pre-labeled spectral reflectance labeling results of the image content attributes;

[0096] a correction module 205, configured to correct the reflectance image by using the spectral reflectance to obtain a corrected reflectance image;

[0097] a fusion module 206, configured to fuse the corrected reflectance image and the illumination image to obtain the brightness corrected image.

[0098] It should be noted that the above modules and the examples and application scenarios realized by the corresponding steps are the same, but are not limited to the content disclosed in the above embodiments.

[0099] It should be noted that the above modules as part of the device can be implemented by software or by hardware, and the hardware environment includes a network environment.

[0100] The embodiment of the present application also provides a computer device, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete mutual communication through the communication bus, the memory is used for storing a computer program, and the processor is used for executing the method in any one of the above embodiments by running the computer program stored on the memory.

[0101] Fig. 3 is a structural block diagram of an optional computer device according to the embodiment of the present application, as shown in Fig. 3, including a processor 10, a communication interface 20, a memory 30 and a communication bus 40, wherein the processor 10, the communication interface 20 and the memory 30 complete mutual communication through the communication bus 40, and

[0102] the memory 30 is used for storing a computer program;

[0103] the processor 10 is used for executing the computer program stored on the memory 30 to realize the construction method of the disease database in any one of the above embodiments.

[0104] Optionally, in the embodiment, the communication bus described above can be a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, or the like. The communication bus can be divided into an address bus, a data bus, a control bus, and the like. For the convenience of representation, only one thick line is used in FIG. 3, but it does not mean that there is only one bus or only one type of bus.

[0105] The communication interface is used for communication between the computer device and other devices.

[0106] The memory can include a RAM, and can also include a non-volatile memory, for example, at least one disk memory. Optionally, the memory can also be at least one storage device located away from the aforementioned processor.

[0107] The processor described above can be a general-purpose processor, which can include but is not limited to: a CPU (Central Processing Unit), an NP (Network Processor), and the like; and can also be a DSP (Digital Signal Processing), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component.

[0108] Optionally, the specific examples in the embodiment can refer to the examples described in the above-described embodiments, and the embodiment will not be described here.

[0109] As can be understood by those skilled in the art, the structure shown in FIG. 3 is only schematic, and the device for implementing the method of any one of the above-described embodiments can be a terminal device, which can be a smart phone (such as an Android phone, an IOS phone, and the like), a tablet computer, a palm computer, a Mobile Internet Device (MID), a PAD, or the like. FIG. 3 does not limit the structure of the electronic device. For example, the terminal device can further include more or fewer components (such as a network interface, a display device, and the like) than those shown in FIG. 3, or have a different configuration from that shown in FIG. 3.

[0110] Those skilled in the art can understand that all or part of the steps of various methods in the above embodiments can be completed by instructing the terminal device related hardware through a program, and the program can be stored in a computer readable storage medium, which can include a flash disk, a ROM, a RAM, a magnetic disk or an optical disk, etc.

[0111] As an exemplary embodiment, the present application also provides a computer readable storage medium, which stores a computer program, wherein the computer program is configured to execute the method steps of any one of the embodiments when running.

[0112] Optionally, in the embodiment, the storage medium can be used for the program code for executing the method steps of the embodiments.

[0113] Optionally, in the embodiment, the storage medium can be located on at least one of the network devices in the network shown in the above embodiments.

[0114] Optionally, in the embodiment, the storage medium is configured to store the method for executing the above embodiments.

[0115] Optionally, the specific examples in the embodiment can refer to the examples described in the above embodiments, and the embodiment will not be described here.

[0116] Optionally, in the embodiment, the storage medium can include but is not limited to a U disk, a ROM, a RAM, a mobile hard disk, a magnetic disk or an optical disk, etc. various media that can store program codes.

[0117] The serial numbers of the above embodiments of the present application are only for description, not representing the advantages and disadvantages of the embodiments.

[0118] The integrated units in the above embodiments, if realized in the form of software function units and sold or used as independent products, can be stored in the above computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the parts that make contributions to the prior art or the whole or part of the technical solutions can be embodied in the form of software products, which are stored in the storage medium and include a number of instructions for making one or more computer devices (which can be personal computers, servers or network devices, etc.) execute all or part of the steps of the methods in the above embodiments.

[0119] In the above embodiments of the present application, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can refer to the related description of other embodiments.

[0120] In several embodiments provided in the present application, it should be understood that the disclosed client can be implemented in other manners. Of course, the described apparatus embodiments are merely schematic, and the division of units is merely logical function division, and there can be another division manner in actual implementation. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections can be indirect couplings or communication connections through some interfaces, units or modules, and can be in electrical, mechanical or other forms.

[0121] The units described as separated components can or can not be physically separated, and the components displayed as units can or can not be physical units, i.e., can be located in one place or distributed on a plurality of network units. In actual implementation, some or all of the units can be selected according to actual needs to achieve the purposes of the solutions provided in the embodiments.

[0122] In addition, each functional unit in the embodiments of the present application can be integrated in a processing unit, or each unit can exist physically as a separate unit, or two or more units can be integrated in one unit. The integrated unit can be implemented in the form of hardware, or in the form of a software functional unit.

[0123] The above are only the preferred embodiments of the present application, and it should be pointed out that, for those skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, which should be considered as the protection scope of the present application.

[0124] In the above-described embodiments of the present application, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the relevant description of other embodiments.

[0125] The above are only the preferred embodiments of the present application, and it should be pointed out that, for those skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, which should be considered as the protection scope of the present application.

Claims

An endoscope image brightness correction method characterized by, The endoscope image brightness correction method comprises: brightness of the endoscope image to be corrected is acquired; image decomposition is performed on the brightness of the endoscope image to be corrected to obtain a reflection map and an illumination map; image segmentation is performed on the reflection map according to image content attributes of an endoscope application scenario to obtain target regions with different image content attributes; spectral reflectivity of the target regions is determined based on pre-labeled spectral reflectivity labeling results of the image content attributes; the reflection map is corrected using the spectral reflectivity to obtain a corrected reflection map; the corrected reflection map and the illumination map are fused to obtain the brightness corrected image. The endoscope image brightness correction method according to claim 1, wherein The reflection map is corrected using the spectral reflectivity to obtain a corrected reflection map, comprising: a spectral reflectivity preset weight corresponding to the spectral reflectivity labeling result is acquired; the reflection map is corrected based on the spectral reflectivity preset weight and the spectral reflectivity labeling result to obtain a first corrected reflection map; the first corrected reflection map is taken as the corrected reflection map. The endoscope image brightness correction method according to claim 2, characterized in that, The reflection map is corrected using the spectral reflectivity to obtain a corrected reflection map, further comprising: segmentation confidence degrees of the target regions when image segmentation is performed are acquired; segmentation confidence degree influence weights and / or noise influence weights for correcting the reflection map are determined based on the segmentation confidence degrees and noise interference of the reflection map, respectively; wherein the segmentation confidence degree influence weight is proportional to the segmentation confidence degree, and the noise influence weight is proportional to the noise interference of the reflection map; the reflection map is corrected based on the segmentation confidence degree influence weight and / or the noise influence weight and the spectral reflectivity labeling result to obtain a second corrected reflection map; the first corrected reflection map and the second corrected reflection map are fused to obtain the corrected reflection map. The endoscope image brightness correction method according to claim 3, characterized in that, The reflection map is corrected using the spectral reflectivity to obtain a corrected reflection map, further comprising: spectral reflectivity labeling smoothing values corresponding to region boundary pixels and boundary regions of the target regions are acquired; boundary region pixel influence weights for correcting the reflection map are determined based on the region boundary pixels; the reflection map is corrected based on the boundary region pixel influence weights and the spectral reflectivity labeling smoothing values to obtain a third corrected reflection map; the first corrected reflection map, the second corrected reflection map and the third corrected reflection map are fused to obtain the corrected reflection map. The endoscope image brightness correction method according to claim 1, wherein The endoscope image brightness correction method further comprises: spectral reflectivity curves of target objects corresponding to the image content attributes are acquired; wherein the spectral reflectivity curves are obtained by pre-spectrally measuring the target objects; spectral power distribution weights corresponding to wavelengths are determined based on the spectral reflectivity curves; tristimulus values are obtained based on the spectral reflectivity curves, the spectral power distribution weights and a preset standard observer function; the tristimulus values are converted to an sRGB color space to obtain a corresponding relationship between the image content attributes and RGB labeling values as the reflectivity labeling result. The endoscope image brightness correction method according to claim 1, wherein The brightness of the endoscope image to be corrected is acquired, comprising: obtaining preset lens correction parameters, preset illumination correction parameters of the endoscope, and a collection image of the endoscope; wherein the preset lens correction parameters are obtained by lens calibration of the endoscope in a closed self-light source and an external environment with uniform illumination; correcting the collection image based on the preset lens correction parameters to obtain a lens correction image; correcting the lens correction image based on the preset illumination correction parameters to obtain an illumination correction image; taking the illumination correction image as the brightness correction image. The endoscope image brightness correction method according to claim 1, wherein Before the fusing the corrected reflection image and the illumination image to obtain the brightness correction image, the method further comprises: obtaining a preset brightness correction process; when the preset brightness correction process is gamma front brightness correction, obtaining a preset gamma correction curve, and correcting the illumination image based on the preset gamma correction curve. An endoscope image brightness correction device characterized by comprising: The apparatus comprises: an acquisition module configured to acquire a brightness correction image collected by an endoscope; wherein the brightness correction image is obtained by image collection after lens correction and illumination correction of the endoscope in sequence; an image decomposition module configured to perform image decomposition on the brightness correction image to obtain a reflection image and an illumination image; an image segmentation module configured to perform image segmentation on the reflection image according to image content attributes of an application scenario of the endoscope to obtain target regions with different image content attributes; a spectral reflectance determination module configured to determine spectral reflectance of the target regions based on pre-labeled spectral reflectance labeling results of the image content attributes; a correction module configured to correct the reflection image by using the spectral reflectance to obtain a corrected reflection image; a fusion module configured to fuse the corrected reflection image and the illumination image to obtain the brightness correction image. A computer device, characterized in that comprise: a memory and a processor, which are communicatively connected, and the memory stores computer instructions, and the processor executes the computer instructions to perform the endoscope image brightness correction method in any one of claims 1 to 7. A computer-readable storage medium, characterized by The computer readable storage medium stores computer instructions, and the computer instructions are used to make a computer execute the endoscope image brightness correction method in any one of claims 1 to 7.

Images