Fundus color photo synthesis method, electronic equipment and computer readable medium

Abstract

The invention discloses a fundus color photograph synthesis method, electronic equipment and a computer readable medium, and the method comprises the steps: extracting an infrared light blood vessel feature image from an infrared light collection fundus image, and extracting a visible light blood vessel feature image from a visible light collection fundus image; based on the infrared light blood vessel feature image and the visible light blood vessel feature image, aligning the infrared light collection eye fundus image to the visible light collection eye fundus image to obtain a registration infrared light eye fundus image; generating an RGB fusion image according to the visible light collection eye fundus image and the registration infrared light eye fundus image; and adjusting the color information of the RGB fusion image based on a standard fundus image to obtain a target fundus color photo. Enough diagnosis information can be provided, so that a doctor can accurately identify and analyze the fundus diseases, and the doctor can read and understand the fundus diseases, so that fundus disease identification and analysis results can be quickly given.

Description

Technical Field

[0001] The present application relates to the technical field of image processing, and specifically, to a method for synthesizing fundus color photos, an electronic device, and a computer-readable medium. Background Art

[0002] Fundus color photos refer to the fundus color photos taken by professional devices such as fundus cameras. They can clearly show the fundus structures such as the retina, optic disc, and macula area. Through fundus color photos, doctors can observe the morphology and changes of blood vessels, nerves, pigments, etc. in the fundus. Fundus color photos can be used to detect and evaluate various fundus diseases, such as diabetic retinopathy, age-related macular degeneration, retinal vein occlusion, etc., and are of great significance in ophthalmic clinical diagnosis, disease monitoring, and research, thus providing important basis for the diagnosis and treatment of diseases.

[0003] However, the images obtained by current fundus image processing technologies cannot provide sufficient diagnostic information, thus limiting the accurate identification and analysis of fundus diseases by doctors. Or although they can provide sufficient diagnostic information, due to being only grayscale images without true color results, it limits doctors' understanding of reading films.

[0004] Therefore, there is an urgent need for a better method for synthesizing fundus color photos. Summary of the Invention

[0005] The present application aims to solve one of the technical problems in the related technologies to a certain extent. For this purpose, the present application provides a method for synthesizing fundus color photos, an electronic device, and a computer-readable medium.

[0006] As the first aspect of the present application, there is provided a method for synthesizing fundus color photos, wherein the method includes: Extracting an infrared light blood vessel feature image from the fundus image collected by infrared light, and extracting a visible light blood vessel feature image from the fundus image collected by visible light; Aligning the fundus image collected by infrared light to the fundus image collected by visible light based on the infrared light blood vessel feature image and the visible light blood vessel feature image to obtain a registered infrared light fundus image; Generating an RGB fusion image according to the fundus image collected by visible light and the registered infrared light fundus image; Adjusting the color information of the RGB fusion image based on a standard fundus image to obtain a target fundus color photo.

[0007] Optionally, the generating an RGB fusion image according to the fundus image collected by visible light and the registered infrared light fundus image includes: Based on a preset minimum radius and a preset maximum radius, both the visible light acquired fundus image and the registered infrared light fundus image are divided into a central region, a transition region, and a peripheral region; Generate a color transition region image according to the transition region of the visible light acquired fundus image and the transition region of the registered infrared light fundus image; Map the pixel values of the central region of the visible light acquired fundus image to the RGB three channels to obtain a color central region image; and map the pixel values of the peripheral region of the registered infrared light fundus image to the RGB three channels to obtain a color peripheral region image; Stitch the color central region image, the color transition region image, and the color peripheral region image to obtain an RGB fusion image.

[0008] Optionally, the generating a color transition region image according to the transition region of the visible light acquired fundus image and the transition region of the registered infrared light fundus image includes: For two pixels with the same pixel coordinates in the transition region of the visible light acquired fundus image and the transition region of the registered infrared light fundus image, determine the corresponding RGB three-channel values for their pixel coordinates according to the weight parameters of the two pixels and their respective pixel values; wherein, the weight parameter linearly changes with the distance between the pixel coordinates and the center point of the image; Generate a color transition region image according to each pixel coordinate in the transition region and its corresponding RGB three-channel values.

[0009] Optionally, the aligning the infrared light acquired fundus image to the visible light acquired fundus image based on the infrared light blood vessel feature image and the visible light blood vessel feature image to obtain a registered infrared light fundus image includes: Optimize the parameters of the initial affine matrix according to the infrared light blood vessel feature image and the visible light blood vessel feature image to obtain an optimized target affine matrix; Apply the target affine matrix to the infrared light acquired fundus image to obtain a registered infrared light fundus image.

[0010] Optionally, the optimizing the parameters of the initial affine matrix according to the infrared light blood vessel feature image and the visible light blood vessel feature image to obtain an optimized target affine matrix includes: Apply the initial affine matrix to the infrared light blood vessel feature image to obtain a projected feature image; Optimize the parameters of the initial affine matrix according to the gradient descent algorithm until the optimization loss is minimized to obtain a target affine matrix; wherein, the optimization loss is determined by the coincidence rate between the projected feature image and the visible light blood vessel feature image.

[0011] Optionally, extracting a vascular feature image from the acquired fundus image, including: Adjusting the brightness of the acquired fundus image; Optimizing the vascular network pixels in the image after brightness adjustment; According to a preset segmentation ratio, respectively extracting an upper segmentation sub-image and a lower segmentation sub-image from the optimized image; wherein, the value range of the preset segmentation ratio includes [2 / 3, 1); Respectively extracting the largest sub-vascular network from the upper segmentation sub-image and the lower segmentation sub-image and superimposing them to obtain a vascular feature image.

[0012] Optionally, the optimizing the vascular network pixels in the image after brightness adjustment includes: Using a ridge filter to identify ridge feature pixels in the image after brightness adjustment, and determining the ridge feature pixels with a confidence level higher than the confidence threshold as vascular pixels; wherein, the confidence threshold is determined by the global mean and standard deviation of the pixel values of the image after brightness adjustment; Performing dilation and contraction processing on the image after brightness adjustment to fill in the areas where the vascular pixels are discontinuous.

[0013] Optionally, the adjusting the color information of the RGB fusion image based on the standard fundus image to obtain a target fundus color photo includes: Determining the global mean and standard deviation of each of the RGB three channels of the standard fundus image to linearly scale the RGB three channel values of the RGB fusion image; Converting the linearly scaled RGB fusion image into a first Lab image and converting the standard fundus image into a second Lab image; Determining the global mean of the L channel of the second Lab image to linearly scale the L channel value of the first Lab image, and linearly scaling the histograms of the a channel and b channel of the first Lab image according to the histograms of the a channel and b channel of the second Lab image to obtain a target fundus color photo.

[0014] As a second aspect of the present application, there is provided an electronic device, wherein the electronic device includes: One or more processors; A memory storing one or more computer programs thereon, when the one or more computer programs are executed by the one or more processors, enabling the one or more processors to implement the fundus color photo synthesis method according to the first aspect of the present application.

[0015] As a third aspect of the present application, there is provided a computer-readable medium having a computer program stored thereon, wherein when the computer program is executed by a processor, it implements the fundus color photo synthesis method according to the first aspect of the present application.

[0016] The fundus color photo synthesis method provided by the embodiments of the present application extracts an infrared light blood vessel feature image from the fundus image collected by infrared light, and extracts a visible light blood vessel feature image from the fundus image collected by visible light. Based on the infrared light blood vessel feature image and the visible light blood vessel feature image, the fundus image collected by infrared light is aligned with the fundus image collected by visible light to obtain a registered infrared fundus image. According to the fundus image collected by visible light and the registered infrared fundus image, an RGB fusion image is generated. Based on the standard fundus image, the color information of the RGB fusion image is adjusted to obtain a target fundus color photo. By performing spatial fusion and color conversion on the multi-band light source images, high-quality fundus color photos are synthesized, which can not only provide sufficient diagnostic information to facilitate the accurate identification and analysis of fundus diseases by doctors, but also facilitate doctors' reading and understanding of the images, so as to quickly give the identification and analysis results of fundus diseases. Brief Description of the Drawings

[0017] The following further describes the present application with reference to the drawings: Figure 1 It is a flowchart of an implementation manner of the fundus color photo synthesis method provided by the embodiments of the present application; Figure 2 It is a flowchart of an implementation manner of generating an RGB fusion image provided by the embodiments of the present application; Figure 3 It is a flowchart of an implementation manner of generating a color transition region image provided by the embodiments of the present application; Figure 4 It is a flowchart of an implementation manner of obtaining a registered infrared fundus image provided by the embodiments of the present application; Figure 5 It is a flowchart of an implementation manner of obtaining an optimized target affine matrix provided by the embodiments of the present application; Figure 6 It is a flowchart of an implementation manner of extracting a blood vessel feature image from the collected fundus image provided by the embodiments of the present application; Figure 7 It is a flowchart of an implementation manner of optimizing the blood vessel network pixels in the image after adjusting the brightness provided by the embodiments of the present application; Figure 8 It is a flowchart of an implementation manner of obtaining a target fundus color photo based on the RGB fusion image provided by the embodiments of the present application; Figure 9It is a schematic diagram of a specific implementation manner of the fundus color photo synthesis method provided by an embodiment of the present application; Figure 10 It is a module diagram of an implementation manner of an electronic device provided by an embodiment of the present application; Figure 11 It is a schematic diagram of a computer-readable medium provided by an embodiment of the present application.

[0018] Description of the reference numerals 101: Processor 102: Memory 103: I / O interface 104: Bus Specific implementation manner The embodiments of the present application will be described in detail below. The examples of the embodiments are shown in the drawings, where the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. Based on the embodiments in the implementation manner, it is intended to explain the present application and should not be construed as a limitation to the present application.

[0019] As used herein, the phrase "one embodiment" or "example" or "instance" means that a particular feature, structure, or characteristic described in connection with the embodiment itself may be included in at least one embodiment of the present disclosure. The appearances of the phrase "in one embodiment" in various places in the specification do not necessarily refer to the same embodiment.

[0020] Fundus color photos refer to the fundus color photos taken by professional devices such as fundus cameras. They can clearly show the fundus structures such as the retina, optic nerve papilla, and macula area. Through fundus color photos, doctors can observe the morphology and changes of blood vessels, nerves, pigments, etc. in the fundus. Fundus color photos can be used to detect and evaluate a variety of fundus diseases, such as diabetic retinopathy, age-related macular degeneration, retinal vein occlusion, etc., and are of great significance in ophthalmic clinical diagnosis, disease monitoring and research, thus providing an important basis for the diagnosis and treatment of diseases.

[0021] However, the images obtained by current fundus image processing technologies cannot provide sufficient diagnostic information, thus limiting the accurate identification and analysis of fundus diseases by doctors. Or although they can provide sufficient diagnostic information, since they are only grayscale images without true color results, it limits the doctors' understanding of the films.

[0022] Therefore, there is an urgent need for a better fundus color photo synthesis method.

[0023] In response to this, the applicant of the present application proposes that by fusing the fundus images collected by infrared light and the fundus images collected by visible light and performing color space conversion, high-quality color image synthesis can be achieved, and it is particularly suitable for the synthesis of fundus color photos.

[0024] As a first aspect of the embodiments of the present application, a method for synthesizing fundus color photos is provided. As Figure 1 shown, the method includes: Step S110: Extract an infrared light blood vessel feature image from the fundus image collected under infrared light, and extract a visible light blood vessel feature image from the fundus image collected under visible light; Step S120: Align the fundus image collected under infrared light with the fundus image collected under visible light based on the infrared light blood vessel feature image and the visible light blood vessel feature image to obtain a registered infrared light fundus image; Step S130: Generate an RGB fusion image according to the fundus image collected under visible light and the registered infrared light fundus image; Step S140: Adjust the color information of the RGB fusion image based on a standard fundus image to obtain a target fundus color photo.

[0025] Among them, the fundus image collected under infrared light and the fundus image collected under visible light are images obtained by collecting the fundus of the same patient under the light source environment of infrared light and the light source environment of visible light respectively. The embodiments of the present application do not make specific limitations on the wavelengths of infrared light and visible light. For example, the wavelengths of infrared light and visible light can be 785 nm and 525 nm respectively.

[0026] Among them, the embodiments of the present application do not make special limitations on the execution order between the two sub-steps in Step S110. Either sub-step can be executed first and the other sub-step can be executed later, or they can be executed simultaneously.

[0027] Among them, in the embodiments of the present application, by extracting the blood vessel feature image to register and align the fundus image collected under infrared light with the fundus image collected under visible light, the spatial deviation of the multimodal collected fundus images can be eliminated, the complementary information of infrared light and visible light can be integrated, and the role of multimodal imaging can be maximized.

[0028] Among them, in the embodiments of the present application, through spatial fusion and color coding of the fundus image collected under visible light and the registered infrared light fundus image, the "surface color feature" exhibited by the fundus under the excitation of the visible light source and the "deep gray feature" exhibited by the fundus under the excitation of the infrared light source are integrated into an RGB fusion image, which not only retains the intuitiveness of traditional fundus color photos but also supplements the penetration advantage of infrared light, making the information content contained in the RGB fusion image higher.

[0029] Among them, in the embodiment of the present application, the color information of the RGB fusion image is adjusted based on the standard fundus image to obtain the target fundus color photo, converting the RGB fusion image, which may have problems such as color distortion and color overload due to light source differences, into the target fundus color photo, realizing the key conversion from technical implementation to clinical application of multimodal imaging. By establishing a unified color diagnosis language, problems such as color deviation, device differences, and visual cognitive conflicts in cross-modal imaging are solved, enabling the finally obtained target fundus color photo to not only retain the deep information of infrared light but also conform to the doctor's visual habits of traditional fundus color photos, ultimately facilitating the improvement of the accuracy and repeatability of fundus disease diagnosis.

[0030] The fundus color photo synthesis method provided by the embodiment of the present application extracts the infrared light blood vessel feature image from the fundus image collected by infrared light, and extracts the visible light blood vessel feature image from the fundus image collected by visible light. Based on the infrared light blood vessel feature image and the visible light blood vessel feature image, the fundus image collected by infrared light is aligned with the fundus image collected by visible light to obtain the registered infrared fundus image. According to the fundus image collected by visible light and the registered infrared fundus image, an RGB fusion image is generated. Based on the standard fundus image, the color information of the RGB fusion image is adjusted to obtain the target fundus color photo. By performing spatial fusion and color conversion on multi-band light source images, high-quality fundus color photos are synthesized, which can not only provide sufficient diagnostic information to facilitate the accurate identification and analysis of fundus diseases by doctors but also facilitate doctors' reading and understanding of the images, enabling them to quickly give the results of fundus disease identification and analysis.

[0031] In some embodiments, generating the RGB fusion image according to the fundus image collected by visible light and the registered infrared fundus image (i.e., involved in step S130) is as Figure 2 shown and includes: Step S210, based on a preset minimum radius and a preset maximum radius, both the fundus image collected by visible light and the registered infrared fundus image are divided into a central region, a transition region, and a peripheral region; Step S220, according to the transition region of the fundus image collected by visible light and the transition region of the registered infrared fundus image, a color transition region image is generated; Step S230, mapping the pixel values of the central region of the fundus image collected by visible light to the RGB three channels to obtain a color central region image; and mapping the pixel values of the peripheral region of the registered infrared fundus image to the RGB three channels to obtain a color peripheral region image; Step S240, splicing the color central region image, the color transition region image, and the color peripheral region image to obtain the RGB fusion image.

[0032] Among them, in the embodiments of the present application, the values of the preset minimum radius r and the preset maximum radius R are not specifically limited. For example, they can be determined according to the actual size of the fundus image collected by infrared light / the fundus image collected by visible light. And it can be understood that the value of the preset minimum radius r is less than the value of the preset maximum radius R.

[0033] Among them, the central region refers to the part that does not exceed r from the center point of the image, the transition region refers to the part that exceeds r and does not exceed R from the center point of the image, and the peripheral region refers to the part that exceeds R from the center point of the image.

[0034] Among them, it can be understood that the RGB fusion image obtained in the embodiments of the present application also includes a central region, a transition region, and a peripheral region, that is, a color central region image, a color transition region image, and a color peripheral region image. The color central region image is the part of the RGB fusion image that does not exceed r from the center point of the image and is generated according to the central region of the fundus image collected by visible light. The color transition region image is the part of the RGB fusion image that exceeds r and does not exceed R from the center point of the image and is generated according to the transition region of the fundus image collected by visible light and the transition region of the registered infrared fundus image. The color peripheral region image is the part of the RGB fusion image that exceeds R from the center point of the image and is generated according to the peripheral region of the registered infrared fundus image.

[0035] Among them, in the embodiments of the present application, there is no specific limitation on how to map pixel values to the RGB three channels. For example, they can be copied to the R, G, and B channels, or distributed according to requirements.

[0036] In some embodiments, generating a color transition region image according to the transition region of the fundus image collected by visible light and the transition region of the registered infrared fundus image (i.e., involved in step S220), as Figure 3 shown, includes: Step S310, for two pixels with the same pixel coordinates in the transition region of the fundus image collected by visible light and the transition region of the registered infrared fundus image, determine the corresponding RGB three-channel values for their pixel coordinates according to the weight parameters of the two pixels and their respective pixel values; wherein, the weight parameter linearly changes with the distance between the pixel coordinates and the center point of the image; Step S320, generate a color transition region image according to each pixel coordinate in the transition region and its corresponding RGB three-channel values.

[0037] Among them, in the embodiments of the present application, by setting weight parameters for each pixel coordinate in the transition region, linear superposition and gradual change processing of pixels in the two transition regions are realized.

[0038] Among them, the embodiments of the present application do not specifically limit how the weight parameter linearly changes with the distance between the pixel coordinates and the center point of the image. For example, for any pixel in the transition region, the distance from it to the center of the image is d (r ≤ d ≤ R). For the transition region of the visible light acquired fundus image, the weight parameter decreases linearly with the distance increasing, from 1 (when d = r) to 0 (when d = R). For the transition region of the registered infrared light fundus image, the weight parameter increases linearly with the distance increasing.

[0039] In some embodiments, based on the infrared light blood vessel feature image and the visible light blood vessel feature image, aligning the infrared light acquired fundus image with the visible light acquired fundus image to obtain a registered infrared light fundus image (i.e., involved in step S120), as Figure 4 shown, includes: Step S410, optimizing the parameters of the initial affine matrix according to the infrared light blood vessel feature image and the visible light blood vessel feature image to obtain an optimized target affine matrix; Step S420, applying the target affine matrix to the infrared light acquired fundus image to obtain a registered infrared light fundus image.

[0040] The applicant of the present application proposes that the challenge in registering and aligning fundus images is that the blood vessels in the outermost periphery of the field of view are basically straight, without bifurcation points or corner points of blood vessels, so they are not suitable for key point registration. Therefore, the applicant of the present application proposes to iteratively optimize the initial affine matrix, and then register the infrared light acquired fundus image with the visible light acquired fundus image through the target affine matrix.

[0041] Among them, the embodiments of the present application do not specifically limit the initial affine matrix. For example, a two-dimensional affine matrix (Affine Matrix) can be used, which can be represented as a 3×3 matrix, that is , and the optimization target is the six parameters a, b, c, d, e, f of the Affine Matrix.

[0042] Among them, applying the affine matrix to an image means using the parameters of the affine matrix to transform the initial pixel coordinates (x, y) to obtain the transformed pixel coordinates ( , ). For example, applying a two-dimensional affine matrix to an image, the coordinates (x, y) of the pixel can be transformed using the parameters a, b, c to obtain the transformed coordinates , and the coordinates (x, y) of the pixel can be transformed using the parameters d, e, f to obtain the transformed coordinates 。

[0043] In some embodiments, according to the infrared light blood vessel feature image and the visible light blood vessel feature image, the parameters of the initial affine matrix are optimized to obtain the optimized target affine matrix (i.e., involved in step S410), as Figure 5 shown, including: Step S510, applying the initial affine matrix to the infrared light blood vessel feature image to obtain a projection feature image; Step S510, according to the gradient descent algorithm, optimizing the parameters of the initial affine matrix until the optimization loss is minimized, and obtaining the target affine matrix; wherein, the optimization loss is determined by the coincidence rate between the projection feature image and the visible light blood vessel feature image.

[0044] It can be understood that to obtain the target affine matrix with the minimum optimization loss, multiple gradient descent iterations may be required. The optimization loss function is the coincidence rate between the projection feature image and the visible light blood vessel feature image. When the coincidence rate is the smallest, it represents the minimum optimization loss function and the maximum optimization loss. On the contrary, when the coincidence rate is the largest, it represents the maximum optimization loss function and the minimum optimization loss.

[0045] In some embodiments, extracting the blood vessel feature image from the collected fundus image (i.e., involved in step S110), as Figure 6 shown, including: Step S610, adjusting the brightness of the collected fundus image; Step S620, optimizing the blood vessel network pixels in the image after adjusting the brightness; Step S630, respectively extracting the upper segmentation sub-image and the lower segmentation sub-image from the optimized image according to a preset segmentation ratio; wherein, the value of the preset division ratio includes [2 / 3, 1); Step S640, respectively extracting the largest sub-blood vessel network from the upper segmentation sub-image and the lower segmentation sub-image and superimposing them to obtain the blood vessel feature image.

[0046] It can be understood that the above step S110 involves two sub-steps, namely extracting the visible light blood vessel feature image from the visible light collected fundus image, and extracting the infrared light blood vessel feature image from the infrared light collected fundus image. However, the execution principles of the two sub-steps can be the same, that is, both are extracted through steps S610 - S640.

[0047] Among them, the embodiments of the present application do not specifically limit how to adjust the brightness of the acquired fundus image. For example, the brightness can be adjusted by performing Contrast Limited Adaptive Histogram Equalization (CLAHE) on the acquired fundus image.

[0048] Among them, vascular network pixels refer to the pixels that make up the vascular network. Optimizing the vascular network pixels in the image after brightness adjustment involves both identifying the vascular network pixels in the image after brightness adjustment and filling in the discontinuous parts of the vascular network.

[0049] The applicant of the present application proposes that considering that the discontinuity of the blood vessels at the middle optic cup and disc may cause the largest sub-vascular network and the second largest sub-vascular network in an image to be the upper half blood vessels and the lower half blood vessels respectively, by first dividing the optimized image into two sub-images according to a preset segmentation ratio and then extracting the largest sub-vascular network, such a situation can be effectively avoided, so that a clear vascular network can be obtained, and thus a clearer and more accurate vascular feature image can be obtained.

[0050] Among them, the embodiments of the present application do not specifically limit the value of the preset segmentation ratio, as long as the value is in the interval [2 / 3, 1). As an optimal implementation, the value of the preset segmentation ratio can be 2 / 3.

[0051] Among them, the upper segmented sub-image refers to the sub-image with a larger area obtained by dividing from the upper end to the lower end of the image and dividing when reaching the preset segmentation ratio (for example, when the preset segmentation ratio is 2 / 3, the upper 2 / 3 part of the image) is the upper segmented sub-image. The lower segmented sub-image can be understood in the same way.

[0052] In some embodiments, the optimization of the vascular network pixels in the image after brightness adjustment (i.e., involved in step S620), as Figure 7 shown, includes: Step S710, using a ridge filter to identify ridge feature pixels from the image after brightness adjustment, and determining the ridge feature pixels with a confidence level higher than the confidence threshold as vascular pixels; among them, the confidence threshold is determined by the global mean and standard deviation of the pixel values of the image after brightness adjustment; Step S720, performing dilation and contraction processing on the image after brightness adjustment to fill in the discontinuous areas of the vascular pixels.

[0053] Among them, in the embodiment of the present application, the ridge filter is first used to perform pattern recognition on the reticular blood vessel region features. For the recognized ridge feature pixel regions, the regions with a confidence level higher than the confidence level threshold are recognized as blood vessels. Then, post-processing of dilation and contraction is performed on the image after adjusting the brightness, and the discontinuous regions of blood vessel pixels are filled.

[0054] Among them, the embodiment of the present application does not specifically limit how to determine the confidence level threshold based on the global mean and standard deviation of the pixel values of the image after adjusting the brightness. For example, the confidence level threshold can be set to , where represents the global mean, and represents the standard deviation.

[0055] Among them, it can be understood that before performing dilation and contraction processing on the image, the image is usually binarized first.

[0056] In some embodiments, based on the standard fundus image, the color information of the RGB fusion image is adjusted to obtain the target fundus color photo (i.e., related to step S140), as Figure 8 shown, including: Step S810, determining the global mean and standard deviation of each of the RGB three channels of the standard fundus image to linearly scale the RGB three-channel values of the RGB fusion image; Step S820, converting the linearly scaled RGB fusion image into a first Lab image and converting the standard fundus image into a second Lab image; Step S830, determining the global mean of the L channel of the second Lab image to linearly scale the L channel value of the first Lab image, and linearly scaling the histograms of the a channel and b channel of the first Lab image according to the histograms of the a channel and b channel of the second Lab image to obtain the target fundus color photo.

[0057] Among them, the standard fundus image is the template fundus image. In the embodiment of the present application, the global mean and standard deviation of each of the RGB three channels of the standard fundus image are first obtained, and the RGB three-channel values of the RGB fusion image are linearly scaled to the global mean and standard deviation of each of the RGB three channels of the standard fundus image to achieve preliminary color adjustment of the RGB fusion image.

[0058] Furthermore, in the embodiments of the present application, both the RGB fusion image and the standard fundus image are converted into the Lab color space, and the conversion results are represented by the first Lab image and the second Lab image respectively. The global mean value of the L channel of the second Lab image is obtained, and the L channel values of the first Lab image are linearly scaled to the global mean value of the L channel of the second Lab image, which can ensure the consistency of the overall brightness of the finally obtained target fundus color photo.

[0059] In the embodiments of the present application, the histograms of the a channel and the b channel of the first Lab image are also linearly scaled and fitted to the histograms of the a channel and the b channel of the second Lab image, which can ensure that the color distribution of the finally obtained target fundus color photo is highly consistent with the color distribution of the standard fundus image.

[0060] In addition, the applicant of the present application also proposes that after step S140, the area that is completely un-overlapped at the edge of the target fundus color photo can be cropped, which can ensure that the image structure of the finally obtained target fundus color photo is complete and the visual effect is better.

[0061] The following refers to Figure 9 as shown, and in combination with a specific embodiment, the fundus color photo synthesis method provided by the embodiments of the present application will be described again. As Figure 9 shown, mainly through steps such as ridge feature extraction, binarization, and maximum subgraph extraction, vascular feature images are respectively extracted from the fundus image collected by infrared light (IR image) and the fundus image collected by visible light (VL image). Furthermore, according to the two-dimensional affine matrix (Affine Matrix) , the infrared light fundus image, and the visible light vascular feature image, the infrared light fundus image is aligned with the visible light fundus image to obtain a registered infrared light fundus image. Furthermore, according to the visible light fundus image (VL image) and the registered infrared light fundus image (IR image), an RGB fusion image is generated. Furthermore, the RGB fusion image is subjected to Lab color conversion to the standard fundus image to obtain a target fundus color photo. Furthermore, the edge of the target fundus color photo is cropped.

[0062] As the second aspect of the embodiments of the present application, an electronic device is provided, where, as Figure 10 shown, the electronic device includes: One or more processors 101; A memory 102, on which one or more computer programs are stored. When the one or more computer programs are executed by the one or more processors 101, the one or more processors 101 implement the fundus color photo synthesis method provided by the first aspect of the embodiments of the present application.

[0063] The electronic device may further include one or more I / O interfaces 103, which are connected between the processor 101 and the memory 102 and are configured to implement information interaction between the processor 101 and the memory 102.

[0064] Among them, the processor 101 is a device with data processing capabilities, including but not limited to a central processing unit (CPU), etc.; the memory 102 is a device with data storage capabilities, including but not limited to a random access memory (RAM), more specifically such as SDRAM, DDR, etc., a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), and a flash memory (FLASH); the I / O interface (read / write interface) is connected between the processor and the memory and can implement information interaction between the processor and the memory, including but not limited to a data bus (Bus), etc.

[0065] In some embodiments, the processor 101, the memory 102, and the I / O interface 103 are interconnected through a bus 104 and are further connected to other components of the computing device.

[0066] As the third aspect of the embodiments of the present application, as Figure 11 shown, a computer-readable medium is provided, on which a computer program is stored, where the computer program, when executed by a processor, implements the fundus color photo synthesis method provided in the first aspect of the embodiments of the present application.

[0067] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. Accordingly, the computer program can be stored in a non-volatile computer-readable storage medium, and when the computer program is executed, it can implement the methods of any of the above embodiments. Among them, any reference to a memory, storage, database, or other medium used in the embodiments provided by the present application can include non-volatile and / or volatile memories. Non-volatile memories can include read-only memories (ROMs), programmable ROMs (PROMs), electrically programmable ROMs (EPROMs), electrically erasable programmable ROMs (EEPROMs), or flash memories. Volatile memories can include random access memories (RAMs) or external cache memories. By way of illustration and not limitation, RAMs are available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and Rambus dynamic RAM (RDRAM), etc.

[0068] The above is only the specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Those skilled in the art should understand that the present application includes but is not limited to the content described in the drawings and the above specific implementation manner. Any modification that does not deviate from the functional and structural principles of the present application will be included in the scope of the claims.

Claims

1. A method for synthesizing fundus color photographs, characterized in that, The method includes: extracting an infrared light blood vessel feature image from a fundus image acquired with infrared light, and extracting a visible light blood vessel feature image from a fundus image acquired with visible light; aligning the infrared light acquired fundus image to the visible light acquired fundus image based on the infrared light blood vessel feature image and the visible light blood vessel feature image to obtain a registered infrared light fundus image; generating an RGB fusion image according to the visible light acquired fundus image and the registered infrared light fundus image; adjusting the color information of the RGB fusion image based on a standard fundus image to obtain a target fundus color photo.

2. The method according to claim 1, characterized in that The generating an RGB fusion image according to the visible light acquired fundus image and the registered infrared light fundus image includes: dividing both the visible light acquired fundus image and the registered infrared light fundus image into a central region, a transition region, and a peripheral region based on a preset minimum radius and a preset maximum radius; generating a color transition region image according to the transition region of the visible light acquired fundus image and the transition region of the registered infrared light fundus image; mapping the pixel values of the central region of the visible light acquired fundus image to the RGB three channels to obtain a color central region image; and mapping the pixel values of the peripheral region of the registered infrared light fundus image to the RGB three channels to obtain a color peripheral region image; stitching the color central region image, the color transition region image, and the color peripheral region image to obtain an RGB fusion image.

3. The method according to claim 2, wherein The generating a color transition region image according to the transition region of the visible light acquired fundus image and the transition region of the registered infrared light fundus image includes: for two pixels with the same pixel coordinates in the transition region of the visible light acquired fundus image and the transition region of the registered infrared light fundus image, determining corresponding RGB three-channel values for their pixel coordinates according to the weight parameters of the two pixels and their respective pixel values; wherein the weight parameter linearly changes with the distance between the pixel coordinates and the center point of the image; generating a color transition region image according to each pixel coordinate in the transition region and its corresponding RGB three-channel values.

4. The method according to claim 1, characterized in that The aligning the infrared light acquired fundus image to the visible light acquired fundus image based on the infrared light blood vessel feature image and the visible light blood vessel feature image to obtain a registered infrared light fundus image includes: optimizing the parameters of an initial affine matrix according to the infrared light blood vessel feature image and the visible light blood vessel feature image to obtain an optimized target affine matrix; applying the target affine matrix to the infrared light acquired fundus image to obtain a registered infrared light fundus image.

5. The method according to claim 4, wherein The optimizing the parameters of an initial affine matrix according to the infrared light blood vessel feature image and the visible light blood vessel feature image to obtain an optimized target affine matrix includes: applying the initial affine matrix to the infrared light blood vessel feature image to obtain a projected feature image; According to the gradient descent algorithm, optimize the parameters of the initial affine matrix until the optimization loss is minimized to obtain the target affine matrix; wherein, the optimization loss is determined by the coincidence rate between the projected feature image and the visible light blood vessel feature image.

6. The method according to claim 1, wherein Extract the blood vessel feature image from the collected fundus image, including: Adjust the brightness of the collected fundus image; Optimize the blood vessel network pixels in the image after brightness adjustment; According to a preset segmentation ratio, respectively extract an upper segmentation sub-image and a lower segmentation sub-image from the optimized image; wherein, the value range of the preset segmentation ratio is [2 / 3, 1); Extract the largest sub-blood vessel network from the upper segmentation sub-image and the lower segmentation sub-image respectively and superimpose them to obtain the blood vessel feature image.

7. The method according to claim 6, characterized in that The optimization of the blood vessel network pixels in the image after brightness adjustment includes: Use a ridge filter to identify ridge feature pixels from the image after brightness adjustment, and determine the ridge feature pixels with a confidence level higher than the confidence threshold as blood vessel pixels; wherein, the confidence threshold is determined by the global mean and standard deviation of the pixel values of the image after brightness adjustment; Perform dilation and contraction processing on the image after brightness adjustment to fill the discontinuous areas of the blood vessel pixels.

8. The method according to claim 1, characterized in that The adjustment of the color information of the RGB fusion image based on the standard fundus image to obtain the target fundus color photo includes: Determine the global mean and standard deviation of each of the RGB three channels of the standard fundus image to linearly scale the RGB three channel values of the RGB fusion image; Convert the linearly scaled RGB fusion image into a first Lab image and convert the standard fundus image into a second Lab image; Determine the global mean of the L channel of the second Lab image to linearly scale the L channel value of the first Lab image, and linearly scale the histograms of the a channel and b channel of the first Lab image according to the histograms of the a channel and b channel of the second Lab image to obtain the target fundus color photo.

9. An electronic device, characterized in that, The electronic device includes: One or more processors; A memory, on which one or more computer programs are stored. When the one or more computer programs are executed by the one or more processors, the one or more processors implement the fundus color photo synthesis method according to any one of claims 1-8.

10. A computer-readable medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the fundus color photo synthesis method according to any one of claims 1-8.

Images