Image fusion method, image fusion device and computer readable storage medium

Abstract

The application provides an image fusion method, an image fusion device and a computer readable storage medium. The image fusion method comprises: adjusting a first channel value of a visible light image by using a fourth channel value of a fluorescence image and a first channel fusion curve to obtain a first fusion channel value; fusing the fourth channel value, a second channel value of the visible light image and / or a third channel value of the visible light image by using a second channel fusion curve to obtain a second fusion channel value; fusing the fourth channel value, the second channel value of the visible light image and / or the third channel value of the visible light image by using a third channel fusion curve to obtain a third fusion channel value; and generating a fusion image of the visible light image and the fluorescence image. In this way, the image fusion device fuses the visible light image and the fluorescence image, so that the fused image has both texture information of the visible light and fluorescence information, and the display effect and details of the image are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to an image fusion method, an image fusion device and a computer readable storage medium. BACKGROUND

[0002] Fluorescent imaging agents can be injected into the body to automatically gather in the tumor and other lesions. Fluorescent imaging agents injected into target blood vessels and tissues emit different wavelengths of fluorescence after absorbing near-infrared light, and the fluorescent image is obtained by capturing the fluorescent signal using a lens receptor.

[0003] The fluorescent image can well reflect the size and location of the lesion, but cannot directly show the current scene; the visible light image can well reflect the texture and details of the current scene, but cannot present the image information outside the visible spectrum, and doctors often need to combine the information of both to make a more accurate diagnosis. However, the current fusion image of the visible light image and the fluorescent image has the problems of uncontrollable color, weak interpretability of the fluorescent path in the fusion image, and missing texture features after fusion, which greatly reduces the display effect and details after image fusion. SUMMARY

[0004] The present application provides an image fusion method, an image fusion device and a computer readable storage medium.

[0005] The present application provides an image fusion method, which comprises:

[0006] obtaining a visible light image and a fluorescent image of the same area;

[0007] adjusting the first channel value of the visible light image using the fourth channel value of the fluorescent image and a first channel fusion curve to obtain a first fusion channel value;

[0008] fusing the fourth channel value of the fluorescent image, the second channel value of the visible light image, and / or the third channel value of the visible light image through a second channel fusion curve to obtain a second fusion channel value;

[0009] fusing the fourth channel value of the fluorescent image, the second channel value of the visible light image, and / or the third channel value of the visible light image through a third channel fusion curve to obtain a third fusion channel value;

[0010] generating a fusion image of the visible light image and the fluorescent image using the first fusion channel value, the second fusion channel value and the third fusion channel value.

[0011] The first channel fusion curve comprises a first channel adjustment curve and a first channel mapping curve.

[0012] The fourth channel value of the fluorescence image is fused with the first channel value of the visible light image to obtain a first fused channel value, including:

[0013] The fourth channel value is adjusted by using the first channel adjustment curve to obtain a fourth channel intensity;

[0014] The first channel value is mapped by using the first channel mapping curve and the fourth channel intensity to obtain a mapped first fused channel value.

[0015] The curve type of the first channel adjustment curve and the first channel mapping curve is an inverse curve, a monotonic curve, an inverse curve, a logarithmic curve, an exponential curve, or a fused curve of any two or more of the above.

[0016] The fourth channel value of the fluorescence image, the second channel value of the visible light image, and / or the third channel value of the visible light image are fused by a second channel fusion curve to obtain a second fused channel value, including:

[0017] The fourth channel value of the fluorescence image is taken as input data of the second channel fusion curve to obtain the second fused channel value by fusion;

[0018] Or, the second channel value of the visible light image is taken as input data of the second channel fusion curve to obtain the second fused channel value by fusion;

[0019] Or, the third channel value of the visible light image is taken as input data of the second channel fusion curve to obtain the second fused channel value by fusion;

[0020] Or, the fourth channel value of the fluorescence image, the second channel value of the visible light image, and the third channel value of the visible light image are fused according to a preset weight to obtain a weighted fused channel value, and the weighted fused channel value is taken as input data of the second channel fusion curve to obtain the second fused channel value by fusion.

[0021] The input of the second channel fusion curve and the input of the third channel fusion curve are different, and / or the curve type of the second channel fusion curve and the curve type of the third channel fusion curve are different.

[0022] After the visible light image and the fluorescence image of the same area are obtained, the image fusion method further includes:

[0023] The visible light image is converted from a first color gamut to a second color gamut for image fusion;

[0024] After the generating the fusion image of the visible light image and the fluorescence image by using the first fusion channel value, the second fusion channel value and the third fusion channel value, the image fusion method further comprises:

[0025] Converting the fusion image from the second color gamut to the first color gamut;

[0026] Fusing the fusion image of the first color gamut with the visible light image of the first color gamut to obtain a final fusion image.

[0027] The fusing the fusion image of the first color gamut with the visible light image of the first color gamut to obtain a final fusion image comprises:

[0028] Mapping the fourth channel value of the fluorescence image through a fourth channel fusion curve to obtain a fusion weight;

[0029] Fusing the fusion image of the first color gamut with the visible light image of the first color gamut according to the fusion weight to obtain a final fusion image.

[0030] After the obtaining the visible light image and the fluorescence image of the same area, the image fusion method further comprises:

[0031] Obtaining a highlight area of each channel of the visible light image;

[0032] Fusing the highlight area of each channel to obtain a highlight area of the visible light image;

[0033] Performing image fusion on an area other than the highlight area of the visible light image.

[0034] The obtaining the highlight area of each channel of the visible light image comprises:

[0035] Obtaining an upper limit and a lower limit of a transition band of a second channel of the visible light image;

[0036] Setting an area of a pixel point whose second channel value is lower than the lower limit of the transition band as a highlight area;

[0037] Setting an area of a pixel point whose second channel value is higher than the upper limit of the transition band as a non-highlight area;

[0038] Setting a pixel point in a range between the upper limit and the lower limit of the transition band of the second channel according to a highlight probability of a preset transition curve as a highlight area or a non-highlight area.

[0039] The application further provides an image fusion device, comprising a processor and a memory, wherein the memory stores program data, and the processor is used to execute the program data to realize the image fusion method as described above.

[0040] The application further provides a computer readable storage medium, which is used to store program data, and the program data is used to realize the image fusion method as described above when executed by a processor.

[0041] The application has the beneficial effect that the image fusion device acquires a visible light image and a fluorescent image of the same region; adjusts a first channel value of the visible light image by using a fourth channel value of the fluorescent image and a first channel fusion curve to obtain a first fusion channel value; fuses the fourth channel value of the fluorescent image, a second channel value of the visible light image and / or a third channel value of the visible light image by a second channel fusion curve to obtain a second fusion channel value; fuses the fourth channel value of the fluorescent image, the second channel value of the visible light image and / or the third channel value of the visible light image by a third channel fusion curve to obtain a third fusion channel value; and generates a fusion image of the visible light image and the fluorescent image by using the first fusion channel value, the second fusion channel value and the third fusion channel value. In this way, the image fusion device fuses the visible light image and the fluorescent image, so that the fused image has both texture information of the visible light and fluorescent information, and the display effect and details of the image are improved. BRIEF DESCRIPTION OF DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the application, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor. Among them:

[0043] Figure 1 is a flowchart of an embodiment of the image fusion method provided by the application;

[0044] Figure 2 is a general flowchart of the image fusion method provided by the application;

[0045] Figure 3 is Figure 1 is a specific flowchart of the image fusion method step S11 shown in the figure;

[0046] Figure 4 is a visible light channel S transition band diagram provided by the application;

[0047] Figure 5 This is a schematic diagram of the visible light path V-channel transition zone provided in this application;

[0048] Figure 6 This is a simulation comparison diagram of adding a highlighting strategy and not adding a highlighting strategy as provided in this application;

[0049] Figure 7 This is a schematic diagram of an embodiment of the H-channel mapping curve provided in this application;

[0050] Figure 8 This is a schematic diagram of an embodiment of the S-channel mapping curve provided in this application;

[0051] Figure 9 This is a schematic diagram of an embodiment of the V-channel mapping curve provided in this application;

[0052] Figure 10 This is a flowchart illustrating another embodiment of the image fusion method provided in this application;

[0053] Figure 11 This is a comparison image provided in this application showing the difference between RGB domain blending without and with RGB domain blending;

[0054] Figure 12 This is a schematic diagram of the structure of an embodiment of the image fusion apparatus provided in this application;

[0055] Figure 13 This is a schematic diagram of another embodiment of the image fusion apparatus provided in this application;

[0056] Figure 14 This is a schematic diagram of an embodiment of the computer-readable storage medium provided in this application. Detailed Implementation

[0057] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0058] In medical settings, fusing visible light images with fluorescence images produces a fused image that reflects both the size and location of lesions and the texture and details of the current scene. Therefore, fusing visible light images with fluorescence images is particularly important.

[0059] To this end, the application provides a fusion method of a visible light image and a fluorescent image, so that the fused image has both the texture information of the visible light and the fluorescent information, realizes effective display of a lesion or normal tissue, and solves the problems of uncontrollable color of the fused image, weak explainability of fluorescent paths in the fused image, missing texture features after fusion, unnatural color transition of the fused image, and unnatural highlight area of the fused image in the prior art.

[0060] Specifically refer to Figure 1 and Figure 2 , Figure 1 is a flowchart of an embodiment of the image fusion method provided by the application, Figure 2 is a general flowchart of the image fusion method provided by the application.

[0061] Among them, the image fusion method of the application is applied to an image fusion device, wherein the image fusion device of the application can be a server, or a system cooperated by a server and a terminal device. Accordingly, each part of the image fusion device, such as each unit, subunit, module, and sub-module, can be all arranged in the server, or can be arranged in the server and the terminal device respectively.

[0062] Further, the server can be hardware or software. When the server is hardware, it can be implemented as a distributed server cluster composed of multiple servers, or as a single server. When the server is software, it can be implemented as multiple software or software modules, such as software or software modules for providing a distributed server, or as a single software or software module, which is not limited here. In some possible implementation manners, the image fusion method of the embodiment of the application can be realized by a processor calling computer readable instructions stored in a memory.

[0063] Specifically, as shown in Figure 1 , the image fusion method of the embodiment of the application specifically includes the following steps:

[0064] Step S11: Obtain a visible light image and a fluorescent image of the same area.

[0065] In the embodiment of the application, the image fusion device collects a visible light image of the same area by using a visible light camera, and collects a fluorescent image of the area at the same time by using a fluorescent camera.

[0066] As Figure 2As shown, the image fusion device converts the visible light image from the RGB color gamut to the HSV color gamut, and fuses the visible light image with the fluorescent image under the HSV color gamut. Further, the image fusion device fuses the visible light image and the fluorescent image using a highlight strategy, that is, the pixel value of the highlight region remains unchanged in the fused image, and this strategy can preserve the true features of the highlight region, so that the fused image is more natural.

[0067] Specifically, the determination method of the highlight region in the highlight strategy mentioned in the present application is described in detail with reference to Figure 3 , Figure 3 is Figure 1 a specific flowchart of the image fusion method step S11.

[0068] As shown in Figure 3 , the image fusion method of the embodiment of the present application specifically includes the following steps:

[0069] Step S111: Obtain the highlight region of each channel of the visible light image.

[0070] In the embodiment of the present application, the highlight region has the characteristic of a small S channel value, that is, a small second channel value, and according to this characteristic and a given S channel highlight determination threshold, the highlight region of the visible light path S channel can be determined.

[0071] In order to make the transition between the highlight region and the non-highlight region smooth and natural, a transition band is added between the highlight region and the non-highlight region, as shown in Figure 4 The part below the lower limit of the transition band is determined as the highlight region, the part above the upper limit of the transition band is determined as the non-highlight region, and the middle part is filled with a continuous curve or a straight line. The curve includes but is not limited to an inverse curve, a monotonic curve, an inverse 2 number curve, a logarithmic curve, an exponential curve, or a fusion curve of any two or more of the above curves, etc. The three parts together constitute the highlight region based on the S channel of the visible light path.

[0072] Among them, the highlight probability between the upper limit of the transition band and the lower limit of the transition band represents the probability that the corresponding S channel value of the visible light path belongs to the highlight region, so that the pixel points in the transition region can be divided into the highlight region and the non-highlight region.

[0073] Meanwhile, the highlight region has the characteristic of a large V channel value, that is, a large third channel value, and according to this characteristic and a given V channel highlight determination threshold, the highlight region of the visible light path V channel can be determined.

[0074] In order to make the transition between the highlight region and the non-highlight region smooth and natural, a transition band is added between the highlight region and the non-highlight region, as shown in Figure 5As shown, the part higher than the upper limit of the transition zone is determined as the highlight region, the part lower than the lower limit of the transition zone is determined as the non-highlight region, and the intermediate part is filled with a continuous curve or a straight line, the curve including but not limited to an inverse curve, a monotonic curve, an inverse curve, a logarithmic curve, an exponential curve, or a fusion curve of any two or more of the above curves, etc., and the three parts together constitute the highlight region based on the visible light path V channel.

[0075] The highlight probability between the upper limit of the transition zone and the lower limit of the transition zone represents the probability that the corresponding visible light path V channel value belongs to the highlight region, so that the pixel points in the transition zone can be divided into the highlight region and the non-highlight region.

[0076] Step S112: Fuse the highlight region of each channel to obtain the highlight region of the visible light image.

[0077] In the embodiment of the present application, the image fusion device fuses the highlight region determined according to the visible light path S channel and the highlight region determined according to the visible light path V channel to obtain the final highlight region, and the fusion manner can be the union of the highlight regions determined by the two channels as the final highlight region.

[0078] The embodiment of the present application uses multiple methods to jointly calculate the highlight region of the visible light path, which can avoid or reduce the highlight region error caused by single threshold segmentation, thereby reducing the influence on subsequent image fusion.

[0079] Step S113: Perform image fusion on the region outside the highlight region of the visible light image.

[0080] In the embodiment of the present application, the image fusion device only performs image fusion with the fluorescent image on the region outside the highlight region of the visible light image, and keeps the pixel value of the pixel point in the highlight region unchanged.

[0081] The simulation comparison chart of the highlight strategy added and the highlight strategy not added is as shown in Figure 6 As shown, when the highlight strategy is not added, the highlight region of the fused image is also colored, which is inconsistent with the actual lighting scene. Therefore, adding the highlight strategy can preserve the true features of the highlight region, making the fused image more natural.

[0082] Step S12: Adjust the first channel value of the visible light image by using the fourth channel value of the fluorescent image and the first channel fusion curve to obtain a first fused channel value.

[0083] In the embodiment of the present application, before fusion, the image fusion device can also pre-process the fluorescent image, and the specific process is as follows:

[0084] The fluorescence path actual response region is obtained by comparing the fluorescence path response intensity with a given fluorescence path threshold. The fluorescence path image is subjected to filtering processing, including but not limited to mean filtering, Gaussian filtering, median filtering and the like, to reduce the influence of fluorescence path noise on the fusion image. The fluorescence path image is subjected to normalization operation to enhance the fluorescence response contrast.

[0085] Further, the image fusion device completes the HSV image fusion by adjusting the HSV channel data of the visible light path according to the fluorescence response intensity. The image fusion process of each color gamut channel of the visible light image is described as follows:

[0086] The H channel value of the HSV domain fusion image is controlled by the fluorescence path Y channel value (i.e. the fourth channel value), the H channel adjustment curve and the H channel mapping curve, so that the color of the fusion image is within the specified color range, and no other color affecting visual judgment appears. The H channel adjustment curve can be obtained by linear fitting or fusion of a single or multiple curves or broken lines, and the curve includes but is not limited to an inverse curve, a logarithmic curve, an inversion curve, an exponential curve and a fused curve thereof. To ensure the continuity of the hue of the fusion image, the H channel mapping curve should be a monotonic curve, including but not limited to an inversion curve, a monotonic broken line, an inverse curve, a logarithmic curve and a fused curve thereof.

[0087] Taking the fluorescence path Y channel value as data, the mapped fluorescence path intensity Y' is obtained through the H channel adjustment curve. The H channel adjustment curve can modify the H channel value of the fluorescence path, change the effect of the fusion region in the fusion image by enhancing or suppressing the contrast or fluorescence response intensity of the specified data range; the H channel mapping curve can convert the color of the fluorescence response region from the original hue to other specified hue, and realize the conversion between different hues by a simple method, thereby highlighting the fluorescence response region in the fusion image. When the H channel mapping curve has monotonicity, the fusion image has fluorescence interpretability. As shown in Figure 7 Figure 7 is a schematic diagram of an embodiment of the H channel mapping curve provided by the present application.

[0088] Step S13: the fourth channel value of the fluorescence image, the second channel value of the visible light image, and / or the third channel value of the visible light image are fused through the second channel fusion curve to obtain a second fusion channel value.

[0089] ​In the embodiment of the present application, the S channel value of the HSV domain fusion image is obtained by the joint action of the visible light path S channel value (second channel value), the visible light path V channel value (third channel value) and the fluorescent light path Y channel value data through a plurality of adjustment curves. The joint regulation of the plurality of curves can enhance the adjustability of the S channel, and the number of curves is determined according to actual needs. Each adjustment curve can be obtained by linear fitting or fusion of a single or multiple curves or broken lines, and the curves include but are not limited to inverse curves, logarithmic curves, inverse curves, exponential curves and their fusion curves.

[0090] It should be noted that the data inputting the plurality of mapping curves of the S channel is determined by the natural light path S channel value, the natural light path V channel value and the fluorescent light path Y channel value, and the determination method includes but is not limited to the following methods: 1) using only the fluorescent light path Y channel data as the data of the mapping curve; 2) using only the visible light path S channel data as the input data of the mapping curve; 3) using only the visible light path V channel as the input data of the mapping curve, as shown in Figure 8 Figure 8 is a schematic diagram of an embodiment of the S channel mapping curve provided by the present application; 4) fusing the fluorescent light path Y channel data, the visible light path S channel data and the visible light path V channel data with a certain weight to obtain the fusion data as the input data of the mapping curve; 5) the plurality of mapping curves of the S channel, and the input of each curve can be any one of the above four cases.

[0091] By the above method, the S channel mapping curve and the input data of the mapping curve are determined, and the S channel of the HSV domain fusion image can be obtained.

[0092] Step S14: fusing the fourth channel value of the fluorescent image, the second channel value of the visible light image and / or the third channel value of the visible light image through a third channel fusion curve to obtain a third fusion channel value.

[0093] In the embodiment of the present application, the V channel value of the HSV domain fusion image is obtained by the joint action of the visible light path S channel value (second channel value), the visible light path V channel value (third channel value) and the fluorescent light path Y channel value data through a plurality of adjustment curves. The joint regulation of the plurality of curves can enhance the adjustability of the V channel, and the number of curves is determined according to actual needs. Each adjustment curve can be obtained by linear fitting or fusion of a single or multiple curves or broken lines, and the curves include but are not limited to inverse curves, logarithmic curves, inverse curves, exponential curves and their fusion curves.

[0094] ​It should be noted that the data of the plurality of mapping curves of the V channel is determined by the natural light path S channel value, the natural light path V channel value and the fluorescent light path Y channel value, and the determination method includes but is not limited to the following methods: 1) only using the fluorescent light path Y channel data as the data of the mapping curve; 2) only using the visible light path S channel data as the input data of the mapping curve; 3) only using the visible light path V channel as the input data of the mapping curve, as shown in Figure 9 Figure 9 is a schematic diagram of an embodiment of the V channel mapping curve provided by the present application; 4) fusing the fluorescent light path Y channel data, the visible light path S channel data and the visible light path V channel data with a certain weight to obtain the fusion data as the input data of the mapping curve; 5) a plurality of mapping curves of the S channel, and the input of each curve can be any one of the above four cases.

[0095] By the above method, the V channel mapping curve and the input data of the mapping curve are determined, and the V channel of the HSV domain fusion image can be obtained.

[0096] It should be noted that when the S channel and the V channel of the HSV domain fusion image are calculated, the visible light path S channel value (the second channel value), the visible light path V channel value (the third channel value) and the fluorescent light path Y channel value data are combined through a plurality of adjustment curves. The image fusion device can control the mapping curve to output different data by adjusting the input of the mapping curve, the curve type of the mapping curve and the curve parameters of the mapping curve, that is, outputting the S channel value and the V channel value respectively.

[0097] Step S15: generating a fusion image of the visible light image and the fluorescent light image by using the first fusion channel value, the second fusion channel value and the third fusion channel value.

[0098] In the embodiment of the present application, the image fusion device generates the fusion image of the visible light image and the fluorescent light image by combining the H channel value, the S channel value and the V channel value of the fusion image calculated by the steps S12 to S14.

[0099] ​In the embodiment of the present application, the image fusion device acquires a visible light image and a fluorescence image of the same region; adjusts a first channel value of the visible light image by using a fourth channel value of the fluorescence image and a first channel fusion curve to obtain a first fused channel value; fuses a second channel value of the visible light image and / or a third channel value of the visible light image by using the fourth channel value of the fluorescence image, the second channel value of the visible light image and / or the third channel value of the visible light image through a second channel fusion curve to obtain a second fused channel value; fuses the second channel value of the visible light image and / or the third channel value of the visible light image by using the fourth channel value of the fluorescence image, the second channel value of the visible light image and / or the third channel value of the visible light image through a third channel fusion curve to obtain a third fused channel value; and generates a fused image of the visible light image and the fluorescence image by using the first fused channel value, the second fused channel value and the third fused channel value. In this way, the image fusion device fuses the visible light image and the fluorescence image, so that the fused image has both texture information of the visible light and fluorescence information, improves the display effect and details of the image, and has fluorescence path interpretability and can simply and quickly realize conversion between different color phases.

[0100] Further, the image fusion device can further fuse the fused image obtained in the above embodiment with the original visible light image, and details are described in the following Figure 10 , Figure 10 is a flowchart of another embodiment of the image fusion method provided by the present application.

[0101] As shown in Figure 10 , the image fusion method of the embodiment of the present application specifically includes the following steps:

[0102] Step S16: converting the fused image from the second color gamut to the first color gamut.

[0103] Step S17: fusing the fused image of the first color gamut with the visible light image of the first color gamut to obtain a final fused image.

[0104] In the embodiment of the present application, the image fusion device converts the fused HSV image, i.e. the fused image obtained in the above embodiment, to the RGB domain to obtain an RGB domain fused image FUSE1. Then, the image fusion device fuses the visible light RGB image with FUSE1 to obtain a final fused image, so that the fused edge transition is more natural.

[0105] As shown in Figure 11 , Figure 11 is a comparison diagram of the RGB domain fusion without adding and the RGB domain fusion with adding provided by the present application, and the color transition of the fused region is smooth and natural through the secondary fusion in the RGB domain.

[0106] Specifically, the fusion weight is obtained by mapping the fluorescence channel Y data through a fusion curve. In order to make the fusion effect natural, the fusion curve needs to have monotonicity. The fusion curve can be obtained by linear fitting or fusion of a single curve or multiple curves, and the fusion curve includes but is not limited to an inverse curve, a logarithmic curve, an inverse curve, an exponential curve and a fused curve thereof.

[0107] The present application fuses the visible light image and the fluorescent image in multiple color gamuts, which makes the fused image color controllable, the fluorescent path interpretable, the texture details rich, the color natural and the transition smooth.

[0108] The present application fuses in the HSV domain and the RGB domain in turn. In the fusion in the HSV domain, the color of the fusion region is controlled by controlling the pixel value of the H channel, avoiding the appearance of similar colors to the environment in the fusion region to interfere with the judgment, and being able to simply and quickly regulate the hue of the fusion region, realizing the conversion between different hues, and making the fusion image retain as many natural light path textures and details as possible. Through the secondary fusion in the RGB domain, the color transition of the fusion region is smooth and natural.

[0109] The present application fuses the H channel of the image with the fluorescent response intensity, making the fusion region have fluorescent interpretability.

[0110] The present application uses a highlight strategy, and the pixel value of the highlight region in the fused image remains unchanged. This strategy can retain the true characteristics of the highlight region, making the fused image more natural.

[0111] The present application uses two highlight determination methods to determine the final highlight region, avoiding the problem of highlight determination failure that may occur due to a single threshold determination.

[0112] Those skilled in the art can understand that in the above method of the specific embodiments, the writing order of each step does not mean a strict execution order and does not constitute any limitation on the implementation process, and the specific execution order of each step should be determined by its function and possible internal logic.

[0113] To realize the image fusion method of the above-mentioned embodiments, the present application further provides an image fusion device, please refer to Figure 12 , Figure 12 is a structure schematic diagram of an embodiment of the image fusion device provided by the present application.

[0114] The image fusion device 200 of the embodiment of the present application comprises an acquisition module 21, a channel module 22 and an image module 23.

[0115] The acquisition module 21 is configured to acquire a visible light image and a fluorescent image of the same region.

[0116] The channel module 22 is configured to adjust the first channel value of the visible light image by using the fourth channel value of the fluorescence image and a first channel fusion curve to obtain a first fusion channel value.

[0117] The channel module 22 is configured to fuse the fourth channel value of the fluorescence image, the second channel value of the visible light image and / or the third channel value of the visible light image by a second channel fusion curve to obtain a second fusion channel value.

[0118] The channel module 22 is configured to fuse the fourth channel value of the fluorescence image, the second channel value of the visible light image and / or the third channel value of the visible light image by a third channel fusion curve to obtain a third fusion channel value.

[0119] The image module 23 is configured to generate a fusion image of the visible light image and the fluorescence image by using the first fusion channel value, the second fusion channel value and the third fusion channel value.

[0120] To implement the image fusion method of the above-mentioned embodiments, the present application further provides another image fusion device, and details are referred to Figure 13 , Figure 13 is a structural schematic diagram of another embodiment of the image fusion device provided by the present application.

[0121] The image fusion device 300 of the embodiment of the present application comprises a memory 31 and a processor 32, wherein the memory 31 and the processor 32 are coupled.

[0122] The memory 31 is configured to store program data, and the processor 32 is configured to execute the program data to implement the image fusion method described in the above-mentioned embodiments.

[0123] In the embodiment, the processor 32 can also be referred to as a CPU (Central Processing Unit). The processor 32 can be an integrated circuit chip with a signal processing capability. The processor 32 can also be a general-purpose processor, a DSP (Digital Signal Process), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor 32 can also be any conventional processor.

[0124] To realize the image fusion method of the above-mentioned embodiments, the present application further provides a computer readable storage medium, such as a computer readable storage medium 400 as shown in the figure. Figure 14 The computer readable storage medium 400 is used to store program data 41, which when executed by a processor, is used to realize the image fusion method as described in the above-mentioned embodiments.

[0125] The present application further provides a computer program product, wherein the above-mentioned computer program product comprises a computer program, and the above-mentioned computer program is operable to cause a computer to execute the image fusion method as described in the embodiments of the present application. The computer program product can be a software installation package.

[0126] The image fusion method described in the above-mentioned embodiments of the present application, when realized in the form of a software functional unit and sold or used as an independent product, can be stored in a device, such as a computer readable storage medium. Based on this understanding, the technical solutions of the present application, essentially or in other words, the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor execute all or part of the steps of the method described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various program code storage media.

[0127] The above-mentioned is only the embodiment of the present application, and does not limit the patent scope of the present application, and any equivalent structure or equivalent process transformation using the content of the specification and the drawings, or direct or indirect application in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. An image fusion method, characterized by, The image fusion method comprises: obtaining a visible light image and a fluorescence image of the same region; adjusting an H channel value of the visible light image by using a Y channel value of the fluorescence image and a first channel fusion curve to obtain a first fusion channel value; fusing by a second channel fusion curve using the Y channel value of the fluorescence image, an S channel value of the visible light image, and / or a V channel value of the visible light image to obtain a second fusion channel value; fusing by a third channel fusion curve using the Y channel value of the fluorescence image, the S channel value of the visible light image, and / or the V channel value of the visible light image to obtain a third fusion channel value; generating a fusion image of the visible light image and the fluorescence image by using the first fusion channel value, the second fusion channel value, and the third fusion channel value; after the visible light image and the fluorescence image of the same region are obtained, the image fusion method further comprises: converting the visible light image from a first color gamut to a second color gamut for image fusion; after the fusion image of the visible light image and the fluorescence image is generated by using the first fusion channel value, the second fusion channel value, and the third fusion channel value, the image fusion method further comprises: converting the fusion image from the second color gamut to the first color gamut; fusing the fusion image of the first color gamut with the visible light image of the first color gamut to obtain a final fusion image.

2. The image fusion method according to claim 1, wherein the first channel fusion curve comprises a first channel adjustment curve and a first channel mapping curve; the adjusting of the H channel value of the visible light image by using the Y channel value of the fluorescence image and the first channel fusion curve to obtain the first fusion channel value comprises: adjusting the Y channel value by using the first channel adjustment curve to obtain a Y channel intensity; mapping the H channel value by using the first channel mapping curve and the Y channel intensity to obtain a mapped first fusion channel value.

3. The image fusion method according to claim 2, wherein the curve types of the first channel adjustment curve and the first channel mapping curve are: an inversion curve, a monotonic curve, an inverse curve, a logarithmic curve, an exponential curve, or a fusion curve of any two or more of the above.

4. The image fusion method according to claim 1, wherein the fusing by the second channel fusion curve using the Y channel value of the fluorescence image, the S channel value of the visible light image, and / or the V channel value of the visible light image to obtain the second fusion channel value comprises: taking the Y channel value of the fluorescence image as input data of the second channel fusion curve to fuse to obtain the second fusion channel value; or, taking the S channel value of the visible light image as input data of the second channel fusion curve to fuse to obtain the second fusion channel value; or, taking the V channel value of the visible light image as input data of the second channel fusion curve to fuse to obtain the second fusion channel value. Or, the Y channel value of the fluorescence image, the S channel value of the visible light image and the V channel value of the visible light image are fused according to a preset weight to obtain a weighted fusion channel value, the weighted fusion channel value is taken as input data of the second channel fusion curve, and the second fusion channel value is obtained through fusion.

5. The image fusion method of claim 4, wherein, the input of the second channel fusion curve is different from the input of the third channel fusion curve, and / or the curve type of the second channel fusion curve is different from the curve type of the third channel fusion curve.

6. The image fusion method of claim 5, wherein, the fusion of the first color gamut fusion image and the visible light image of the first color gamut to obtain the final fusion image comprises: a fusion weight is obtained by mapping the Y channel value of the fluorescence image through a fourth channel fusion curve; the first color gamut fusion image and the visible light image of the first color gamut are fused according to the fusion weight to obtain the final fusion image.

7. The image fusion method of claim 1, wherein, after the visible light image and the fluorescence image of the same area are obtained, the image fusion method further comprises: obtaining a highlight area of each channel of the visible light image; fusing the highlight area of each channel to obtain a highlight area of the visible light image; performing image fusion on the area other than the highlight area of the visible light image.

8. The image fusion method of claim 7, wherein, the obtaining of the highlight area of each channel of the visible light image comprises: obtaining an upper limit and a lower limit of a transition band of the S channel of the visible light image; setting the area of the pixel points with the S channel value lower than the lower limit of the transition band as the highlight area; setting the area of the pixel points with the S channel value higher than the upper limit of the transition band as a non-highlight area; setting the pixel points in the range between the upper limit and the lower limit of the transition band of the S channel according to a highlight probability of a preset transition curve as the highlight area or the non-highlight area.

9. An image fusion apparatus characterized by comprising: The image fusion device comprises a processor and a memory, the memory stores program data, and the processor is used to execute the program data to realize the image fusion method of any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium is used to store program data, and the program data is used to realize the image fusion method of any one of claims 1-8 when executed by a processor.

Images