Image fusion method and device and endoscope

Abstract

The invention discloses an image fusion method and device and an endoscope, and relates to the field of image processing, and the method comprises the steps: obtaining a visible light image and a fluorescence image collected in the same region; filtering the visible light image to obtain a filtered image and a texture image; carrying out image fusion on the filtered image and the fluorescence image to obtain a preliminary fusion image; and fusing the preliminary fusion image and the texture image to obtain a final output image. The fluorescence image and the visible light image are image acquisition of the same area, so that although the representation modes are different, the essential area content is consistent. The visible light image is separated into the filtered image and the texture image through filtering and then fused, the accuracy of the fused image can be improved, the filtered image and the fluorescence image are fused and then added into the texture image, texture information can be effectively protected, and the stability of the fused image is improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular, to a method, device, and endoscope for image fusion. Background Art

[0002] Fluorescence imaging technology mainly labels human tissues with fluorescent dyes, and then irradiates the area with light of a specific wavelength to induce the fluorescent dyes to emit fluorescence. Endoscope systems integrated with fluorescence imaging technology have characteristics such as high definition, high sensitivity, and high safety, and are applied to early cancer screening or resection applications in multiple medical fields, including hepatobiliary surgery, gastrointestinal surgery, thoracic surgery, urology, and gynecology. During the operation, such endoscopes can help doctors quickly and accurately identify and resect the labeled tumor area, reducing the possibility of misresection. However, fluorescence imaging technology can only display the fluorescence intensity distribution image and cannot reflect the texture information of human tissues. Visible light imaging technology can clearly display the color and texture information of human tissues irradiated by visible light sources. Displaying the visible light image and the fluorescence image on the display simultaneously not only reduces the display size of the image or increases the volume of the display, but also requires doctors to compare the two images based on experience, reducing the operation efficiency of the endoscope. Summary of the Invention

[0003] The object of the present invention is to provide a method, device, and endoscope for image fusion. By filtering the visible light image into a filtered image and a texture image and then fusing them, the accuracy of the fused image can be improved. After fusing the filtered image with the fluorescence image and then adding the texture image, the texture information can be effectively protected and the stability of the fused image can be improved.

[0004] To solve the above technical problems, the present invention provides a method for image fusion, including:

[0005] Obtaining a visible light image and a fluorescence image collected for the same area;

[0006] Filtering the visible light image to obtain a filtered image and a texture image;

[0007] Performing image fusion on the filtered image and the fluorescence image to obtain a preliminary fused image;

[0008] Fusing the preliminary fused image with the texture image to obtain a final output image.

[0009] On the other hand, a visible light image acquisition module and a fluorescence image acquisition module are jointly provided on the endoscope;

[0010] Obtaining a visible light image and a fluorescence image collected for the same area includes:

[0011] Control the visible light image acquisition module and the fluorescence image acquisition module to acquire images of the acquisition range where the endoscope is located;

[0012] Obtain the visible light image acquired by the visible light image acquisition module and the fluorescence image acquired by the fluorescence module.

[0013] On the other hand, filter the visible light image to obtain a filtered image and a texture image, including:

[0014] Select a pixel point P in the visible light image i,j , the pixel point P i,j is a pixel point with the upper left corner of the visible light image as the origin and relative coordinates (i, j);

[0015] Select another pixel point Q from the visible light image u,v , the pixel point Q u,v is a pixel point with the upper left corner of the visible light image as the origin and relative coordinates (u, v);

[0016] Control the pixel point P i,j to remain stationary, move the pixel point Q in the visible light image u,v , and calculate the filtering values of each color channel of the pixel point P i,j based on the pixel point Q u,v until the pixel point Q u,v traverses each pixel point in the visible light image;

[0017] Move the position of the pixel point P i,j , and return to control the pixel point P i,j to remain stationary, move the pixel point Q in the visible light image u,v , and calculate the filtering values of each color channel of the pixel point P i,j based on the pixel point Q u,v until the pixel point Q u,v traverses each pixel point in the visible light image;

[0018] According to the filtering values of the pixel point P i, at various positions in the visible light image, obtain the filtered image of the visible light image;

[0019] Subtract the filtered image from the visible light image for each color channel respectively to obtain the texture image of the visible light image.

[0020] On the other hand, control the pixel point P i,j to remain stationary, move the pixel point Q in the visible light image u,v , and calculate the pixel point P i,j based on the pixel point Qu,v filtering values of each color channel until pixel point Q u,v Traverse each pixel point in the visible light image, including:

[0021] Determine area A centered on pixel point P i,j and area B centered on pixel point Q u,v The first area and the second area are of the same size;

[0022] Determine the filtering weight of pixel point Q according to the similarity between area A and area B u,v of the filtering weight;

[0023] Control pixel point P i,j to remain stationary, move pixel point Q in the visible light image u,v , and calculate the pixel point P according to the filtering weight of the pixel point Q u,v based on the filtering values of each color channel of pixel point Q i,j until pixel point Q u,v Traverse each pixel point in the visible light image. u,v On the other hand, determine the filtering weight of pixel point Q according to the similarity between area A and area B, including:

[0024] Determine the similarity between area A and area B according to the similarity relationship formula, and the similarity relationship formula is u,v ;

[0025] wherein, ;

[0026] where, is the similarity between area A and area B, k represents the row coordinate with the upper left corner of the area as the zero point in area A and area B, l represents the column coordinate with the upper left corner of the area as the zero point in area A and area B, cn represents the number of color channels in the area, A(k, l, cn) represents the pixel value when the number of color channels is cn at a point with relative coordinates (k, l) with the upper left corner of area A as the zero point, B(k, l, cn) represents the pixel value when the number of color channels is cn at a point with relative coordinates (k, l) with the upper left corner of area B as the zero point, the number of rows of pixel points occupied by area A and area B is m, and the number of columns is n;

[0027] Determine the filtering weight of pixel point Q according to the similarity, and the expression of the filtering weight is u,v ; ;

[0028] wherein, is the filtering weight of pixel point Q u,v , and h is the filtering coefficient.

[0029] On the other hand, calculate the pixel point P according to the filtering weight of the pixel point Q u,v based on the filtering values of each color channel of the pixel point Q i,j including: u,v

[0030] Calculate the pixel point P according to the filtering weight of the pixel point Q u,v based on the filtering values of each color channel of the pixel point Q i,j The expression of the filtering value is u,v ;

[0031] where fI t (i, j, cn) is the filtering value of the pixel point P i,j i represents the row number of the pixel point P i,j in the visible light image, j represents the column number of the pixel point P i,j in the visible light image, cn represents the number of color channels, IH represents the height of the visible light image I t IW represents the width of the visible light image I t is the filtering weight of the pixel point Q u,v I t (i, j, cn) is the filtering value of the pixel point Q u,v t

[0032]

[0033] On the other hand, perform image fusion on the filtered image and the fluorescence image to obtain a preliminary fusion image, including:

[0033] Fuse the filtered image with the fluorescence image for each color channel respectively to obtain a preliminary fusion image, and the expression of the preliminary fusion image is ;

[0034] where is the pixel value of a point with relative coordinates (i, j) and color channel number cn in the preliminary fusion image with the upper left corner of the image as the origin, fI t (i, j, cn) is the filtering value of the pixel point P i,j is the preset fusion coefficient when the color channel number is cn, G t (i, j) represents the pixel value of a point with relative coordinates (i, j) and color channel number cn in the fluorescence image t with the upper left corner of the image as the origin.

[0035] On the other hand, fuse the preliminary fusion image with the texture image to obtain a final output image, including:

[0036] Fuse the preliminary fusion image and the texture image to obtain a final output image, and the expression of the final output image is ;

[0037] wherein, is the final output image The pixel value at a point with a relative coordinate of (i, j) and a color channel number of cn in the final output image with the upper left corner of the image as the zero point, is the preliminary fusion image The pixel value at a point with a relative coordinate of (i, j) and a color channel number of cn in the preliminary fusion image with the upper left corner of the image as the zero point, dI t (i, j, cn) is the pixel value at a point with a relative coordinate of (i, j) and a color channel number of cn in the texture image dI t with the upper left corner of the image as the zero point.

[0038] To solve the above technical problems, the present invention also provides an image fusion device, including:

[0039] A memory for storing a computer program;

[0040] A processor for implementing the steps of the above image fusion method when executing the computer program.

[0041] To solve the above technical problems, the present invention also provides an endoscope, including the above image fusion device, and the lens of the endoscope is provided with a visible light image acquisition module and a fluorescence image acquisition module.

[0042] The present invention discloses a method, device and endoscope for image fusion, relating to the field of image processing, including: acquiring a visible light image and a fluorescence image collected for the same region; filtering the visible light image to obtain a filtered image and a texture image; performing image fusion on the filtered image and the fluorescence image to obtain a preliminary fusion image; fusing the preliminary fusion image and the texture image to obtain a final output image. The fluorescence image and the visible light image are images collected for the same region. Therefore, although the representation methods are different, the essential regional content is the same. By filtering the visible light image into a filtered image and a texture image and then performing fusion, the accuracy of the fusion image can be improved. Adding the texture image after fusing the filtered image and the fluorescence image can effectively protect the texture information and improve the stability of the fusion image. Description of the Drawings

[0043] To more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required in the prior art and the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0044] Figure 1 Flowchart of a method for image fusion provided by the present invention;

[0045] Figure 2 Flowchart of a method for image fusion provided by the present invention;

[0046] Figure 3 Structural schematic diagram of an image fusion device provided by the present invention;

[0047] Figure 4 Structural schematic diagram of an endoscope provided by the present invention. Detailed implementation manners

[0048] The core of the present invention is to provide a method, device and endoscope for image fusion. By filtering, the visible light image is separated into a filtered image and a texture image and then fused, which can improve the accuracy of the fused image. After fusing the filtered image with the fluorescence image and then adding the texture image, the texture information can be effectively protected and the stability of the fused image can be improved.

[0049] To make the objectives, technical solutions and advantages of the embodiments of the present invention clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention.

[0050] Figure 1 Flowchart of a method for image fusion provided by the present invention. The method for image fusion includes:

[0051] S11: Obtain the visible light image and the fluorescence image collected for the same region;

[0052] Fluorescence imaging technology mainly labels human tissues with fluorescent dyes, and then irradiates this area with light of a specific wavelength to induce the fluorescent dyes to emit fluorescence. The endoscope system integrated with fluorescence imaging technology has the characteristics of high definition, high sensitivity, high safety, etc., and is applied to early cancer screening or resection applications in multiple medical fields, including hepatobiliary surgery, gastrointestinal surgery, thoracic surgery, urological surgery, and gynecology, etc. During the operation, this type of endoscope can help doctors quickly and accurately identify and resect the marked tumor area, reducing the possibility of misresection. However, fluorescence imaging technology can only display the fluorescence intensity distribution image and cannot reflect the texture information of human tissues, while visible light imaging technology can clearly display the color and texture information of human tissues irradiated by visible light sources, forming complementary information with the fluorescence image. Therefore, a method for fusing visible light images and fluorescence images with high precision and high stability is needed to enrich the image information obtained by the endoscope and provide effective guarantee for the medical diagnosis and treatment process.

[0053] For the visible light image and the fluorescence image collected from the same area, the sizes of the visible light image and the fluorescence image are the same, and the number of pixel points in the visible light image and the fluorescence image is the same and each pixel point corresponds one by one.

[0054] S12: Filter the visible light image to obtain a filtered image and a texture image;

[0055] The filtered image can be understood as a visible light image with lower clarity, while the texture image can reflect the collected texture. The fluorescence image is a fluorescence intensity distribution image.

[0056] S13: Fuse the filtered image and the fluorescence image to obtain a preliminary fused image;

[0057] S14: Fuse the preliminary fused image and the texture image to obtain the final output image.

[0058] When fusing with the fluorescence image, there is no need to use the texture image because the texture level clarity of the texture image will decrease during the fusion process. Therefore, fusing the filtered image and the fluorescence image to obtain a preliminary fused image, and then fusing the preliminary fused image and the texture image to obtain the final output image can effectively protect the texture information and improve the stability and precision of the final output image.

[0059] The present invention discloses a method, apparatus and endoscope for image fusion, relating to the field of image processing, including: acquiring a visible light image and a fluorescence image collected for the same region; filtering the visible light image to obtain a filtered image and a texture image; fusing the filtered image and the fluorescence image to obtain a preliminary fused image; and fusing the preliminary fused image with the texture image to obtain a final output image. Since the fluorescence image and the visible light image are images collected for the same region, although the representation methods are different, the essential regional contents are the same. By filtering the visible light image into a filtered image and a texture image and then performing fusion, the accuracy of the fused image can be improved. Adding the texture image after fusing the filtered image and the fluorescence image can effectively protect the texture information and improve the stability of the fused image.

[0060] Based on the above embodiments:

[0061] In some embodiments, the visible light image acquisition module and the fluorescence image acquisition module are jointly arranged on the endoscope;

[0062] Acquiring a visible light image and a fluorescence image collected for the same region includes:

[0063] Controlling the visible light image acquisition module and the fluorescence image acquisition module to perform image acquisition on the acquisition range where the endoscope is located;

[0064] Acquiring the visible light image collected by the visible light image acquisition module and the fluorescence image collected by the fluorescence module.

[0065] The endoscope of the present invention includes a fluorescence image acquisition module, a visible light image acquisition module and an image fusion device. The fluorescence image acquisition module is installed on the endoscope for acquiring fluorescence images. The visible light image acquisition module is installed on the endoscope for acquiring visible light images. The fluorescence image acquisition module and the visible light acquisition module share an imaging optical path and have the same field of view. The image fusion device is connected to the fluorescence image acquisition module and the visible light image acquisition module by wire and obtains the fluorescence image and the visible light image from the fluorescence image acquisition module and the visible light image acquisition module respectively.

[0066] The fusion module controls the fluorescence image acquisition module to acquire 1 frame of fluorescence image G t , controls the visible light image acquisition module to acquire 1 frame of visible light image I t , where t represents the time point.

[0067] In some embodiments, filtering the visible light image to obtain a filtered image and a texture image includes:

[0068] Selecting a pixel point P in the visible light image i,j , pixel point P i,jis a pixel point with the upper left corner of the visible light image as the zero point and relative coordinates of (i, j);

[0069] Select another pixel point Q from the visible light image u,v , the pixel point Q u,v is a pixel point with the upper left corner of the visible light image as the zero point and relative coordinates of (u, v);

[0070] Control the pixel point P i,j to remain stationary, and move the pixel point Q in the visible light image u,v , and calculate the pixel point P i,j based on the filtering values of each color channel of the pixel point Q u,v until the pixel point Q u,v traverses each pixel point in the visible light image;

[0071] Move the position of the pixel point P i,j , and return to control the pixel point P i,j to remain stationary, and move the pixel point Q in the visible light image u,v , and calculate the pixel point P i,j based on the filtering values of each color channel of the pixel point Q u,v until the pixel point Q u,v traverses each pixel point in the visible light image;

[0072] According to the filtering values of the pixel point P i, at various positions in the visible light image, obtain the filtered image of the visible light image;

[0073] Subtract the filtered image from the visible light image for each color channel respectively to obtain the texture image of the visible light image.

[0074] In some embodiments, control the pixel point P i,j to remain stationary, and move the pixel point Q in the visible light image u,v , and calculate the pixel point P i,j based on the filtering values of each color channel of the pixel point Q u,v until the pixel point Q u,v traverses each pixel point in the visible light image, including:

[0075] Determine the A area centered on the pixel point P i,j and the B area centered on the pixel point Q u,v , and the sizes of the first area and the second area are the same;

[0076] Determine the filtering weight of the pixel point Q u,v according to the similarity between the A area and the B area;

[0077] Control the pixel point P i,jKeep still and move the pixel point Q in the visible light image u,v and calculate the pixel point P according to the filtering weight of the pixel point Q u,v Based on the filtering values of each color channel of the pixel point Q i,j until the pixel point Q traverses every pixel point in the visible light image. u,v u,v t i,j

[0078] The fusion module performs a filtering operation on a pixel point P in the visible light image I t where i represents the row number of the point in the image and j represents the column number of the point in the image. First, take an m×n area A centered on the point P i,j from the visible light image I i,j where m represents the number of rows occupied by the area A and n represents the number of columns occupied by the area A; then, take an m×n area B centered on another pixel point Q t in the visible light image I t where m represents the number of rows occupied by the area B and n represents the number of columns occupied by the area B, u represents the row number of the point in the image, and v represents the column number of the point in the image. u,v t i,j t

[0079] The pixel point P i,j remains still and the pixel point Q is moved sequentially in the visible light image I t and the filtering weight is calculated according to step 3 until all points in the visible light image I u,v are traversed by the pixel point Q t and the filtering values of each color channel of the pixel point P u,v are calculated. The pixel point P i,j is moved sequentially in the visible light image I t and the filtering values of each color channel of each pixel point P i,j are calculated respectively until all pixel points in the visible light image I i,j are traversed by the pixel point P t i,j and the filtered image fI t is obtained. t

[0080] Subtract the filtered image fI t from the visible light image I separately for each color channel t to obtain the texture image dI t .

[0081] It should be noted that the visible light image is composed of three colors: RGB (Red, Green, Blue), so there are three color channels, namely red, green, and blue. Calculations are performed separately for each color channel, and the results obtained are then fused.

[0082] In some embodiments, the filtering weight of pixel point Q is determined according to the similarity between region A and region B, including: u,v

[0083] Determine the similarity between region A and region B according to the similarity relationship formula, and the similarity relationship formula is ;

[0084] wherein, is the similarity between region A and region B, k represents the row coordinate with the upper left corner of the region as the zero point in regions A and B, l represents the column coordinate with the upper left corner of the region as the zero point in regions A and B, cn represents the number of color channels in the region, A(k, l, cn) represents the pixel value when the number of color channels is cn at a point with relative coordinates (k, l) and the upper left corner of the region as the zero point in region A, B(k, l, cn) represents the pixel value when the number of color channels is cn at a point with relative coordinates (k, l) and the upper left corner of the region as the zero point in region B, the number of rows of pixel points occupied by regions A and B is m, and the number of columns is n;

[0085] Determine the filtering weight of pixel point Q u,v according to the similarity, and the expression of the filtering weight is ;

[0086] wherein, is the filtering weight of pixel point Q u,v , and h is the filtering coefficient.

[0087] It can be understood that the higher the similarity between regions A and B, the higher the filtering weight of pixel point Q u,v .

[0088] The method of first fixing pixel point P i,j , moving pixel point Q u,v , and then moving pixel point P i,j until every point in the visible light image is traversed is local filtering, which can improve the calculation efficiency.

[0089] In some embodiments, calculate the filtering value of pixel point P u,v based on each color channel of pixel point Q i,j according to the filtering weight of pixel point Q u,v , including:

[0090] Calculate the filtering value of pixel point P u,v based on each color channel of pixel point Q i,j according to the filtering weight of pixel point Q u,v , and the expression of the filtering value is ;

[0091] ​Among them, fI t (i, j, cn) is the filtering value of pixel point P i,j where i represents the row number of pixel point P i,j in the visible light image, j represents the column number of pixel point P i,j in the visible light image, cn represents the number of color channels, IH represents the height of the visible light image I t and IW represents the width of the visible light image I t . is the filtering weight of pixel point Q u,v and I t (i, j, cn) is the filtering value of pixel point Q u,v .

[0092] During the calculation process, the filtering values of the three color channels are calculated separately to obtain the filtering values of the three color channels of fI t (i, j, 1), fI t (i, j, 2), and fI t (i, j, 3). Then, the visible light image I t is subtracted from the filtering image fI t for each color channel respectively to obtain the texture image dI t .

[0093] In some embodiments, the filtering image and the fluorescence image are subjected to image fusion to obtain a preliminary fusion image, including:

[0094] The filtering image is fused with the fluorescence image for each color channel respectively to obtain a preliminary fusion image. The expression of the preliminary fusion image is ;

[0095] Among them, is the pixel value of a point with a relative coordinate of (i, j) and a color channel number of cn in the preliminary fusion image with the upper left corner of the image as the zero point, fI t (i, j, cn) is the filtering value of pixel point P i,j , is the preset fusion coefficient when the color channel number is cn, and G t (i, j) represents the pixel value of a point with a relative coordinate of (i, j) in the fluorescence image t with the upper left corner of the image as the zero point.

[0096] According to the color channels, the filtering images of red, green, and blue are respectively fused with the fluorescence image to obtain the preliminary fusion images of red, green, and blue, and each color channel has its own preset fusion coefficient.

[0097] In some embodiments, fusing the preliminary fusion image with the texture image to obtain a final output image, including:

[0098] Fusing the preliminary fusion image with the texture image to obtain a final output image, and the expression of the final output image is ;

[0099] where is the pixel value of the point with relative coordinates (i, j) and color channel number cn in the final output image with the upper left corner of the image as the zero point, is the preliminary fusion image the pixel value of the point with relative coordinates (i, j) and color channel number cn in it, dI t (i, j, cn) is the pixel value of the point with relative coordinates (i, j) and color channel number cn in the texture image dI t with the upper left corner of the image as the zero point.

[0100] Similarly, when performing image fusion, according to the color channels, the preliminary fusion images of red, green, and blue are respectively fused with the texture image to obtain the final output images of red, green, and blue.

[0101] Figure 3 FIG.

[0102] is a schematic structural diagram of an image fusion device provided by the present invention. The image fusion device includes:

[0103] A memory 31 for storing a computer program;

[0104] For the introduction of the image fusion device provided by the present application, please refer to the above embodiments, and details will not be described herein.

[0105] Figure 4 FIG.

[0106] is a schematic structural diagram of an endoscope provided by the present invention. The endoscope includes the above image fusion device 3, and the lens 4 of the endoscope is provided with a visible light image acquisition module 1 and a fluorescence image acquisition module 2.

[0107] The endoscope of the present invention at least includes a visible light image acquisition module 1, a fluorescence image acquisition module 2, and an image fusion device 3. The fluorescence image acquisition module 2 is installed on the lens 4 of the endoscope for acquiring fluorescence images. The visible light image acquisition module 1 is installed on the lens 4 of the endoscope for acquiring visible light images. The fluorescence image acquisition module 2 and the visible light acquisition module 1 share an imaging optical path and have the same field of view. The image fusion device 3 is connected to the fluorescence image acquisition module 2 and the visible light image acquisition module 1 by wire, and obtains the fluorescence image and the visible light image from the fluorescence image acquisition module 2 and the visible light image acquisition module 1 respectively.

[0108] It should also be noted that in this specification, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising a..." does not exclude the existence of additional identical elements in the process, method, article or device comprising the element.

[0109] Those skilled in the art can further realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. To clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described according to functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention.

[0110] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be obvious to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but will be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for image fusion, characterized in that: include: Obtaining visible light images and fluorescence images collected from the same area; Filtering the visible light image to obtain a filtered image and a texture image; Performing image fusion on the filtered image and the fluorescent image to obtain a preliminary fused image; The preliminary fused image is fused with the texture image to obtain a final output image.

2. The image fusion method according to claim 1, characterized in that: The visible light image acquisition module and the fluorescence image acquisition module are jointly arranged on the endoscope; Obtain visible light images and fluorescence images collected from the same area, including: Controlling the visible light image acquisition module and the fluorescent image acquisition module to acquire images within the acquisition range where the endoscope is located; The visible light image acquired by the visible light image acquisition module and the fluorescent image acquired by the fluorescent module are acquired.

3. The image fusion method according to claim 1, characterized in that: Filtering the visible light image to obtain a filtered image and a texture image includes: Select a pixel point P in the visible light image i,j , the pixel point P i,j is a pixel point with the upper left corner of the visible light image as the zero point and the relative coordinates as (i, j); Select another pixel Q from the visible light image u,v , the pixel point Q u,v is a pixel point with the upper left corner of the visible light image as the zero point and relative coordinates as (u, v); Control the pixel point P i,j Do not move, move the pixel Q in the visible light image u,v , and calculate the pixel point P i,j Based on the pixel point Q u,v The filtering values ​​of each color channel until the pixel point Q u,v Traversing each pixel in the visible light image; Move the pixel point P i,j The position of the pixel P is returned. i,j Do not move, move the pixel Q in the visible light image u,v Steps, and calculate the pixel point P i,j Based on the pixel point Q u,v The filtering values ​​of each color channel until the pixel point Q u,v A step of traversing each pixel in the visible light image; According to the pixel point P i, A filtering value at each position in the visible light image is used to obtain a filtered image of the visible light image; The filtered image is subtracted from the visible light image according to each color channel to obtain a texture image of the visible light image.

4. The image fusion method according to claim 3, characterized in that: Control the pixel point P i,j Do not move, move the pixel Q in the visible light image u,v , and calculate the pixel point P i,j Based on the pixel point Q u,v The filtering values ​​of each color channel until the pixel point Q u,v Traversing each pixel in the visible light image includes: Determine the pixel point P i,j The area A as the center and the pixel point Q u,v A region B is centered, and the first region and the second region have the same size; Determine pixel point Q according to the similarity between region A and region B u,v The filter weights of Control the pixel point P i,j Do not move, move the pixel Q in the visible light image u,v , and according to the pixel point Q u,v The filter weight is calculated for the pixel P i,j Based on the pixel point Q u,v The filtering values ​​of each color channel until the pixel point Q u,v Traverse each pixel in the visible light image.

5. The image fusion method according to claim 4, characterized in that: Determine pixel point Q according to the similarity between region A and region B u,v The filter weights include: The similarity between the region A and the region B is determined according to a similarity relationship formula, where: ; in, is the similarity between region A and region B, k represents the row coordinates of region A and region B with the upper left corner of the region as the zero point, l represents the column coordinates of region A and region B with the upper left corner of the region as the zero point, cn represents the number of color channels in the region, A(k,l,cn) represents the pixel value of a point with the upper left corner of the region as the zero point and the relative coordinates (k,l) in region A when the number of color channels is cn, B(k,l,cn) represents the pixel value of a point with the upper left corner of the region as the zero point and the relative coordinates (k,l) in region B when the number of color channels is cn, the number of rows and columns of pixels occupied by region A and region B is m, and the number of columns is n; Determine the pixel point Q according to the similarity u,v The filter weight of the filter is expressed as ; in, is the pixel Q u,v The filter weight is , and h is the filter coefficient.

6. The image fusion method according to claim 5, characterized in that: According to the pixel point Q u,v The filter weight is calculated for the pixel P i,j Based on pixel Q u,v The filter values ​​of each color channel include: According to the pixel point Q u,v The filter weight is calculated for the pixel P i,j Based on pixel Q u,v The filter value of each color channel is expressed as ; Among them, fI t (i,j,cn) is the pixel point P i,j The filter value of i represents the pixel point P i,j The number of rows in the visible light image, j represents the pixel point P i,j The number of columns in the visible light image, cn represents the number of color channels, and IH represents the visible light image I t Height, IW represents the visible light image I t width, is the pixel Q u,v The filter weight, I t (i,j,cn) is the pixel Q u,v The filter value of .

7. The image fusion method according to claim 1, characterized in that: The filtered image and the fluorescent image are fused to obtain a preliminary fused image, including: The filtered image is fused with the fluorescent image according to each color channel to obtain a preliminary fused image. The expression of the preliminary fused image is: ; in, For the initial fusion image The pixel value of a point with the upper left corner of the image as zero and the relative coordinates (i, j) when the number of color channels is cn, fI t (i,j,cn) is the pixel point P i,j The filter value of The preset fusion coefficient when the number of color channels is cn, G t (i, j) represents the fluorescence image G t The pixel value of a point with relative coordinates (i, j) at the upper left corner of the image and zero as the point.

8. The image fusion method according to any one of claims 1 to 7, characterized in that: The preliminary fused image is fused with the texture image to obtain a final output image, including: The preliminary fusion image is fused with the texture image to obtain a final output image. The expression of the final output image is: ; in, For the final output image The pixel value when the number of color channels is cn at a point with relative coordinates (i, j) in the upper left corner of the image as zero point, For the initial fusion image The pixel value of a point with the upper left corner of the image as zero and the relative coordinates (i, j) when the number of color channels is cn, dI t (i,j,cn) is the texture image dI t The pixel value when the number of color channels is cn at a point with relative coordinates (i, j) and the upper left corner of the image as zero.

9. An image fusion device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the image fusion method according to any one of claims 1 to 8 when executing the computer program.

10. An endoscope, characterized in that: It comprises the image fusion device as claimed in claim 9, wherein the lens of the endoscope is provided with a visible light image acquisition module and a fluorescent image acquisition module.

Images