Dual fluorescence and visible light image fusion method and device, electronic device and medium

Abstract

The present disclosure provides a dual fluorescence and visible light image fusion method and device, electronic equipment and medium, relating to the field of image processing technology. The present disclosure specifically implements the following scheme: simultaneously acquiring a visible light image and a fluorescence original image of a target scene; obtaining a conversion hue, a first saturation and a first brightness based on the fluorescence original image, wherein the conversion hue is a value obtained by linearly converting the hue value of the fluorescence original image; identifying a green component and a blue component based on the conversion hue and the fluorescence original image, and obtaining a mask image having information of the green component and the blue component; and obtaining a fused image based on the visible light image, the mask image, the first saturation and the first brightness, thereby improving the fusion effect of the visible light image and the dual fluorescence images.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to a method and device for fusing dual fluorescence and visible light images, an electronic device, and a computer-readable storage medium, which are applicable to the fields of medical imaging, biological detection, industrial detection, and the like. Background Art

[0002] In recent years, fluorescence navigation endoscopes, with their precise guidance capabilities, have been widely used in surgical procedures, particularly in gynecological and hepatobiliary surgeries for intraoperative tumor marking and cholangiography, becoming a valuable aid for doctors. With the advancement of medical technology, dual-fluorescence navigation endoscopes have emerged. These systems utilize ICG and MB dyes, combined with a dedicated dual-fluorescence navigation system, to achieve simultaneous dual-color marking of two target areas, significantly expanding the application of fluorescence navigation.

[0003] However, the dual-fluorescence navigation system achieves dual-color labeling based on ICG (indocyanine green) and MB (methylene blue) dyes. Although it has expanded the application scenarios of fluorescence navigation, the following core problems still exist in actual applications: First, ICG fluorescence appears purple-blue and MB fluorescence appears purple-red. The two are similar in hue. The human eye has limited ability to distinguish such similar colors. In addition, the fluorescence signal needs to be fused with the white light image to locate the anatomical structure during surgery. The strong brightness of white light will "dilute" the saturation of the fluorescence color, making the purple-blue and purple-red more difficult to distinguish. Especially in complex tissue or bleeding scenarios, the boundary judgment error increases significantly; secondly, the purple-red color of the MB fluorescence original signal is highly similar to the red in the abdominal cavity. The traditional fusion algorithm simply weightedly superimposes the fluorescence and white light signals, and cannot effectively suppress the interference of the tissue's own color, resulting in blurred boundaries of the MB marked area.

[0004] To address the above issues, existing algorithms typically distinguish ICG and MB signals using a single channel in the HSV domain and then perform color overlay. However, this may result in blurred boundaries between the two signals during initial differentiation and be susceptible to background noise, hindering subsequent operations. Summary of the Invention

[0005] The present disclosure provides a method and device for fusing dual fluorescence and visible light images, an electronic device, and a computer-readable storage medium.

[0006] According to a first aspect, a method for fusing dual fluorescence and visible light images is provided, the method comprising: simultaneously acquiring a visible light image and a fluorescence original image of a target scene; obtaining a conversion hue, a first saturation, and a first brightness based on the fluorescence original image, wherein the conversion hue is a value obtained by linearly converting the hue value of the fluorescence original image; identifying a green component and a blue component based on the conversion hue and the fluorescence original image, and obtaining a mask image having information of the green component and the blue component; and obtaining a fused image based on the visible light image, the mask image, the first saturation, and the first brightness.

[0007] According to a second aspect, a dual fluorescence and visible light image fusion device is provided, which includes: an acquisition unit, configured to simultaneously acquire a visible light image and a fluorescence original image of a target scene; an obtaining unit, configured to obtain a conversion hue, a first saturation, and a first brightness based on the fluorescence original image, wherein the conversion hue is a value obtained by linearly converting the hue value of the fluorescence original image; an identification unit, configured to identify the green component and the blue component in the image based on the conversion hue and the fluorescence original image, and obtain a mask image having information of the green component and the blue component; and a fusion unit, configured to obtain a fused image based on the visible light image, the mask image, the first saturation, and the first brightness.

[0008] According to a third aspect, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method described in any implementation manner of the first aspect.

[0009] According to a fourth aspect, a non-transitory computer-readable storage medium storing computer instructions is provided, where the computer instructions are used to cause a computer to execute the method as described in any implementation of the first aspect.

[0010] Embodiments of the present disclosure provide a dual fluorescence and visible light image fusion method and apparatus. First, a visible light image and a fluorescent original image of a target scene are simultaneously acquired. Second, a converted hue, a first saturation, and a first brightness are obtained based on the fluorescent original image. Then, based on the converted hue and the fluorescent original image, the green and blue components are identified to obtain a mask image containing information about the green and blue components. Finally, a fused image is obtained based on the visible light image, the mask image, the first saturation, and the first brightness. Thus, by converting the hue and the fluorescent original image, the green and blue components are accurately identified to form a mask image containing the green and blue components. This effectively distinguishes the foreground from the background, and the mask image allows processing to be performed only on the stained area without affecting other areas. The brightness and saturation of the fluorescent original image are then combined with the saturation and brightness of the visible light image within the area defined by the mask image, maintaining color contrast and avoiding overexposure of the color contrast in the target area corresponding to the mask image, thereby improving the fusion effect of the visible light image and the dual fluorescence image.

[0011] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0013] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0014] Figure 1 is a flow chart of an embodiment of a method for fusing dual fluorescence and visible light images according to the present disclosure;

[0015] Figure 2 is a schematic diagram of a visible light image in the present disclosure;

[0016] Figure 3 is a schematic diagram of a fluorescent raw image in the present disclosure;

[0017] Figure 4 is a schematic diagram of a mask image in the present disclosure;

[0018] Figure 5 This is a structural diagram of the present disclosure using a neural network model to generate a mask image;

[0019] Figure 6 It is a structural schematic diagram of an embodiment of the dual fluorescence and visible light image fusion device disclosed in the present invention;

[0020] Figure 7 4 is a block diagram of an electronic device used to implement the dual fluorescence and visible light image fusion method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0021] Unless expressly stated otherwise, throughout the specification and claims, the term “comprise” or variations such as “include” or “comprising” will be understood to include the stated elements or components but not to exclude other elements or components.

[0022] The technical solutions of the present disclosure are described below through specific examples. It should be understood that one or more steps mentioned in the present disclosure do not exclude the existence of other methods and steps before and after the combination step, or other methods and steps may be inserted between these explicitly mentioned steps. It should also be understood that these examples are only used to illustrate the present disclosure and are not used to limit the scope of the present disclosure. Unless otherwise specified, the numbering of each method step is only for the purpose of identifying each method step, and does not limit the order of arrangement of each method or limit the scope of implementation of the present disclosure. Changes or adjustments in their relative relationships can also be regarded as the scope of implementation of the present disclosure without substantial changes in the technical content.

[0023] The sources of the raw materials and instruments used in the examples are not particularly limited and can be purchased from the market or prepared according to conventional methods known to those skilled in the art.

[0024] In response to the defects in traditional technologies, this paper proposes a method for fusing dual fluorescence and visible light images. By collaboratively optimizing cross-image-saturation and adaptively adjusting the dual fluorescence fusion color, it effectively solves the above problems, improves the accuracy, visual effect and adaptability of the fused image, and meets the needs of different application scenarios. Figure 1 A process 100 of an embodiment of a dual fluorescence and visible light image fusion method according to the present disclosure is shown. The dual fluorescence and visible light image fusion method includes the following steps:

[0025] Step 101 : simultaneously acquire a visible light image and a fluorescent original image of a target scene.

[0026] In this embodiment, a visible light image is an image formed by recording light radiation reflected or emitted by an object within the electromagnetic wavelength range perceptible to the human eye (wavelength range 380-780 nm). A fluorescent raw image is an image generated by the phenomenon of fluorescence (photoluminescence). Fluorescent raw images rely on the photoluminescence properties of fluorescent substances and achieve specific, high-contrast imaging of microscopic targets by precisely controlling the excitation and emission wavelengths.

[0027] In this embodiment, the target scene is a scene represented by a visible light image and a fluorescent original image at the same time, such as a gynecological, hepatobiliary surgery, endoscopy, etc. scene at a certain time. The visible light image and the fluorescent original image can be obtained from a dual-light path optical system that collects images of the target scene. In the dual-light path optical system, the incident light is divided into two paths, one of which is used to collect the visible light image. The visible light image is also a white light original image, that is, an image belonging to the 400-700nm band, such as Figure 2 As shown; the other is used to collect the original fluorescence image, such as Figure 3 As shown in Figure 1, the original fluorescence image is an image in the near-infrared band. In a dual-light path optical system, parallel image acquisition can be achieved through synchronized sensors, eliminating displacement errors caused by time differences.

[0028] Alternatively, visible light images and fluorescence raw images can be obtained from a multispectral sensor set, which uses a single sensor to integrate multi-band pixels (such as a sensor that mimics the compound eye of a mantis shrimp) to simultaneously capture visible light and specific fluorescence wavelengths, thereby obtaining visible light images and fluorescence raw images.

[0029] Step 102: Obtain a converted hue, a first saturation, and a first brightness based on the original fluorescent image.

[0030] In this embodiment, the first hue is the hue value of the original fluorescent image (i.e., the specific value of the hue), the first saturation is the saturation value of the original fluorescent image (i.e., the specific value of the saturation), and the first brightness is the brightness value of the original fluorescent image (i.e., the specific value of the brightness). The hue value range of the original fluorescent image in various color models is the same, which is 0-360 degrees.

[0031] In this embodiment, the original fluorescent image is converted to the hexagonal pyramid color space (HSV space) to obtain a first hue, a first saturation, and a first lightness. The first hue, used for angular measurement, ranges from 0° to 360°, starting from red and counting counterclockwise, with red at 0°, green at 120°, and blue at 240°. The first saturation indicates the degree to which the color is close to a spectral color, ranging from 0% to 100%, with larger values ​​indicating more saturated colors. The first lightness indicates the brightness of the color, ranging from 0% (black) to 100% (white).

[0032] In this embodiment, step 102 includes converting the original fluorescent image from an RGB or BGR color space to an HSV color space using a color space conversion tool. In the HSV color space, a first hue is calculated using a hue calculation formula, a first saturation is calculated using a saturation calculation formula, and a first brightness is calculated using a brightness calculation formula. The hue calculation formula, the saturation calculation formula, and the brightness calculation formula are all well-established formulas.

[0033] In this embodiment, the converted hue is a value obtained by linearly converting the hue value of the original fluorescent image. For example, if the hue value of the original fluorescent image is a first hue, the first hue is linearly converted to obtain the converted hue. It should be noted that linearly converting the hue value refers to converting the numerical range of the hue value in a color space (such as HSV) to the numerical range of the color space (the original color space of the original fluorescent image, such as RGB or BGR) before the hue value is converted to the original fluorescent image. For example, converting the hue value from 0-360 to within the range of 0-255.

[0034] In this embodiment, the original fluorescence image is converted to the HSV domain and the H (hue) channel value is extracted to obtain the first hue, first saturation, and first brightness. Based on the RGB to HSV conversion formula, the hue is converted to the first hue. The hue calculation formula is shown in Equation (1).

[0035]

[0036] In formula (1), G, B, and R represent the values ​​of the green, blue, and red components of the color, respectively. max and min represent the maximum and minimum values ​​of these three color components, respectively. H is the calculated first hue, in degrees. When G (green) is at its maximum value, the calculation formula for H is shown in formula (2):

[0037] H=(GB) / (max-min)×60°(2)

[0038] When B (blue) is at its maximum value, the calculation formula of H is shown in formula (3):

[0039] H=(2+(BR) / (max-min))×60°(3)

[0040] When R (red) is the maximum value, the calculation formula of hue H is shown in formula (4):

[0041] H=(4+(RG) / (max-min))×60°(4)

[0042] In this embodiment, H is converted from the value range of [0, 360] to the value range of [0, 255] to obtain the converted hue, as shown in formula (5);

[0043] (5)

[0044] In formula (5), H' represents the conversion hue. The first hue is linearly converted by formula (5) to obtain the conversion hue. By performing a linear conversion on the hue value of the original fluorescent image as shown in formula (5), the numerical range of the hue value can be made consistent with the numerical range of the current color space (such as RGB or BGR) of the original fluorescent image, which facilitates the superposition of the two, thereby effectively obtaining the mask image.

[0045] Step 103 : Based on the converted hue and the original fluorescent image, the green component and the blue component are identified to obtain a mask image having information of the green component and the blue component.

[0046] In this embodiment, the green and blue components are dyes in the original fluorescent image. A dye (stain / dye) is a chemical substance that binds to a specific target and changes its optical properties. Dyes are used to mark the original fluorescent image. The green component is ICG (indocyanine green) and the blue component is MB (methylene blue). The mask image is a binary matrix (typically 0 or 255) generated by a specific algorithm. A mask image containing information about the green and blue components means that points of interest in the mask image represent either the green or blue component. Thus, all points of interest in the entire mask image represent both the green and blue components.

[0047] In this embodiment, the converted hue and the original fluorescent image are superimposed to obtain a superimposed image; the green component and the blue component in the superimposed image are identified to generate a mask image having information of the green component and the blue component. Specifically, the identification step includes: converting the superimposed image from color to a grayscale image. The superimposed image may be interfered by noise, such as scattered light, background fluorescence, etc., and using methods such as median filtering or Gaussian filtering to remove noise in the grayscale image to obtain a smoothed image. The contrast of the smoothed image is enhanced by methods such as histogram equalization or linear contrast stretching to make the fluorescence signals of the green component and the blue component more obvious. The image with increased contrast is input into a pre-trained binary classification model so that the binary classification model classifies the green component and the blue component and generates a mask image. Specifically, the mask image is as follows: Figure 4 shown.

[0048] Optionally, the step of identifying the green and blue components in the overlay image and generating a mask image containing information about the green and blue components further includes: selecting an appropriate threshold based on the intensity of the overlay image to segment the overlay image into foreground (fluorescent signal) and background. A global threshold (such as the Otsu method) or a local threshold method can be used. The segmented foreground region is labeled with its green and blue components. Specifically, a connected component analysis method can be used to label each fluorescent signal region as an independent region. A trained classifier is then used to classify each fluorescent signal region into its green and blue components, generating a mask image for both the green and blue components.

[0049] Step 104 : obtaining a fused image based on the visible light image, the mask image, the first saturation, and the first brightness.

[0050] In this embodiment, the above-mentioned step 104 includes: based on the mask image, selecting a region of the visible light image to obtain a regional image; performing image conversion on the regional image to obtain a second hue, a second saturation and a second brightness; based on the first brightness and the second brightness, obtaining a fused brightness; based on the first saturation and the second saturation, obtaining a fused saturation; based on the second hue, the fused saturation and the fused brightness, obtaining a reconstructed image; and fusing the reconstructed image with an image in the visible light image that does not belong to the mask image area to obtain a fused image.

[0051] The embodiments of the present disclosure provide a dual fluorescence and visible light image fusion method. First, a visible light image and a fluorescence original image of a target scene are simultaneously acquired. Second, based on the fluorescence original image, a conversion hue, a first saturation, and a first brightness are obtained. Then, based on the conversion hue and the fluorescence original image, the green component and the blue component are identified to obtain a mask image containing information about the green and blue components. Finally, a fused image is obtained based on the visible light image, the mask image, the first saturation, and the first brightness. Thus, by converting the hue and the fluorescence original image, the green component and the blue component are accurately identified to form a mask image containing the green and blue components. This effectively distinguishes the foreground from the background, and the mask image allows processing only on the stained area without affecting other areas. The brightness and saturation of the fluorescence original image are used with the saturation and brightness of the visible light image to fuse within the area defined by the mask image, maintaining color contrast and avoiding overexposure of color contrast in the target area corresponding to the mask image, thereby improving the fusion effect of the visible light image and the dual fluorescence image.

[0052] In existing techniques, the original fluorescence image cannot effectively distinguish detailed features such as texture, and the fused image requires edge refinement to restore detailed features. In some optional implementations of the present disclosure, the method further includes: performing edge detection on the mask image to obtain an edge detection result; and adding the edge detection result to the fused image to obtain the target image.

[0053] In this optional implementation, multiple methods are available for edge detection on the mask image, such as the Sobel operator and Laplacian edge detection. The Sobel operator is a gradient-based edge detection method that detects edges by calculating the horizontal and vertical gradients of the image. The Laplacian edge detection method is a second-order derivative-based edge detection method that detects edges by calculating the Laplacian operator of the image.

[0054] In this optional implementation, adding the edge detection result to the fused image to obtain the target image includes: enhancing the edges in the edge detection result to obtain enhanced edge information, wherein the edge enhancement can be achieved by adjusting the intensity or contrast of the edges in the edge detection result; and adding the enhanced edge information to the fused image, wherein the edge information can be added to the fused image by weighted superposition of the edge information and the fused image.

[0055] The dual fluorescence and visible light image fusion method provided in this embodiment adds the edge detection result of the mask image to the fused image, thereby enhancing the edge of the fused image through edge enhancement technology and avoiding edge blurring caused by traditional pixel-level fusion.

[0056] In some optional implementations of the present disclosure, the above-mentioned dual fluorescence and visible light image fusion method also includes: using two different colors to identify the green component and the blue component in the mask image to obtain a component identification image; adding the component identification image to the fused image to obtain a result image in which the green component and the blue component are identified.

[0057] In this embodiment, the component identification image is a multi-color mask image, and the multi-color mask image can be obtained through a variety of methods such as threshold segmentation, clustering algorithm, and deep learning. Among them, threshold segmentation refers to: by setting different thresholds when generating the mask image, different grayscale or color areas in the image are segmented and marked with different colors. The clustering algorithm refers to: using a clustering algorithm such as K-means, when generating the mask image, the pixels in the mask image are divided into different categories and represented with different colors. The deep learning method refers to: by training a neural network model, different areas in the image are automatically identified and a multi-color mask image is generated. For example, a semantic segmentation model can mark different objects in an image with different colors.

[0058] In this embodiment, the component identification image is added to the fused image, which can effectively highlight the green component and the blue component in the image, thereby improving the identification effect of multiple components in the fused image.

[0059] In one embodiment of the present disclosure, the above-mentioned dual fluorescence and visible light image fusion method also includes: using a trained human visual sensitive area prediction model to estimate the user's gaze area in the fused image in real time to obtain a gaze estimation result; based on the gaze estimation result, assigning dynamic fusion weights to different areas in the mask image; and obtaining a new fused image based on the change of the dynamic fusion weight.

[0060] In this embodiment, the attention levels of different regions are determined by predicting the areas where people look at the fused image, and the display brightness of the fused image is dynamically adjusted by adjusting the dynamic fusion weights of different regions of the mask image.

[0061] The dual fluorescence and visible light image fusion method provided in this embodiment introduces the human physiological vision mechanism for dynamic weight control. In scenarios such as fluorescence-enhanced medical images, it can significantly improve the recognizability of the region of interest and enhance the user experience.

[0062] Optionally, since image fusion methods are usually optimized for static images and ignore the fusion stability problem in video applications, the above-mentioned dual fluorescence and visible light image fusion method also includes: when the fused image is a video frame in a video stream, performing three-frame joint fusion of the video frame and its adjacent video frames; based on optical flow field estimation, performing inter-frame alignment on the mask image; and generating a new fused image of the video frame through the aligned mask image.

[0063] The dual fluorescence and visible light image fusion method provided in this embodiment introduces temporal consistency constraints and inter-frame mask registration strategies to ensure that the fused image has no obvious flicker or drift in dynamic scenes, significantly expanding the practical boundaries of fusion technology.

[0064] Optionally, the above-mentioned dual fluorescence and visible light image fusion method also includes: using the statistical noise distribution of each channel image before fusion to construct a noise probability map; inputting the noise probability map and the fused image into a pre-trained ghost detection model to obtain a ghost detection result of the ghost detection model; in response to the ghost detection result indicating that the fused image has serious deviations in the reverse mapping, regenerating the mask image, and obtaining a new fused image based on the visible light image, the regenerated mask image, the first saturation and the first brightness; in response to the ghost detection result indicating that the fused image has no serious deviations in the direction mapping, determining that the fused image is qualified.

[0065] A "reverse consistency discriminator" is introduced in the neural network fusion process to perform reprojection verification on the output image; if it is found that the fused image has serious deviations in the reverse mapping, it will automatically fall back and recalibrate the fusion weights.

[0066] This solution integrates the ideas of reverse reasoning and discriminant supervision in computer vision and provides the ability to actively correct the output image quality.

[0067] In some optional implementations of the present disclosure, obtaining the converted hue based on the original fluorescence image includes: performing mean filtering on the original fluorescence image to obtain a processed image; converting the processed image into a hexagonal pyramid model and extracting a first hue; and obtaining the converted hue based on the first hue.

[0068] In this embodiment, mean filtering is a typical linear filtering algorithm. It involves applying a template to the target pixel in the image. This template includes the target pixel's surrounding pixels (the eight pixels surrounding the target pixel constitute the filtering template, i.e., the target pixel itself). The original pixel value is then replaced with the average value of all pixels in the template. The size of the filter window for mean filtering can match the resolution of the original fluorescence image. For example, the original fluorescence image is a grayscale image with a resolution of 1024×1024 pixels. Fluorescently labeled proteins within cells exhibit varying degrees of brightness, representing the distribution of fluorescence intensity. When performing mean filtering on the original fluorescence image, a 3×3 filter window can be selected. The grayscale values ​​of each pixel in the image and its eight surrounding pixels are averaged to obtain a new pixel value. For example, for a pixel at coordinates (500, 500) in the image, its original grayscale value is 120, and the grayscale values ​​of its eight surrounding pixels are 110, 115, 125, 120, 130, 120, 115, and 110, respectively. After mean filtering, the new grayscale value of the pixel is:

[0069] (120+110+115+125+120+130+120+115+110) / 9=118.89≈119

[0070] After mean filtering, the noise of the entire fluorescence original image is effectively suppressed, the image becomes smoother, and the distribution of the fluorescence signal becomes clearer and more stable, providing a better foundation for subsequent hue extraction.

[0071] The mean-filtered image (processed image) is converted to a hexagonal pyramid model. The hexagonal pyramid model is a color space model that represents color as a hexagonal pyramid structure consisting of three components: hue, saturation, and lightness. In this model, hue represents the type of color, saturation represents the purity of the color, and lightness represents the brightness of the color. During the conversion process, the grayscale value range of the processed image (0-255) is first mapped to the lightness axis of the hexagonal pyramid model. Assuming that the grayscale value range of the processed image is 0-255, the lightness axis of the hexagonal pyramid model also ranges from 0-255, allowing a direct linear mapping. Next, the hue value of each pixel is calculated based on the grayscale value distribution of the processed image. In the hexagonal pyramid model, the calculation of hue is dependent on the pixel's position in color space. For fluorescence images, since they primarily exhibit a single fluorescent color (for example, green), their hue values ​​are relatively concentrated. Assume that the primary hue of a fluorescent image is green, and the corresponding hue value in the hexagonal pyramid model falls within a specific angular range (e.g., 120°-150°). By calculating the position of each pixel in the hexagonal pyramid model, its corresponding first hue can be extracted, and then a linear transformation is performed on the first hue to obtain the transformed hue. For example, for the pixel at coordinates (500, 500) in the processed image, its grayscale value after mean filtering is 119. After transformation and calculation, its hue value in the hexagonal pyramid model is 135°, indicating that this pixel falls within the hue range of green fluorescence.

[0072] The method for obtaining the conversion hue provided by this optional implementation first performs mean filtering on the original fluorescent image to obtain a processed image; the processed image is converted into a hexagonal pyramid model, and the first hue is extracted, and the conversion hue is obtained based on the first hue, thereby improving the reliability of the conversion hue extraction and improving the accuracy of the conversion hue.

[0073] In some optional implementations of the present disclosure, the above-mentioned identifying the green component and the blue component based on the converted hue and the fluorescent original image to obtain a mask image with information of the green component and the blue component includes: splicing the converted hue and the fluorescent original image to obtain a spliced ​​image; inputting the spliced ​​image into a pre-trained neural network model to obtain a recognition image including the green component and the blue component; and obtaining a mask image with information of the green component and the blue component based on the recognition image.

[0074] In this optional implementation, the converted hue and the original fluorescent image are spliced ​​to obtain a spliced ​​image, which includes: splicing the converted hue and the original fluorescent image by pixel value to obtain a spliced ​​image, and inputting the spliced ​​image into a pre-trained neural network model as the input of the neural network model, so that the neural network model can perceive color priors at the initial stage.

[0075] In this optional implementation, the neural network model is trained using a set of labeled images. The labeled images are images that have been labeled for green and blue components. The labeled images are obtained by obtaining video data and obtaining labeled images based on the video data. The video data includes at least one video frame, each of which has a green component and a blue component. Specifically, several frames of images are selected from the video data, and the green and blue components in these images are manually labeled.

[0076] In this optional implementation, a neural network model is used to identify spatial features (such as edges and textures) and segment these spatial features from the original fluorescent image. However, relying solely on spatial information may lead to mis-segmentation in areas with similar textures (such as blue objects of different materials). The color prior of converted hue can provide semantic clues independent of the spatial structure, making up for the limitations of spatial features. Since the green and blue components have obvious hue characteristics, it is considered feasible to use hue prior for assistance.

[0077] In this optional implementation, the neural network model includes an encoder and a decoder. The encoder extracts image features from the spliced ​​image, while the decoder uses guided connected attention to focus on color and edge texture features within the image to produce a recognition image. A neural network model consisting of an encoder and a decoder is used. The spliced ​​image is input into the trained neural network model, which outputs a recognition image containing green and blue components. This recognition image is generated by the model's decoder, where the guided connected attention mechanism ensures accurate recognition of color and edge texture features.

[0078] like Figure 5 As shown in the figure, a CGBA-Net neural network model can be used. This CGBA-Net consists of an encoder B and a decoder J. Taking the original fluorescent image Y and the superposition of the converted hue Z as input, the encoder B extracts the input image's features through five layers of convolution. The decoder J adds guided connection attention, focusing on the input's color features and edge texture features to achieve accurate segmentation. After each convolutional layer in the encoding stage, there is a guided connection attention (GCA) module. This module enhances feature representation through a bidirectional attention (BA) mechanism.

[0079] The GCA module receives the feature map F from the previous layer h and F l After upsampling and convolution, a weight matrix is ​​generated through the bidirectional attention mechanism to adjust the weight of the feature map. In the GCA module, the feature map F h and F l Feature map F generated by the bidirectional attention mechanism effFuse with the original feature map to generate the enhanced feature map F GCA .

[0080] Enhanced feature map F GCA The image is decoded through a series of convolutional layers to gradually restore the spatial resolution of the image. The decoding stage may include upsampling operations to increase the size of the feature map.

[0081] In this optional implementation, the recognition image is to use a neural network model to respectively recognize an image including ICG signals and MB signals. By recognizing the image, the following can be obtained: Figure 4 or Figure 5 The mask image T is shown. In the mask image, the green component and the blue component are both represented by the same pixel value, for example, 1.

[0082] The method for obtaining a mask image provided by this optional implementation uses a neural network model and image processing technology to accurately identify specific green and blue components from the original fluorescent image and the visible light image, and accurately segments the identified image including the green and blue components through a lightweight deep learning algorithm with a hue prior, avoiding confusion between the two signals and the influence of background noise.

[0083] Optionally, the above-mentioned identification of the green and blue components based on the converted hue and the original fluorescence image to obtain a mask image containing information about the green and blue components includes: stitching the converted hue and the original fluorescence image to obtain a stitched image; extracting morphological features of the green and blue components in the fluorescence signal region of the stitched image, such as area, perimeter, aspect ratio, circularity, etc.; extracting intensity features of the green and blue components in the fluorescence signal region of the stitched image, such as mean grayscale value, maximum grayscale value, standard deviation, etc.; and extracting texture features of the green and blue components in the fluorescence signal region of the stitched image, such as features of the grayscale co-occurrence matrix, including contrast, correlation, energy, etc.; and obtaining a mask image based on the morphological features, intensity features, and texture features.

[0084] In some optional implementations of the present disclosure, the above-mentioned obtaining of a fused image based on the visible light image, the mask image, the first saturation and the first brightness includes: obtaining a second hue, a second saturation and a second brightness based on the visible light image; obtaining a fused brightness based on the first brightness, the second brightness and the mask image; obtaining a fused saturation based on the first saturation, the second saturation and the mask image; obtaining an output image based on the second hue, the fused saturation and the fused brightness; and fusing the image in the output image belonging to the mask image area with the image in the visible light image that does not belong to the mask image area to obtain a fused image.

[0085] In this optional implementation, obtaining the fused luminance based on the first luminance, the second luminance, and the mask image includes: weightedly summing the first luminance and the second luminance to obtain the summed luminance; extracting the luminance in the summed luminance that belongs to the corresponding area of ​​the mask image to obtain the fused luminance.

[0086] In this optional implementation, based on the first saturation, the second saturation and the mask image, the fused saturation is obtained, including: weighted summing of the first saturation and the second saturation to obtain the summed saturation; extracting the saturation in the corresponding area of ​​the mask image from the summed saturation to obtain the fused saturation.

[0087] In this optional implementation, the fused luminance is the luminance obtained by fusing the first luminance and the second luminance corresponding to the mask image. By fusing the luminance, we can only focus on the fused luminance of the luminance of the corresponding area of ​​the mask image; the fused saturation is the saturation obtained by fusing the first saturation and the second saturation corresponding to the mask image. By fusing the saturation, we can only focus on the saturation of the corresponding area of ​​the mask image.

[0088] In this optional implementation, the output image is an image under the HSV model, and the output image is an image generated using the second hue, the fused saturation, and the fused brightness. That is, the output image uses the second hue as its hue, the fused saturation as its saturation, and the fused brightness as its brightness.

[0089] In this optional implementation, the above-mentioned fusing of the image belonging to the mask image area in the output image with the image not belonging to the mask image area in the visible light image to obtain a fused image includes: converting the output image to a three-primary color light mode to obtain a converted image; using a fusion formula to fuse the converted image with the mask image to obtain an image belonging to the mask image area; using a fusion formula to fuse the visible light image with the mask image to obtain a first fused image, and removing the first fused image from the visible light image to obtain an image not belonging to the mask image area in the visible light image, and performing weighted summation of the image belonging to the mask image area and the image not belonging to the mask image area in the visible light image to obtain a fused image.

[0090] This optional implementation provides a method for obtaining a fused image. Based on the visible light image, a second hue, a second saturation, and a second brightness are obtained; based on the first brightness, the second brightness, and the mask image, a fused brightness is obtained; based on the first saturation, the second saturation, and the mask image, a fused saturation is obtained; based on the second hue, the fused saturation, and the fused brightness, an output image is obtained; and the image in the output image belonging to the mask image region is fused with the image in the visible light image not belonging to the mask image region to obtain a fused image. This provides a reliable implementation method for obtaining a fused image and improves the reliability of fused image acquisition. Using cross-image brightness-saturation collaborative optimization, the dual-fluorescence fusion color is adaptively adjusted to avoid overexposure in the target area and maintain color contrast.

[0091] In some optional implementations of the present disclosure, the above-mentioned fused luminance is obtained based on the first luminance, the second luminance and the mask image, including: based on the first luminance, counting the first luminance mean and the first luminance variance of the first luminance in the mask image; based on the second luminance, counting the second luminance mean and the second luminance variance of the second luminance in the mask image; substituting the second luminance, the luminance mean and the luminance variance into the cross-image luminance matching formula to obtain the fused luminance; the cross-image luminance matching formula represents the product of the luminance matching parameter, the difference between the second luminance and the second luminance mean, the ratio of the first luminance variance to the second luminance variance and the sum of the first luminance mean, wherein the luminance matching parameter is a pre-set parameter.

[0092] In this optional implementation, the cross-image brightness matching formula is shown in formula (6):

[0093] (6)

[0094] In formula (6), is the first brightness mean of the first brightness in the mask image, is the first brightness variance of the first brightness in the mask image, is the second brightness mean of the second brightness in the mask image, is the second brightness variance of the second brightness in the mask image; is the fused image, a is the custom parameter for brightness matching, and the value of a is shown in formula (7).

[0095] (7)

[0096] In formula (7), The value between .

[0097] In this optional implementation, V1 and V2 in the cross-image brightness matching formula may represent local regions of the two images, respectively. The cross-image brightness matching formula adjusts the local deviations of V2 and combines them with the global trend of V1 to generate a new fused image, V′. This adjustment of local deviations can enhance image structural details to a certain extent, while the introduction of global trends maintains overall consistency.

[0098] The method for obtaining fused brightness provided by this optional implementation adopts adaptive gradient fusion technology. While retaining the color characteristics of the target area, it enhances structural details through gradient domain fusion, avoids the edge blurring caused by traditional pixel-level fusion, and improves the fusion effect of dual fluorescence images and visible light images.

[0099] In some optional implementations of the present disclosure, the above-mentioned fused saturation is obtained based on the first saturation, the second saturation and the mask image, including: based on the first saturation, counting the first saturation mean of the first saturation in the mask image; based on the second saturation, counting the second saturation mean of the second saturation in the mask image; multiplying the first saturation by the ratio of the second saturation mean to the first saturation mean to obtain the fused saturation.

[0100] In this optional implementation, the first saturation mean value and the second saturation mean value may be calculated using an averaging formula.

[0101] In this optional implementation, the formula for calculating the fused saturation by multiplying the first saturation by the ratio of the second saturation mean to the first saturation mean is a formula for standardizing or normalizing the first saturation mean and the second saturation mean. Through this formula, the value of the first saturation can be adjusted to a level that matches the mean of the second saturation.

[0102] The method for obtaining the fused saturation provided by this optional implementation obtains the fused saturation by multiplying the first saturation by the ratio of the second saturation mean to the first saturation mean, providing a reliable implementation method for obtaining the fused saturation.

[0103] For the target scene of the endoscope, in a specific example, the dual fluorescence and visible light image fusion method provided by the present disclosure can be an endoscope-based visible light and dual fluorescence image enhancement fusion algorithm. The fluorescence image is processed by a deep learning algorithm with prior knowledge of hue, and the green component and the blue component are accurately segmented to form a corresponding high-priority mask image. At the same time, cross-image brightness-saturation collaborative optimization is adopted to fuse the images within the corresponding masks to maintain color contrast and avoid overexposure in the target area. Finally, structural details are enhanced through gradient domain fusion to avoid edge blurring caused by traditional pixel-level fusion, realize the fusion of dual fluorescence images and visible light images, and obtain a dual fluorescence fused image.

[0104] Further references Figure 6 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a dual fluorescence and visible light image fusion device. Figure 1 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0105] like Figure 6 As shown, the dual fluorescence and visible light image fusion device 600 provided in this embodiment includes: an acquisition unit 601, a obtaining unit 602, an identification unit 603, and a fusion unit 604. The acquisition unit 601 can be configured to simultaneously obtain a visible light image and a fluorescence original image of a target scene. The obtaining unit 602 can be configured to obtain a converted hue, a first saturation, and a first brightness based on the fluorescence original image. The converted hue is a value obtained by linearly converting the hue value of the fluorescence original image. The identification unit 603 can be configured to identify the green and blue components in the image based on the converted hue and the fluorescence original image, and obtain a mask image containing information about the green and blue components. The fusion unit 604 can be configured to obtain a fused image based on the visible light image, the mask image, the first saturation, and the first brightness.

[0106] In this embodiment, the specific processing of the dual fluorescence and visible light image fusion device 600: the acquisition unit 601, the obtaining unit 602, the identification unit 603, and the fusion unit 604 and the technical effects thereof can be referred to respectively. Figure 1 The relevant descriptions of step 101, step 102, step 103, and step 104 in the corresponding embodiment are not repeated here.

[0107] In one embodiment of the present disclosure, the above-mentioned device 600 also includes: a detection unit (not shown in the figure), which is configured to: perform edge detection on the mask image to obtain an edge detection result; and add the edge detection result to the fused image to obtain a target image.

[0108] In some embodiments of the present disclosure, the obtaining unit 602 is configured to: perform mean filtering on the original fluorescence image to obtain a processed image; convert the processed image into a hexagonal pyramid model and extract the first hue; and obtain a converted hue based on the first hue.

[0109] In some embodiments of the present disclosure, the above-mentioned recognition unit 603 is configured to: stitch the converted hue and the fluorescent original image to obtain a stitched image; input the stitched image into a pre-trained neural network model to obtain a recognition image including a green component and a blue component, the neural network model includes an encoder and a decoder, the encoder is used to extract image features of the stitched image, and the decoder obtains the recognition image by guiding the connection attention to focus on the color features and edge texture features in the image features; based on the recognition image, a mask image with information of the green component and the blue component is obtained; the neural network model is obtained by training a set of labeled images, the labeled image is an image obtained after labeling the green component and the blue component, and the labeled image is obtained by the following steps: obtaining video data, and obtaining a labeled image based on the video data, the video data including at least one video frame, and each video frame has a green component and a blue component.

[0110] In some embodiments of the present disclosure, the above-mentioned fusion unit 604 is configured to: obtain the second hue, second saturation and second brightness based on the visible light image; obtain the fused brightness based on the first brightness, second brightness and mask image; obtain the fused saturation based on the first saturation, second saturation and mask image; obtain the output image based on the second hue, fused saturation and fused brightness; and fuse the image in the output image belonging to the mask image area with the image in the visible light image that does not belong to the mask image area to obtain a fused image.

[0111] In some embodiments of the present disclosure, the above-mentioned fusion unit 604 is further configured to: based on the first luminance, count the first luminance mean and the first luminance variance of the first luminance in the mask image; based on the second luminance, count the second luminance mean and the second luminance variance of the second luminance in the mask image; substitute the second luminance, the luminance mean and the luminance variance into the cross-image luminance matching formula to obtain the fused luminance; the cross-image luminance matching formula represents the product of the luminance matching parameter, the difference between the second luminance and the second luminance mean, the ratio of the first luminance variance to the second luminance variance and the sum of the first luminance mean, wherein the luminance matching parameter is a pre-set parameter.

[0112] In some embodiments of the present disclosure, the above-mentioned fusion unit 604 is further configured to: based on the first saturation, count the first saturation mean of the first saturation in the mask image; based on the second saturation, count the second saturation mean of the second saturation in the mask image; multiply the first saturation by the ratio of the second saturation mean to the first saturation mean to obtain the fused saturation.

[0113] The document generation device provided by the embodiment of the present disclosure, first, the acquisition unit 601 simultaneously acquires the visible light image and the fluorescent original image of the target scene; secondly, the acquisition unit 602 obtains the converted hue, the first saturation and the first brightness based on the fluorescent original image; then, the identification unit 603 identifies the green component and the blue component in the image based on the converted hue and the fluorescent original image, and obtains a mask image with information of the green component and the blue component; finally, the fusion unit 604 obtains the fused image based on the visible light image, the mask image, the first saturation and the first brightness; thus, by accurately identifying the green component and the blue component through the conversion hue and the fluorescent original image, and forming a mask image with green component and blue component, the foreground and background can be effectively distinguished, and the mask image can be used to process only the stained area without affecting other areas; the brightness and saturation of the fluorescent original image are used with the saturation and brightness of the visible light image to be fused within the area defined by the mask image, thereby maintaining the color contrast, avoiding overexposure of the color contrast in the target area corresponding to the mask image, and improving the fusion effect of the visible light image and the dual fluorescent image.

[0114] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0115] Figure 7 A schematic block diagram of an example electronic device 700 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their modes are provided for example only and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0116] like Figure 7As shown, electronic device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. Various programs and data required for the operation of electronic device 700 may also be stored in RAM 703. Computing unit 701, ROM 702, and RAM 703 are interconnected via a bus 704. An input / output (I / O) interface 705 is also connected to bus 704.

[0117] Multiple components in the electronic device 700 are connected to the I / O interface 705, including an input unit 706, such as a keyboard, a mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a magnetic disk, an optical disk, etc.; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the electronic device 700 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0118] The computing unit 701 can be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above, such as the dual-fluorescence and visible light image fusion method. For example, in some embodiments, the dual-fluorescence and visible light image fusion method can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 708. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the computing unit 701, one or more steps of the dual-fluorescence and visible light image fusion method described above can be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to execute the dual fluorescence and visible light image fusion method in any other appropriate manner (eg, by means of firmware).

[0119] Various embodiments of the systems and techniques described above can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0120] The program code for implementing the methods of the present disclosure can be written in any combination of one or more programming languages. Such program code can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable dual-fluorescence and visible light image fusion device, such that when executed by the processor or controller, the program code implements the modes / operations specified in the flowcharts and / or block diagrams. The program code can be executed entirely on the machine, partially on the machine, as a standalone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0121] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fibers, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0122] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0123] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0124] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not limited herein.

[0125] The foregoing descriptions of specific exemplary embodiments of the present disclosure are for purposes of illustration and description. These descriptions are not intended to limit the present disclosure to the precise forms disclosed, and it is apparent that many variations and modifications are possible in light of the foregoing teachings. The exemplary embodiments have been selected and described for the purpose of explaining the specific principles of the present disclosure and their practical application, thereby enabling those skilled in the art to realize and utilize a variety of exemplary embodiments of the present disclosure and various options and modifications. The scope of the present disclosure is intended to be defined by the claims and their equivalents.

[0126] The above are merely embodiments of the present disclosure and are not intended to limit the present disclosure. Any modifications, equivalent replacements, improvements, etc. made within the principles of the present disclosure should be included in the scope of protection of the present disclosure.

Claims

1. A dual fluorescence and visible light image fusion method, characterized in that: The method comprises: Simultaneously acquire visible light images and original fluorescence images of the target scene; Based on the original fluorescence image, a conversion hue, a first saturation, and a first lightness are obtained, wherein the conversion hue is a value obtained by linearly converting the hue value of the original fluorescence image; obtaining the conversion hue based on the original fluorescence image includes: performing mean filtering on the original fluorescence image to obtain a processed image; converting the processed image into a hexagonal pyramid model and extracting a first hue; and obtaining the conversion hue based on the first hue; identifying a green component and a blue component based on the converted hue and the fluorescent original image, and obtaining a mask image having information of the green component and the blue component; A fused image is obtained based on the visible light image, the mask image, the first saturation, and the first brightness.

2. The method according to claim 1, characterized in that The method further comprises: Performing edge detection on the mask image to obtain an edge detection result; adding the edge detection result to the fused image to obtain a target image; Alternatively, two different colors are used to identify the green component and the blue component in the mask image to obtain a component identification image; the component identification image is added to the fused image to obtain a result image in which the green component and the blue component are identified; Alternatively, a trained human visual sensitive area prediction model is used in real time to estimate the user's gaze area in the fused image to obtain a gaze estimation result; based on the gaze estimation result, dynamic fusion weights are assigned to different areas in the mask image; and based on the change of the dynamic fusion weights, a new fused image is obtained.

3. The method according to claim 1 or 2, characterized in that The step of identifying a green component and a blue component based on the converted hue and the fluorescent original image to obtain a mask image having information of the green component and the blue component includes: splicing the converted color tone and the original fluorescent image to obtain a spliced ​​image; Inputting the stitched image into a pre-trained neural network model to obtain a recognition image including a green component and a blue component, the neural network model including an encoder and a decoder, the encoder being configured to extract image features of the stitched image, and the decoder being configured to obtain a recognition image by directing connected attention to focus on color features and edge texture features among the image features; Based on the recognition image, obtaining a mask image having information of the green component and the blue component; The neural network model is obtained by training a set of labeled images, and the labeled images are images obtained after labeling the green component and the blue component. The labeled images are obtained by the following steps: acquiring video data, and obtaining a labeled image based on the video data, wherein the video data includes at least one video frame, and each video frame has a green component and a blue component.

4. The method according to claim 1 or 2, characterized in that The obtaining of a fused image based on the visible light image, the mask image, the first saturation, and the first brightness includes: obtaining a second hue, a second saturation, and a second brightness based on the visible light image; Obtaining a fused luminance based on the first luminance, the second luminance, and the mask image; Obtaining a fused saturation based on the first saturation, the second saturation, and the mask image; obtaining an output image based on the second hue, the fused saturation, and the fused lightness; The image corresponding to the mask image area in the output image is fused with the image not belonging to the mask image area in the visible light image to obtain a fused image.

5. The method according to claim 4, characterized in that The obtaining of a fused luminance based on the first luminance, the second luminance, and the mask image includes: Calculating a first brightness mean and a first brightness variance of the first brightness in the mask image based on the first brightness; Based on the second brightness, calculating a second brightness mean and a second brightness variance of the second brightness in the mask image; Substitute the second luminance, the luminance mean, and the luminance variance into a cross-image luminance matching formula to obtain a fused luminance; the cross-image luminance matching formula represents the sum of the product of a luminance matching parameter, a difference between the second luminance and the second luminance mean, and a ratio of the first luminance variance to the second luminance variance, and the first luminance mean, wherein the luminance matching parameter is a pre-set parameter.

6. The method according to claim 4, characterized in that The obtaining of a fused saturation based on the first saturation, the second saturation, and the mask image includes: Based on the first saturation, calculating a first saturation mean value of the first saturation in the mask image; Based on the second saturation, calculating a second saturation mean value of the second saturation in the mask image; The first saturation is multiplied by the ratio of the second saturation mean to the first saturation mean to obtain a fused saturation.

7. A dual fluorescence and visible light image fusion device, characterized in that: The device comprises: an acquisition unit configured to simultaneously acquire a visible light image and a fluorescent original image of a target scene; The obtaining unit is configured to obtain a conversion hue, a first saturation, and a first brightness based on the original fluorescence image, wherein the conversion hue is a value obtained by linearly converting the hue value of the original fluorescence image; the obtaining unit is further configured to: perform mean filtering on the original fluorescence image to obtain a processed image; convert the processed image into a hexagonal pyramid model and extract a first hue; and obtain the conversion hue based on the first hue; an identification unit configured to identify a green component and a blue component in an image based on the converted hue and the fluorescent original image, and obtain a mask image having information of the green component and the blue component; The fusion unit is configured to obtain a fused image based on the visible light image, the mask image, the first saturation, and the first brightness.

8. An electronic device, characterized in that: include: at least one processor; and a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to enable the computer to execute the method according to any one of claims 1 to 6.

Images