Dual-fluorescence and visible light image fusion method and device, electronic equipment and medium

Abstract

The invention provides a dual-fluorescence and visible light image fusion method and device, electronic equipment and a medium, and relates to the technical field of image processing, and the method comprises the steps: obtaining a visible light image and a fluorescence original image in a target scene at the same time; based on the original fluorescence image, obtaining a conversion hue, a first saturation and a first brightness, the conversion hue being a value obtained after linear conversion of a hue value of the original fluorescence image; identifying a green component and a blue component based on the converted hue and the original fluorescence image to obtain a mask image having information of the green component and the blue component; based on the visible light image, the mask image, the first saturation and the first brightness, the fusion image is obtained, and the fusion effect of the visible light image and the dual-fluorescence image is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technologies, and in particular, to a method and apparatus for fusing dual-fluorescence and visible light images, an electronic device, and a computer-readable storage medium, which are applicable to fields such as medical imaging, biological detection, and industrial detection. Background Art

[0002] In recent years, the fluorescence navigation endoscope system has been deeply applied in the field of surgical operations due to its precise guiding ability. Especially in the intraoperative marking of tumors and cholangiography in gynecological and hepatobiliary surgeries, it has become a powerful assistant for doctors. With the development of medical technology, the dual-fluorescence navigation endoscope system has emerged. It uses two dyes, ICG and MB, and cooperates with a dedicated dual-fluorescence navigation system to achieve synchronous dual-color marking of two target regions, significantly expanding the application boundary of fluorescence navigation.

[0003] However, the dual-fluorescence navigation system achieves dual-color marking based on ICG (indocyanine green) and MB (methylene blue) dyes. Although it expands the application scenarios of fluorescence navigation, there are still the following core problems in practical applications: First, the ICG fluorescence appears purple-blue, and the MB fluorescence appears purple-red. The two are close in hue, and the human eye has limited ability to distinguish such similar colors. Moreover, during the operation, the fluorescence signal needs to be fused with the white light image to locate the anatomical structure. The strong brightness of the white light will "dilute" the saturation of the fluorescence color, making it even more difficult to distinguish purple-blue from purple-red. Especially in complex or bleeding tissue scenarios, the boundary judgment error increases significantly. Second, the original signal of the MB fluorescence appears purple-red, which is highly similar to the red color in the abdominal cavity. Traditional fusion algorithms simply weighted and superimposed the fluorescence and white light signals, unable to effectively suppress the interference of the tissue's own color, resulting in blurred boundaries of the MB marked area.

[0004] To solve the above problems, existing algorithms usually follow the method of distinguishing the ICG and MB signals through a single channel in the HSV domain and then performing color superposition. However, this may lead to blurred boundaries during the initial differentiation of the two signals and be easily affected by background noise, which is not conducive to subsequent operations. Summary of the Invention

[0005] The present disclosure provides a method and apparatus for fusing dual-fluorescence and visible light images, an electronic device, and a computer-readable storage medium.

[0006] According to a first aspect, a method for fusing dual fluorescence and visible light images is provided. The method includes: simultaneously acquiring a visible light image and a raw fluorescence image under a target scene; obtaining a converted hue, a first saturation, and a first lightness based on the raw fluorescence image, where the converted hue is a value obtained after linearly converting the hue value of the raw fluorescence image; identifying a green component and a blue component based on the converted hue and the raw fluorescence image to obtain a mask image with information on the green component and the blue component; and obtaining a fused image based on the visible light image, the mask image, the first saturation, and the first lightness.

[0007] According to a second aspect, a dual fluorescence and visible light image fusion device is provided. The device includes: an acquisition unit configured to simultaneously acquire a visible light image and a raw fluorescence image under a target scene; a obtaining unit configured to obtain a converted hue, a first saturation, and a first lightness based on the raw fluorescence image, where the converted hue is a value obtained after linearly converting the hue value of the raw fluorescence image; an identification unit configured to identify a green component and a blue component in the image based on the converted hue and the raw fluorescence image to obtain a mask image with information on the green component and the blue component; and a fusion unit configured to obtain a fused image based on the visible light image, the mask image, the first saturation, and the first lightness.

[0008] According to a third aspect, an electronic device is provided. The electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor, where the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method described in any implementation manner of the first aspect.

[0009] According to a fourth aspect, a non-transitory computer-readable storage medium storing computer instructions is provided, where the computer instructions are used to cause a computer to execute the method described in any implementation manner of the first aspect.

[0010] An approach and apparatus for fusing dual-fluorescence and visible-light images provided by embodiments of the present disclosure. First, a visible-light image and a raw fluorescence image of a target scene are acquired simultaneously. Second, a converted hue, a first saturation, and a first lightness are obtained based on the raw fluorescence image. Then, a green component and a blue component are identified based on the converted hue and the raw fluorescence image, and a mask image with information on the green component and the blue component is obtained. Finally, a fused image is obtained based on the visible-light image, the mask image, the first saturation, and the first lightness. Thus, by converting the hue and using the raw fluorescence image, the green component and the blue component are accurately identified to form a mask image with the green component and the blue component, which can effectively distinguish the foreground from the background, and the mask image can be used to process only the stained area without affecting other areas. The lightness and saturation of the raw fluorescence image are fused with the saturation and lightness of the visible-light image within the area defined by the mask image, maintaining the color contrast and avoiding overexposure of the color contrast in the target area corresponding to the mask image, thereby improving the fusion effect of the visible-light image and the dual-fluorescence image.

[0011] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it used to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] To more clearly illustrate the technical solutions in the embodiments of the present disclosure, the following briefly introduces the drawings required for the description of the embodiments. Obviously, the following-described drawings are only some embodiments of the present disclosure. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0013] The drawings are used to better understand the solution and do not limit the present disclosure.

[0014] Figure 1 is a flowchart according to an embodiment of the method for fusing dual-fluorescence and visible-light images of the present disclosure; Figure 2 is a schematic diagram of a visible-light image in the present disclosure; Figure 3 is a schematic diagram of a raw fluorescence image in the present disclosure; Figure 4 is a schematic diagram of a mask image in the present disclosure; Figure 5 is a schematic structural diagram of generating a mask image using a neural network model in the present disclosure; Figure 6 is a schematic structural diagram of an embodiment of the apparatus for fusing dual-fluorescence and visible-light images of the present disclosure; Figure 7 It is a block diagram of an electronic device for implementing the dual-fluorescence and visible-light image fusion method of the embodiments of the present disclosure. Detailed implementation manners

[0015] Unless otherwise clearly stated, throughout the specification and claims, the term "comprising" or its variations such as "including" or "having" will be understood to include the stated elements or components, without excluding other elements or other components.

[0016] The technical solutions of the present disclosure are described below through specific embodiments. It should be understood that one or more steps mentioned in the present disclosure do not exclude the existence of other methods and steps before and after the combined steps, or other methods and steps can be inserted between these clearly mentioned steps. It should also be understood that these examples are only used to illustrate the present disclosure and not to limit the scope of the present disclosure. Unless otherwise specified, the numbers of the method steps are only for the purpose of identifying each method step, rather than limiting the arrangement order of each method or the implementation scope of the present disclosure. The change or adjustment of their relative relationships can also be regarded as the scope in which the present disclosure can be implemented under the condition of no substantial change in technical content.

[0017] For the raw materials and instruments used in the embodiments, there is no specific limitation on their sources, and they can be purchased in the market or prepared according to the conventional methods well-known to those skilled in the art.

[0018] Aiming at the defects in the traditional technology, the present disclosure proposes a dual-fluorescence and visible-light image fusion method. By cross-image-saturation collaborative optimization, the dual-fluorescence fusion color is adaptively adjusted, effectively solving the above problems, improving the accuracy, visual effect and adaptability of the fused image, and meeting the requirements of different application scenarios. Figure 1 Flow 100 of an embodiment of the dual-fluorescence and visible-light image fusion method according to the present disclosure is shown. The above dual-fluorescence and visible-light image fusion method includes the following steps: Step 101, simultaneously obtain a visible-light image and a fluorescence original image of a target scene.

[0019] In this embodiment, the visible-light image refers to an image formed by recording the light radiation reflected or emitted by an object using the electromagnetic wave band (wavelength range 380 - 780 nm) that can be perceived by the human eye. The fluorescence original image is an image generated using the fluorescence phenomenon (photoluminescence). The fluorescence original image depends on the photoluminescence characteristics of the fluorescent substance, and by precisely controlling the excitation and emission light wavelengths, specific and high-contrast imaging of microscopic targets is achieved.

[0020] In this embodiment, the target scene is the scene at the same moment represented by the visible light image and the fluorescence original image, such as scenes of gynecology, hepatobiliary surgery, endoscopy, etc. at a certain moment. The visible light image and the fluorescence original image can be obtained from a dual-path optical system that collects images of the target scene. In the dual-path optical system, the incident light is divided into two paths. One path is used to collect the visible light image, which is also the white light original image, that is, an image belonging to the wavelength band of 400 - 700 nm, as Figure 2 shown; the other path is used to collect the fluorescence original image, as Figure 3 shown. The fluorescence original image is an image in the near-infrared band. In the dual-path optical system, parallel image acquisition can be achieved through a synchronization sensor to eliminate the displacement error caused by time difference.

[0021] Optionally, the visible light image and the fluorescence original image can also be obtained from a multi-spectral sensor set. The multi-spectral sensor set uses a single sensor to integrate multi-band pixels (such as a sensor imitating the compound eye of a mantis shrimp) to simultaneously capture visible light and specific fluorescence wavelengths, thereby obtaining the visible light image and the fluorescence original image.

[0022] Step 102: Based on the fluorescence original image, obtain the converted hue, the first saturation, and the first lightness.

[0023] In this embodiment, the first hue is the hue value of the fluorescence original image (i.e., the specific value of the hue), the first saturation is the saturation value of the fluorescence original image (i.e., the specific value of the saturation), and the first lightness is the lightness value of the fluorescence original image (i.e., the specific value of the lightness). The range of the hue value of the fluorescence original image in various color models is the same, all being 0~360°.

[0024] In this embodiment, the fluorescence original image is converted to the hexagonal cone color space (hereinafter referred to as the HSV space) to obtain the first hue, the first saturation, and the first lightness. The first hue is used for angular measurement and belongs to 0~360°, calculated counterclockwise starting from red, with red being 0°, green being 120°, and blue being 240°. The first saturation represents the degree to which the color approaches the spectral color, with a range of 0%~100%. The larger the value, the more saturated the color. The first lightness represents the degree of color brightness, with a range of 0% (black) to 100% (white).

[0025] In this embodiment, the above step 102 includes: using a color space conversion tool to convert the fluorescence original image from the RGB or BGR color space to the HSV space. In the HSV space, the first hue is calculated through the hue calculation formula, the first saturation is obtained through the saturation calculation formula, and the first lightness is calculated through the lightness calculation formula. Among them, the hue calculation formula, the saturation calculation formula, and the lightness calculation formula are mature formulas.

[0026] In this embodiment, the converted hue is the value obtained after linearly converting the hue value of the original fluorescence image. For example, if the hue value of the original fluorescence image is the first hue, the first hue is linearly converted to obtain the converted hue. It should be noted that linearly converting the hue value means converting the numerical range of the color space (such as HSV) of the hue value to the numerical range of the color space (the original color space of the original fluorescence image, such as RGB or BGR) before the converted hue value of the original fluorescence image. For example, the hue value is converted from 0-360 to the range of 0-255.

[0027] In this embodiment, the original fluorescence image is converted to the HSV domain and the H (hue) channel value is extracted to obtain the first hue, the first saturation, and the first brightness respectively. Based on the formula for converting RGB to HSV, it is converted to the first hue, and the hue calculation formula is shown in Equation (1).

[0028]

[0029] In Equation (1), G, B, and R respectively represent the values of the green, blue, and red components of the color. max and min respectively represent the maximum and minimum values among these three color components. H is the calculated first hue, and the unit is °. When G (green) is the maximum value, the calculation formula of H is shown in Equation (2): H=(G-B) / (max-min)×60° (2) When B (blue) is the maximum value, the calculation formula of H is shown in Equation (3): H=(2+(B-R) / (max-min))×60° (3) When R (red) is the maximum value, the calculation formula of the hue H is shown in Equation (4): H=(4+(R-G) / (max-min))×60° (4) In this embodiment, H is converted from the numerical range of [0, 360] to the numerical range of [0, 255] to obtain the converted hue, as shown in Equation (5); (5) In Equation (5), H’ represents the converted hue. By linearly converting the first hue through Equation (5), the converted hue is obtained. By linearly converting the hue value of the original fluorescence image as shown in Equation (5), the numerical range of the hue value can be made consistent with the numerical range of the current color space (such as RGB or BGR) of the original fluorescence image, which is convenient for the superposition of the two, and thus the mask image can be effectively obtained.

[0030] Step 103: Based on the converted hue and the original fluorescence image, identify the green component and the blue component to obtain a mask image with information on the green component and the blue component.

[0031] In this embodiment, the green component and the blue component are the stains in the original fluorescence image. A stain / dye is a chemical substance that can bind to a specific target and change its optical properties. The original fluorescence image can be marked with a stain. The green component is the ICG (indocyanine green) stain, and the blue component is the MB (methylene blue) stain. The mask image is a binary matrix (usually 0 or 255) generated by a specific algorithm. A mask image with the information of the green component and the blue component means that the points of interest in the mask image represent the green component or the blue component. Thus, all the points of interest in the entire mask image are the points of interest representing the green component and the blue component.

[0032] In this embodiment, the converted hue and the original fluorescence image are superimposed to obtain a superimposed image; the green component and the blue component in the superimposed image are identified to generate a mask image with the information of the green component and the blue component. Specifically, the identification steps include: converting the superimposed image from color to grayscale. The superimposed image may be disturbed by noise, such as scattered light, background fluorescence, etc. Methods such as median filtering or Gaussian filtering are used to remove the noise in the grayscale image to obtain a smoothed image. By methods such as histogram equalization or linear contrast stretching, the contrast of the smoothed image is enhanced to make the fluorescence signals of the green component and the blue component more obvious. The image with increased contrast is input into a pre-trained binary classification model, and the binary classification model classifies the green component and the blue component and generates a mask image. Specifically, the mask image is as Figure 4 shown.

[0033] Optionally, the step of identifying the green component and the blue component in the superimposed image and generating a mask image with the information of the green component and the blue component further includes: according to the intensity of the superimposed image, selecting an appropriate threshold to segment the superimposed image into a foreground (fluorescence signal) and a background. Global thresholding (such as the Otsu method) or local thresholding methods can be used. The foreground region after segmentation is marked with the green component and the blue component. Specifically, the connected component analysis method can be used to mark each fluorescence signal region as an independent region, and a trained classifier is used to classify the green component and the blue component for each fluorescence signal region and generate a mask image of both the green component and the blue component.

[0034] Step 104, based on the visible light image, the mask image, the first saturation, and the first lightness, obtain a fused image.

[0035] In this embodiment, the above step 104 includes: based on the mask image, selecting a region from the visible light image to obtain a region image; performing image conversion on the region image to obtain a second hue, a second saturation, and a second lightness; obtaining a fused lightness based on the first lightness and the second lightness; obtaining a fused saturation based on the first saturation and the second saturation; obtaining a recombined image based on the second hue, the fused saturation, and the fused lightness; and fusing the recombined image with the image in the visible light image that does not belong to the region of the mask image to obtain a fused image.

[0036] The dual-fluorescence and visible light image fusion method provided by the embodiments of the present disclosure first simultaneously obtains a visible light image and a fluorescence original image of a target scene; secondly, obtains a converted hue, a first saturation, and a first lightness based on the fluorescence original image; then, based on the converted hue and the fluorescence original image, identifies green components and blue components to obtain a mask image with information on the green components and the blue components; and finally, obtains a fused image based on the visible light image, the mask image, the first saturation, and the first lightness. Thus, by converting the hue and the fluorescence original image, the green components and the blue components are accurately identified to form a mask image with the green components and the blue components, which can effectively distinguish the foreground and the background, and the mask image can be used to process only the stained area without affecting other areas; the lightness and saturation of the fluorescence original image are fused with the saturation and lightness of the visible light image within the region defined by the mask image, maintaining the color contrast and avoiding overexposure of the color contrast in the target region corresponding to the mask image, thereby improving the fusion effect of the visible light image and the dual-fluorescence image.

[0037] In the prior art, the fluorescence original image cannot well distinguish detailed features such as textures, and the fused image needs to be edge-refined to restore the detailed features. In some alternative implementation manners of the present disclosure, the above method further includes: performing edge detection on the mask image to obtain an edge detection result; and adding the edge detection result to the fused image to obtain a target image.

[0038] In this alternative implementation manner, there are multiple methods for performing edge detection on the mask image. For example: Sobel operator, Laplacian edge detection. The Sobel operator is an edge detection method based on gradient calculation, which detects edges by calculating the horizontal and vertical gradients of the image. Laplacian edge detection is an edge detection method based on the second derivative, which detects edges by calculating the Laplacian operator of the image.

[0039] In this alternative implementation, adding the edge detection result to the fused image to obtain the target image includes: enhancing the edges in the edge detection result to obtain enhanced edge information, where edge enhancement can be achieved by adjusting the intensity or contrast of the edges in the edge detection result; adding the enhanced edge information to the fused image, where the edge information can be added to the fused image by weighted superposition of the edge information and the fused image.

[0040] The dual-fluorescence and visible-light image fusion method provided in this embodiment adds the edge detection result of the mask image to the fused image, achieving enhancement of the edges of the fused image through edge enhancement technology and avoiding edge blurring caused by traditional pixel-level fusion.

[0041] In some alternative implementations of the present disclosure, the above dual-fluorescence and visible-light image fusion method further includes: using two different colors to identify the green component and the blue component in the mask image to obtain a component identification image; adding the component identification image to the fused image to obtain a result image that identifies the green component and the blue component.

[0042] In this embodiment, the component identification image is a multi-color mask image, and the multi-color mask image can be obtained through various methods such as threshold segmentation, clustering algorithms, and deep learning. Among them, threshold segmentation means: when generating the mask image, different gray-level or color regions in the image are segmented by setting different thresholds and marked with different colors. The clustering algorithm means: using clustering algorithms such as K-means, when generating the mask image, the pixel points in the mask image are divided into different categories and represented by different colors. The deep learning method means: by training a neural network model, different regions in the image are automatically identified and a multi-color mask image is generated. For example, a semantic segmentation model can mark different objects in the image with different colors.

[0043] In this embodiment, adding the component identification image to the fused image can effectively highlight the green component and the blue component in the image and improve the identification effect of multiple components in the fused image.

[0044] In an embodiment of the present disclosure, the above dual-fluorescence and visible-light image fusion method further includes: estimating the user's gaze area in the fused image in real time using a trained human visual sensitive area prediction model to obtain a gaze estimation result; based on the gaze estimation result, assigning dynamic fusion weights to different regions in the mask image; and obtaining a new fused image based on the change of the dynamic fusion weights.

[0045] In this embodiment, by predicting the user's gaze area in the fused image, the attention degree of different regions is determined, and by adjusting the dynamic fusion weights of different regions in the mask image, the display brightness of the fused image is dynamically adjusted through the dynamic fusion weights.

[0046] The dual-fluorescence and visible-light image fusion method provided in this embodiment introduces the human physiological visual mechanism for dynamic weight regulation. In scenarios such as fluorescence-enhanced medical images, it can significantly improve the recognizability of the region of interest and enhance the user experience.

[0047] Optionally, since image fusion methods are usually optimized for static images and ignore the fusion stability problem in video applications, the above dual-fluorescence and visible-light image fusion method further includes: when the fused image is a video frame in a video stream, performing three-frame joint fusion on the video frame and its adjacent video frames; performing inter-frame registration on the mask image based on optical flow field estimation; and generating a new fused image of the video frame through the registered mask image.

[0048] The dual-fluorescence and visible-light image fusion method provided in this embodiment introduces a temporal consistency constraint and an inter-frame mask registration strategy to ensure that the fused image has no obvious flicker or drift in a dynamic scene, significantly expanding the practical boundary of the fusion technology.

[0049] Optionally, the above dual-fluorescence and visible-light image fusion method further includes: using the statistical noise distribution of each channel image before fusion to construct a noise probability map; inputting the noise probability map and the fused image into a pre-trained ghost detection model to obtain the ghost detection result of the ghost detection model; in response to the ghost detection result indicating a serious deviation in the reverse mapping of the fused image, regenerating the mask image, and obtaining a new fused image based on the visible-light image, the regenerated mask image, the first saturation, and the first lightness; in response to the ghost detection result indicating no serious deviation in the direction mapping of the fused image, determining that the fused image is qualified.

[0050] Introduce a "reverse consistency discriminator" during the neural network fusion process to perform reprojection verification on the output image; if a serious deviation in the reverse mapping of the fused image is found, automatically roll back and re-correct the fusion weights.

[0051] This solution integrates the ideas of reverse reasoning and discriminative supervision in computer vision, providing an active correction ability for the quality of the output image.

[0052] In some alternative implementation manners of the present disclosure, obtaining the converted hue based on the original fluorescence image includes: performing mean filtering on the original fluorescence image to obtain a processed image; converting the processed image into a hexagonal pyramid model and extracting the first hue; and obtaining the converted hue based on the first hue.

[0053] In this embodiment, mean filtering is a typical linear filtering algorithm. It means that a template is given to the target pixel on the image. This template includes its surrounding neighboring pixels (the 8 pixels around the target pixel, forming a filtering template, that is, including the target pixel itself), and then the average value of all the pixels in the template is used to replace the original pixel value. The size of the filtering window for mean filtering processing can match the resolution of the original fluorescence image. For example, if the original fluorescence image is a grayscale image with a resolution of 1024×1024 pixels, the fluorescently labeled proteins in the cells in the image show different levels of brightness, representing the distribution of fluorescence intensity. When performing mean filtering on the original fluorescence image, a filtering window of size 3×3 can be selected, and the gray values of each pixel in the image and its surrounding 8 neighboring pixels are averaged to obtain a new pixel value. For example, for the pixel point with coordinates (500, 500) in the image, its original gray value is 120, and the gray values of its surrounding 8 neighboring pixels are 110, 115, 125, 120, 130, 120, 115, and 110 respectively. After mean filtering, the new gray value of this pixel point is: (120 + 110 + 115 + 125 + 120 + 130 + 120 + 115 + 110) / 9 = 118.89 ≈ 119 After mean filtering, the noise in the entire original fluorescence image is effectively suppressed, the image becomes smoother, and the distribution of the fluorescence signal is clearer and more stable, providing a better basis for subsequent hue extraction.

[0054] Convert the image after mean filtering (processed image) into a hexagonal pyramid model. The hexagonal pyramid model is a color space model that represents colors as a hexagonal pyramid structure, which includes three components: hue, saturation, and lightness. In this model, hue represents the type of color, saturation represents the purity of the color, and lightness represents the brightness of the color. During the conversion process, first map the grayscale value range (0 - 255) of the processed image to the lightness axis of the hexagonal pyramid model. Assuming the grayscale value range of the processed image is 0 - 255 and the lightness axis range of the hexagonal pyramid model is also 0 - 255, a direct linear mapping can be performed. Next, calculate the hue value of each pixel point according to the grayscale value distribution of the processed image. In the hexagonal pyramid model, the calculation of hue is related to the position of the pixel point in the color space. For a fluorescence image, since it mainly presents a single fluorescence color (such as green fluorescence), its hue value is relatively concentrated. Assuming the main hue of the fluorescence image is green, the corresponding hue value is within a specific angular range (such as 120° - 150°) in the hexagonal pyramid model. By calculating the position of each pixel point in the hexagonal pyramid model, the corresponding first hue can be extracted, and then a linear transformation is performed on the first hue to obtain the transformed hue. For example, for the pixel point with coordinates (500, 500) in the processed image, its grayscale value after mean filtering is 119. After conversion and calculation, its hue value in the hexagonal pyramid model is 135°, indicating that this pixel point belongs to the hue range of green fluorescence.

[0055] The method for obtaining the transformed hue provided by this optional implementation first performs mean filtering on the original fluorescence image to obtain a processed image; converts the processed image into a hexagonal pyramid model, extracts the first hue, and obtains the transformed hue based on the first hue, which improves the reliability of the transformed hue extraction and the accuracy of the transformed hue.

[0056] In some optional implementations of the present disclosure, the above-mentioned method of identifying the green component and the blue component based on the transformed hue and the original fluorescence image to obtain a mask image with information on the green component and the blue component includes: splicing the transformed hue and the original fluorescence image to obtain a spliced image; inputting the spliced image into a pre-trained neural network model to obtain an identification image including the green component and the blue component; and obtaining a mask image with information on the green component and the blue component based on the identification image.

[0057] In this optional implementation, splicing the transformed hue and the original fluorescence image to obtain a spliced image includes: performing pixel value splicing on the transformed hue and the original fluorescence image to obtain a spliced image, and inputting the spliced image into a pre-trained neural network model as the input of the neural network model, so that the neural network model can perceive the color prior at the initial stage.

[0058] In this alternative implementation, the neural network model is trained using an annotated image set. The annotated images are obtained by annotating the green and blue components, and the annotated images are obtained through the following steps: Obtain video data, and based on the video data, obtain the annotated images. The video data includes at least one video frame, and each video frame has green and blue components. Specifically, select several frame images from the video data and manually annotate the green and blue components in these images.

[0059] In this alternative implementation, the neural network model is used to identify spatial features (such as edges and textures) and segment the spatial features from the fluorescence raw images. However, relying solely on spatial information may result in missegmentation in regions with similar textures (such as blue objects of different materials). The color prior of the converted hue can provide semantic clues independent of the spatial structure and compensate for the limitations of spatial features. Since the green and blue components have obvious hue characteristics, it is considered feasible to use the hue prior for assistance.

[0060] In this alternative implementation, the neural network model includes an encoder and a decoder. The encoder is used to extract the image features of the spliced image, and the decoder obtains the recognition image by guiding the connection attention to focus on the color features and edge texture features in the image features. Use a neural network model including an encoder and a decoder. Input the spliced image into the trained neural network model, and the model outputs a recognition image containing green and blue components. This recognition image is generated by the decoder of the model, where the guided connection attention mechanism ensures the accurate recognition of color features and edge texture features.

[0061] As Figure 5 shown, the neural network model can select CGBA-Net, which consists of an encoder B and a decoder J. Using the fluorescence raw image Y and the converted hue Z superimposed as the input, the encoder B extracts the features of the input image through five layers of convolution. The decoder J adds guided connection attention to focus on the input color features and edge texture features to achieve the purpose of accurate segmentation. After each convolutional layer in the encoding stage, there is a guided connection attention (GCA) module. This module enhances the feature representation through a bidirectional attention mechanism (BA).

[0062] The GCA module receives the feature map F h and F l , and after upsampling and convolutional processing, generates a weight matrix through the bidirectional attention mechanism to adjust the weights of the feature map. In the GCA module, the feature map F h and F l generate the feature map F effFuse with the original feature map to generate an enhanced feature map F GCA 。

[0063] Enhanced feature map F GCA Decode through a series of convolutional layers to gradually restore the spatial resolution of the image. The decoding stage may include upsampling operations to increase the size of the feature map.

[0064] In this alternative implementation, the recognition image is an image that separately recognizes the ICG signal and the MB signal through a neural network model. Through the recognition image, such as Figure 4 or Figure 5 The mask image T shown. In the mask image, both the green component and the blue component are represented by the same pixel value, for example, represented by 1.

[0065] The method for obtaining the mask image provided by this alternative implementation accurately recognizes specific green and blue components from the fluorescence original image and the visible light image through a neural network model and image processing technology, and precisely segments the recognition image including the green and blue components through a lightweight deep learning algorithm with hue prior, avoiding the confusion of the two signals and the influence of background noise.

[0066] Optionally, the above-mentioned method for recognizing the green and blue components based on the converted hue and the fluorescence original image to obtain a mask image including the green and blue components' information includes: splicing the converted hue and the fluorescence original image to obtain a spliced image; extracting the morphological features of the green and blue components in the fluorescence signal region of the spliced image, such as area, perimeter, aspect ratio, circularity, etc. Extract the intensity features of the green and blue components in the fluorescence signal region of the spliced image, such as average gray value, maximum gray value, standard deviation, etc. Extract the texture features of the green and blue components in the fluorescence signal region of the spliced image, such as the features of the gray-level co-occurrence matrix, including contrast, correlation, energy, etc. Based on the morphological features, intensity features, and texture features, obtain the mask image.

[0067] In some alternative implementations of the present disclosure, the above-mentioned method for obtaining a fused image based on the visible light image, the mask image, the first saturation, and the first lightness includes: obtaining a second hue, a second saturation, and a second lightness based on the visible light image; obtaining a fused lightness based on the first lightness, the second lightness, and the mask image; obtaining a fused saturation based on the first saturation, the second saturation, and the mask image; obtaining an output image based on the second hue, the fused saturation, and the fused lightness; fusing the image in the region belonging to the mask image in the output image with the image in the region not belonging to the mask image in the visible light image to obtain a fused image.

[0068] In this alternative implementation, obtaining the fused lightness based on the first lightness, the second lightness, and the mask image includes: performing a weighted sum of the first lightness and the second lightness to obtain a summed lightness; and extracting the lightness in the summed lightness that belongs to the corresponding region of the mask image to obtain the fused lightness.

[0069] In this alternative implementation, obtaining the fused saturation based on the first saturation, the second saturation, and the mask image includes: performing a weighted sum of the first saturation and the second saturation to obtain a summed saturation; and extracting the saturation in the summed saturation that belongs to the corresponding region of the mask image to obtain the fused saturation.

[0070] In this alternative implementation, the fused lightness is the lightness obtained after fusing the first lightness and the second lightness corresponding to the mask image. Through the fused lightness, one can focus only on the fused lightness of the lightness in the corresponding region of the mask image; the fused saturation is the saturation obtained after fusing the first saturation and the second saturation corresponding to the mask image. Through the fused saturation, one can focus only on the fused saturation of the saturation in the corresponding region of the mask image.

[0071] In this alternative implementation, the output image is an image in the HSV model. The output image is an image generated using the second hue, the fused saturation, and the fused lightness. That is, the output image uses the second hue as its hue, the fused saturation as its saturation, and the fused lightness as its lightness.

[0072] In this alternative implementation, fusing the image in the output image that belongs to the region of the mask image with the image in the visible light image that does not belong to the region of the mask image to obtain a fused image includes: converting the output image to the three-primary color light mode to obtain a converted image; using a fusion formula to fuse the converted image with the mask image to obtain the image that belongs to the region of the mask image; using a fusion formula to fuse the visible light image with the mask image to obtain a first fused image, and removing the first fused image from the visible light image to obtain the image in the visible light image that does not belong to the region of the mask image, and performing a weighted sum of the image that belongs to the region of the mask image and the image in the visible light image that does not belong to the region of the mask image to obtain the fused image.

[0073] The method for obtaining a fused image provided by this alternative implementation obtains a second hue, a second saturation, and a second brightness based on a visible light image; obtains a fused brightness based on a first brightness, a second brightness, and a mask image; obtains a fused saturation based on a first saturation, a second saturation, and the mask image; obtains an output image based on the second hue, the fused saturation, and the fused brightness; and fuses the image in the region of the mask image in the output image with the image in the visible light image that does not belong to the region of the mask image to obtain a fused image, providing a reliable implementation for obtaining the fused image and improving the reliability of obtaining the fused image. By using cross-image brightness-saturation co-optimization, the dual-fluorescence fusion color is adaptively adjusted to avoid overexposure in the target region and maintain color contrast.

[0074] In some alternative implementations of the present disclosure, the obtaining of the fused brightness based on the first brightness, the second brightness, and the mask image includes: statistically calculating a first brightness mean and a first brightness variance of the first brightness in the mask image based on the first brightness; statistically calculating a second brightness mean and a second brightness variance of the second brightness in the mask image based on the second brightness; substituting the second brightness, the brightness mean, and the brightness variance into a cross-image brightness matching formula to obtain the fused brightness; the cross-image brightness matching formula represents the sum of the product of a brightness matching parameter, the difference between the second brightness and the second brightness mean, and the ratio of the first brightness variance to the second brightness variance and the first brightness mean, where the brightness matching parameter is a preset parameter.

[0075] In this alternative implementation, the cross-image brightness matching formula is as shown in Equation (6), (6) In Equation (6), is the first brightness mean of the first brightness in the mask image, is the first brightness variance of the first brightness in the mask image, is the second brightness mean of the second brightness in the mask image, is the second brightness variance of the second brightness in the mask image; is the fused image, a represents a custom brightness matching parameter, and the value of a is as shown in Equation (7).

[0076] (7) In Equation (7), Values between.

[0077] In this alternative implementation, in the above cross-image brightness matching formula, V1 and V2 may respectively represent local regions of two images. The cross-image brightness matching formula generates a new fused image V′ by adjusting the local deviation of V2 and combining the global trend of V1. This adjustment of the local deviation can enhance the structural details of the image to a certain extent, while maintaining the overall consistency by introducing the global trend.

[0078] The method for obtaining the fused lightness provided in this alternative implementation uses an adaptive gradient fusion technique. While retaining the color features of the target region, it enhances the structural details through gradient domain fusion, avoids the edge blurring caused by traditional pixel-level fusion, and improves the fusion effect of the dual-fluorescence image and the visible light image.

[0079] In some alternative implementations of the present disclosure, obtaining the fused saturation based on the first saturation, the second saturation, and the mask image includes: statistically calculating the first saturation mean of the first saturation in the mask image based on the first saturation; statistically calculating the second saturation mean of the second saturation in the mask image based on the second saturation; multiplying the first saturation by the ratio of the second saturation mean to the first saturation mean to obtain the fused saturation.

[0080] In this alternative implementation, the first saturation mean and the second saturation mean can be calculated through an averaging formula.

[0081] In this alternative implementation, the formula for calculating the fused saturation by multiplying the first saturation by the ratio of the second saturation mean to the first saturation mean is a formula for normalizing or standardizing the first saturation mean and the second saturation mean. Through this formula, the value of the first saturation can be adjusted to a level that matches the mean of the second saturation.

[0082] The method for obtaining the fused saturation provided in this alternative implementation provides a reliable implementation for obtaining the fused saturation by multiplying the first saturation by the ratio of the second saturation mean to the first saturation mean.

[0083] For the target scene of the endoscope, in a specific example, the dual-fluorescence and visible-light image fusion method provided by the present disclosure can be a visible-light and dual-fluorescence image enhancement fusion algorithm based on the endoscope. The fluorescence image is processed by a deep learning algorithm with hue prior knowledge to accurately segment the green component and the blue component, forming corresponding high-priority mask images. At the same time, cross-image brightness-saturation collaborative optimization is adopted to fuse the images within the corresponding masks, maintaining color contrast and avoiding overexposure in the target area. Finally, gradient-domain fusion is used to enhance the structural details, avoiding edge blurring caused by traditional pixel-level fusion, realizing the fusion of the dual-fluorescence image and the visible-light image, and obtaining the dual-fluorescence fusion image.

[0084] Further referring to Figure 6 , as an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a dual-fluorescence and visible-light image fusion device. This device embodiment corresponds to Figure 1 the method embodiment shown, and this device can be specifically applied to various electronic devices.

[0085] As Figure 6 shown, the dual-fluorescence and visible-light image fusion device 600 provided in this embodiment includes: an acquisition unit 601, a obtaining unit 602, an identification unit 603, and a fusion unit 604. Among them, the above acquisition unit 601 can be configured to simultaneously acquire a visible-light image and a fluorescence original image under the target scene. The above obtaining unit 602 can be configured to obtain a converted hue, a first saturation, and a first lightness based on the fluorescence original image. The converted hue is a value obtained after linearly converting the hue value of the fluorescence original image. The above identification unit 603 can be configured to identify the green component and the blue component in the image based on the converted hue and the fluorescence original image, obtaining a mask image with information on the green component and the blue component. The above fusion unit 604 can be configured to obtain a fusion image based on the visible-light image, the mask image, the first saturation, and the first lightness.

[0086] In this embodiment, in the dual-fluorescence and visible-light image fusion device 600: the specific processing of the acquisition unit 601, the obtaining unit 602, the identification unit 603, and the fusion unit 604 and the technical effects brought by them can respectively refer to Figure 1 the relevant descriptions of steps 101, 102, 103, and 104 in the corresponding embodiments, which will not be elaborated here.

[0087] In an embodiment of the present disclosure, the above device 600 further includes: a detection unit (not shown in the figure). The above detection unit is configured to: perform edge detection on the mask image to obtain an edge detection result; add the edge detection result to the fusion image to obtain a target image.

[0088] In some embodiments of the present disclosure, the obtaining unit 602 is configured to: perform mean filtering on the original fluorescence image to obtain a processed image; convert the processed image into a hexagonal pyramid model and extract a first hue; and obtain a converted hue based on the first hue.

[0089] In some embodiments of the present disclosure, the recognition unit 603 is configured to: splice the converted hue and the original fluorescence image to obtain a spliced image; input the spliced image into a pre-trained neural network model to obtain a recognition image including a green component and a blue component. The neural network model includes an encoder and a decoder. The encoder is used to extract the image features of the spliced image, and the decoder obtains the recognition image by guiding the connection attention to focus on the color features and edge texture features in the image features; obtain a mask image with information of the green component and the blue component based on the recognition image; the neural network model is trained by a labeled image set, and the labeled image is an image obtained after labeling the green component and the blue component. The labeled image is obtained through the following steps: obtain video data, and obtain the labeled image based on the video data. The video data includes at least one video frame, and each video frame has a green component and a blue component.

[0090] In some embodiments of the present disclosure, the fusion unit 604 is configured to: obtain a second hue, a second saturation, and a second lightness based on the visible light image; obtain a fusion lightness based on the first lightness, the second lightness, and the mask image; obtain a fusion saturation based on the first saturation, the second saturation, and the mask image; obtain an output image based on the second hue, the fusion saturation, and the fusion lightness; and fuse the image in the region of the mask image in the output image with the image in the region not belonging to the mask image in the visible light image to obtain a fusion image.

[0091] In some embodiments of the present disclosure, the fusion unit 604 is further configured to: based on the first lightness, statistically calculate a first lightness mean and a first lightness variance of the first lightness in the mask image; based on the second lightness, statistically calculate a second lightness mean and a second lightness variance of the second lightness in the mask image; substitute the second lightness, the lightness mean, and the lightness variance into a cross-image brightness matching formula to obtain the fusion lightness; the cross-image brightness matching formula represents the sum of the product of a brightness matching parameter, the difference between the second lightness and the second lightness mean, and the ratio of the first lightness variance to the second lightness variance and the first lightness mean, where the brightness matching parameter is a pre-set parameter.

[0092] In some embodiments of the present disclosure, the above-mentioned fusion unit 604 is further configured to: based on the first saturation, statistically calculate the first saturation mean of the first saturation in the mask image; based on the second saturation, statistically calculate the second saturation mean of the second saturation in the mask image; multiply the first saturation by the ratio of the second saturation mean to the first saturation mean to obtain the fusion saturation.

[0093] For the document generation device provided by the embodiments of the present disclosure, first, the acquisition unit 601 simultaneously acquires a visible light image and a fluorescence original image in a target scene; secondly, the obtaining unit 602 obtains a converted hue, a first saturation, and a first lightness based on the fluorescence original image; then, the recognition unit 603 recognizes the green component and the blue component in the image based on the converted hue and the fluorescence original image to obtain a mask image with information on the green component and the blue component; finally, the fusion unit 604 obtains a fused image based on the visible light image, the mask image, the first saturation, and the first lightness. Thus, by converting the hue and the fluorescence original image, the green component and the blue component can be accurately recognized to form a mask image with the green component and the blue component, which can effectively distinguish the foreground and the background, and only the stained area can be processed through the mask image without affecting other areas; the lightness and saturation of the fluorescence original image are fused with the saturation and lightness of the visible light image within the area defined by the mask image, maintaining the color contrast and avoiding overexposure of the color contrast in the target area corresponding to the mask image, thereby improving the fusion effect of the visible light image and the dual-fluorescence image.

[0094] According to the embodiments of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0095] Figure 7 FIG. shows a schematic block diagram of an exemplary electronic device 700 that can be used to implement the embodiments of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as, a laptop computer, a desktop computer, a workbench, a personal digital assistant, a server, a blade server, a mainframe computer, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as, a personal digital processor, a cellular phone, a smart phone, a wearable device, and other similar computing devices. The components shown herein, their connections and relationships, and their modes are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0096] As Figure 7As shown, the electronic device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to computer programs stored in a read-only memory (ROM) 702 or computer programs loaded from a storage unit 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the electronic device 700 can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0097] Multiple components in the electronic device 700 are connected to the I / O interface 705, including: an input unit 706, such as a keyboard, a mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a magnetic disk, an optical disc, etc.; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the electronic device 700 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0098] The computing unit 701 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 701 executes the various methods and processes described above, such as the dual fluorescence and visible light image fusion method. For example, in some embodiments, the dual fluorescence and visible light image fusion method can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as the storage unit 708. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the computing unit 701, one or more steps of the dual fluorescence and visible light image fusion method described above can be executed. Alternatively, in other embodiments, the computing unit 701 can be configured to execute the dual fluorescence and visible light image fusion method by any other appropriate means (e.g., by means of firmware).

[0099] The various embodiments of the systems and techniques described above in this document can be implemented in digital electronic circuitry, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on a chip (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special-purpose or general-purpose programmable processor that receives data and instructions from a storage system, at least one input device, and at least one output device, and transmits the data and instructions to the storage system, the at least one input device, and the at least one output device.

[0100] The program code for implementing the methods of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable dual-fluorescence and visible light image fusion devices, such that when the program codes are executed by the processor or controller, the patterns / operations specified in the flowcharts and / or block diagrams are implemented. The program code can be executed entirely on the machine, partially on the machine, executed partially on the machine as an independent software package and partially on a remote machine, or executed entirely on a remote machine or server.

[0101] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media would include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0102] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and a pointing device (e.g., a mouse or a trackball) through which the user can provide input to the computer. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, speech input, or tactile input).

[0103] The systems and techniques described herein can be implemented in a computing system including backend components (e.g., as a data server), or a computing system including middleware components (e.g., an application server), or a computing system including frontend components (e.g., a user computer having a graphical user interface or a web browser through which the user can interact with an implementation of the systems and techniques described herein), or a computing system including any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0104] It should be understood that the various forms of the processes shown above can be used, with steps reordered, added, or deleted. For example, the steps recited in this disclosure can be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this is not limited herein.

[0105] The foregoing description of specific exemplary embodiments of the present disclosure is for purposes of illustration and exemplification. These descriptions are not intended to limit the present disclosure to the precise forms disclosed, and obviously, many changes and variations are possible in light of the above teachings. The purpose of selecting and describing the exemplary embodiments is to explain the specific principles of the present disclosure and its practical applications, so that those skilled in the art can implement and utilize the various different exemplary embodiments of the present disclosure, as well as various different selections and changes. The scope of the present disclosure is intended to be defined by the claims and their equivalents.

[0106] The above are only embodiments of the present disclosure and are not intended to limit the present disclosure. Any modifications, equivalent replacements, improvements, etc. made within the principles of the present disclosure shall be included within the protection scope of the present disclosure.

Claims

1. A method for fusing dual fluorescence and visible light images, characterized in that The method includes: Obtaining a visible light image and a fluorescence original image in a target scenario simultaneously; Based on the fluorescence original image, obtaining a converted hue, a first saturation, and a first lightness, where the converted hue is a value obtained after linearly converting the hue value of the fluorescence original image; Based on the converted hue and the fluorescence original image, identifying a green component and a blue component, and obtaining a mask image with information of the green component and the blue component; Based on the visible light image, the mask image, the first saturation, and the first lightness, obtaining a fused image.

2. The method according to claim 1, wherein The method further includes: Performing edge detection on the mask image to obtain an edge detection result; adding the edge detection result to the fused image to obtain a target image; Alternatively, using two different colors to identify the green component and the blue component in the mask image to obtain a component identification image; adding the component identification image to the fused image to obtain a result image for identifying the green component and the blue component; Alternatively, using a trained human visual sensitive area prediction model to estimate the user's gaze area in the fused image in real time to obtain a gaze estimation result; based on the gaze estimation result, assigning dynamic fusion weights to different regions in the mask image; based on the change of the dynamic fusion weights, obtaining a new fused image.

3. The method according to claim 1 or 2, characterized in that, The obtaining the converted hue based on the fluorescence original image includes: Performing mean filtering on the fluorescence original image to obtain a processed image; Converting the processed image into a hexagonal pyramid model and extracting a first hue; Based on the first hue, obtaining the converted hue.

4. The method according to claim 1 or 2, characterized in that, The identifying the green component and the blue component based on the converted hue and the fluorescence original image to obtain a mask image with information of the green component and the blue component includes: Stitching the converted hue and the fluorescence original image to obtain a stitched image; Inputting the stitched image into a pre-trained neural network model to obtain an identification image including the green component and the blue component, where the neural network model includes an encoder and a decoder, the encoder is used to extract the image features of the stitched image, and the decoder obtains the identification image by guiding the connection attention to focus on the color features and edge texture features in the image features; Based on the identification image, obtaining a mask image with information of the green component and the blue component; The neural network model is trained by an annotated image set, the annotated image is an image obtained after annotating the green component and the blue component, and the annotated image is obtained through the following steps: obtaining video data, and based on the video data, obtaining the annotated image, where the video data includes at least one video frame, and each video frame has a green component and a blue component.

5. The method according to claim 1 or 2, characterized in that The obtaining the fused image based on the visible light image, the mask image, the first saturation, and the first lightness includes: Based on the visible light image, obtaining a second hue, a second saturation, and a second lightness; Based on the first lightness, the second lightness, and the mask image, obtaining a fused lightness; Based on the first saturation, the second saturation, and the mask image, a fused saturation is obtained; Based on the second hue, the fused saturation, and the fused lightness, an output image is obtained; The image in the output image corresponding to the region of the mask image is fused with the image in the visible light image that does not belong to the region of the mask image to obtain a fused image.

6. The method according to claim 5, wherein The obtaining the fused lightness based on the first lightness, the second lightness, and the mask image includes: Based on the first lightness, the first lightness mean and the first lightness variance of the first lightness in the mask image are statistically calculated; Based on the second lightness, the second lightness mean and the second lightness variance of the second lightness in the mask image are statistically calculated; The second lightness, the lightness mean, and the lightness variance are substituted into a cross-image brightness matching formula to obtain the fused lightness; the cross-image brightness matching formula represents the sum of the product of a brightness matching parameter, the difference between the second lightness and the second lightness mean, and the ratio of the first lightness variance to the second lightness variance, and the first lightness mean, where the brightness matching parameter is a pre-set parameter.

7. The method according to claim 5, characterized in that, The obtaining the fused saturation based on the first saturation, the second saturation, and the mask image includes: Based on the first saturation, the first saturation mean of the first saturation in the mask image is statistically calculated; Based on the second saturation, the second saturation mean of the second saturation in the mask image is statistically calculated; The first saturation is multiplied by the ratio of the second saturation mean to the first saturation mean to obtain the fused saturation.

8. A dual-fluorescence and visible-light image fusion device, characterized in that, The device includes: An acquisition unit configured to simultaneously acquire a visible light image and a fluorescence original image in a target scene; A obtaining unit configured to obtain a converted hue, a first saturation, and a first lightness based on the fluorescence original image, where the converted hue is a value obtained by linearly converting the hue value of the fluorescence original image; An identification unit configured to identify a green component and a blue component in the image based on the converted hue and the fluorescence original image to obtain a mask image with information on the green component and the blue component; A fusion unit configured to obtain a fused image based on the visible light image, the mask image, the first saturation, and the first lightness.

9. An electronic device, characterized in that, Includes: At least one processor; And a memory communicatively connected to the at least one processor; Wherein, the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method according to any one of claims 1-7.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to execute the method according to any one of claims 1-7.

Images