A fluorescent and visible light image fusion imaging method for an intraoperative environment

Abstract

The application relates to a fluorescence and visible light image fusion imaging method for an intraoperative environment and relates to the technical field of surgical imaging processing. The application acquires a visible light image of an intraoperative target imaging area and a fluorescence image of the target imaging area by using a plurality of cameras; the visible light image and the fluorescence image are input into a pre-trained image matching model to obtain a matching transformation matrix of the two images; wherein the image matching model comprises a feature extractor of an image and a matcher of a cascaded feature extractor; the matcher acquires the matching transformation matrix of the visible light image and the fluorescence image according to semantic feature maps extracted by the feature extractor; the fluorescence image is subjected to affine transformation according to the matching transformation matrix to obtain a transformed fluorescence image; the transformed fluorescence image and the visible light image are input into a pre-trained image fusion model to be fused to obtain a fusion image of the target imaging area. The application utilizes a multi-mode light source and image fusion surgery to provide an image with guiding significance.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a fluorescence and visible light image fusion imaging method for intraoperative environments. Background Technology

[0002] The statements in this section are merely background information relating to this disclosure and do not necessarily constitute prior art.

[0003] Specific human organs / glands exhibit unique reflection effects under specific lighting conditions, producing imaging results distinct from normal imaging. Therefore, during surgery, to better perceive organs / glands, multiple modalities of light, including specific lighting, and corresponding optical elements are used for comprehensive imaging. This technology has already been applied to specific organs such as the breast and parathyroid glands. However, this process typically involves separate imaging through multiple channels. During surgery, surgeons manually compare the results from different channels to make judgments and perform surgical procedures. This approach is not conducive to real-time differentiation of the target organ and gland during surgery. Furthermore, separate imaging through multiple channels hinders surgeons from acquiring more comprehensive image information in real time, and the overall effect is not significantly different from single-modal imaging. Therefore, it is necessary to provide a fluorescence and visible light image fusion imaging method that can fuse images under different lighting modalities, and whose fused images provide support for surgery and facilitate convenient identification of the fused images, thus addressing the intraoperative environment. Summary of the Invention

[0004] To solve the above-mentioned technical problems, or at least partially solve them, the present invention provides a fluorescence and visible light image fusion imaging method for intraoperative environments.

[0005] In a first aspect, the present invention provides a fluorescence and visible light image fusion imaging method for intraoperative environments, comprising: Use a multi-channel camera to acquire visible light images and fluorescence images of the target imaging area during surgery; The visible light image and fluorescence image of the target imaging region are input into a pre-trained image matching model to obtain the matching transformation matrix of the two images; wherein, the image matching model includes a feature extractor of an image and a matcher of a cascaded feature extractor; the matcher obtains the matching transformation matrix of the visible light image and the fluorescence image based on the semantic feature map extracted by the feature extractor; The fluorescence image is then subjected to an affine transformation based on the matching transformation matrix to obtain the transformed fluorescence image. The transformed fluorescence image and the visible light image are input into a pre-trained image fusion model for fusion to obtain a fused image of the target imaging region.

[0006] Furthermore, the feature extractor of the image matching model extracts the semantic feature map of the image to obtain the semantic feature vector of each pixel. The image matching model's matcher uses the obtained semantic feature vectors to calculate the matching score matrix of two images; it iterates through the matching score matrix and the two original semantic feature maps to calculate the probability of matching two pixels; it selects a set number of candidate matching points with the highest probability values; for each candidate matching point, it selects its local window map; it uses a lightweight image model to extract the semantic features of the local window map, generating local feature maps of the visible light image and the fluorescence image; it performs an autocorrelation operation on the local feature maps and uses a softmax operation to determine the two most relevant points, and uses the most relevant points to solve for the matching transformation matrix of the visible light image and the fluorescence image.

[0007] Furthermore, the visible light image A and the fluorescence image B are input into the feature extractor to obtain their semantic feature maps. The semantic feature vector corresponding to each pixel (u, v) in the semantic feature map of visible light image A is recorded as follows: The semantic feature vector corresponding to each pixel (u, v) on the semantic feature map of fluorescence image B is recorded as follows: .

[0008] Furthermore, the image matching model is trained using a training dataset labeled with matching point pairs, optimized according to the following loss function: ; The first term of the loss function represents the positional error between the predicted matching point and the actual matching point. These are the actual coordinates of the matching point measured by the robotic arm system; These are the coordinates of the most relevant matching point, i.e., the predicted value, which is the final output of the image matching model. Indicates L2 distance; The variance represents the uncertainty of the corresponding point; the second term of the loss function is the logarithmic sum of the probabilities given by the image matching model for all the finally selected matching point pairs (i,j), which encourages the image matching model to make matching decisions with high confidence.

[0009] Furthermore, the image fusion model includes a convolution-based fusion convolution module and a ResNet module based on a residual convolutional network for auxiliary training; for the input visible light image and the transformed fluorescence image, the image fusion model uses the fusion convolution module to obtain the fused image of the visible light image. Image before fusion with fluorescence image The two original images are added together to generate the fused image. .

[0010] Furthermore, in order to achieve the fusion of the graph To provide better visual results for surgical imaging, the training constraints for the training fusion convolutional module are constructed and trained as follows: First, use the image before fusion of visible light images Before fusion of fluorescence images The merged image is generated by adding the two original images together. HED edge loss and grayscale loss functions are constructed and trained to adjust the parameters in the fusion convolution module; Then, the fused graph is processed by a ResNet module formed by a residual convolutional network. The analysis and processing are performed to extract the fusion semantics, and the semantic Dice loss of the semantics is calculated by combining the semantic labels of the labeled data to train and adjust the parameters in the fusion convolution module.

[0011] Furthermore, the grayscale loss function is as follows: ; Where H and W represent the height and width of the image, Indicates grayscale; The HED edge loss function is as follows: ; in, This represents the HED edge detection operator, which converts the image into pixel gradients for output.

[0012] Secondly, the present invention provides a fluorescence and visible light image fusion imaging device for intraoperative environments, comprising: at least one processing unit, the processing unit being connected to a storage unit and a multi-channel camera via a bus unit, the multi-mode light source required for imaging by the multi-channel camera including a visible light source and a fluorescence light source; the storage unit storing a computer program, the computer program being executed by the processing unit to realize the fluorescence and visible light image fusion imaging method for intraoperative environments.

[0013] Thirdly, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the fluorescence and visible light image fusion imaging method for intraoperative environments as described above.

[0014] The technical solutions provided in the embodiments of the present invention have the following advantages compared with the prior art: This invention uses multiple cameras to acquire images of the target imaging area in a book under multi-mode light sources. It employs a pre-trained image matching model for registration of visible light and fluorescence images, supporting cross-device and cross-time dimension image matching. This model can match visible light and fluorescence images of different resolutions and viewing angles together before subsequent fusion, reducing limitations on hardware products and usage scenarios.

[0015] This application employs a pre-trained image fusion model for fusion imaging. By performing bilateral image fusion, this invention enhances the convenience for operators, expands the information dimensions of the images, and provides stronger real-time guidance for common surgical procedures such as the removal of target organs or glands and plastic surgery. It improves the success rate and time efficiency of intraoperative procedures by providing additional data content.

[0016] The method of this invention can make full use of the different optical properties of human organs or glands under different light sources for imaging and fusion, which improves the incompatibility of traditional medical imaging systems under complex scenes with a single light source. It enables users to see both the macroscopic global information in the visible light image and the image information of specific areas highlighted by the fluorescence image, which facilitates users to perform fine operations in subsequent surgical procedures and provides technical support for high-precision and intelligent surgical operations. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 A flowchart of a fluorescence and visible light image fusion imaging method for intraoperative environment provided in an embodiment of the present invention; Figure 2 An architecture diagram of the image matching model and its training provided in an embodiment of the present invention; Figure 3 The image fusion model and its training architecture provided in the embodiments of the present invention; Figure 4 A schematic diagram of a fluorescence and visible light image fusion imaging system for intraoperative environments provided in an embodiment of the present invention; Figure 5This is a schematic diagram of a fluorescence and visible light image fusion imaging device for intraoperative environments, provided as an embodiment of the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0022] Example 1 like Figure 1 As shown, this invention provides a fluorescence and visible light image fusion imaging method for intraoperative environments, comprising: S100, under multi-mode light source, uses multiple cameras to acquire visible light images and fluorescence images of the target imaging area during surgery; A multi-channel camera is used to acquire images of the target imaging area of ​​organs / glands during surgery. The multi-channel camera consists of two cameras, one of which uses ordinary RGB visible light to capture the target imaging area, and the other camera uses an infrared fluorescent light source to capture images.

[0023] S200 pre-trains a specific image matching model for matching visible light and fluorescence images. The visible light and fluorescence images of the target imaging region are input into the image matching model to obtain the matching transformation matrix of the two images.

[0024] like Figure 2 As shown, the image matching model includes an image feature extractor and a matcher of a cascaded feature extractor for image matching. For example, the image matching model uses the DinoV2 large visual model as the feature extractor to extract the basic visual feature vectors of the visible light and fluorescence images. Then, a matcher of a cascaded feature extractor is used to output the pixel coordinates of the most relevant feature points, and the matching transformation matrix is ​​further calculated.

[0025] Specifically, the image matching model takes two images as input each time. In this application, one image is a visible light image and the other is a fluorescence image. Let the input visible light image be represented as A and the fluorescence image as B. For the input visible light image A and fluorescence image B, a feature extractor is used to obtain the semantic feature maps of both. The semantic feature vector corresponding to each pixel (u, v) in the semantic feature map of the visible light image A is recorded as follows: The semantic feature vector corresponding to each pixel (u, v) on the semantic feature map of fluorescence image B is recorded as follows: .

[0026] The matching score matrix between the two images is calculated based on the semantic feature vectors. The elements of the example matching score matrix are calculated as follows: ; in, This is a weight parameter used to mark the semantic weight of the current region where the pixel is located; it is generally set to 2. It is the vector inner product.

[0027] The formula for calculating the probability P of two matching pixels by traversing the matching score matrix and the two original semantic feature maps is as follows: ; This formula is used to calculate the probability P that the i-th pixel in the visible light image and the j-th pixel in the fluorescence image are a matching pair. i and j are the pixel indices during the traversal process, and each pixel index (i.e., i or j) corresponds to a set of pixel coordinates (u, v). is an indicator variable representing whether the i-th pixel in the visible light image matches the j-th pixel in the fluorescence image. C is the total number of channels considered.

[0028] By summing and aggregating the matching score matrix and the information from the two original semantic feature maps, a comprehensive judgment is made as to whether two pixels truly correspond. The ultimate goal is to select the top K pixels with the highest probability values ​​from all pixels as candidate matching points. In this example, K is set to fifty. Then, for each pixel, a local window image consisting of its surrounding w×w pixels (w is 3 in this example) is selected. A lightweight image model is used to extract the semantic features of the local window image, generating local feature maps for visible light image A and fluorescence image B, respectively represented as follows. and The lightweight image model used in the example is the lightweight Mobilenetv3; then, autocorrelation operations are performed on the local feature maps of visible light image A and fluorescence image B, and a softmax operation is used to determine the two most correlated points. .

[0029] The matching transformation matrix between the visible light image and the fluorescence image is solved using the most relevant points.

[0030] The training process of an image matching model includes: The training dataset for the image matching model was obtained as follows: Two robotic arms mounted on fixed bases, equipped with various types of cameras, were used to capture videos containing marked matching points to collect training data. The self-position information of the real images was derived based on the real-time motion records of the two robotic arms. and The true image conversion information of the two was obtained by using quaternion transformation calculation. A large amount of conversion information from real images was obtained. After processing the data, the matching relationships of the markers in the real images are calculated, and M matching point pairs are extracted from each image, denoted as MM. .

[0031] The image matching model was trained using the training dataset and optimized with the following loss function: ; The first term of the loss function represents the positional error between the predicted matching point and the actual matching point. These are the coordinates of the actual matching point measured by the robotic arm system. These are the coordinates of the most relevant matching point, i.e., the predicted value, which is the final output of the image matching model. This represents the L2 distance. This represents the variance, indicating the uncertainty at a given point. If the measurement error (large variance) at a point is large, then... A large reciprocal and a small inverse reduce its impact on the total loss, encouraging the image model to focus more on points with high location confidence. The second term of the loss function is the logarithmic sum of the probabilities given by the image matching model for all finally selected matching point pairs (i,j). This encourages the image matching model to make matching decisions with high confidence.

[0032] S300: The fluorescence image is affinely transformed according to the matching transformation matrix to obtain the transformed fluorescence image. In surgical observation systems with multi-mode light sources and multiple cameras, since the visible light camera and the fluorescence camera cannot be integrated into one location for observation, two cameras are generally used for separate observation. For example, two medical robotic arms are used, one equipped with a visible light camera and the other with a fluorescence camera, and then they are used to capture images of the same target imaging area. This operation will cause a certain visual deviation between the two cameras. For example, if the visible light camera is to the left of the fluorescence camera, the image will be shifted to the right. This parallax between the two images makes subsequent fusion processing inconvenient. The image matching model of this invention can still be fused even if there is a large positional difference between the two cameras, or it can still work normally in some special cases where the observation position and observation area need to be changed due to operational needs.

[0033] S400: The transformed fluorescence image and visible light image are used as inputs to the image fusion model to construct the fused image.

[0034] A specific image fusion model is pre-trained to fuse the transformed fluorescence image and the visible light image to obtain the fused image.

[0035] The significance of image fusion lies in its ability to display the highlight response characteristics of fluorescence on RGB visible light images, facilitating intraoperative observation, marking, and other procedures. Practice has shown that simply weighting and adding two images together does not yield readily observable results. Therefore, this application employs an image fusion model to fuse visible light and fluorescence images.

[0036] The image fusion model consists of a convolution-based fusion convolutional module and a ResNet module based on a residual convolutional network used for auxiliary training. The ResNet module provides Dice loss during the training of the fusion convolutional module and is discarded during inference.

[0037] Given an input visible light image and a transformed fluorescence image, the image fusion model uses a fusion convolution module to obtain the pre-fusion image of the visible light image. Image before fusion with fluorescence image The two original images are added together to generate the fused image. .

[0038] In order to make the fusion image To provide better visual results for surgical imaging, the training constraints for the fusion convolutional module are constructed and trained as follows: First, use the image before fusion of visible light images Before fusion of fluorescence images The merged image is generated by adding the two original images together. HED edge loss and grayscale loss functions are constructed and trained to adjust the parameters in the fusion convolution module; The grayscale loss function is as follows: ; Where H and W represent the height and width of the image. This represents the grayscale conversion operator.

[0039] The HED edge loss function is as follows: ; Same as above. This represents the HED edge detection operator. The HED edge detection algorithm converts the image into pixel gradients for output.

[0040] Then, the fused graph is processed by a ResNet module formed by a residual convolutional network. The analysis and processing are performed to extract the fusion semantics, and the Dice loss is calculated by combining the semantic labels of the labeled data to train and adjust the parameters in the fusion convolution module.

[0041] The image fusion model was trained on two datasets, MSRS and RoadScene.

[0042] The registered fluorescence image and the visible light image have significant imaging differences and cannot be simply added together. This invention provides an image fusion model for two-light images, which can effectively avoid observational discomfort caused by the differences between the two images after superposition. For example, excessively strong highlights in the fluorescence image can obscure the image content in the visible light image, making the overall image blurry. Conversely, excessive color in the visible light image can mask the clear gradient information in the fluorescence image. This allows users to obtain a fused image that fully preserves image features such as fluorescence highlight regions, visible light semantic information, and the gradient field of the fluorescence image.

[0043] Certain organs / glands exhibit high brightness under fluorescence, while visible light provides clear and bright imaging of the entire imaging area. Combining these two methods helps surgeons identify specific organs or glands within a complex surgical field. This specialized optical imaging system fully utilizes the different optical properties of human organs or glands under various light sources, improving upon the limitations of traditional medical imaging systems that rely on a single light source for complex scenarios. It allows users to see both macroscopic global information and specific regional image information, facilitating precise operations during subsequent surgery and mitigating errors caused by single-modal imaging. This provides a technological guarantee for high-precision, intelligent surgical procedures.

[0044] Example 2 like Figure 4 As shown, this invention provides a fluorescence and visible light image fusion imaging system for intraoperative environments, comprising: The system comprises an image acquisition module, a registration module, a fusion module, and a user interaction module. The image acquisition module uses multiple cameras to acquire visible light and fluorescence images of the target imaging area during surgery, and transmits these images to the registration and fusion modules for processing. The registration module receives the visible light and fluorescence images of the target imaging area from the image acquisition module, extracts and matches feature points, and outputs a matching transformation matrix for the two images. This matrix is ​​then used to homography the fluorescence image before being output to the fusion module. The fusion module receives the transformed fluorescence image and the original visible light image, performs image fusion based on deep learning, and outputs the fused image to the user interaction module. The user interaction module primarily provides functions for saving images and videos, changing image resolution, and viewing historical images, mainly to meet the personal needs of the operator during surgery.

[0045] Example 3 See Figure 5 As shown, this embodiment of the invention provides a fluorescence and visible light image fusion imaging device for intraoperative environments, characterized by comprising: at least one processing unit, the processing unit being connected to a storage unit and a multi-channel camera via a bus unit, the multi-channel camera imaging requiring a multi-mode light source, the multi-mode light source including a visible light source and a fluorescence light source; the storage unit, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as the software program, computer-executable program, and module corresponding to the fluorescence and visible light image fusion imaging method for intraoperative environments in this embodiment of the invention. The processing unit implements the aforementioned fluorescence and visible light image fusion imaging method for intraoperative environments by running the software program, computer-executable program, and module stored in the storage unit, including: Use a multi-channel camera to acquire visible light images and fluorescence images of the target imaging area during surgery; The visible light image and fluorescence image of the target imaging region are input into a pre-trained image matching model to obtain the matching transformation matrix of the two images; wherein, the image matching model includes a feature extractor of an image and a matcher of a cascaded feature extractor; the matcher obtains the matching transformation matrix of the visible light image and the fluorescence image based on the semantic feature map extracted by the feature extractor; The fluorescence image is then subjected to an affine transformation based on the matching transformation matrix to obtain the transformed fluorescence image. The transformed fluorescence image and the visible light image are input into a pre-trained image fusion model for fusion to obtain a fused image of the target imaging region.

[0046] Of course, the computer program stored in the storage unit of the fluorescence and visible light image fusion imaging device for intraoperative environment provided in the embodiments of the present invention is not limited to the method operation described above, but can also execute related operations in the fluorescence and visible light image fusion imaging method for intraoperative environment provided in any embodiment of the present invention.

[0047] Example 4 This invention provides a computer-readable storage medium storing a computer program that, when executed, implements the described fluorescence and visible light image fusion imaging method for intraoperative environments, including: Use a multi-channel camera to acquire visible light images and fluorescence images of the target imaging area during surgery; The visible light image and fluorescence image of the target imaging region are input into a pre-trained image matching model to obtain the matching transformation matrix of the two images; wherein, the image matching model includes a feature extractor of an image and a matcher of a cascaded feature extractor; the matcher obtains the matching transformation matrix of the visible light image and the fluorescence image based on the semantic feature map extracted by the feature extractor; The fluorescence image is then subjected to an affine transformation based on the matching transformation matrix to obtain the transformed fluorescence image. The transformed fluorescence image and the visible light image are input into a pre-trained image fusion model for fusion to obtain a fused image of the target imaging region.

[0048] The computer-readable storage medium provided in the embodiments of the present invention stores a computer program that is not limited to the method operation described above, but can also execute related operations in the fluorescence and visible light image fusion imaging method for intraoperative environment provided in any embodiment of the present invention.

[0049] In the embodiments provided by this invention, it should be understood that the disclosed structures and methods can be implemented in other ways. For example, the structural embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces, structures, or units, and may be electrical, mechanical, or other forms.

[0050] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0051] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0052] The above description is merely a specific embodiment of the present invention, enabling those skilled in the art to understand or implement the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A fluorescence and visible light image fusion imaging method for intraoperative environment, characterized in that, include: Use a multi-channel camera to acquire visible light images and fluorescence images of the target imaging area during surgery; The visible light image and fluorescence image of the target imaging region are input into a pre-trained image matching model to obtain the matching transformation matrix of the two images. The image matching model includes a feature extractor for one image and a matcher for a cascaded feature extractor. The feature extractor of the image matching model extracts the semantic feature map of the image, obtaining the semantic feature vector of each pixel. The matcher of the image matching model uses the obtained semantic feature vector to calculate the matching score matrix of the two images. The matching score matrix and the two original semantic feature maps are traversed to calculate the probability of matching between two pixels. A set number of candidate matching points with the highest probability values ​​are selected. For each candidate matching point, its local window map is selected. A lightweight image model is used to extract the semantic features of the local window map, generating local feature maps of the visible light image and the fluorescence image. The local feature maps are subjected to autocorrelation operations and softmax operations are used to determine the two most relevant points. The matching transformation matrix of the visible light image and the fluorescence image is solved using the most relevant points. The image matching model was trained using a training dataset labeled with matching point pairs, and the following loss function was used to optimize the model: ； The first term of the loss function represents the positional error between the predicted matching point and the actual matching point. These are the actual coordinates of the matching point measured by the robotic arm system; These are the coordinates of the most relevant matching point, i.e., the predicted value, which is the final output of the image matching model. Indicates L2 distance; The variance represents the uncertainty of the corresponding point; the second term of the loss function is the logarithmic sum of the probabilities given by the image matching model for all the finally selected matching point pairs (i,j), which encourages the matching decisions made by the image matching model to have high confidence. The indicator variable represents whether the i-th pixel in the visible light image matches the j-th pixel in the fluorescence image; The fluorescence image is then subjected to an affine transformation based on the matching transformation matrix to obtain the transformed fluorescence image. The transformed fluorescence image and the visible light image are input into a pre-trained image fusion model for fusion to obtain a fused image of the target imaging region. The image fusion model includes a convolution-based fusion convolution module and a ResNet module based on a residual convolutional network for auxiliary training. For the input visible light image and the transformed fluorescence image, the image fusion model uses the fusion convolution module to obtain the pre-fusion image of the visible light image. Image before fusion with fluorescence image The two original images are added together to generate the fused image. ; In order to make the fusion image To provide better visual results for surgical imaging, the training constraints for the training fusion convolutional module are constructed and trained as follows: First, use the image before fusion of visible light images Before fusion of fluorescence images The merged image is generated by adding the two original images together. HED edge loss and grayscale loss functions are constructed and trained to adjust the parameters in the fusion convolution module; the grayscale loss function is as follows: ； Where H and W represent the height and width of the image, Indicates grayscale; The HED edge loss function is as follows: ； in, This represents the HED edge detection operator, which converts the image into pixel gradients for output. Then, the fused graph is processed by a ResNet module formed by a residual convolutional network. The analysis and processing are performed to extract the fusion semantics, and the semantic Dice loss of the semantics is calculated by combining the semantic labels of the labeled data to train and adjust the parameters in the fusion convolution module.

2. The fluorescence and visible light image fusion imaging method for intraoperative environment according to claim 1, characterized in that, Input visible light image A and fluorescence image B into the feature extractor to obtain their semantic feature maps. Record the semantic feature vector corresponding to each pixel (u, v) in the semantic feature map of visible light image A. The semantic feature vector corresponding to each pixel (u, v) on the semantic feature map of fluorescence image B is recorded as follows: ; The matching score matrix between the two images is calculated based on the semantic feature vectors. The elements of the matching score matrix are calculated as follows: ； in, These are weight parameters used to label the semantic weight of the current region where the pixel is located; It is the dot product of vectors; The formula for calculating the probability P of two matching pixels by traversing the matching score matrix and the two original semantic feature maps is as follows: ； The formula is used to calculate the probability P that the i-th pixel in the visible light image and the j-th pixel in the fluorescence image are a pair of matching points, where i and j are the pixel indices during the traversal process, and each pixel index i or j corresponds to a set of pixel coordinates. is an indicator variable, indicating whether the i-th pixel in the visible light image matches the j-th pixel in the fluorescence image, and C is the total number of channels considered.

3. A fluorescence and visible light image fusion imaging device for intraoperative environments, characterized in that, include: At least one processing unit is provided, which is connected to a storage unit and a multi-channel camera via a bus unit. The multi-channel camera imaging requires a multi-mode light source, which includes a visible light source and a fluorescence light source. The storage unit stores a computer program, which, when executed by the processing unit, implements the fluorescence and visible light image fusion imaging method for intraoperative environment as described in claim 1 or 2.

4. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the fluorescence and visible light image fusion imaging method for intraoperative environment as described in any one of claims 1 or 2.

Images