Method and system for determining the fused image of white light and fluorescence images of an object to be imaged.
By simultaneously acquiring and processing white light and fluorescence images, and combining vascular network enhancement and ROI recognition technologies, the problem of image spatial mismatch was solved, achieving precise fusion of white light and fluorescence images and providing an intuitive quantitative mapping of cervical tissue metabolic abnormalities.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SHANGHAI FIRST MATERNITY & INFANT HOSPITAL
- Filing Date
- 2026-04-07
- Publication Date
- 2026-07-03
Smart Images
Figure CN122335567A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of this application relate to the field of image data processing technology, specifically to a method and system for determining a fusion image of a white light image and a fluorescence image of an object to be imaged. Background Technology
[0002] The statements herein are provided merely as background information in connection with this application and do not necessarily constitute prior art.
[0003] In existing technologies, white light images of predetermined sites are acquired to obtain high-resolution cervical surface morphology and vascular information. However, white light images are difficult to reflect early metabolic changes in cervical tissue. While cervical tissue fluorescence images obtained by tissue autofluorescence imaging technology can reflect metabolic status, they lack clear anatomical structural references and make it difficult to accurately map the abnormal metabolic areas reflected in the fluorescence images onto white light images. Summary of the Invention
[0004] A brief overview of this application is provided below to offer a basic understanding of certain aspects thereof. It should be understood that this overview is not an exhaustive summary of the application. It is not intended to identify key or essential parts of the application, nor is it intended to limit its scope. Its purpose is merely to present certain concepts in a simplified form as a prelude to the more detailed description that follows.
[0005] In a first aspect, embodiments of this application provide a method for determining a fusion image of a white light image and a fluorescence image of an object to be imaged, comprising the following steps: S10: synchronously acquiring a white light image and a fluorescence image of the same predetermined location of the object to be imaged; S20: processing the white light image, performing vascular network enhancement and grayscale processing on the white light image at the predetermined location to obtain a white light grayscale image; S30: processing the fluorescence image, identifying the fluorescence image at the predetermined location, and obtaining the ROI of the fluorescence image; S40: determining the blue channel B and green channel G in the ROI of the fluorescence image, determining the redox ratio map of the ROI of the fluorescence image based on the blue channel B and green channel G, extracting the texture features of the redox ratio map, and determining a fixed-size suspicious region coordinate array and a score array of the fluorescence image based on the texture features; S50: based on the ROI coordinates of the fluorescence image, S60: Determine the ROI of the white light grayscale image. Based on the ROI coordinates of the fluorescence image and the ROI of the white light grayscale image, determine the coordinate compensation between the ROI of the white light grayscale image and the ROI of the fluorescence image; S70: Based on the score array determined in step S40, determine the different color value arrays corresponding to different scores. Based on the fixed-size suspicious region coordinate array of the fluorescence image determined in step S40 and the coordinate compensation determined in step S50, determine the suspicious region coordinate array of the ROI of the white light grayscale image; S80: Based on the suspicious region coordinate array of the ROI of the white light grayscale image and the different color value arrays determined in step S60, generate a discrete point set color block; S90: After linearly smoothing the discrete point set color block, obtain a heatmap of the discrete point set color block. Based on the heatmap and the ROI of the white light grayscale image, determine the fusion map of the white light image and the fluorescence image.
[0006] The method for determining the fusion image of a white light image and a fluorescence image of an object to be imaged, provided in the embodiments of this application, ensures the spatiotemporal consistency of the white light and fluorescence images by synchronously acquiring white light and fluorescence images of the same predetermined location of the object to be imaged, reducing spatial mismatch between the acquired white light and fluorescence images caused by differences in acquisition time or offset of acquisition site, and improving image fusion accuracy. It enhances the vascular network of the white light image to enhance the detail information of blood vessels and reduce background noise, and identifies the ROI of the fluorescence image to eliminate artifact interference caused by the operating instruments, facilitating the extraction of image features. Furthermore, it determines the coordinate array and fraction array of a fixed-size suspicious region in the fluorescence image by using the texture features of the redox ratio map of the ROI of the fluorescence image, and utilizes the R... OI coordinates determine the coordinate compensation of the ROI of the white light grayscale image and the ROI of the fluorescence image, providing an accurate data basis for mapping the metabolic abnormality areas reflected in the fluorescence image to the white light image. This helps to further eliminate the spatial misalignment between the white light image and the fluorescence image caused by differences in acquisition time or acquisition site offset, thereby determining a more accurate coordinate array of the suspicious ROI of the white light grayscale image. Based on this, a discrete point set of color blocks is generated according to the coordinate array of the suspicious ROI of the white light grayscale image. After linear gradient smoothing, a heatmap is obtained, which is fused with the ROI of the white light grayscale image to obtain an accurate fused image of the white light grayscale image and the fluorescence image. This allows the location, range, and degree of abnormality reflected in the fluorescence image to be mapped onto the white light image in an intuitive and quantitative form.
[0007] Secondly, embodiments of this application provide a system for determining a fused image of a white light image and a fluorescence image of an object to be imaged. The system includes: an image acquisition module, an image processing module, an image feature extraction and analysis module, a coordinate mapping module, a color block generation module, and a fusion module. The image acquisition module is configured to simultaneously acquire a white light image and a fluorescence image of the same predetermined location of the object to be imaged. The image processing module is configured to perform vascular network enhancement and grayscale processing on the white light image at the predetermined location to obtain a white light grayscale image, and to identify the fluorescence image at the predetermined location to obtain the Region of Interest (ROI) of the fluorescence image. The image feature extraction and analysis module is configured to determine the blue channel B and the green channel G in the ROI of the fluorescence image, determine the redox ratio map of the ROI of the fluorescence image based on the blue channel B and the green channel G, extract the texture features of the redox ratio map, and determine a fixed-size coordinate array and a score array of suspicious regions in the fluorescence image based on the texture features. The coordinate mapping module is configured to determine the ROI of the white light grayscale image based on the ROI coordinates of the fluorescence image, and to determine the coordinate compensation between the ROIs of the white light grayscale image and the fluorescence image based on the ROI coordinates of both. The color block generation module is configured to determine different color value arrays corresponding to different scores based on the score array, and to determine the coordinate array of the suspected ROI of the white light grayscale image based on the coordinate array of the fixed-size suspected region of the fluorescence image and the coordinate compensation; and to generate a discrete point set color block based on the coordinate array of the suspected ROI of the white light grayscale image and the different color value arrays. The fusion module is configured to obtain a heatmap of the discrete point set color blocks after linear gradient smoothing, and to determine the fused image of the white light image and the fluorescence image based on the heatmap and the ROI of the white light grayscale image. Attached Figure Description
[0008] Other objects and advantages of this application will become apparent from the following description of embodiments of this application with reference to the accompanying drawings, and will help to provide a comprehensive understanding of this application.
[0009] Figure 1 This is a flowchart of a method for determining a fusion map of a white light image and a fluorescence image of an object to be imaged according to an embodiment of this application; Figure 2 It is a comparison diagram between the fluorescence image and the white light image of the object to be imaged according to an embodiment of this application, and the fused image of the determined white light image and fluorescence image; Figure 3 It is a comparison diagram between a fluorescence image and a white light image of another object to be imaged according to an embodiment of this application, and a fused diagram of the determined white light image and fluorescence image.
[0010] It should be noted that the accompanying drawings are not necessarily drawn to scale, but are shown only in a schematic manner without affecting the reader's understanding. Detailed Implementation
[0011] Exemplary embodiments of this application will be described below with reference to the accompanying drawings. For clarity and brevity, not all features of actual implementations are described in the specification. However, it should be understood that many implementation-specific decisions must be made in the development of any such actual embodiment to achieve the developer's specific goals, such as complying with constraints related to the system and business, and these constraints may vary depending on the implementation. Furthermore, it should be understood that while development work can be very complex and time-consuming, such development work is merely a routine task for those skilled in the art who benefit from the content of this application.
[0012] It should also be noted that, in order to avoid obscuring this application with unnecessary details, only the equipment structure and / or processing steps closely related to the solution according to this application are shown in the accompanying drawings, while other details that are not closely related to this application are omitted.
[0013] The inventors of this application have discovered that in the prior art, in order to map the metabolic abnormality area reflected in the fluorescence image onto the white light image, a simple image overlay or side-by-side display method is usually adopted. However, due to differences in acquisition time or changes in acquisition location, there is a problem of spatial mismatch between the white light image and the fluorescence image, which causes the abnormal area in the fluorescence image to be shifted in position on the white light image.
[0014] Based on this, embodiments of this application provide a method for determining a fusion image of a white light image and a fluorescence image of an object to be imaged, such as... Figure 1 As shown, Figure 1 A flowchart illustrating a method for determining a fused image of a white light image and a fluorescence image of an object to be imaged, according to an embodiment of this application, is provided. The method includes the following steps: S10: Simultaneously acquire white light and fluorescence images of the same predetermined location of the object to be imaged.
[0015] S20: Process the white light image by enhancing the vascular network of the white light image at a predetermined location and performing grayscale processing to obtain a white light grayscale image.
[0016] S30: Process fluorescence images, identify fluorescence images at predetermined locations, and obtain the ROI of the fluorescence images.
[0017] S40: Determine the blue channel B and green channel G in the ROI of the fluorescence image. Based on the blue channel B and green channel G, determine the redox ratio map of the ROI of the fluorescence image. Extract the texture features of the redox ratio map. Based on the texture features, determine the coordinate array and score array of the suspicious region of a fixed size in the fluorescence image.
[0018] S50: Determine the ROI of the white light grayscale image based on the coordinates of the ROI of the fluorescence image, and determine the coordinate compensation between the ROI of the white light grayscale image and the ROI of the fluorescence image based on the ROI coordinates of the fluorescence image and the ROI of the white light grayscale image.
[0019] S60: Based on the score array determined in step S40, determine the different color value arrays corresponding to different scores. Based on the fixed-size suspicious region coordinate array of the fluorescence image determined in step S40 and the coordinate compensation determined in step S50, determine the ROI suspicious region coordinate array of the white light grayscale image.
[0020] S70: Based on the coordinate array of the suspicious ROI region and the array of different color values of the white light grayscale image determined in step S60, generate a set of discrete point color blocks.
[0021] S80: After performing linear gradient smoothing on the discrete point set color blocks, a heatmap of the discrete point set color blocks is obtained. Based on the ROI of the heatmap and the white light grayscale image, the fusion map of the white light image and the fluorescence image is determined.
[0022] The method for determining the fusion image of a white light image and a fluorescence image of an object to be imaged, provided in the embodiments of this application, ensures the spatiotemporal consistency of the white light and fluorescence images by synchronously acquiring white light and fluorescence images of the same predetermined location of the object to be imaged, reducing spatial mismatch between the acquired white light and fluorescence images caused by differences in acquisition time or offset of acquisition site, and improving image fusion accuracy. It enhances the vascular network of the white light image to enhance the detail information of blood vessels and reduce background noise, and identifies the ROI of the fluorescence image to eliminate artifact interference caused by the operating instruments, facilitating the extraction of image features. Furthermore, it determines the coordinate array and fraction array of a fixed-size suspicious region in the fluorescence image by using the texture features of the redox ratio map of the ROI of the fluorescence image, and utilizes the R... OI coordinates determine the coordinate compensation of the ROI of the white light grayscale image and the ROI of the fluorescence image, providing an accurate data basis for mapping the metabolic abnormality areas reflected in the fluorescence image to the white light image. This helps to further eliminate the spatial misalignment between the white light image and the fluorescence image caused by differences in acquisition time or acquisition site offset, thereby determining a more accurate coordinate array of the suspicious ROI of the white light grayscale image. Based on this, a discrete point set of color blocks is generated according to the coordinate array of the suspicious ROI of the white light grayscale image. After linear gradient smoothing, a heatmap is obtained, which is fused with the ROI of the white light grayscale image to obtain an accurate fused image of the white light grayscale image and the fluorescence image. This allows the location, range, and degree of abnormality reflected in the fluorescence image to be mapped onto the white light image in an intuitive and quantitative form.
[0023] ROI stands for Region of Interest, which is a specific area in an image that contains key information and needs to be analyzed or measured.
[0024] In some embodiments, the object to be imaged may be the cervix or other similar site. In step S10, the predetermined location of the object to be imaged is the surface of the cervix.
[0025] like Figure 2 and Figure 3 As shown, Figure 2 This diagram illustrates a comparison between a fluorescence image and a white light image of the object to be imaged, and a fused image of the determined white light image and fluorescence image, representing an embodiment of this application. Figure 3 This diagram illustrates a comparison between a fluorescence image and a white light image of another object to be imaged, as well as a fused image of the determined white light and fluorescence images, according to an embodiment of this application. Figure 2 and Figure 3 In the image, from left to right, are the fluorescence image, the white light image, and the fusion image of the white light and fluorescence images. The region represented by 10 is the ROI of the fluorescence image, and the region represented by 20 is the ROI of the white light grayscale image. The color blocks in region 20 that are significantly different from the background color represent the color blocks of the discrete point set color block heatmap.
[0026] In some embodiments, in step S10, an endoscope or microscope system with white light imaging and specific wavelength fluorescence excitation / acquisition functions can be used to simultaneously acquire white light and fluorescence images of the same predetermined position of the object to be imaged, so as to ensure the spatiotemporal consistency of the white light and fluorescence images as much as possible, reduce the spatial misalignment between the two, and thus facilitate obtaining a more accurate fusion image.
[0027] In some embodiments, in step S20, the Frangi filtering method can be used to enhance the vascular network of the white light image. The Frangi filtering method is based on the eigenvalue response of the Hessian matrix, which can selectively enhance the tubular vascular structure in the white light image and suppress non-vascular background noise, making the vascular display clearer and more continuous. This facilitates the acquisition of a white light grayscale image with prominent vascular network, effectively preserving key anatomical structural information, and making the image features more stable, thus providing a reliable reference for subsequent image fusion.
[0028] In some embodiments, step S30 may further include the following steps: identifying the upper and lower bright connected regions of the fluorescence image, with the middle part of the upper and lower bright connected regions being taken as the ROI region of the fluorescence image; if there are no upper and lower bright connected regions, then the entire image is taken as the ROI region of the fluorescence image.
[0029] Because the instruments used during image acquisition (such as expanders, pliers, etc.) often obstruct part of the field of view and produce bright reflective artifacts, this embodiment identifies the upper and lower bright connected regions of the acquired fluorescence image as artifact regions produced by the instrument, and determines the middle part of the upper and lower bright connected regions as the Region of Interest (ROI) of the fluorescence image. If no upper and lower bright connected regions are identified in the fluorescence image, it means that the instrument has not produced obvious artifacts in the fluorescence image, and the entire image is used as the ROI of the fluorescence image. This method can clearly eliminate the interference of artifacts caused by the instrument, which is beneficial for subsequent extraction of accurate image features.
[0030] In some embodiments, step S30 may further include the following steps: S31: Perform noise reduction processing on the fluorescence image.
[0031] S32: Using an adaptive threshold segmentation method, extract regions in the denoised fluorescence image whose brightness is significantly higher than the background, and determine the fluorescence binarized image of the extracted regions.
[0032] S33: Perform connected component analysis on the fluorescence binarized image to identify all highlighted connected components.
[0033] S34: Based on the instrument position characteristics, determine the regions in all highlighted connected regions that are instrument artifacts.
[0034] S35: The area between the highlighted areas identified as instrument artifacts is the ROI region of the fluorescence image; if no obvious highlighted area is detected, the entire image is taken as the ROI region of the fluorescence image.
[0035] In this embodiment, threshold segmentation is performed on the fluorescence image after noise reduction to extract regions with significantly higher brightness than the background. The fluorescence binarized image of these regions is then determined, and connected component analysis is performed to identify all bright connected components, thereby extracting all artifact regions that may be caused by the operating instruments. Based on the instrument's position characteristics, regions within the bright connected components that are determined to be caused by the operating instruments are identified, further improving the accuracy of eliminating artifact interference caused by the operating instruments. This avoids artifacts misleading subsequent image feature extraction, reduces the possibility of misjudgment, and prevents the omission of bright regions not caused by instruments, thus avoiding the loss of key information. This effectively improves the accuracy and reliability of subsequent redox ratio calculation, texture feature extraction, and suspicious region identification.
[0036] In some embodiments, in step S34, the instrument location feature is located at the upper and lower edges of the fluorescence image, and based on this location feature, the regions located at the upper and lower edges in the bright connected region are determined as regions of instrument artifacts.
[0037] In some embodiments, in step S40, determining the redox ratio map of the ROI of the fluorescence image may further include the steps of: separating the blue channel B and the green channel G from the ROI of the fluorescence image; determining the B / (B+G) value of each pixel in the ROI of the fluorescence image based on the blue channel B and the green channel G, wherein this ratio is related to the concentration ratio of NADH and FAD in the cell and can reflect the redox metabolic state of the tissue; and generating the redox ratio map of the ROI of the fluorescence image based on the B / (B+G) value of each pixel to facilitate the extraction of its texture features.
[0038] In some embodiments, in step S40, the extracted texture features may include at least one of the following features from the gray-level co-occurrence matrix (GLCM) features of the fluorescence image: contrast, correlation, energy, and homogeneity. These texture features can characterize the microscopic spatial distribution pattern of tissue metabolic state, wherein contrast reflects sharpness, correlation reflects linearity, energy reflects uniformity, and homogeneity reflects local uniformity.
[0039] In this embodiment, these texture features of the redox ratio map are extracted to transform the fluorescence image ROI image data into quantifiable numerical indicators, so as to provide a data basis for subsequently determining the coordinate array and score array of all suspicious regions of the fluorescence image, for example, to provide effective input for the model.
[0040] In some embodiments, step S40 may further include the following steps: sequentially extracting a series of fixed-size suspicious region feature arrays from the redox ratio map of the ROI of the fluorescence image; using the SVM model and PlattScaling method, and the suspicious region feature arrays, determining the coordinate array and score array of all suspicious regions of the fluorescence image based on the texture features of the extracted redox ratio map.
[0041] To accurately identify suspicious regions in fluorescence images, this embodiment utilizes an SVM model and the Platt Scaling method, along with a series of fixed-size suspicious region feature arrays extracted from the redox ratio map of the ROI of the fluorescence image. This establishes a relationship between features, coordinate arrays, and score arrays. Consequently, based on the texture features of the redox ratio map of the ROI of the extracted fluorescence image of the object to be imaged, suspicious regions in the fluorescence image can be accurately identified and located, and confidence scores can be determined, effectively improving the standardization and efficiency of suspicious region identification.
[0042] Specifically, in some embodiments, multiple fluorescence images confirmed by pathological results can be acquired, and the redox ratio maps of the ROIs of the multiple fluorescence images can be determined. From the redox ratio maps of the ROIs of the multiple fluorescence images, a series of fixed-size image patches are extracted sequentially. The gray-level co-occurrence matrix texture features of the image patches are determined, including at least one feature among contrast, correlation, energy, and homogeneity, forming a series of fixed-size suspicious region feature arrays. Using the suspicious region feature arrays, an SVM model is trained, and then the Platt Scaling method is used to generate the model. Based on the generated model and the texture features of the redox ratio maps of the ROIs of the extracted fluorescence images of the object to be imaged, the coordinate arrays and score arrays of all suspicious regions in the fluorescence images are determined. In this embodiment, by embedding the doctor's diagnostic experience into the model, it is beneficial to more accurately identify and locate suspicious regions in fluorescence images, further improving the standardization and efficiency of suspicious region identification.
[0043] The coordinate array of the suspicious region in the fluorescence image represents the location of the suspicious region, which can be represented by the coordinates of the center point of the rectangle; the score array of the suspicious region in the fluorescence image represents the degree of suspicion, which can be represented by a value between 0 and 1.
[0044] Furthermore, when generating the model, the feature array of the suspicious region can be used as input and its corresponding pathological results as labels to train the SVM model. Then, the Platt Scaling method can be used to convert it into a probability output to generate a model suitable for determining the coordinate array and score array of all suspicious regions in the fluorescence image.
[0045] In some embodiments, determining the ROI of the white light grayscale image in step S50 may further include the following steps: generating a binary image of the white light grayscale image using the OTSU threshold segmentation algorithm based on the white light grayscale image; performing noise reduction processing on the fluorescence image, using an adaptive threshold segmentation method to extract regions in the denoised fluorescence image whose brightness is significantly higher than the background, and determining the fluorescence binarized image of the extracted regions; based on the fluorescence binarized image and the ROI coordinates of the fluorescence image, sequentially traversing the images to be confirmed on the binary image of the white light grayscale image that are of the same size as the ROI of the fluorescence binarized image, performing XOR processing, accumulating the XOR values, determining the image to be confirmed with the smallest accumulated XOR value as the ROI of the binary image of the white light grayscale image, and obtaining the ROI of the white light grayscale image through ROI coordinate mapping.
[0046] Because the OTSU threshold segmentation algorithm can adapt to different lighting conditions and work stably under various brightness levels, and because the two images are captured in a short time interval with only slight positional shifts, this embodiment uses the OTSU threshold segmentation algorithm to obtain high-quality binary images of the white light grayscale image to be confirmed and fluorescence binarized images. Based on this, the images to be confirmed, with ROIs of the same size as the fluorescence binarized image, are sequentially traversed on the binary image of the white light grayscale image and XORed, and the XOR values are accumulated to quickly obtain the results. Then, the ROI of the white light grayscale image is determined based on the accumulated XOR value, which facilitates the subsequent accurate determination of the coordinate compensation between the ROI of the white light grayscale image and the ROI of the fluorescence image, achieving fast and accurate registration and effectively eliminating the spatial misalignment problem between the white light image and the fluorescence image caused by differences in acquisition time or offset of the acquisition location.
[0047] Specifically, in some embodiments, in step S50, based on the ROI coordinates of the fluorescence image, using the size of the binary image of the ROI of the fluorescence image as a template, the binary image of the entire white light grayscale image can be traversed pixel by pixel to sequentially traverse the binary image of the white light grayscale image, thereby determining an image to be confirmed that is the same size as the ROI of the fluorescence image.
[0048] In some embodiments, in step S50, the ROI coordinates of the fluorescence image and the ROI coordinates of the white light grayscale image are determined, and the difference between the two coordinates is the coordinate compensation between the ROI of the white light grayscale image and the ROI of the fluorescence image.
[0049] In some embodiments, when determining the array of different color values corresponding to different scores in step S60, different score ranges can be set to correspond to different color values based on the score array determined in step S40. For example, the score range of 0-0.3 is set to correspond to dark blue, which represents normal or low suspicion; the score range of 0.3-0.5 is set to correspond to cyan; the score range of 0.5-0.6 is set to correspond to yellow; the score range of 0.6-0.8 is set to correspond to orange; and the score range of 0.8-1 is set to correspond to red, which represents high suspicion.
[0050] Furthermore, in step S60, the fixed-size suspicious region coordinate array of the fluorescence image determined in step S40 is added to the coordinate compensation determined in step S50 to obtain the ROI suspicious region coordinate array of the white light grayscale image, so as to map the suspicious region of the fluorescence image onto the white light grayscale image.
[0051] In some embodiments, in step S70, each set of discrete points generated can represent a suspicious location and its degree of suspicion.
[0052] In some embodiments, step S80 may further include the steps of: performing linear gradient smoothing on the discrete point set color blocks to obtain a heatmap of the discrete point set color blocks; and overlaying the heatmap with a predetermined transparency onto the ROI of the white light grayscale image to obtain an overlay image of the white light grayscale image and the heatmap, i.e., obtaining a fused image of the white light image and the fluorescence image. This fused image, with a heatmap reflecting the location and degree of abnormal tissue metabolism overlaid on the clear vascular anatomy background of the white light grayscale image, can intuitively and quantitatively reflect abnormal information.
[0053] For example, in step S80, Gaussian convolution kernels or distance transformation combined with color interpolation can be used to perform linear gradient smoothing on the color blocks of the discrete point set, which helps to make the color transition inside and at the edges of the color blocks more natural and smooth.
[0054] Embodiments of this application also provide a system for determining a fused image of a white light image and a fluorescence image of an object to be imaged. The system includes: an image acquisition module, an image processing module, an image feature extraction and analysis module, a coordinate mapping module, a color block generation module, and a fusion module. The image acquisition module is configured to simultaneously acquire white light and fluorescence images of the same predetermined location of the object to be imaged. The image processing module is configured to perform vascular network enhancement and grayscale processing on the white light image at the predetermined location to obtain a white light grayscale image, and to identify the fluorescence image at the predetermined location to obtain the ROI of the fluorescence image. The image feature extraction and analysis module is configured to determine the blue channel B and green channel G in the ROI of the fluorescence image, determine the redox ratio map of the ROI of the fluorescence image based on the blue channel B and green channel G, extract the texture features of the redox ratio map, and determine a fixed-size coordinate array and score array of suspicious regions in the fluorescence image based on the texture features. The coordinate mapping module is configured to determine the ROI of the white light grayscale image based on the ROI coordinates of the fluorescence image, and to determine the coordinate compensation between the ROIs of the white light grayscale image and the fluorescence image based on the ROI coordinates of both. The color block generation module is configured to determine different color value arrays corresponding to different scores based on the score array, and to determine the coordinate array of the suspected ROI of the white light grayscale image based on the coordinate array of the fixed-size suspected region of the fluorescence image and the coordinate compensation; and to generate a discrete point set color block based on the coordinate array of the suspected ROI of the white light grayscale image and the different color value arrays. The fusion module is configured to obtain a heatmap of the discrete point set color blocks after linear gradient smoothing, and to determine the fused image of the white light image and the fluorescence image based on the heatmap and the ROI of the white light grayscale image.
[0055] The system for determining the fusion image of a white light image and a fluorescence image of an object to be imaged, provided in the embodiments of this application, synchronously acquires white light and fluorescence images of the same predetermined location of the object to be imaged by setting an image acquisition module, ensuring the spatiotemporal consistency of the white light and fluorescence images, reducing spatial mismatch between the acquired white light and fluorescence images caused by differences in acquisition time or offset of acquisition site, and improving image fusion accuracy; by setting an image processing module to enhance the vascular network of the white light image to enhance the detail information of the blood vessels and reduce background noise, and to identify the ROI of the fluorescence image to eliminate artifact interference caused by the operating instruments, facilitating the extraction of image features; and by setting an image feature extraction and analysis module to determine the coordinate array and fraction array of a fixed-size suspicious region of the fluorescence image based on the texture features of the redox ratio map of the ROI of the fluorescence image, and setting a coordinate mapping module, utilizing... The ROI coordinates of the fluorescence image are used to determine the coordinate compensation of the ROIs of the white light grayscale image and the fluorescence image. This provides an accurate data basis for mapping the metabolically abnormal areas reflected in the fluorescence image to the white light image, which helps to further eliminate the spatial misalignment between the white light image and the fluorescence image caused by differences in acquisition time or acquisition site offset. Based on this, a more accurate coordinate array of the suspected ROI of the white light grayscale image is determined. On this basis, a color block generation module generates a set of discrete point color blocks based on the coordinate array of the suspected ROI of the white light grayscale image. Then, a fusion module performs linear gradient smoothing on the discrete point set of color blocks to obtain a heat map, which is fused with the ROI of the white light grayscale image to obtain an accurate fused image of the white light grayscale image and the fluorescence image. This allows the location, range, and degree of abnormality reflected in the fluorescence image to be mapped onto the white light image in an intuitive and quantitative form.
[0056] In some embodiments, the image processing module may be configured to use the Frangi filtering method to enhance the vascular network in a white light image. The Frangi filtering method, based on the eigenvalue response of the Hessian matrix, can selectively enhance tubular vascular structures in a white light image and suppress non-vascular background noise, resulting in clearer and more continuous vascular visualization. This facilitates the acquisition of a white light grayscale image with prominent vascular networks, effectively preserving key anatomical structural information and making image features more stable, thus providing a reliable reference for subsequent image fusion.
[0057] In some embodiments, the image processing module can also be configured to identify the upper and lower bright connected regions of the fluorescence image, with the middle part of the upper and lower bright connected regions serving as the ROI region of the fluorescence image, and if there are no upper and lower bright connected regions, the entire image serving as the ROI region of the fluorescence image.
[0058] In this embodiment, the image processing module is configured to identify the upper and lower bright connected regions of the acquired fluorescence image, thus determining them as artifact regions produced by the operating instrument. The middle portion of the upper and lower bright connected regions is identified as the Region of Interest (ROI) of the fluorescence image. If no upper and lower bright connected regions are identified in the fluorescence image, it indicates that the operating instrument has not produced significant artifacts in the fluorescence image, and the entire image is then used as the ROI of the fluorescence image. This configuration of the image processing module effectively eliminates artifact interference caused by the operating instrument, facilitating the subsequent extraction of accurate image features.
[0059] In some embodiments, the image processing module may also be configured to: perform noise reduction processing on the fluorescence image; use an adaptive threshold segmentation method to extract regions in the denoised fluorescence image whose brightness is significantly higher than the background, and determine the fluorescence binarized image of the extracted regions; perform connected component analysis on the fluorescence binarized image to identify all bright connected components; determine the regions that are instrument artifacts in all bright connected components based on the instrument location characteristics; define the regions between the bright regions that are instrument artifacts as the ROI region of the fluorescence image; if no obvious bright regions are detected, then the entire image is taken as the ROI region of the fluorescence image.
[0060] In this embodiment, the image processing module is configured to perform threshold segmentation on the denoised fluorescence image, extracting regions with significantly higher brightness than the background. The fluorescence binarized image of these regions is then determined, and connected component analysis is performed to identify all bright connected components. This extracts all artifact regions that may be caused by the operating equipment. Furthermore, based on the equipment's position characteristics, regions within the bright connected components that are determined to be caused by the operating equipment are identified. This further improves the accuracy of eliminating artifact interference from the operating equipment, preventing artifacts from misleading subsequent image feature extraction, reducing the possibility of misjudgment, and avoiding the omission of bright regions not caused by the equipment, thus improving the accuracy and reliability of subsequent redox ratio calculation, texture feature extraction, and suspicious region identification.
[0061] In some embodiments, the image feature extraction and analysis module can be configured to extract texture features including at least one of contrast, correlation, energy, and homogeneity from the gray-level co-occurrence matrix (GLCM) features of the fluorescence image. In this embodiment, by configuring the image feature extraction and analysis module to extract these texture features from the redox ratio map, the fluorescence image ROI image data is transformed into quantifiable numerical indicators. This provides a data foundation for subsequently determining the coordinate and score arrays of all suspicious regions in the fluorescence image, for example, providing effective input for the model.
[0062] In some embodiments, the image feature extraction and analysis module can also be configured to sequentially extract a series of fixed-size suspicious region feature arrays from the redox ratio map of the ROI of the fluorescence image; and use the SVM model and PlattScaling method, along with the suspicious region feature arrays, to determine the coordinate arrays and score arrays of all suspicious regions in the fluorescence image based on the texture features of the extracted redox ratio map.
[0063] To accurately identify suspicious regions in fluorescence images, this embodiment sets the image feature extraction and analysis module to utilize the SVM model and Platt Scaling method, along with a series of fixed-size suspicious region feature arrays extracted from the redox ratio map of the ROI of the fluorescence image. This establishes a relationship between features, coordinate arrays, and score arrays. Consequently, based on the texture features of the redox ratio map of the ROI of the fluorescence image of the object to be imaged, suspicious regions in the fluorescence image can be accurately identified and located, and confidence scores can be determined, effectively improving the standardization and efficiency of suspicious region identification.
[0064] In some embodiments, the coordinate mapping module can be configured to: generate a binary image of the white light grayscale image using the OTSU threshold segmentation algorithm based on the white light grayscale image; perform noise reduction processing on the fluorescence image, use an adaptive threshold segmentation method to extract regions in the denoised fluorescence image whose brightness is significantly higher than the background, and determine the fluorescence binarized image of the extracted region; based on the fluorescence binarized image and the ROI coordinates of the fluorescence image, sequentially traverse the images to be confirmed on the binary image of the white light grayscale image that are of the same size as the ROI of the fluorescence binarized image, perform XOR processing, accumulate the XOR values, determine the image to be confirmed with the smallest accumulated XOR value as the ROI of the binary image of the white light grayscale image, and obtain the ROI of the white light grayscale image through ROI coordinate mapping.
[0065] Because the OTSU threshold segmentation algorithm can adapt to different lighting conditions and work stably under various brightness levels, and because the image capture interval between the two images is short with only slight positional shifts, this embodiment sets the coordinate mapping module to use the OTSU threshold segmentation algorithm to generate a binary image, thus obtaining a high-quality binary image of the white light grayscale image to be confirmed and a fluorescence binarized image. Based on this, the images to be confirmed with the same ROI size as the fluorescence binarized image are sequentially traversed on the binary image of the white light grayscale image, and XORed with each other, accumulating the XOR values to quickly obtain the results. Then, the ROI of the white light grayscale image is determined based on the accumulated XOR value, which facilitates accurate coordinate compensation between the ROI of the white light grayscale image and the ROI of the fluorescence image, achieving fast and accurate registration and effectively eliminating the spatial misalignment problem between the white light image and the fluorescence image caused by differences in acquisition time or offset of the acquisition location.
[0066] Regarding the embodiments of this application, it should also be noted that, without conflict, the embodiments of this application and the features in the embodiments can be combined with each other to obtain new embodiments.
[0067] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. The scope of protection of this application shall be determined by the scope of the claims.
Claims
1. A method for determining a fused image of a white light image and a fluorescence image of an object to be imaged, characterized in that, It includes the following steps: S10: Simultaneously acquire white light and fluorescence images of the same predetermined location of the object to be imaged; S20: Process the white light image by enhancing the vascular network of the white light image at the predetermined position and performing grayscale processing to obtain a white light grayscale image; S30: Process the fluorescence image, identify the fluorescence image at the predetermined location, and obtain the ROI of the fluorescence image; S40: Determine the blue channel B and green channel G in the ROI of the fluorescence image; determine the redox ratio map of the ROI of the fluorescence image based on the blue channel B and green channel G; extract the texture features of the redox ratio map; and determine the coordinate array and score array of the fixed-size suspicious region of the fluorescence image based on the texture features. S50: Determine the ROI of the white light grayscale image based on the coordinates of the ROI of the fluorescence image, and determine the coordinate compensation between the ROI of the white light grayscale image and the ROI of the fluorescence image based on the ROI coordinates of the fluorescence image and the ROI of the white light grayscale image. S60: Based on the score array determined in step S40, determine the different color value arrays corresponding to different scores; based on the fixed-size suspicious region coordinate array of the fluorescence image determined in step S40 and the coordinate compensation determined in step S50, determine the ROI suspicious region coordinate array of the white light grayscale image. S70: Based on the ROI suspicious region coordinate array and the different color value array of the white light grayscale image determined in step S60, generate a discrete point set color block; S80: After performing linear gradient smoothing on the discrete point set color blocks, a heatmap of the discrete point set color blocks is obtained. Based on the ROI of the heatmap and the white light grayscale image, a fusion map of the white light image and the fluorescence image is determined.
2. The method according to claim 1, characterized in that, In step S20, the Frangi filtering method is used to perform vascular network enhancement processing on the white light image.
3. The method according to claim 1, characterized in that, Step S30 also includes the following steps: Identify the upper and lower highlighted connected regions of the fluorescence image. The middle part of the upper and lower highlighted connected regions is taken as the ROI region of the fluorescence image. If there are no upper and lower highlighted connected regions, the entire image is taken as the ROI region of the fluorescence image.
4. The method according to claim 3, characterized in that, Step S30 also includes the following steps: S31: Perform noise reduction processing on the fluorescence image; S32: Using an adaptive threshold segmentation method, extract the region in the denoised fluorescence image whose brightness is significantly higher than the background, and determine the fluorescence binarized image of the extracted region; S33: Perform connected component analysis on the fluorescence binarized image to identify all highlighted connected components; S34: Based on the instrument position characteristics, determine the regions in all the highlighted connected regions that are instrument artifacts; S35: Determine the region between the highlighted areas of the instrument artifact as the ROI region of the fluorescence image; if no obvious highlighted area is detected, then take the entire image as the ROI region of the fluorescence image.
5. The method according to claim 1, characterized in that, In step S40, the extracted texture features include at least one of the following features in the gray-level co-occurrence matrix (GLCM) features of the fluorescence image: contrast, correlation, energy, and homogeneity.
6. The method according to claim 1, characterized in that, Step S40 also includes the following steps: From the redox ratio map of the ROI in the fluorescence image, a series of fixed-size feature arrays of suspicious regions are extracted sequentially. Using the SVM model and Platt Scaling method, along with the feature array of the suspected regions, the coordinate array and score array of all suspected regions in the fluorescence image are determined based on the texture features of the extracted redox ratio map.
7. The method according to claim 1, characterized in that, In step S50, determining the ROI of the white light grayscale image further includes the following steps: Based on the white light grayscale image, a binary image of the white light grayscale image is generated using the OTSU threshold segmentation algorithm; The fluorescence image is denoised by using an adaptive threshold segmentation method to extract regions in the denoised fluorescence image whose brightness is significantly higher than the background, and the fluorescence binarized image of the extracted regions is determined. Based on the fluorescence binarized image and the ROI coordinates of the fluorescence image, the images to be confirmed that are of the same size as the ROI of the fluorescence binarized image on the binary image of the white light grayscale image are XORed sequentially, and the XOR values are accumulated. The image to be confirmed with the smallest accumulated XOR value is determined as the ROI of the binary image of the white light grayscale image. The ROI of the white light grayscale image is obtained by mapping the ROI coordinates.
8. A system for determining a fused image of a white light image and a fluorescence image of an object to be imaged, characterized in that, It includes: An image acquisition module is configured to simultaneously acquire a white light image and a fluorescence image of the same predetermined location of the object to be imaged; The image processing module is configured to enhance the vascular network of the white light image at the predetermined location and perform grayscale processing to obtain a white light grayscale image, and to identify the fluorescence image at the predetermined location to obtain the ROI of the fluorescence image; The image feature extraction and analysis module is configured to determine the blue channel B and green channel G in the ROI of the fluorescence image, determine the redox ratio map of the ROI of the fluorescence image based on the blue channel B and green channel G, extract the texture features of the redox ratio map, and determine a fixed-size suspicious region coordinate array and score array of the fluorescence image based on the texture features. The coordinate mapping module is configured to determine the ROI of the white light grayscale image based on the ROI coordinates of the fluorescence image, and to determine the coordinate compensation between the ROI of the white light grayscale image and the ROI of the fluorescence image based on the ROI coordinates of the fluorescence image and the ROI of the white light grayscale image. The color block generation module is configured to determine different color value arrays corresponding to different scores based on the score array, determine the ROI suspicious region coordinate array of the white light grayscale image based on the fixed-size suspicious region coordinate array of the fluorescence image and the coordinate compensation, and generate discrete point set color blocks based on the ROI suspicious region coordinate array of the white light grayscale image and the different color value arrays. The fusion module is configured to perform linear gradient smoothing on the discrete point set color blocks to obtain a heatmap of the discrete point set color blocks, and determine a fusion map of the white light image and the fluorescence image based on the ROI of the heatmap and the white light grayscale image.
9. The system according to claim 8, characterized in that, The image processing module is configured to, The Frangi filtering method is used to enhance the vascular network of the white light image.
10. The system according to claim 8, characterized in that, The image processing module is also configured to, Identify the upper and lower highlighted connected regions of the fluorescence image. The middle part of the upper and lower highlighted connected regions is taken as the ROI region of the fluorescence image. If there are no upper and lower highlighted connected regions, the entire image is taken as the ROI region of the fluorescence image.
11. The system according to claim 10, characterized in that, The image processing module is also configured to, The fluorescence image is then subjected to noise reduction processing; An adaptive threshold segmentation method is used to extract regions in the denoised fluorescence image whose brightness is significantly higher than the background, and to determine the fluorescence binarized image of the extracted regions. Connectivity analysis was performed on the fluorescence binarized image to identify all highlighted connected components; Based on the instrument location characteristics, the regions that are instrument artifacts in all the highlighted connected regions are identified; The region between the highlighted areas of the instrument artifact is determined as the ROI region of the fluorescence image; if no obvious highlighted area is detected, the entire image is taken as the ROI region of the fluorescence image.
12. The system according to claim 8, characterized in that, The image feature extraction and analysis module is configured to, The extracted texture features include at least one of the following features from the gray-level co-occurrence matrix (GLCM) features of the fluorescence image: contrast, correlation, energy, and homogeneity.
13. The system according to claim 8, characterized in that, The image feature extraction and analysis module is also configured to, From the redox ratio map of the ROI in the fluorescence image, a series of fixed-size feature arrays of suspicious regions are extracted sequentially. Using the SVM model and Platt Scaling method, along with the feature array of the suspected regions, the coordinate array and score array of all suspected regions in the fluorescence image are determined based on the texture features of the extracted redox ratio map.
14. The system according to claim 8, characterized in that, The coordinate mapping module is configured such that, Based on the white light grayscale image, a binary image of the white light grayscale image is generated using the OTSU threshold segmentation algorithm; The fluorescence image is denoised by using an adaptive threshold segmentation method to extract regions in the denoised fluorescence image whose brightness is significantly higher than the background, and the fluorescence binarized image of the extracted regions is determined. Based on the fluorescence binarized image and the ROI coordinates of the fluorescence image, the images to be confirmed that are of the same size as the ROI of the fluorescence binarized image on the binary image of the white light grayscale image are XORed sequentially, and the XOR values are accumulated. The image to be confirmed with the smallest accumulated XOR value is determined as the ROI of the binary image of the white light grayscale image. The ROI of the white light grayscale image is obtained by mapping the ROI coordinates.