Tumor detection method and system and computer equipment

Abstract

The invention relates to the technical field of tumor detection, and discloses a tumor detection method and system and computer equipment, and the method comprises the following steps: obtaining autofluorescence decay data of a target biological tissue, and extracting fluorescence intensity and fluorescence lifetime data from the autofluorescence decay data; performing data fusion on the fluorescence lifetime data and the fluorescence intensity data of the target biological tissue to obtain dual-channel fusion data; and inputting the dual-channel fusion data into a tumor detection model pre-trained based on a deep learning network model, outputting the prediction probability of each pixel corresponding to different categories, taking the category with the highest probability value as the tumor detection result of each pixel, and obtaining the distribution positions of tumors and blood vessels in the target biological tissue. According to the method, the fluorescence lifetime data and the fluorescence intensity data are fused and then input into the tumor detection model, so that the model can analyze tissue characteristics from richer and more comprehensive dimensions, the distribution positions of tumors and blood vessels in target biological tissues can be more accurately positioned, and the detection accuracy and the recognition speed are improved.

Description

Technical Field

[0001] The present invention relates to the field of tumor detection and processing, and in particular to a tumor detection method, system and computer equipment. Background Art

[0002] Accurate identification of tumor boundaries is crucial for tumor resection and prognosis. Histological diagnosis is the gold standard for distinguishing cancerous tissue from normal tissue, and it relies heavily on pathologists to analyze morphological features from hematoxylin and eosin (H&E)-stained tissue sections. Moreover, the staining method is complex, time-consuming, and inefficient, making it unsuitable for intraoperative pathological diagnosis.

[0003] With the development of nonlinear optical microscopy and artificial intelligence, high-resolution and stain-free histopathology techniques have long assisted tumor diagnosis, such as multimodal imaging such as second harmonic generation (SHG), two-photon excitation fluorescence (TPEF), and coherent anti-Stokes Raman scattering (CARS). Recently, artificial intelligence-driven stimulated Raman scattering (SRS) microscopy has shown great potential in rapid tumor diagnosis. By selectively detecting proteins and lipids, images comparable to H&E staining are generated, and cancerous areas are automatically identified by convolutional neural networks (CNN). However, the spatiotemporal overlap of Stokes and pump beams is a prerequisite for SRS microscopy to obtain high-fidelity pathological images, which puts additional requirements on optical path design and system stability. Multiphoton fluorescence microscopy is easier to perform than coherent Raman microscopy and can provide fluorescence intensity imaging of brain tumors and central nervous system (CNS) at the subcellular level. By extracting and counting structural information in multi-channel fluorescence images, tumor areas and vascular areas can be identified. However, this multi-channel detection increases the complexity of the imaging system. Summary of the invention

[0004] In view of this, the present invention provides a tumor detection method, system and computer device to solve the problem of how to improve the accuracy of non-staining tumor detection.

[0005] In a first aspect, the present invention provides a tumor detection method, comprising: Obtain the spontaneous fluorescence decay data of the target biological tissue, and extract the fluorescence intensity data and fluorescence lifetime data therefrom; fusing the fluorescence lifetime data and the fluorescence intensity data of the target biological tissue to obtain dual-channel fused data; The dual-channel fusion data is input into a tumor detection and recognition model pre-trained based on a deep learning network model, and the predicted probability of each pixel corresponding to different categories is output. The category with the highest probability value is used as the tumor detection result for each pixel, and the distribution position of the tumor and blood vessels in the target biological tissue is obtained.

[0006] The tumor detection method provided in this embodiment obtains fluorescence lifetime data and fluorescence intensity data, and fuses them to form dual-channel fusion data, which integrates two fluorescence-related information of different natures. Fluorescence lifetime mainly reflects the microenvironment information of the organism, while fluorescence intensity mainly reflects the morphological structure information of the biological tissue. The two characterize the state of the target biological tissue from different angles. After fusing them and inputting them into the tumor detection model, the model can analyze tissue characteristics from a richer and more comprehensive dimension, which helps to more accurately locate the distribution of tumors and blood vessels in the target biological tissue, reduce misjudgments caused by a single data type, and improve detection accuracy and recognition speed.

[0007] In an optional embodiment, the method further includes: converting the tumor detection result into a detection result image in a preset format; generating a fluorescence intensity image based on the fluorescence intensity data of the target biological tissue; The detection result image is superimposed on the fluorescence intensity image, and the final detection result is displayed by controlling the transparency of the detection result image.

[0008] The embodiment of the present invention will superimpose the detection result image with the fluorescence intensity image generated based on the fluorescence intensity data, and can simultaneously display the key information related to tumor detection and the fluorescence intensity distribution of the tissue itself. The fluorescence intensity shows the structure of the tissue. By superimposing the prediction results and adjusting the transparency of the prediction results, the corresponding pathological area can be directly distinguished on the intensity map, and the target area can be visualized intuitively. At the same time, the flexibly adjustable transparency does not require the user to switch back and forth between different images for comparison, and can quickly obtain key information in the same screen, reducing the time cost of information integration during the analysis process, thereby speeding up the analysis of the test results and improving the efficiency of the entire pathological diagnosis process.

[0009] In an optional embodiment, the process of training the tumor detection model includes: Obtaining the autofluorescence decay data of a preset tumor tissue slice sample, and extracting fluorescence intensity data, fluorescence lifetime data, and intensity-weighted lifetime images therefrom; The intensity-weighted fluorescence lifetime image is used as a mask to annotate the tumor, blood vessels, and background to generate a pixel-level labeled image. The fluorescence lifetime data and the fluorescence intensity data are fused to obtain dual-channel fused data; The dual-channel fusion data and the label image constitute a data pair, and the training set and the verification set are divided according to a preset ratio. The preset convolutional neural network is trained based on the training set, and the trained model is verified based on the verification set to obtain a model that meets the preset requirements as a tumor detection model.

[0010] The embodiment of the present invention obtains and fuses the spontaneous fluorescence lifetime data and fluorescence intensity data of the tissue to form dual-channel fused data, taking into account two key information reflecting the characteristics of the tissue. The fused data covers richer tissue details, allowing the model to fully learn the comprehensive characteristics of different tissues in these two aspects during the training process, and improve the model's ability to distinguish various types of tissues such as tumors and blood vessels; the intensity-weighted lifetime image is used as a mask to annotate tumors, blood vessels and background, and then a pixel-level label image is generated. This annotation method is accurate to the pixel level and can carefully divide the boundaries and ranges of different tissues, avoiding the errors caused by rough annotation, so that the model can receive accurate supervision information during training, which is conducive to learning the characteristic differences of different tissues at the microscopic level, thereby improving the accuracy of the model in identifying target tissues such as tumors.

[0011] In an optional embodiment, the obtaining of the fluorescence lifetime data of a preset tumor tissue slice sample includes: The collected fluorescence lifetime data were fitted with a double exponential model by least squares fitting to calculate the fluorescence lifetime distribution, and the average fluorescence lifetime data were obtained based on the fluorescence lifetime distribution.

[0012] The dual exponential model adopted in the embodiment of the present invention can be more in line with the dual-component characteristics that may exist in the fluorescence attenuation process under actual conditions. Compared with a simple single exponential model, it can capture the attenuation law of fluorescent substances in different states in more detail, thereby more accurately reflecting the fluorescence lifetime corresponding to each pixel, reducing the fitting error caused by improper model selection, and improving the accuracy of the acquired fluorescence lifetime data. After obtaining the fluorescence lifetime distribution, the average fluorescence lifetime data is further obtained based on it. This approach fully considers the overall distribution characteristics of the fluorescence lifetime in the sample tissue, so that the final average fluorescence lifetime data can more scientifically and reasonably reflect the overall characteristics of the fluorescence lifetime of the entire tumor tissue section sample, enhance the reliability of the data, and provide a solid data foundation for subsequent tumor detection, analysis and other work based on these data.

[0013] In an optional implementation, the step of using the intensity-weighted lifespan image as a mask to label the tumor, blood vessels, and background to generate a pixel-level label image includes: Obtain H&E images corresponding to preset tumor tissue slice samples; Comparing the features of the H&E image and the intensity-weighted lifetime image, and marking the tumor, blood vessels, and background with different colors as annotation data to obtain an initial pixel-level label image; The labeled data is screened and optimized based on expert experience and knowledge to obtain the final pixel-level labeled image.

[0014] H&E images are classic staining images commonly used in pathological diagnosis, which can clearly present important information such as cell morphology and tissue structure. The embodiment of the present invention uses intensity-weighted fluorescence lifetime images to show tissue characteristics from the perspective of fluorescence lifetime and tissue structure. The combination and comparison of the two can dig out the details of the tissue from different dimensions, so that the labeling personnel can more comprehensively and accurately grasp the specific location and range of tumors, blood vessels and backgrounds, avoid the deviation that may occur when relying on a single image for labeling, thereby improving the accuracy of labeling, and carefully dividing the tissue at the pixel level by color distinction according to different categories. It can clearly present the boundaries and distribution of different tissues at the microscopic level, which helps to more accurately learn the characteristic differences of various tissues in subsequent model training and other links, lay a good data foundation for accurate tumor detection, and use expert experience and knowledge to correct and improve it, so that the final generated pixel-level label image is more in line with the real tissue characteristics and pathological laws, greatly enhancing the reliability and professionalism of the label image.

[0015] In an optional embodiment, the method of training a preset convolutional neural network based on a training set and verifying the trained model based on a verification set to obtain a model that meets preset requirements as a tumor detection model includes: Input the training set into the U-Net network and perform training based on preset training parameters; After the training is completed, the dual-channel fusion data and the corresponding label images in the verification set will be input into the trained network model as verification data. The effect of the model training will be verified by calculating the preset performance indicators and comparing them with the preset verification targets. The model that achieves the preset verification targets will be used as the tumor detection model.

[0016] The embodiment of the present invention selects the U-Net network as the basic architecture for training. The U-Net network has outstanding performance in fields such as medical image segmentation, so that when processing dual-channel fusion data corresponding to tumor tissue slice samples, it can better transmit and utilize feature information at different scales, avoid losing important details in the feature extraction and recovery process, and well meet the needs of tumor detection to accurately distinguish subtle differences between different tissues, thereby enhancing the applicability of the model in this specific application scenario.

[0017] In a second aspect, the present invention provides a tumor detection system, the system comprising: A fluorescence data acquisition module is used to acquire the spontaneous fluorescence decay data of the target biological tissue and extract fluorescence intensity data and fluorescence lifetime data therefrom; A data fusion module, used for fusing the fluorescence lifetime data and the fluorescence intensity data of the target biological tissue to obtain dual-channel fusion data; The tumor detection module is used to input the dual-channel fusion data into a tumor detection model pre-trained based on a deep learning network model, output the predicted probability of each pixel corresponding to different categories, take the category with the highest probability value as the tumor detection result of each pixel, and obtain the distribution position of the tumor and blood vessels in the target biological tissue.

[0018] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the tumor detection method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0019] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the tumor detection method of the first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1 is a schematic diagram of a flow chart of a tumor detection method in an embodiment of the present invention; Figure 2 is a schematic diagram of another tumor detection method in an embodiment of the present invention; Figure 3 It is a schematic diagram of the main framework of the tumor detection model training process according to an embodiment of the present invention; Figure 4 Schematic diagram of the U-Net network structure of an embodiment of the present invention; Figure 5 is a schematic diagram of the detection result of the fusion fluorescence lifetime and fluorescence emphasis information of an embodiment of the present invention; Figure 6 is a schematic diagram of detection results obtained based on fluorescence lifetime or fluorescence emphasis information in an embodiment of the present invention; Figure 7 is a structural block diagram of a tumor detection system according to an embodiment of the present invention; Figure 8 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0023] Tumor detection methods in the prior art include: First, marker-based tumor recognition methods.

[0024] Tissue staining: By fixing, slicing, staining and sealing the sample, cell morphology and tissue structure information are obtained to identify the tumor area. Although this method can provide intuitive histological features, the process is cumbersome, time-consuming and labor-intensive. In addition, the diagnostic criteria of different pathologists may vary, affecting the consistency and accuracy of the results.

[0025] Fluorescent dye staining: The sample is labeled with dyes and the pathological type is distinguished based on the fluorescence emission intensity of the target object. However, the insufficient photostability of fluorescent dyes and factors such as internal heterogeneity of tumors can easily lead to false negative or false positive results, limiting its diagnostic reliability.

[0026] Second, based on marker-free tumor recognition methods.

[0027] Fluorescence intensity image analysis: By extracting local or global information of the fluorescence intensity image, combined with traditional image processing techniques (such as threshold segmentation), the tumor area is segmented. Although this method can reflect the tumor area to a certain extent, due to the uneven distribution of image intensity caused by tumor heterogeneity and changes in excitation light intensity, and the large differences between different samples, traditional algorithms are difficult to accurately segment the tumor area.

[0028] Fluorescence lifetime parameter analysis: Tumor regions are identified based on differences in fluorescence lifetime parameters, but existing methods are mostly limited to statistical analysis, making it difficult to achieve accurate identification at subcellular resolution.

[0029] Third, tumor recognition method based on deep learning.

[0030] Deep learning-based methods use the powerful feature extraction capabilities of convolutional neural networks to analyze the morphological structure information in fluorescence microscopy images or the chemical bond vibration information obtained through stimulated Raman scattering (SRS) images to identify tumor areas. However, these methods usually rely on the classification of image patches and can only output regional-level results, and cannot directly achieve accurate tumor identification at the subcellular resolution level.

[0031] Based on this, an embodiment of the present invention provides a tumor detection method based on autofluorescence lifetime information and autofluorescence intensity information. Fluorescence lifetime is highly sensitive to the interaction between the microenvironment and the surrounding of the fluorophore. Compared with intensity-based imaging, fluorescence lifetime imaging microscopy (FLIM) is not easily affected by artifacts such as fluorophore concentration, photobleaching, changes in excitation intensity, and uneven illumination of the sample. FLIM typically measures endogenous fluorophores associated with redox reactions, such as nicotinamide adenine dinucleotide (NADH) or flavin adenine dinucleotide (FAD), to reveal metabolic heterogeneity within tissues. The embodiment of the present invention uses a deep learning network to analyze dual-channel data fused with fluorescence lifetime information and fluorescence intensity information, which can greatly improve the recognition speed and accuracy of tumor boundaries and blood vessels.

[0032] According to an embodiment of the present invention, an embodiment of a tumor detection method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0033] In this embodiment, a tumor detection method is provided, which can be used in a computer device terminal. Figure 1 is a flow chart of a tumor detection method according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps: Step S101, obtaining spontaneous fluorescence attenuation data of a target biological tissue, and extracting fluorescence intensity data and fluorescence lifetime data therefrom.

[0034] Specifically, a fluorescence lifetime microscopy system based on time-correlated single photon counting is used to obtain the spontaneous fluorescence decay data of the target biological tissue to be detected. In one embodiment, the fluorescence lifetime data is obtained using time-correlated single photon counting technology. The principle is to use a pulsed laser beam to scan the sample to excite fluorescent molecules, and the TCSPC technology uses a high-time resolution detector (such as a photomultiplier tube) to record the arrival time of each fluorescent photon. The system will synchronize with the pulse signal of the excitation light and accurately time the arrival time of each fluorescent photon. By recording the time distribution of a large number of fluorescent photons, the corresponding fluorescence decay data can be obtained. The fluorescence lifetime data and the intensity-weighted fluorescence lifetime image are calculated by fitting and analyzing the fluorescence decay curve of each pixel, and the corresponding fluorescence intensity image can be obtained by summing the fluorescence decay curves of each pixel.

[0035] Step S102 , fusing the fluorescence lifetime data and the fluorescence intensity data of the target biological tissue to obtain dual-channel fused data.

[0036] Specifically, in one embodiment, the data fusion process is to regard the fluorescence lifetime data and the fluorescence intensity data as two independent channels and splice them in a certain order in the data dimension. In the case of image data, if the fluorescence lifetime image and the fluorescence intensity image have the same number of pixels, the fluorescence lifetime value and the fluorescence intensity value corresponding to each pixel can be arranged in sequence to form a new dual-channel data structure. In another embodiment, considering that the fluorescence lifetime data and the fluorescence intensity data may have different importance or contribution to the final result, different weights can be assigned to them, and then weighted average fusion is performed. The weight can be determined based on experience, prior knowledge or by experiment to evaluate the importance of the two data to tumor detection. For example, if it is known that the fluorescence lifetime data is more important in distinguishing tumor tissue from normal tissue, the corresponding weight can be set larger. The above data fusion process is only used as an example and is not limited to this.

[0037] Step S103, input the dual-channel fusion data into a tumor detection model pre-trained based on a deep learning network model, output the predicted probability of each pixel corresponding to different categories, take the category with the highest probability value as the tumor detection result of each pixel, and obtain the distribution position of the tumor and blood vessels in the target biological tissue.

[0038] According to the input requirements of the deep learning network model, the embodiment of the present invention adjusts the format of the dual-channel fusion data and inputs it into the model. The deep learning network model selects a convolutional neural network (CNN). Taking the U-Net architecture as an example, the fluorescence lifetime data and fluorescence intensity data of the sample tissue are fused and first enter the encoder part of the model. Through a series of convolution layers and pooling layers, the features in the data are gradually extracted. In this process, the convolution layer performs a convolution operation with the input data through the convolution kernel to extract local features of different scales, and the pooling layer is used to reduce the data dimension, reduce the amount of calculation and increase. After the feature extraction of the encoder, the data enters the decoder part. The decoder gradually restores the resolution of the data through the upsampling layer and the convolution layer, and fuses and refines the extracted features. Ultimately, the model outputs two main parts: one is the predicted probability of each pixel corresponding to different categories, which is obtained through a classifier (such as a fully connected layer followed by a softmax function). The softmax function can convert the numerical value output by the classifier into a probability distribution so that the sum of the probabilities of each category corresponding to each pixel is 1; the other is the distribution location information of tumors and blood vessels in the target biological tissue, which is usually represented by generating a segmented image of the same size as the input data (or after a certain scaling), in which different pixel values ​​(or colors) represent different tissue categories (such as tumors, blood vessels, background, etc.).

[0039] For each pixel, the category with the highest probability value is selected as the classification result of the pixel from the predicted probabilities of different categories output by the model. The classification results of all pixels are combined to obtain the tumor detection result. For example, if the categories output by the model include tumor, blood vessel and background, and for a certain pixel, the predicted probabilities of tumor, blood vessel and background are 0.7, 0.2 and 0.1 respectively, then the classification result of the pixel is tumor. In this way, the tissue category division of different regions in the entire target biological tissue can be obtained, thereby determining the location and range of the tumor.

[0040] In an embodiment of the present invention, the model outputs the predicted probability of each pixel corresponding to different categories, which provides a way to quantify the uncertainty of the detection results. In practical applications, the complexity and individual differences of biological tissues may lead to certain uncertainties in the detection results. Through the probability output, the user can understand the reliability of each pixel classification. For example, if the predicted probability of the tumor category of a certain pixel is 0.55, the predicted probability of the blood vessel category is 0.4, and the predicted probability of the background category is 0.1, although the pixel is finally classified as a tumor, the lower probability also indicates that there may be a certain risk of misjudgment, which requires further inspection or confirmation in combination with other diagnostic methods. In some scenarios with extremely high requirements for detection accuracy (such as early cancer screening), a higher probability threshold can be set, and only pixels with higher probabilities are judged as tumors to reduce false positive results; in some scenarios that require a comprehensive understanding of the tissue situation (such as tumor boundary research), the threshold can be appropriately lowered to obtain a wider range of areas that may belong to tumors, and then further verified by other methods. This flexibility enables the model to better adapt to different clinical and research needs.

[0041] The tumor detection method provided in this embodiment obtains fluorescence lifetime data and fluorescence intensity data and fuses them to form dual-channel fusion data, which is input into the tumor detection model. This allows the model to analyze tissue characteristics from a richer and more comprehensive dimension, which helps to more accurately locate the distribution of tumors and blood vessels in the target biological tissue, reduce misjudgments caused by a single data type, and improve detection accuracy and recognition speed.

[0042] In one embodiment, the tumor detection method provided by the embodiment of the present invention is as follows: Figure 2 As shown, after step S103, the following steps are also included: Step S104: converting the tumor detection result into a detection result image in a preset format.

[0043] Specifically, the tumor detection results are converted into detection result images in a preset format (for example, the tumor detection results are stored in a PNG image format) so that the results have a unified and standardized presentation format. This facilitates different users, such as doctors and researchers, to quickly view and interpret the detection results according to the established format requirements, thereby improving the efficiency of information transmission.

[0044] Step S105 , generating a fluorescence intensity image based on the fluorescence intensity data of the target biological tissue.

[0045] Step S106, superimposing the detection result image and the fluorescence intensity image, and displaying the final detection result by controlling the transparency of the detection result image.

[0046] In the embodiment of the present invention, the detection result image is superimposed with the fluorescence intensity image generated based on the fluorescence intensity data, so that the key information related to tumor detection and the fluorescence intensity distribution of the tissue itself can be displayed simultaneously. The fluorescence intensity shows the structure of the tissue, superimposed with the prediction result, and by adjusting the transparency of the prediction result, the corresponding area can be directly seen on the intensity map, so that the target area can be directly visualized on the intensity map, which assists in more accurate analysis and judgment.

[0047] The final test results are displayed by controlling the transparency of the test result image, which increases the flexibility of the display. Users can adjust the transparency according to actual needs. For example, in the initial review, the transparency can be appropriately lowered to more comprehensively observe the overall fluorescence intensity background of the tissue and the approximate location of the tumor; when focusing on the details of the tumor, the transparency can be appropriately increased to highlight the specific morphology, boundaries and other features of the tumor, meeting the needs of detailed observation and analysis of the test results in different scenarios.

[0048] The detection result image and fluorescence intensity image comprehensively presented in the embodiment of the present invention can provide doctors and other professionals with richer information for diagnosis. The change in fluorescence intensity can be compared to further confirm the rationality of the tumor detection result, and the characteristics of the two images can be combined to determine whether there is a misjudgment, etc., which helps to improve the accuracy of diagnosis and provide a reliable basis for the subsequent formulation of accurate treatment plans. At the same time, the user does not need to switch back and forth between different images for comparison, and can quickly obtain key information in the same screen, reducing the time cost of information integration during the analysis process, thereby speeding up the analysis of the detection results and improving the efficiency of the entire diagnostic process.

[0049] In one embodiment, the main framework of the tumor detection model training process is as follows: Figure 3As shown in the figure, based on mouse brain slices, TCSPC-FLIM (time-correlated single photon counting-fluorescence lifetime imaging microscope) technology is used to obtain image data of three different modalities: fluorescence intensity, average fluorescence lifetime, and intensity-weighted lifetime image. The fluorescence intensity and average fluorescence lifetime data are image fused to generate dual-channel data, and the intensity-weighted lifetime image is target labeled to mark the area or object of interest, generate a label image, and use the dual-channel data and the corresponding label image to train the U-Net network for image segmentation tasks. The fused image is classified using the trained multi-view classifier, and after obtaining the classification results, the results are visualized for easy observation and analysis.

[0050] In one embodiment, the tumor detection model training process specifically includes the following steps: Step S1, obtaining the autofluorescence attenuation data of a preset tumor tissue slice sample, and extracting fluorescence intensity data, fluorescence lifetime data and intensity-weighted lifetime image therefrom.

[0051] Specifically, the tumor tissue slice samples were obtained from mouse brain tumor tissue slices. The mice were killed by cervical dislocation on the 15th day after tumor inoculation. The brain tissue was then quickly removed, cut in half from the tumor injection site, and formalin-fixed and paraffin-embedded. Serial slices (5 µm thick) were cut from different brain parts using a freezing microtome. One of them was used for H&E staining for histological comparison, and the other slices were dewaxed and used for fluorescence lifetime imaging.

[0052] Furthermore, a fluorescence lifetime microscopy system based on time-correlated single photon counting was used to perform fluorescence lifetime imaging on dewaxed mouse brain tumor tissue sections, and the fluorescence decay data of the endogenous fluorescent substance FAD in the sample was collected and saved as an .sdt file. To ensure the same experimental conditions, the acquisition time of all raw data was the same. The calculation and fitting of fluorescence lifetime were completed by SPCImage 8.9 software (Becker&Hickl GmbH). The fluorescence decay data was corresponding to the fluorescence decay histogram of each pixel in the fluorescence lifetime image, and the least squares fitting was performed using a double exponential model to calculate the fluorescence lifetime distribution, and the average fluorescence lifetime data was obtained based on the fluorescence lifetime distribution. Finally, the average fluorescence lifetime data (.asc file), fluorescence intensity data (.asc file) and intensity-weighted lifetime image (.tif image) were exported for subsequent analysis.

[0053] The embodiment of the present invention adopts a double exponential model that can better fit the multi-component characteristics that may exist in the fluorescence attenuation process under actual conditions. Compared with a simple single exponential model, it can capture the attenuation law of fluorescent substances in different states more finely, thereby more accurately reflecting the fluorescence lifetime corresponding to each pixel, reducing the fitting error caused by improper model selection, and improving the accuracy of the acquired fluorescence lifetime data. Further based on it, the average fluorescence lifetime data is obtained. This approach fully considers the overall distribution characteristics of the fluorescence lifetime in the sample tissue. Instead of simply using a single numerical value to represent, it integrates the different fluorescence lifetimes of each pixel, so that the final average fluorescence lifetime data can more scientifically and reasonably reflect the overall characteristics of the fluorescence lifetime of the entire tumor tissue slice sample, enhance the reliability of the data, and provide a solid data foundation for subsequent tumor detection and analysis based on these data.

[0054] Step S2: using the intensity-weighted lifespan image as a mask to label the tumor, blood vessels, and background, and generating a pixel-level label image.

[0055] In an embodiment of the present invention, the Image Labeler tool of MATLAB R2019B is used to annotate the intensity-weighted lifetime image. Using it as a mask not only provides the morphological and structural information of the tissue, but also contains rich microenvironmental feature information. In the annotation process, the corresponding H&E image is referenced, which is a classic staining image commonly used in pathological diagnosis, and can clearly present important information such as cell morphology and tissue structure. Three semantic categories are pre-defined, namely tumor, vessel, and background. By comparing the detailed features of the two images, the details of the tissue can be mined from different dimensions, and the tumor area and vascular distribution can be annotated. In order to better distinguish these categories, different colors are assigned to each label, and the target area is finely annotated by the polygon tool and the brush tool. By carefully dividing the tissue at the pixel level by color distinction according to different categories, the boundaries and distribution of different tissues at the microscopic level can be clearly presented, which is helpful for subsequent model training and other links to more accurately learn the characteristic differences of various tissues, and lay a good data foundation for accurate tumor detection.

[0056] After the labeling is completed, the label data is exported as a pixel-level label map. Each pixel value of these label maps represents the corresponding category, and the resolution of the label map is consistent with the original image. In addition, the embodiment of the present invention is based on a number of experts with rich experience in the field of tissue preparation and imaging. Based on the expert experience and knowledge, the labeled data has been strictly screened and optimized for multiple rounds, so that the pixel-level label image finally generated is more in line with the real tissue characteristics and pathological laws, greatly enhancing the reliability and professionalism of the label image. It not only improves the accuracy and scientific value of the data, but also lays a solid foundation for the training of tumor and blood vessel recognition models based on deep learning.

[0057] Step S3, fusing the fluorescence intensity data and the fluorescence lifetime data to obtain dual-channel fused data.

[0058] Specifically, the software uses weighted fusion or splicing to merge the fluorescence intensity data and the average fluorescence lifetime data into dual-channel data, where the first channel is the average fluorescence lifetime and the second channel is the fluorescence intensity. The generated dual-channel data can not only retain the average fluorescence lifetime information of the tissue, but also display the fluorescence intensity distribution at the same time, which is convenient for subsequent data analysis. Finally, the synthesized dual-channel data is saved as a .mat format file for subsequent analysis.

[0059] Step S4, the dual-channel fusion data and the label image constitute a data pair, divide the training set and the verification set according to the preset ratio, and train the preset convolutional neural network based on the training set, and verify the trained model based on the verification set to obtain a model that meets the preset requirements as a tumor detection model.

[0060] Specifically, the present invention is based on the U-Net multi-category semantic segmentation model for training, and the network structure is as follows Figure 4As shown in Figure 1, it consists of an encoder and a decoder. The encoder is mainly used to extract the feature information of the image and gradually reduce the spatial resolution. It follows the typical architecture of the convolutional network, including repeated application of two 3×3 convolutional layers (conv), each followed by a rectified linear unit activation layer (ReLu), and a 2-step 2×2 maximum pooling layer (maxpool) for downsampling. After each downsampling, the number of feature channels doubles and the resolution of the image is halved. The decoder is mainly used to gradually restore the spatial resolution of the image and generate an output of the same size as the input image. Each decoder unit performs a 2×2 up-sampling (up-conv) on the feature map and halves the number of feature channels, and performs a jump connection (copy) with the feature map of the same resolution in the encoder to maintain the precise positioning of the extracted data pattern, followed by two 3×3 convolutional layers (conv), each followed by an activation layer (ReLu). In the final layer, a 1×1 convolutional layer (conv) is used to map each 64-dimensional feature vector to the required number of categories. Since zero padding is applied to each convolutional layer, the resolution of the edge area remains consistent.

[0061] Furthermore, the dual-channel data (.mat file) and the corresponding label image are used to form a data pair, which is saved as a training set and a validation set in a ratio of 8:2. During training, the training file (.mat file) and the training target (label image, three-category semantic segmentation) are converted into tensor data as training data, and the training data is input into the fully convolutional neural network built in step 5 for training. The training parameters are: the number of iterations epochs is between 100 and 500 times, the optimizer uses the Adam optimization algorithm, and the classification loss is calculated using the Binary Cross Entropy Loss function. The learning rate lr is between 0.001 and 0.0001, the exponential decay rate beta1 of the first-order moment estimate is 0.9, the exponential decay rate beta2 of the second-order moment estimate is 0.999, and the decay rate weight decay is 0. After the training is completed, the validation file and the corresponding label image in the validation set are converted into tensor data and input into the trained network model as validation data. By comparing with the verification target, calculating performance indicators such as IOU (Intersection over Union) and Dice coefficient, the effect of model training is verified, and the model that reaches the preset verification target is used as the tumor detection model. The data used in the above training parameters are only used as examples and are not limited to this.

[0062] The embodiment of the present invention selects the U-Net network model as the basic architecture for training. The U-Net network has outstanding performance in fields such as medical image segmentation, so that when processing dual-channel fusion data corresponding to tumor tissue slice samples, it can better transmit and utilize feature information at different scales, avoid losing important details in the feature extraction and recovery process, and well meet the needs of tumor detection to accurately distinguish subtle differences between different tissues, thereby enhancing the applicability of the model in this specific application scenario.

[0063] In a specific embodiment, if Figure 5 As shown, each set of images contains four parts, from left to right: τm (fluorescence lifetime), Intensity (fluorescence intensity), H&E (hematoxylin-eosin staining) and Output - overlay (output overlay). Among them: 1. The τm (fluorescence lifetime) image presents different color distributions, showing the changes in fluorescence lifetime in the sample. Tumor tissue, blood vessels, and normal tissue may have differences in fluorescence lifetime. The distribution and pattern of these differences can be observed through these images.

[0064] 2. The Intensity image shows the distribution of fluorescence intensity in the sample. Fluorescence intensity is related to factors such as the concentration of the fluorescent substance. Areas with stronger fluorescence intensity may correspond to certain specific tissue components or cell structures. The light and dark contrast in the image reflects the changes in fluorescence intensity, which helps to identify the characteristics of different areas in the sample.

[0065] 3. H&E (hematoxylin-eosin staining) is used to show the structure and cell morphology of tissues. It can clearly see the morphology of cells, the size and shape of cell nuclei, and the overall structure of tissues, providing important basis for pathological diagnosis.

[0066] 4. FLIM-based (output superposition of the embodiment of the present invention) is the detection result after fusing the information of τm (fluorescence lifetime) and Intensity (fluorescence intensity) images, and superimposing the effect of fluorescence intensity images. Different tissue categories are marked with different colors in the image, which can intuitively show the distribution and position relationship of tumors, blood vessels and normal tissues in the sample, as well as their comprehensive characteristics in terms of fluorescence lifetime, fluorescence intensity and tissue structure, and obtain better recognition results.

[0067] like Figure 6As shown, five groups of different types of images are provided, each group of images contains three rows, showing the conditions of different samples. The images from left to right are: τm (fluorescence lifetime), Intensity (fluorescence intensity), H&E (hematoxylin-eosin staining), FLIM-based (output superposition of the embodiment of the present invention), Intensity-based (output superposition based on fluorescence intensity image). It can be seen that compared with the detection effect based only on fluorescence intensity image, the detection effect of the embodiment of the present invention that combines fluorescence intensity enhancement and fluorescence lifetime information is clearer and more accurate.

[0068] In this embodiment, a tumor detection system is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the system described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0069] This embodiment provides a tumor detection system. Figure 7 As shown, including: The fluorescence data acquisition module 701 is used to acquire the spontaneous fluorescence attenuation data of the target biological tissue and extract the fluorescence intensity data and the fluorescence lifetime data therefrom; A data fusion module 702 is used to fuse the fluorescence lifetime data and the fluorescence intensity data of the target biological tissue to obtain dual-channel fusion data; The tumor detection module 703 is used to input the dual-channel fusion data into a tumor detection model pre-trained based on a deep learning network model, output the predicted probability of each pixel corresponding to different categories, take the category with the highest probability value as the tumor detection result of each pixel, and obtain the distribution position of the tumor and blood vessels in the target biological tissue.

[0070] In an optional embodiment, the above-mentioned detection system also includes: a visualization display module, which is used to convert the tumor detection result into a detection result image in a preset format; generate a fluorescence intensity image based on the fluorescence intensity data of the target biological tissue; superimpose the detection result image with the fluorescence intensity image, and display the final detection result by controlling the transparency of the detection result image.

[0071] In an optional embodiment, the process of training the tumor detection model includes: Obtaining the autofluorescence decay data of a preset tumor tissue slice sample, and extracting fluorescence intensity data, fluorescence lifetime data, and intensity-weighted lifetime images therefrom; The intensity-weighted lifetime image is used as a mask to annotate the tumor, blood vessels, and background, generating a pixel-level labeled image. The fluorescence intensity data and the fluorescence lifetime data are fused to obtain dual-channel fusion data; The dual-channel fusion data and the label image constitute a data pair, and the training set and the verification set are divided according to a preset ratio. The preset convolutional neural network is trained based on the training set, and the trained model is verified based on the verification set to obtain a model that meets the preset requirements as a tumor detection model.

[0072] In an optional embodiment, obtaining fluorescence lifetime data of a preset tumor tissue slice sample includes: The fluorescence lifetime data is corresponding to the fluorescence decay histogram of each pixel in the fluorescence lifetime image, and the double exponential model is used for least square fitting to calculate the fluorescence lifetime distribution, and the average fluorescence lifetime data is obtained based on the fluorescence lifetime distribution.

[0073] In an optional embodiment, the intensity-weighted lifespan image is used as a mask to annotate the tumor, blood vessels, and background to generate a pixel-level label image, including: Obtain H&E images corresponding to preset tumor tissue slice samples; Comparing the features of the H&E image and the intensity-weighted lifetime image, and marking the tumor, blood vessels, and background with different colors as annotation data to obtain an initial pixel-level label image; The labeled data is screened and optimized based on expert experience and knowledge to obtain the final pixel-level labeled image.

[0074] In an optional embodiment, a preset convolutional neural network is trained based on a training set, and the trained model is verified based on a verification set to obtain a model that meets preset requirements as a tumor detection model, including: Input the training set into the U-Net network and perform training based on preset training parameters; After the training is completed, the dual-channel fusion data and the corresponding label images in the verification set will be input into the trained network model as verification data. The effect of the model training will be verified by calculating the preset performance indicators and comparing them with the preset verification targets. The model that achieves the preset verification targets will be used as the tumor detection model.

[0075] The tumor detection system in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC circuit, a processor and a memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0076] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0077] The embodiment of the present invention also provides a computer device having the above Figure 7The tumor detection system shown.

[0078] See also Figure 8 , Figure 8 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 8 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 8 A processor 10 is taken as an example.

[0079] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0080] The memory 20 stores instructions executable by at least one processor 10, so that at least one processor 10 executes the method shown in the above embodiment.

[0081] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0082] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0083] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0084] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0085] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0086] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A tumor detection method, characterized in that: The method comprises: Obtain the time decay data of the autofluorescence of the target biological tissue, and extract the fluorescence intensity data and fluorescence lifetime data from it; fusing the fluorescence lifetime data and the fluorescence intensity data of the target biological tissue to obtain dual-channel fused data; The dual-channel fusion data is input into a tumor detection model pre-trained based on a deep learning network model, and the predicted probability of each pixel corresponding to different categories is output. The category with the highest probability value is used as the tumor detection result for each pixel, and the distribution position of the tumor and blood vessels in the target biological tissue is obtained.

2. The method according to claim 1, characterized in that Also includes: Converting the tumor detection result into a detection result image in a preset format; generating a fluorescence intensity image based on the fluorescence intensity data of the target biological tissue; The detection result image is superimposed on the fluorescence intensity image, and the final detection result is displayed by controlling the transparency of the detection result image.

3. The method according to claim 1, characterized in that: The process of training the tumor detection model includes: Obtaining the autofluorescence decay data of a preset tumor tissue slice sample, and extracting fluorescence intensity data, fluorescence lifetime data, and intensity-weighted lifetime images therefrom; Each pixel is pseudo-colored according to its corresponding fluorescence lifetime and intensity to generate an intensity-weighted lifetime image, which is used as a mask to annotate the tumor, blood vessels, and background to generate a pixel-level label image. The fluorescence intensity data and the fluorescence lifetime data are fused to obtain dual-channel fusion data; The dual-channel fusion data and the label image constitute a data pair, and the training set and the verification set are divided according to a preset ratio. The preset convolutional neural network is trained based on the training set, and the trained model is verified based on the verification set to obtain a model that meets the preset requirements as a tumor detection model.

4. The method according to claim 3, characterized in that: The step of obtaining the fluorescence lifetime data of the preset tumor tissue slice sample comprises: The fluorescence lifetime data is corresponding to the fluorescence decay histogram of each pixel in the fluorescence lifetime image, and the double exponential model is used for least square fitting to calculate the fluorescence lifetime distribution, and the average fluorescence lifetime data is obtained based on the fluorescence lifetime distribution.

5. The method according to claim 3, characterized in that: The step of using the intensity-weighted lifespan image as a mask to label the tumor, blood vessels and background to generate a pixel-level label image includes: Obtain H&E images corresponding to preset tumor tissue slice samples; Comparing the features of the H&E image and the intensity-weighted lifetime image, and marking the tumor, blood vessels, and background with different colors as annotation data to obtain an initial pixel-level label image; The labeled data is screened and optimized based on expert experience and knowledge to obtain the final pixel-level labeled image.

6. The method according to claim 3, characterized in that The method of training a preset convolutional neural network based on a training set and verifying the trained model based on a verification set to obtain a model that meets preset requirements as a tumor detection model includes: Input the training set into the U-Net network and perform training based on preset training parameters; After the training is completed, the dual-channel fusion data and the corresponding label images in the verification set will be input into the trained network model as verification data. The effect of the model training will be verified by calculating the preset performance indicators and comparing them with the preset verification targets. The model that achieves the preset verification targets will be used as the tumor detection model.

7. A tumor detection system, characterized in that: The system comprises: A fluorescence data acquisition module is used to acquire the spontaneous fluorescence decay data of the target biological tissue and extract fluorescence intensity data and fluorescence lifetime data therefrom; A data fusion module, used for fusing the fluorescence lifetime data and the fluorescence intensity data of the target biological tissue to obtain dual-channel fusion data; The tumor detection module is used to input the dual-channel fusion data into a tumor detection model pre-trained based on a deep learning network model, output the predicted probability of each pixel corresponding to different categories, take the category with the highest probability value as the tumor detection result of each pixel, and obtain the distribution position of the tumor and blood vessels in the target biological tissue.

8. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the tumor detection method according to any one of claims 1 to 6 by executing the computer instructions.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the tumor detection method according to any one of claims 1 to 6.

10. A computer program product, characterized in that The method comprises computer instructions for causing a computer to execute the tumor detection method according to any one of claims 1 to 6.

Images