Lung cancer pleural invasion auxiliary diagnosis system and method based on multi-photon microscopy

Abstract

The invention provides a lung cancer pleural invasion auxiliary diagnosis system and method based on a multiphoton microscopy technology, and belongs to the technical field of medical artificial intelligence and intraoperative pathology assistion.The method comprises the steps that multiphoton images of unstained paraffin sections, frozen sections and fresh tissue are collected through the multiphoton microscopy system; constructing a multi-source data set and extracting 142-dimensional tissue features; after cross-domain data enhancement of the generative adversarial network, semantic features extracted by the deep learning pre-training model are fused, and a pleura infringement state classifier is constructed; in the operation, a real-time image is obtained through a portable multi-photon system and input into a classifier to rapidly predict VPI negative and positive. According to the method, rapid, accurate and label-free diagnosis in the operation of the VPI is realized, real-time support is provided for decision-making of the resection range of the lung cancer operation, insufficient treatment or excessive resection is avoided, and the prognosis of a patient is effectively improved.

Description

Technical Field

[0001] This invention relates to the fields of medical artificial intelligence and intraoperative pathology assistance technology, and in particular to an auxiliary diagnostic system and method for lung cancer pleural invasion based on multiphoton microscopy. Background Technology

[0002] Visceral pleural invasion (VPI) is crucial in determining the surgical approach for lung cancer. Lobectomy may be more preferable for VPI-positive patients, while lung-preserving sublobar resection is safer for VPI-negative patients. However, various techniques currently used in clinical practice to diagnose VPI have limitations, as follows: Preoperative CT has an accuracy rate of only 62.7%–72.3%, and reactive pleural changes often cause false positives; intraoperative thoracoscopic gross assessment is highly subjective and difficult to identify early pleural breakthrough lesions; postoperative Elasticavan Gieson (EVG) staining, although the gold standard for VPI diagnosis, requires a complex 24–48 hour processing cycle and depends on expert interpretation, and cannot provide real-time support for intraoperative decision-making.

[0003] None of the aforementioned existing technologies can achieve rapid, accurate, and objective VPI diagnosis during surgery. The choice of surgical plan still depends on the postoperative pathological results, which may lead to insufficient or excessive resection, affecting the patient's prognosis and efficacy.

[0004] Multiphoton microscopy (MPM) combines two-photon excitation fluorescence (TPEF) and second harmonic generation (SHG) for label-free optical biopsy of lung tissue, directly imaging collagen, elastin fibers, and cellular structures. In lung diseases, MPM can distinguish between fibrotic and healthy tissue and improve the detection of normal tumor boundaries in lung adenocarcinoma by combining fibrous patterns with tumor morphology. Machine learning analysis based on MPM images has enabled quantitative classification of interstitial lung disease subtypes and automatic identification of benign and malignant lung lesions, providing a new perspective for precise intraoperative determination of surgical margins.

[0005] However, the application of MPM in VPI detection has not been systematically explored. Since VPI is defined as the infiltration of tumor cells through the elastic layer of the pleura (often masked by reactive inflammation and fibrosis), its diagnosis still relies on postoperative histological examination. Therefore, developing a rapid intraoperative VPI grading method to integrate VPI diagnosis into the clinical workflow, eliminate the delay of traditional histological examination, and provide real-time support for surgical decisions has become an urgent need.

[0006] Therefore, this invention proposes an auxiliary diagnostic system and method for lung cancer pleural invasion based on multiphoton microscopy. Summary of the Invention

[0007] This invention provides an auxiliary diagnostic system and method for lung cancer pleural invasion based on multiphoton microscopy, in order to solve the aforementioned technical problems.

[0008] This invention provides an auxiliary diagnostic system for pleural invasion in lung cancer based on multiphoton microscopy, comprising: The dataset construction module is used to collect multiphoton images of unstained paraffin sections, frozen sections and fresh surgically removed tissues from lung cancer patients based on a multiphoton microscopy system, and selects images at the pleural boundary as the region of interest to construct a multi-source dataset. The tissue feature extraction module is used to analyze each region of interest in the multi-source dataset using existing collagen fiber feature extraction algorithms to obtain 142-dimensional tissue features. The cross-domain data augmentation module is used to perform cross-domain data augmentation between multiphoton images of pathological paraffin sections, frozen sections, and surgically removed fresh tissue using a generative adversarial network model, resulting in several sample images. The classifier building module is used to fuse the high-dimensional tissue features with the semantic feature vectors of sample images extracted based on a deep learning pre-trained model to build a classifier for pleural invasion status, and to complete supervised training on a labeled dataset. The result prediction module is used to acquire real-time images during surgery using a portable multiphoton microscope system, input the real-time images into a trained classification model, predict the VPI positive or negative status, and output the diagnostic result.

[0009] Preferably, the cross-domain data enhancement module includes: The first cross-domain unit is used to input the multiphoton image of the pathological paraffin section into the generative adversarial network model and generate frozen section style images and fresh tissue style images. The second cross-domain unit is used to input multiphoton images of frozen sections and surgically removed fresh tissue into the generative adversarial network model to generate paraffin section-style images. Among them, frozen section style images, fresh tissue style images, and paraffin section style images are sample images.

[0010] Preferably, the classifier construction module includes: Semantic units are used to input image patches of each sample image into a pre-trained model and output a 2048-dimensional semantic feature vector representing spatial structure and histological information. The splicing unit is used to splice the semantic feature vector with the 142-dimensional organizational features to form a composite feature vector; A supervised training unit is used to supervise the training of the composite feature vector to obtain a pleural invasion state classifier.

[0011] Preferably, the supervised training unit includes: A subunit is defined to use the region of interest as a sample bag and the composite feature vector corresponding to the image patch within the region of interest as an instance; The low-dimensional mapping subunit is used to map the concatenated composite feature vector to a low-dimensional embedding space. The weighted subunit is used to weight and aggregate the low-dimensional embedded features according to the importance weight of each instance to the diagnosis, so as to obtain the comprehensive features. The state output subunit is used to output the pleural invasion state based on comprehensive features. During training, a total loss function combining classification loss and clustering regularization loss is used, and end-to-end model optimization is completed based on an adaptive optimization algorithm.

[0012] Preferably, the cross-domain data enhancement module further includes: The point determination unit performs self-enhancing operations (EEAs) with angle transformation, scale scaling, and noise perturbation on multiphoton images of pathological paraffin sections, frozen sections, and surgically excised fresh tissue, respectively, to obtain EEAs. It then performs region feature registration analysis on the three types of EEAs from the same lesion site, calculates the region grayscale matching degree and tissue texture overlap, and divides the enhanced point block sets at different scales. Where L is the number of scale levels; Let j be the number of points at the j-th scale. Let be the three-dimensional coordinates of the k-th point at the j-th scale; The first candidate unit is used for point blocks at each scale level j. The first candidate sample is generated based on the corresponding original image region. ,Will The input generative adversarial network model performs a complete cross-domain transformation, yielding cross-domain candidate samples corresponding to two other image types. ; The coefficient calculation unit is used for... and Perform multi-dimensional structural verification and calculate the enhancement effectiveness coefficient of the corresponding point blocks. ; ; in, for and Feature vector matching degree; For the feature encoding function of the pre-trained model; for The regularity of the three-dimensional structure; , The weighting coefficients are satisfied. ; like Then determine the corresponding point block. corresponding and As valid samples, they are retained, These are preset coefficients; The point mapping unit is used to establish point block mapping associations for frozen sections and surgically removed fresh tissue images, based on effective enhanced point blocks at various scales of paraffin section images. , The multi-scale point mapping topology is obtained based on the intersection operation. ; The second candidate unit is used to construct a second candidate sample based on the mapping topology R, using the point blocks of the effective samples as the reference unit. ; The completion unit is used to include all second candidate samples. The input generative adversarial network model performs a complete cross-domain transformation to obtain cross-domain completion samples. Then, the effective first candidate samples, cross-domain candidate samples, and cross-domain completion samples at all scales are integrated into the sample image set to achieve sample expansion.

[0013] Preferably, the second candidate unit includes: The sub-feature extraction subunit is used to divide the point-block feature vector output by the pre-trained histological feature extraction function into several sub-feature vectors related to lung cancer pleural invasion, including the collagen fiber spatial distribution feature sub-vector. Cell boundary structure feature vector Spatial feature vectors of tissue interstitial space ; The weighting subunit is used to assign weight coefficients based on the contribution of each sub-feature to the determination of pleural invasion status. , , The weighted histological feature distances between corresponding points on paraffin-embedded sections and frozen sections, and between paraffin-embedded sections and freshly excised tissue, were calculated for the same lesion site. ; filter The point blocks form a set of associated units; The constraint alignment sub-unit is used to apply spatial topological constraints to the point blocks corresponding to frozen sections and surgically removed fresh tissue, based on the point correspondence relationship of the pathological paraffin section point blocks in the associated unit set as the reference unit, and to obtain the topologically aligned block combination. An initial generation sub-unit is used to perform feature weighted fusion on the topologically aligned block combination to obtain fused features, and to generate an initial second candidate sample based on the fused features. The final sub-unit is used to calculate the spatial relative position deviation between each point block in the initial second candidate sample and the corresponding block in the original image. If the spatial relative position deviation is less than or equal to a preset deviation, the initial second candidate sample is determined to be the final second candidate sample.

[0014] Preferably, the location determination unit includes: The encoding subunit is used to obtain the space-spectral fusion feature vector of each pixel in the multiphoton image. and will The input encoding grid is used to perform the first encoding, and the encoding formula is: ,in, Encode grids for pixel features; , These are the weight matrix and bias term of the grid, respectively; Use the Sigmoid activation function; The first encoded feature of the i-th pixel; The deviation vector acquisition subunit is used to obtain the first encoded feature of all pixels. Perform adaptive density clustering to obtain the feature center deviation vector for each pixel. ,in, For the first The feature centers of each cluster are formed by all clusters within that cluster. The average value is obtained; Noise extraction subunit, used for each cluster Extracting basic noise features from all feature center deviation vectors. And generate the initial adjustment vector for the corresponding cluster. ; The influence calculation subunit is used to set the pixel adjustment order based on the spatial location correlation between all initial adjustment vectors and the first encoded feature, and to calculate the influence weight of the image adjustment operation effect set before N1 on N1 for the pixel with the current index N1. ; ,in, These are the spatial coordinates of the t-th pixel and the pixel with the current index N1, respectively, and N1 is greater than 1; Cosine similarity of spatial coordinates; To maintain efficiency for the features of the t-th order adjustment operation, and ,in, The t-th order adjusted coding features; The encoding features before the t-th order adjustment; Parameterized fine-tuning subunit, used to adjust according to the influence weights Perform parameterized fine-tuning on the initial adjustment vector for the current index N1: ,in, For fine-tuning the bias term; This is the fine-tuned vector; Feature modulation subunit, used to... Feature modulation is performed on the image after N1-1 initial adjustment vector operations: ,in, The modulation coefficient; Image processing subunit, used to... The image is processed using the LeakyReLU activation function to obtain the currently adjusted image. This process continues until all pixel adjustments are complete, at which point a self-enhancing subimage is obtained. , This is the LeakyReLU activation function.

[0015] Preferably, the noise extraction subunit includes: ,in, To adjust the vector generation network; , They are respectively The weight matrix and bias terms; This is the ReLU activation function.

[0016] This invention provides a method for the auxiliary diagnosis of pleural invasion in lung cancer based on multiphoton microscopy, comprising: Step 1: Collect multiphoton images of unstained paraffin sections, frozen sections, and freshly excised tissue from lung cancer patients using a multiphoton microscopy system. Select the image at the pleural boundary as the region of interest to construct a multi-source dataset. Step 2: Using existing collagen fiber feature extraction algorithms, analyze each region of interest in the multi-source dataset to obtain 142-dimensional tissue features; Step 3: Using a generative adversarial network model, cross-domain data augmentation is performed between multiphoton images of pathological paraffin sections, frozen sections, and surgically removed fresh tissue to obtain several sample images; Step 4: Fuse the high-dimensional tissue features with the semantic feature vectors of the sample images extracted based on the deep learning pre-trained model to construct a pleural invasion state classifier, and complete supervised training on the labeled dataset; Step 5: During the operation, real-time images are acquired using a portable multiphoton microscope system. The real-time images are then input into the trained classification model to predict the VPI positive or negative status and output the diagnostic results.

[0017] Compared with the prior art, the beneficial effects of this application are as follows: This multi-module collaborative auxiliary diagnostic system enables rapid, accurate, and label-free intraoperative diagnosis of pleural invasion in lung cancer. Multi-source datasets provide a rich sample foundation for the model, while the fusion of 142-dimensional tissue features and deep learning semantic features improves classification accuracy. Cross-domain data enhances the model's generalization ability, and the portable system and rapid inference ensure real-time intraoperative application. It effectively addresses the shortcomings of traditional diagnostic techniques, such as low accuracy, long processing time, and high subjectivity. It can provide surgeons with real-time decision support, optimize the surgical resection range, avoid under- or over-resection, and improve patient prognosis.

[0018] Other features and advantages of the invention will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written description and the accompanying drawings.

[0019] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0020] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a structural diagram of an auxiliary diagnostic system for lung cancer pleural invasion based on multiphoton microscopy, as described in an embodiment of the present invention. Figure 2 This is a flowchart of an auxiliary diagnostic method for lung cancer pleural invasion based on multiphoton microscopy, as described in an embodiment of the present invention. Detailed Implementation

[0021] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0022] This invention provides an auxiliary diagnostic system for pleural invasion in lung cancer based on multiphoton microscopy, such as... Figure 1 As shown, it includes: The dataset construction module is used to collect multiphoton images of unstained paraffin sections, frozen sections and fresh surgically removed tissues from lung cancer patients based on a multiphoton microscopy system, and selects images at the pleural boundary as the region of interest to construct a multi-source dataset. The tissue feature extraction module is used to analyze each region of interest in the multi-source dataset using existing collagen fiber feature extraction algorithms to obtain 142-dimensional tissue features. The cross-domain data augmentation module is used to perform cross-domain data augmentation between multiphoton images of pathological paraffin sections, frozen sections, and surgically removed fresh tissue using a generative adversarial network model, resulting in several sample images. The classifier building module is used to fuse the high-dimensional tissue features with the semantic feature vectors of sample images extracted based on a deep learning pre-trained model to build a classifier for pleural invasion status, and to complete supervised training on a labeled dataset. The result prediction module is used to acquire real-time images during surgery using a portable multiphoton microscope system, input the real-time images into a trained classification model, predict the VPI positive or negative status, and output the diagnostic result.

[0023] Preferably, the classifier construction module includes: Semantic units are used to input image patches of each sample image into a pre-trained model and output a 2048-dimensional semantic feature vector representing spatial structure and histological information. The splicing unit is used to splice the semantic feature vector with the 142-dimensional organizational features to form a composite feature vector; A supervised training unit is used to supervise the training of the composite feature vector to obtain a pleural invasion state classifier.

[0024] In this embodiment, the multiphoton microscopy system refers to an optical imaging system that integrates two-photon excitation fluorescence (TPEF) and second harmonic generation (SHG) mechanisms.

[0025] In this embodiment, unstained paraffin pathological sections refer to pathological tissue sections of lung cancer patients prepared by paraffin embedding and sectioning without being stained with exogenous dyes; frozen sections refer to unstained tissue sections prepared by rapidly freezing fresh tissue and then sectioning; and surgically removed fresh tissue refers to parapleural tissue that has not undergone fixation and embedding treatment and is directly removed during lung cancer surgery.

[0026] The region of interest (ROI) refers to a key area in multiphoton images that is relevant to the diagnosis of pleural invasion in lung cancer. It typically includes the pleural boundary and surrounding tissue, containing information about the pleural elastic layer, collagen fiber network, and potential tumor cell infiltration. For example, in acquired multiphoton images, the pleural boundary (a densely distributed area of ​​collagen fibers with significantly higher SHG signal intensity than surrounding tissue) is identified using an image segmentation algorithm. A region with a width of 200 mm is then selected centered on the pleural boundary. Length is 500 The rectangular region is designated as the Region of Interest (ROI), which covers the elastic layer of the pleura and 100 mm on each side. Lung tissue and tumor tissue (if infiltrated). A deep learning-based U-Net image segmentation model was used to automatically extract the Region of Interest (ROI) from multiphoton images. This model was trained on a dataset of 1000 labeled multiphoton images. For images with poor automatic segmentation (such as blurred pleural boundaries), the ROI boundaries were adjusted manually.

[0027] A multi-source dataset refers to a dataset that integrates multiphoton images and corresponding Regions of Interest (ROIs) of three types of samples: unstained paraffin-embedded pathological sections, frozen sections, and fresh surgically excised tissue. This provides rich sample types and diagnostic information for subsequent model training. For example, collecting samples from 500 lung cancer patients, including 300 paraffin-embedded pathological sections, 100 frozen sections, and 100 fresh surgically excised tissues, and acquiring 3-5 multiphoton images from each sample, extracting 1-2 ROIs from each image, ultimately constructs a multi-source dataset containing 2000 ROIs. Each ROI is labeled with the VPI status (invasion / non-invasion) confirmed by a pathologist.

[0028] In this embodiment, the existing collagen fiber feature extraction algorithm refers to a set of algorithms that have been publicly applied in the field of medical image analysis and are capable of quantitatively extracting collagen fiber-related features from tissue images, including morphological feature extraction algorithms, texture feature extraction algorithms, etc., which can be directly applied without the need to develop new algorithms.

[0029] In this embodiment, an image patch refers to a sub-image obtained by uniformly dividing each sample image (ROI) into sub-images of a fixed size. Each image patch contains local histological structural information. Dividing the image patch allows for the capture of more refined spatial features, improving the model's ability to identify local lesions. For example, a 200-pixel image patch can be divided into sub-images. ×200 A ROI image (corresponding to 1024×1024 pixels), according to 50 ×50 The size (corresponding to 256×256 pixels) is divided into non-overlapping parts. Each ROI image can be divided into 4×10=40 image blocks. Each image block corresponds to the collagen fiber and cell structure information of the local area.

[0030] In this embodiment, the 142-dimensional tissue features refer to the 142 quantitative feature parameters extracted from each ROI by the collagen fiber feature extraction algorithm. These features describe the structural characteristics of collagen fibers from different dimensions, including 8 morphological features (collagen area percentage, fiber density, average fiber length, average fiber width, average fiber straightness, crosslink density, average crosslink spacing and orientation) and 134 texture features (6 histogram-based features: mean, variance, skewness, kurtosis, energy and entropy; 80 GLCM features; 48 Gabor wavelet-based features).

[0031] In this embodiment, the generative adversarial network model refers to a deep learning model composed of a generator and a discriminator. Through adversarial training between the two, it can learn the distribution characteristics of data in different domains and realize the conversion of data between different domains. Here, it is used to realize style conversion of three types of images: paraffin sections, frozen sections, and fresh tissue. Specifically, a multi-generator architecture is constructed, comprising generator G1 (paraffin section → frozen section), generator G2 (paraffin section → fresh tissue), generator G3 (frozen section → paraffin section), and generator G4 (fresh tissue → paraffin section). Discriminators D1, D2, and D3 are used to determine the authenticity of frozen sections, fresh tissue, and paraffin sections, respectively. The training dataset uses ROI images from a multi-source dataset, with the image size uniformly adjusted to 256×256 pixels. The batch size is set to 8, the learning rate is 0.0002, and the Adam optimizer is used. The training iterations are 100,000. The model parameters are jointly optimized using cycle consistency loss and adversarial loss to ensure the authenticity and domain adaptability of the generated images. It should be noted that, since actual frozen and fresh tissues are relatively rare, actual frozen sections and fresh tissues are combined and converted to paraffin sections.

[0032] In this embodiment, cross-domain data augmentation refers to using a generative adversarial network (GAN) model to transform images of one type of tissue sample (such as paraffin sections) into the image style of another type of tissue sample (such as frozen sections or fresh tissue), while maintaining the histological structure and diagnostic information in the images. This achieves complementarity and expansion between samples from different domains, solving the problem of insufficient or unevenly distributed samples in a single domain. For example, 1,000 paraffin section ROI images from a multi-source dataset are input into a trained CycleGAN model to generate 1,000 frozen section and fresh tissue style images; simultaneously, 500 frozen section ROI images and 500 fresh tissue ROI images are input into the model to generate 1,000 paraffin section style images. A total of 3,000 new sample images are added through cross-domain transformation.

[0033] In this embodiment, the sample image refers to the collective image of the original multi-source dataset and the generated cross-domain transformed image obtained after cross-domain data augmentation. These images are all ROI images labeled with VPI states and are the core data for classifier training.

[0034] In this embodiment, the deep learning pre-trained model refers to a deep convolutional neural network model pre-trained on a large-scale general image dataset. Specifically, a pre-trained ResNet50 model is loaded from the torchvision library of the PyTorch framework, the parameters of the first 40 layers are frozen, and only the parameters of the last 10 convolutional and fully connected layers are fine-tuned; the images in the sample image set are adjusted to 224×224 pixels and normalized before being used as model input.

[0035] In this embodiment, the semantic feature vector refers to the high-dimensional vector obtained by extracting features from the sample image through a deep learning pre-trained model. It can characterize the spatial structure and histological semantic information of the tissue in the image. Here, it is a 1×2048-dimensional vector output by the fully connected layer of the ResNet50 model, which quantitatively characterizes the spatial distribution and interrelationship of cells and collagen fibers in the image.

[0036] In this embodiment, feature fusion refers to combining a 142-dimensional organizational feature vector and a 2048-dimensional semantic feature vector into a higher-dimensional composite feature vector through concatenation. This achieves complementarity between traditional manual feature extraction and deep learning-automated feature extraction, improving the recognition accuracy of the classification model. For example, for a sample image, its 142-dimensional organizational feature vector is [F1,F2,...,F142], and its 2048-dimensional semantic feature vector is [S1,S2,...,S2048]. Through concatenation, a composite feature vector [F1,F2,...,F142,S1,S2,...,S2048] is obtained, with a dimension of 1×2190 (142+2048=2190).

[0037] In this embodiment, the pleural invasion status classifier refers to a machine learning model built based on composite feature vectors, capable of determining the pleural invasion status (invaded / non-invaded) of a sample based on input features. It employs an instance-learning model architecture and possesses effective learning and classification capabilities for high-dimensional features. The classifier is built on the PyTorch framework and includes a feature encoding module (using a 3-layer fully connected network to map 2190-dimensional composite features to a 512-dimensional low-dimensional embedding space), an attention weighting module (a single-layer neural network that takes 512-dimensional embedding features as input and outputs instance weights in the 0-1 interval, with a sigmoid activation function), and a classification module (a single-layer fully connected network that takes 512-dimensional comprehensive features as input and outputs VPI positive / negative binary classification probabilities, with a softmax activation function). During training, a total loss function is composed of classification loss (cross-entropy loss) and clustering regularization loss (which clusters similar instance features and separates dissimilar instance features). The three-layer fully connected network is as follows: First layer: Input 2190 dimensions → Output 1024 dimensions, activation function ReLU, dropout rate 0.3; Second layer: Input 1024 dimensions → Output 512 dimensions, activation function ReLU, dropout rate 0.3; Third layer: Input 512 dimensions → Output 512 dimensions, activation function ReLU.

[0038] The labeled dataset refers to a dataset containing sample images and their corresponding labels for pleural invasion status. The labels are jointly determined by two or more physicians with more than five years of experience in pathological diagnosis based on the results of postoperative EVG staining.

[0039] In this embodiment, the portable multiphoton microscopy system refers to a device that is small in size, easy to move and deploy during surgery, and can quickly acquire multiphoton images of fresh tissue or frozen sections at the surgical site. Its core imaging principle is the same as that of the laboratory multiphoton microscopy system.

[0040] In this embodiment, real-time images refer to multiphoton images of fresh tissue or frozen sections acquired on-site using a portable multiphoton microscopy system during lung cancer surgery, without any post-processing. The images contain real-time histological structural information of the pleural boundary region.

[0041] In this embodiment, predicting the VPI positive / negative status involves inputting the real-time ROI image extracted during surgery into a trained classification model. The model infers the corresponding VPI status (negative = no invasion, positive = invasion) of the sample. The inference process is fast and automated, requiring no manual interpretation. For example, the real-time ROI image extracted during surgery is preprocessed according to the training dataset (size adjusted to 224×224 pixels, normalized), and then sequentially input into the tissue feature extraction algorithm package to obtain a 142-dimensional tissue feature vector. This 142-dimensional feature vector is then input into the fine-tuned ResNet50 model to obtain a 2048-dimensional semantic feature vector. These vectors are concatenated into a 2190-dimensional composite feature vector, which is then input into the trained classifier. The model outputs the VPI positive probability (range 0-1). If the probability is ≥0.5, the sample is considered VPI positive; otherwise, it is considered VPI negative.

[0042] Outputting diagnostic results refers to presenting the positive or negative VPI prediction results to surgeons in an intuitive and easy-to-understand format, including visualization and simplified report output. For example, the tablet interface uses red and green to indicate the prediction results (red for positive VPI, green for negative VPI), and displays the confidence level as a percentage (e.g., 92% confidence level for positive VPI). At the same time, the diagnostic report (PDF format) is automatically saved locally and transmitted to the electronic medical record system via the hospital intranet.

[0043] The beneficial effects of the above technical solution are as follows: the auxiliary diagnostic system constructed through multi-module collaboration enables rapid, accurate, and label-free intraoperative diagnosis of pleural invasion in lung cancer; multi-source datasets provide a rich sample foundation for the model; the fusion of 142-dimensional tissue features and deep learning semantic features improves classification accuracy; cross-domain data enhances the model's generalization ability; and the portable system and rapid inference ensure real-time intraoperative application. It effectively solves the shortcomings of traditional diagnostic techniques, such as low accuracy, long time consumption, and strong subjectivity, and can provide surgeons with real-time decision support, optimize the surgical resection range, avoid under-treatment or over-resection, and improve patient prognosis.

[0044] This invention provides an auxiliary diagnostic system for pleural invasion in lung cancer based on multiphoton microscopy, wherein the cross-domain data enhancement module includes: The first cross-domain unit is used to input the multiphoton image of the pathological paraffin section into the generative adversarial network model and generate frozen section style images and fresh tissue style images. The second cross-domain unit is used to input multiphoton images of frozen sections and surgically removed fresh tissue into the generative adversarial network model to generate paraffin section-style images. Among them, frozen section style images, fresh tissue style images, and paraffin section style images are sample images.

[0045] In this embodiment, the paraffin section style image has a stronger TPEF signal and a weaker SHG signal, and is clearer overall; the frozen section style image and the fresh tissue style image are the opposite.

[0046] In this embodiment, the first cross-domain unit corresponds to the generators G1 (paraffin section → frozen section) and G2 (paraffin section → fresh tissue) in the CycleGAN model. During training, the paraffin section image is used as the source domain, and the frozen section and fresh tissue images are the target domains, respectively. Cyclic consistency loss is used to ensure that the style of the generated image is consistent with that of the target domain image. At the same time, perceptual loss (based on the loss calculated by extracting features from the VGG16 model) is used to ensure that the histological structure of the generated image is consistent with that of the source domain image. The size of the generated image is consistent with that of the input paraffin section image (1024×1024 pixels).

[0047] In this embodiment, the generator G2 is trained using the same optimization strategy as G1.

[0048] In this embodiment, the generators G3 and G4 of the second cross-domain unit have the same architecture as the generator of the first cross-domain unit (both use residual networks as the backbone network and contain 6 residual blocks). During training, frozen sections and fresh tissue images are used as the source domains and paraffin sections as the target domains, respectively. The same cycle consistency loss and perceptual loss are used for joint optimization. After the generated images undergo the same quality screening process, they together with the images generated by the first cross-domain unit to achieve a comprehensive expansion of the three types of tissue images.

[0049] The beneficial effects of the above technical solution are as follows: through bidirectional cross-domain conversion between the first cross-domain unit and the second cross-domain unit, comprehensive style transfer is achieved among the three types of images: pathological paraffin sections, frozen sections, and fresh tissues. This effectively expands the quantity and diversity of sample images, solves the problem of insufficient samples of a single type, and ensures that the generated images retain core histological structural information and diagnostic features. This provides richer and more balanced training data for subsequent classification models, further improving the generalization ability and diagnostic accuracy of the models.

[0050] This invention provides an auxiliary diagnostic system for pleural invasion in lung cancer based on multiphoton microscopy, wherein the supervised training unit includes: A subunit is defined to use the region of interest as a sample bag and the composite feature vector corresponding to the image patch within the region of interest as an instance; The low-dimensional mapping subunit is used to map the concatenated composite feature vector to a low-dimensional embedding space. The weighted subunit is used to weight and aggregate the low-dimensional embedded features according to the importance weight of each instance to the diagnosis, so as to obtain the comprehensive features. The state output subunit is used to output the pleural invasion state based on comprehensive features. During training, a total loss function combining classification loss and clustering regularization loss is used, and end-to-end model optimization is completed based on an adaptive optimization algorithm.

[0051] In this embodiment, the region of interest (ROI) is used as a sample bag. In the instance learning model, each ROI is treated as an independent sample bag, and the label of the sample bag is the VPI state label (positive / negative) corresponding to that ROI. The training and inference of the model are based on sample bags as the basic unit, and the classification results are output at the sample bag level. For example, one VPI negative ROI (containing 40 image patches) in the training set is defined as one sample bag, labeled as VPI negative; one VPI positive ROI (containing 40 image patches) is defined as another sample bag, labeled as VPI positive. The model learns the feature patterns of multiple sample bags to predict the label of unknown sample bags.

[0052] In this embodiment, the composite feature vector corresponding to an image patch within a region of interest is considered an instance. This means that the 2190-dimensional composite feature vector corresponding to each image patch within a sample package is treated as an instance. Each instance does not have an independent label; its feature pattern is associated with the overall label of the sample package. The model classifies the sample package by learning the association between instances and the sample package label. For example, in a VPI-positive sample package with 40 instances, the feature patterns of some instances (corresponding to areas where tumor cells invade the pleura) are highly correlated with VPI positivity. These instances are identified by the model as key instances, and their features play a dominant role in the classification result of the sample package. The feature patterns of the remaining instances (corresponding to normal tissue areas) are less correlated with VPI positivity and have a smaller impact on the classification result.

[0053] In this embodiment, mapping to a low-dimensional embedding space refers to converting a high-dimensional composite feature vector (2190 dimensions) into a low-dimensional embedding vector (with dimensions much lower than 2190 dimensions) through a feature encoding module. The low-dimensional embedding space can more efficiently represent the core features of an instance while removing redundant information. For example, the 2190-dimensional composite feature vector is mapped to a 512-dimensional low-dimensional embedding space through a 3-layer fully connected network. The embedding vector retains the key information related to the VPI state in the original composite features (such as collagen fiber breakage features and tumor cell infiltration features) while removing redundant information such as image noise and irrelevant structures.

[0054] In this embodiment, the importance weight of each instance to the diagnosis refers to a numerical value (range 0-1) calculated through an attention mechanism, reflecting the contribution of each instance to the diagnosis of the VPI status of the sample package. Key instances (such as image patches corresponding to the invaded area) have a weight close to 1, while non-key instances (such as image patches corresponding to normal tissue) have a weight close to 0. For example, in a VPI-positive sample package with 40 instances, after the attention mechanism calculation, the weights of the 5 instances corresponding to tumor cells breaching the pleural elastic layer are 0.95, 0.92, 0.88, 0.85, and 0.81, respectively, while the weights of the remaining 35 instances are all below 0.3, indicating that these 5 instances are key to diagnosing the sample package as VPI-positive.

[0055] In this embodiment, the output of pleural invasion status based on comprehensive features refers to inputting the comprehensive feature vector of the sample package into the classification module to obtain the VPI status prediction result (positive / negative) corresponding to the sample package. The prediction result is output in the form of probability, which is convenient for doctors to refer to the confidence level. The probability value is the confidence level.

[0056] In this embodiment, the classification loss uses cross-entropy loss, which calculates the difference between the model's predicted probability and the true label; the clustering regularization loss uses contrastive loss, which calculates the similarity of instance features within the same sample bag and the difference of instance features across different sample bags. The total loss = classification loss + 0.1 × clustering regularization loss (0.1 is the regularization coefficient). The classification loss function uses PyTorch's built-in CrossEntropyLoss; the clustering regularization loss is implemented through a custom function, calculating the first average distance between all instance features within the same sample bag and the second average distance between instance features across different sample bags. The loss value = second average distance - first average distance; the coefficient of the total loss function is adjusted based on validation set performance, and the final weight coefficient of the clustering regularization loss is determined to be 0.1.

[0057] In this embodiment, AdamW is selected as the adaptive optimization algorithm. During training, the algorithm adaptively adjusts the learning rate according to the gradient information of each parameter. For parameters with large gradients (such as parameters of the classification module), a smaller learning rate is used for updating, and for parameters with small gradients (such as deep parameters of the feature encoding module), a larger learning rate is used for updating, ensuring that the model converges quickly and is not prone to overfitting.

[0058] The beneficial effects of the above technical solution are as follows: by setting sub-units to clarify the basic data structure of instance learning, the low-dimensional mapping sub-unit achieves effective dimensionality reduction of high-dimensional features, the weighted sub-unit focuses on key instances through the attention mechanism, and the state output sub-unit adopts a joint loss function and adaptive optimization algorithm to achieve end-to-end optimization, thus refining the training process of the instance learning model. This enables the model to more accurately capture key features related to pleural invasion, significantly improve the diagnostic accuracy and generalization ability of the classifier, and at the same time reduce the computational complexity of the model, ensuring the feasibility of rapid intraoperative reasoning.

[0059] This invention provides an auxiliary diagnostic system for pleural invasion in lung cancer based on multiphoton microscopy, wherein the cross-domain data enhancement module further includes: The point determination unit performs self-enhancing operations (EEAs) with angle transformation, scale scaling, and noise perturbation on multiphoton images of pathological paraffin sections, frozen sections, and surgically excised fresh tissue, respectively, to obtain EEAs. It then performs region feature registration analysis on the three types of EEAs from the same lesion site, calculates the region grayscale matching degree and tissue texture overlap, and divides the enhanced point block sets at different scales. Where L is the number of scale levels; Let j be the number of points at the j-th scale. Let be the three-dimensional coordinates of the k-th point at the j-th scale; The first candidate unit is used for point blocks at each scale level j. The first candidate sample is generated based on the corresponding original image region. ,Will The input generative adversarial network model performs a complete cross-domain transformation, yielding cross-domain candidate samples corresponding to two other image types. ; The coefficient calculation unit is used for... and Perform multi-dimensional structural verification and calculate the enhancement effectiveness coefficient of the corresponding point blocks. ; ; in, for and Feature vector matching degree; For the feature encoding function of the pre-trained model; for The regularity of the three-dimensional structure; , The weighting coefficients are satisfied. ; like Then determine the corresponding point block. corresponding and As valid samples, they are retained, These are preset coefficients; The point mapping unit is used to establish point block mapping associations for frozen sections and surgically removed fresh tissue images, based on effective enhanced point blocks at various scales of paraffin section images. , The multi-scale point mapping topology is obtained based on the intersection operation. ; The second candidate unit is used to construct a second candidate sample based on the mapping topology R, using the point blocks of the effective samples as the reference unit. ; The completion unit is used to include all second candidate samples. The input generative adversarial network model performs a complete cross-domain transformation to obtain cross-domain completion samples. Then, the effective first candidate samples, cross-domain candidate samples, and cross-domain completion samples at all scales are integrated into the sample image set to achieve sample expansion.

[0060] In this embodiment, the core objective is to maximize the feature vector matching degree of cross-domain samples. This is because the feature vector matching degree directly determines whether cross-domain samples retain the original tissue's key diagnostic information, which is the foundation for subsequent accurate training of the classifier. In reality, only a single core factor needs to be addressed. Set the level, Depend on Automatic derivation, based on 1 factor ( The experimental design with four levels, combined with the advantages of orthogonal experiments (fewer trials and higher representativeness), adopts the L4 orthogonal array, meaning that four experiments cover four levels, eliminating the need for full factorial experiments. Each group / In the combined dataset, 100 cross-domain samples (paraffin → frozen, paraffin → fresh tissue) were selected from the multi-source dataset. The feature vector matching degree of each group of samples is calculated based on cosine similarity, and the three-dimensional structural regularity of each group of samples is calculated through spatial topological matching degree. The average feature vector matching degree (core evaluation metric) of each combination was recorded, and the results are shown in Table 1: Table 1 Experimental Results Through the above experimental comparisons , The matching degree is highest when the value is 0.6 and 0.4 respectively, and the three-dimensional structural regularity still meets the requirement of no distortion in cross-domain sample structure.

[0061] In this embodiment, the self-enhanced sub-image refers to the image generated by the self-enhancing operation that originates from the same lesion site as the original image, but after geometric transformation or noise perturbation, the collagen fibers in the original paraffin section image are horizontally distributed, while in the self-enhanced sub-image after 90° rotation, the collagen fibers are vertically distributed. In the self-enhanced sub-image after scaling by 1.2 times, the collagen fiber details are clearer. After noise analysis, the self-enhanced sub-image is obtained, but the structure of collagen fibers and cells can still be clearly identified.

[0062] In this embodiment, the regional feature registration analysis is implemented using Python's OpenCV library. First, the SIFT algorithm is used to extract feature points and descriptors of three types of self-enhancing subgraphs. Then, the FLANN matcher is used to match feature points and filter out feature point pairs with a matching degree higher than 0.7. The regional grayscale matching degree is calculated using the normalized cross-correlation coefficient (NCC, with a value range of -1 to 1, the closer to 1, the higher the matching degree). The tissue texture overlap degree is calculated using the correlation parameter of the grayscale co-occurrence matrix (with a value range of -1 to 1, the closer to 1, the higher the overlap degree). Only regions with NCC ≥ 0.6 and texture overlap degree ≥ 0.5 are retained as valid registration regions.

[0063] In this embodiment, based on the size of the effective registration region, it is assumed to be 400. ×600 The corresponding pixel size is 2048×3072. The blocks are divided according to the scale level. The first scale (256×256 pixels) is divided in a non-overlapping manner, while the second scale (128×128 pixels) and the third scale (64×64 pixels) are divided in an overlapping manner (overlap rate of 50%) to ensure the continuity of point coverage. The three-dimensional coordinates of each point are determined by the pixel position and the scale level number. The x and y coordinates are the center pixel position of the block, and the z coordinate is equal to the scale level j. All point coordinates are stored in a CSV file to form an enhanced point block set.

[0064] In this embodiment, the first candidate sample refers to an image sample cropped from the original image region corresponding to each point block in the enhanced point block set. Each first candidate sample corresponds to a histological region of a specific point block, and its size is consistent with the block size of the corresponding scale level. For example, the original image region corresponding to the 10th point block of the second scale level is a 128×128 pixel region (center coordinates x=384, y=512) in the frozen section self-enhanced subimage. The first candidate sample is obtained by cropping this region. The sample contains information on the collagen fibers and cell structure corresponding to that location.

[0065] Complete cross-domain transformation refers to inputting the first candidate sample into a generative adversarial network model and converting it into the style of two other image types. This ensures that the transformed cross-domain candidate sample is stylistically consistent with the target image type while preserving the core histological structure of the original image regions. For example, the first candidate sample generated from a self-enhanced subimage of a paraffin section. (Scale 1, Point 3) Input to the generative adversarial network model, and convert it into cross-domain candidate samples of frozen section style and fresh tissue style through generators G1 and G2 respectively. These two cross-domain candidate samples have the image styles of frozen section and fresh tissue respectively, and are consistent with the histological structure of the original first candidate sample.

[0066] Cross-domain candidate samples refer to synthetic samples obtained through complete cross-domain transformation. These samples originate from the same location block as the first candidate sample but have different image types. Each first candidate sample corresponds to two cross-domain candidate samples (if the original sample is a paraffin section, it corresponds to a frozen section or fresh tissue; if the original sample is either a frozen section or fresh tissue, it corresponds to a paraffin section). For example, the original first candidate sample is fresh tissue (scale 3, location 20), which is converted into a paraffin section-style cross-domain candidate sample using generator G4. This sample corresponds to the same tissue location as the original sample, with the same core structure, but the style is consistent with the paraffin section image.

[0067] In this embodiment, multi-dimensional structural verification refers to verifying the structural consistency between the first candidate sample and the cross-domain candidate sample from two dimensions: feature vector matching degree and three-dimensional structural regularity. This ensures that the cross-domain transformation does not change the core histological structure, but only alters the image style. For example, for the first candidate sample... With the corresponding cross-domain candidate samples Structural verification is performed by calculating the feature vector matching degree (reflecting the consistency at the feature level) and the three-dimensional structural regularity (reflecting the consistency of the spatial structure). If both indicators meet the requirements, the structural verification is deemed successful.

[0068] The three-dimensional structural regularity is calculated by the average similarity of the three-dimensional structural parameters (such as depth, volume, surface area, etc.) between the cross-domain candidate sample and the first candidate sample.

[0069] In this embodiment, the point block mapping association refers to establishing a one-to-one correspondence between valid point blocks in paraffin section images and valid point blocks in frozen sections and fresh tissue images based on the point information of the registered valid region and valid samples. This ensures that blocks of the same tissue point in different types of images can be correlated with each other. For example, the 15th valid point block at the second scale level of a paraffin section image. By performing regional feature registration analysis, the corresponding effective location blocks in the frozen section image were found. Establish mapping association (Including the coordinate correspondence of the point block), similarly, establish a mapping association with the point blocks in the fresh tissue image. .

[0070] In this embodiment, the multi-scale point mapping topology R is obtained by... and The topological structure obtained by performing intersection operations contains the common correspondences of effective location blocks in three types of images: paraffin sections, frozen sections, and fresh tissue. This ensures that blocks of the same tissue location in the three types of images can be accurately matched. For example, Contains 100 mapping associations, It contains 95 mapping associations, and the ice intersection operation yields 88 common mapping associations, meaning that these 88 point blocks have accurate corresponding valid samples in all three types of images.

[0071] In this embodiment, the reference unit refers to the effective point block (i.e., the reference point block) and its corresponding effective samples that are common to the three types of images in the multi-scale point mapping topology R. Typically, the effective point block of a paraffin section image is selected as the reference unit because its histological structure is more completely preserved and its stability is higher. For example, the reference unit in the multi-scale point mapping topology R is selected as (paraffin point block B → ice point block B → new point block B!), where the effective sample corresponding to the paraffin point block B is the first candidate sample. The cross-domain candidate samples corresponding to the freezing point location block B are: The new location, block B, has the following cross-domain candidate samples: ,by Construct a second candidate sample for the baseline unit. It should be noted that the selection of the baseline unit prioritizes effective samples with an enhanced validity coefficient ≥ 0.85, where 0.85 is obtained based on the validation set, that is, when K0 is 0.85, the effective sample retention rate is 80% or higher.

[0072] Based on the valid samples of wax spot block B, sub-features of the three types of samples are extracted and weighted. Spatial structure consistency is ensured through constraint alignment, and then feature weighted fusion is performed to generate the second candidate sample. This sample retains the structural stability of paraffin section samples while incorporating the detailed features of frozen sections and fresh tissue samples, making it more suitable as a training sample for classification models.

[0073] In this embodiment, cross-domain completion samples refer to inputting the second candidate sample into the generative adversarial network model, converting it into samples of styles from the other two image types, and using these samples to complete the missing valid samples in the three image types, ensuring a balanced number of images from the three types in the sample set. For example, the second candidate sample... (Based on paraffin section baseline units) Input generators G1 and G2, which are converted into cross-domain completion samples in frozen section style and fresh tissue style, respectively. and Complete the frozen section and fresh tissue sample corresponding to the location block; if the second candidate sample is constructed based on the frozen section reference unit, it is converted into a paraffin section style cross-domain completion sample to complete the paraffin section sample.

[0074] In this embodiment, integration into the sample image set refers to merging all valid first candidate samples, cross-domain candidate samples, and cross-domain completion samples that have passed quality screening with the original multi-source dataset and the generated cross-domain samples to form the final expanded sample image set.

[0075] The beneficial effects of the above technical solution are as follows: the addition of six units enables more refined and higher-quality cross-domain data augmentation; the point determination unit ensures the accuracy and multi-scale coverage of the augmented points; the first candidate unit and the coefficient calculation unit select high-quality initial cross-domain samples; the point mapping unit establishes a precise correspondence between the three types of images; the second candidate unit generates high-quality samples that integrate the advantages of multiple types of images; and the completion unit achieves a comprehensive expansion of samples, effectively improving the diversity, balance, and quality of samples, providing higher-quality training data for the classification model, further enhancing the model's generalization ability and diagnostic accuracy, while ensuring the structural integrity and diagnostic relevance of the cross-domain samples.

[0076] This invention provides an auxiliary diagnostic system for pleural invasion in lung cancer based on multiphoton microscopy, wherein the second candidate unit includes: The sub-feature extraction subunit is used to divide the point-block feature vector output by the pre-trained histological feature extraction function into several sub-feature vectors related to lung cancer pleural invasion, including the collagen fiber spatial distribution feature sub-vector. Cell boundary structure feature vector Spatial feature vectors of tissue interstitial space ; The weighting subunit is used to assign weight coefficients based on the contribution of each sub-feature to the determination of pleural invasion status. , , The weighted histological feature distances between corresponding points on paraffin-embedded sections and frozen sections, and between paraffin-embedded sections and freshly excised tissue, were calculated for the same lesion site. ; filter The point blocks form a set of associated units; The constraint alignment sub-unit is used to apply spatial topological constraints to the point blocks corresponding to frozen sections and surgically removed fresh tissue, based on the point correspondence relationship of the pathological paraffin section point blocks in the associated unit set as the reference unit, and to obtain the topologically aligned block combination. An initial generation sub-unit is used to perform feature weighted fusion on the topologically aligned block combination to obtain fused features, and to generate an initial second candidate sample based on the fused features. The final sub-unit is used to calculate the spatial relative position deviation between each point block in the initial second candidate sample and the corresponding block in the original image. If the spatial relative position deviation is less than or equal to a preset deviation, the initial second candidate sample is determined to be the final second candidate sample.

[0077] In this embodiment, ,and Collagen fiber characteristics are a key indicator for determining pleural invasion.

[0078] In this embodiment, ,in, These are the corresponding locations for pathological paraffin sections, frozen sections, and surgically removed fresh tissue.

[0079] In this embodiment, the pre-trained histological feature extraction function refers to a function that has been pre-trained on a large-scale medical tissue image dataset and is capable of extracting feature vectors related to histological attributes from point block images. It can be used directly without additional training.

[0080] In this embodiment, the spatial distribution feature subvector of collagen fibers This refers to the feature vector obtained by splitting it from the histological feature vector of the point block. The feature dimensions in this sub-vector include related parameters such as the area ratio of collagen fibers, pixel density, and distribution density.

[0081] Cell boundary structure feature vector This refers to the feature vector obtained by splitting it from the histological feature vector. The feature dimensions in this sub-vector include edge strength, gradient magnitude, contour smoothness, etc.

[0082] Spatial feature vectors of tissue gaps This refers to the feature vector obtained by splitting it from the histological feature vector. The feature dimensions in this sub-vector include the proportion of interstitial area, the number of interstitials, and the average size of interstitials.

[0083] In this embodiment, a preset threshold is used. This refers to a threshold value used to judge the similarity of histological features between two loci. A threshold less than or equal to this value indicates that the histological features of two loci are relatively similar, with little difference between them. The preset threshold is determined using validation set data. This embodiment sets... =1.2.

[0084] In this embodiment, spatial topological constraint alignment refers to adjusting the spatial positions of frozen sections and fresh tissue sites based on the point correspondence in the mapped topology R through geometric transformations (such as translation, rotation, and scaling). This ensures that the histological structures (such as the orientation of collagen fibers and the distribution of cells) in the three types of sites are precisely aligned spatially, maintaining consistent topological relationships. For example, in the associated unit, the collagen fibers in the paraffin section site are distributed in a V-shape, while the collagen fibers in the frozen section site are distributed in an inverted V-shape due to imaging angle deviation. Through spatial topological constraint alignment, a 180° rotation and translation operation is performed on the frozen section site to make its collagen fiber distribution consistent with the V-shaped topological structure of the paraffin section site.

[0085] In this embodiment, the topologically aligned block assembly refers to an assembly composed of three types of point blocks—paraffin sections, frozen sections, and fresh tissue—after spatial topological constraint alignment. For example... , , We sequentially obtain vectors of different classes, calculate the average value of each class vector, and then compare it with the corresponding... , , The weighted multiplication yields the fusion vector.

[0086] In this embodiment, the spatial relative position deviation refers to the average difference in spatial coordinates between each pixel in the initial second candidate sample and the corresponding pixel in the original image (paraffin slice point block). It reflects the spatial consistency between the generated sample and the original image; the smaller the deviation, the more consistent the spatial structure. For example, if the coordinates of a pixel in the initial second candidate sample are (x1, y1), and the coordinates of the corresponding pixel in the original paraffin slice block are (x0, y0), then the position deviation of this pixel is calculated. The spatial relative position deviation is obtained by calculating the position deviations of all pixels and taking the average.

[0087] In this embodiment, if the spatial relative position deviation of an initial second candidate sample is 0.8 pixels (≤1 pixel), it is determined to be the final second candidate sample; if the deviation of another initial sample is 1.5 pixels (>1 pixel), after re-topological alignment, the deviation is reduced to 0.9 pixels, and it is finally determined to be the final second candidate sample. The preset deviation value is 1 pixel, which depends on the spatial topological alignment accuracy requirement to ensure that the position deviation of the tissue structure is within the clinically acceptable range.

[0088] The beneficial effects of the above technical solution are as follows: it achieves refined feature decomposition through sub-feature extraction sub-units, highlights the contribution of key features through weight allocation sub-units, ensures sample similarity by screening related unit sets, achieves spatial structure consistency by constraining aligned sub-units, integrates the advantages of three types of samples in the initial generation sub-units, ensures sample quality by finally determining sub-units, refines the construction process of the second candidate sample, and generates a second candidate sample with higher structural integrity and diagnostic relevance, further improving the quality of cross-domain data, providing higher quality training samples for the classification model, and helping to improve the diagnostic accuracy of the model.

[0089] This invention provides an auxiliary diagnostic system for pleural invasion in lung cancer based on multiphoton microscopy, wherein the location determination unit includes: The encoding subunit is used to obtain the space-spectral fusion feature vector of each pixel in the multiphoton image. and will The input encoding grid is used to perform the first encoding, and the encoding formula is: ,in, Encode grids for pixel features; , These are the weight matrix and bias term of the grid, respectively; Use the Sigmoid activation function; The first encoded feature of the i-th pixel; The deviation vector acquisition subunit is used to obtain the first encoded feature of all pixels. Perform adaptive density clustering to obtain the feature center deviation vector for each pixel. ,in, For the first The feature centers of each cluster are formed by all clusters within that cluster. The average value is obtained; Noise extraction subunit, used for each cluster Extracting basic noise features from all feature center deviation vectors. And generate the initial adjustment vector for the corresponding cluster. ; The influence calculation subunit is used to set the pixel adjustment order based on the spatial location correlation between all initial adjustment vectors and the first encoded feature, and to calculate the influence weight of the image adjustment operation effect set before N1 on N1 for the pixel with the current index N1. ; ,in, These are the spatial coordinates of the t-th pixel and the pixel with the current index N1, respectively, and N1 is greater than 1; Cosine similarity of spatial coordinates; To maintain efficiency for the features of the t-th order adjustment operation, and ,in, The t-th order adjusted coding features; The encoding features before the t-th order adjustment; Parameterized fine-tuning subunit, used to adjust according to the influence weights Perform parameterized fine-tuning on the initial adjustment vector for the current index N1: ,in, For fine-tuning the bias term; This is the fine-tuned vector; Feature modulation subunit, used to... Feature modulation is performed on the image after N1-1 initial adjustment vector operations: ,in, The modulation coefficient; Image processing subunit, used to... The image is processed using the LeakyReLU activation function to obtain the currently adjusted image. This process continues until all pixel adjustments are complete, at which point a self-enhancing subimage is obtained. , This is the LeakyReLU activation function.

[0090] Preferably, the noise extraction subunit includes: ,in, To adjust the vector generation network; , They are respectively The weight matrix and bias terms; This is the ReLU activation function.

[0091] In this embodiment, the spatial-spectral fusion feature vector This refers to a high-dimensional vector formed by fusing the spatial coordinates (x, y, z) and spectral feature information (SHG signal intensity, TPEF signal intensity) of each pixel. It comprehensively characterizes the spatial location and histological properties of each pixel. For example, if a pixel has spatial coordinates (512, 256, 1) (x = 512 pixels, y = 256 pixels, z = 1 is the image layer number), an SHG signal intensity of 200 (grayscale value, 0-255), and a TPEF signal intensity of 150, then the spatial-spectral fusion feature vector of that pixel is... =(512,256,1,200,150), with a dimension of 5.

[0092] In this embodiment, for example, the encoding grid is a single-layer fully connected network with 5 input dimensions and 10 output dimensions. Given a 10×5 matrix, randomly initialized values ​​between [-0.1, 0.1]. A 10×1 vector (initialized to 0); a pixel =(512,256,1,200,150), calculated after encoding A 10-dimensional vector is obtained, which is then mapped to the [0,1] interval using the Sigmoid activation function to obtain the first encoded feature. =(0.62,0.58,0.71,0.45,0.69,0.53,0.75,0.41,0.67,0.59). In this embodiment, the DBSCAN algorithm is used to perform adaptive density clustering on the 1024×1024 pixel encoded feature matrix. In this embodiment, the feature center deviation vector refers to the difference vector between the first encoded feature of each pixel and the feature center of its cluster, which can reflect the degree of deviation of the pixel from the features of pixels of the same type. The deviation vector is calculated for each cluster. Feature center That is, the first encoded feature of all pixels within the cluster. The mean (averaged over feature dimensions) is then calculated for each pixel. and The difference between the values ​​is used to obtain the feature center deviation vector, and the deviation vectors of all pixels form the deviation vector matrix.

[0093] In this embodiment, the basic noise characteristics This refers to the feature vector extracted from the feature center deviation vector of each cluster, which characterizes the random fluctuations (noise) of pixel features within that cluster. It is obtained by calculating the statistical characteristics of the deviation vector. For example, the first... The feature center deviation vector matrix (10000×10) of the =1 cluster is used to calculate the standard deviation, maximum value, and minimum value of each dimension, thus obtaining the basic noise features. =(0.02,0.03,0.025,0.03,0.02,0.028,0.022,0.031,0.024,0.026) (Each element is the standard deviation of the corresponding dimension).

[0094] In this embodiment, the spatial location correlation setting of pixel adjustment order refers to calculating the spatial distance between each pixel and the center pixel of the image, sorting the pixels by distance from smallest to largest to determine the pixel adjustment order, with pixels closer to each other adjusted first, ensuring the spatial continuity of the adjustment operation. For example, if the coordinates of the center pixel of the image are (512, 512), the coordinates of pixel A are (512, 512) (distance 0), the coordinates of pixel B are (512, 513) (distance 1), and the coordinates of pixel C are (510, 510), then the adjustment order is A→B→C, arranged in ascending order of spatial distance.

[0095] In this embodiment, .

[0096] In this embodiment, the value of the fine-tuning bias term is 0.001 to avoid overfitting of the adjustment vector and improve the diversity of the self-enhancing subgraph. The value of the modulation coefficient is 0.95 to balance the influence of the original image features and the adjustment vector and ensure that the feature preservation efficiency is ≥0.9.

[0097] The beneficial effects of the above technical solution are as follows: pixel-level feature representation is achieved through the coding subunit; the deviation vector acquisition subunit accurately divides tissue clusters; the noise extraction subunit captures effective noise features; the impact calculation subunit dynamically evaluates and adjusts the impact; the parameterized fine-tuning subunit adapts to pixel adjustment requirements; the feature modulation subunit achieves precise feature adjustment; the image processing subunit optimizes the output of the self-enhanced subgraph; the generation process of the self-enhanced subgraph is refined; the generated self-enhanced subgraph can more realistically simulate tissue images under different imaging conditions, improve the effectiveness and sample diversity of self-enhancing operations, and provide high-quality basic samples for subsequent cross-domain data enhancement.

[0098] This invention provides an auxiliary diagnostic method for pleural invasion in lung cancer based on multiphoton microscopy, such as... Figure 2 As shown, it includes: Step 1: Collect multiphoton images of unstained paraffin sections, frozen sections, and freshly excised tissue from lung cancer patients using a multiphoton microscopy system. Select the image at the pleural boundary as the region of interest to construct a multi-source dataset. Step 2: Using existing collagen fiber feature extraction algorithms, analyze each region of interest in the multi-source dataset to obtain 142-dimensional tissue features; Step 3: Using a generative adversarial network model, cross-domain data augmentation is performed between multiphoton images of pathological paraffin sections, frozen sections, and surgically removed fresh tissue to obtain several sample images; Step 4: Fuse the high-dimensional tissue features with the semantic feature vectors of the sample images extracted based on the deep learning pre-trained model to construct a pleural invasion state classifier, and complete supervised training on the labeled dataset; Step 5: During the operation, real-time images are acquired using a portable multiphoton microscope system. The real-time images are then input into the trained classification model to predict the VPI positive or negative status and output the diagnostic results.

[0099] The beneficial effects of the above technical solution are as follows: the auxiliary diagnostic system constructed through multi-module collaboration enables rapid, accurate, and label-free intraoperative diagnosis of pleural invasion in lung cancer; multi-source datasets provide a rich sample foundation for the model; the fusion of 142-dimensional tissue features and deep learning semantic features improves classification accuracy; cross-domain data enhances the model's generalization ability; and the portable system and rapid inference ensure real-time intraoperative application. It effectively solves the shortcomings of traditional diagnostic techniques, such as low accuracy, long time consumption, and strong subjectivity, and can provide surgeons with real-time decision support, optimize the surgical resection range, avoid under-treatment or over-resection, and improve patient prognosis.

[0100] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A lung cancer pleural invasion auxiliary diagnostic system based on multiphoton microscopy, characterized in that, include: The dataset construction module is used to collect multiphoton images of unstained paraffin sections, frozen sections and fresh surgically removed tissues from lung cancer patients based on a multiphoton microscopy system, and selects images at the pleural boundary as the region of interest to construct a multi-source dataset. The tissue feature extraction module is used to analyze each region of interest in the multi-source dataset using existing collagen fiber feature extraction algorithms to obtain 142-dimensional tissue features. The cross-domain data augmentation module is used to perform cross-domain data augmentation between multiphoton images of pathological paraffin sections, frozen sections, and surgically removed fresh tissue using a generative adversarial network model, resulting in several sample images. The classifier building module is used to fuse the high-dimensional tissue features with the semantic feature vectors of sample images extracted based on a deep learning pre-trained model to build a classifier for pleural invasion status, and to complete supervised training on a labeled dataset. The result prediction module is used to acquire real-time images during surgery using a portable multiphoton microscope system, input the real-time images into a trained classification model, predict the VPI positive or negative status, and output the diagnostic result.

2. The lung cancer pleural invasion auxiliary diagnostic system based on multiphoton microscopy as described in claim 1, characterized in that, The cross-domain data enhancement module includes: The first cross-domain unit is used to input the multiphoton image of the pathological paraffin section into the generative adversarial network model and generate frozen section style images and fresh tissue style images. The second cross-domain unit is used to input multiphoton images of frozen sections and surgically removed fresh tissue into the generative adversarial network model to generate paraffin section-style images. Among them, frozen section style images, fresh tissue style images, and paraffin section style images are sample images.

3. The lung cancer pleural invasion auxiliary diagnostic system based on multiphoton microscopy as described in claim 1, characterized in that, The classifier construction module includes: Semantic units are used to input image patches of each sample image into a pre-trained model and output a 2048-dimensional semantic feature vector representing spatial structure and histological information. The splicing unit is used to splice the semantic feature vector with the 142-dimensional organizational features to form a composite feature vector; A supervised training unit is used to supervise the training of the composite feature vector to obtain a pleural invasion state classifier.

4. The lung cancer pleural invasion auxiliary diagnostic system based on multiphoton microscopy as described in claim 3, characterized in that, The supervised training unit includes: A subunit is defined to use the region of interest as a sample bag and the composite feature vector corresponding to the image patch within the region of interest as an instance; The low-dimensional mapping subunit is used to map the concatenated composite feature vector to a low-dimensional embedding space. The weighted subunit is used to weight and aggregate the low-dimensional embedded features according to the importance weight of each instance to the diagnosis, so as to obtain the comprehensive features. The state output subunit is used to output the pleural invasion state based on comprehensive features. During training, a total loss function combining classification loss and clustering regularization loss is used, and end-to-end model optimization is completed based on an adaptive optimization algorithm.

5. The method for auxiliary diagnosis of pleural invasion in lung cancer based on multiphoton microscopy according to claim 1, characterized in that, The cross-domain data enhancement module also includes: The point determination unit performs self-enhancing operations (EEAs) with angle transformation, scale scaling, and noise perturbation on multiphoton images of pathological paraffin sections, frozen sections, and surgically excised fresh tissue, respectively, to obtain EEAs. It then performs region feature registration analysis on the three types of EEAs from the same lesion site, calculates the region grayscale matching degree and tissue texture overlap, and divides the enhanced point block sets at different scales. Where L is the number of scale levels; Let j be the number of points at the j-th scale. Let be the three-dimensional coordinates of the k-th point at the j-th scale; The first candidate unit is used for point blocks at each scale level j. The first candidate sample is generated based on the corresponding original image region. ,Will The input generative adversarial network model performs a complete cross-domain transformation, yielding cross-domain candidate samples corresponding to two other image types. ; The coefficient calculation unit is used for... and Perform multi-dimensional structural verification and calculate the enhancement effectiveness coefficient of the corresponding point blocks. ; ； in, for and Feature vector matching degree; For the feature encoding function of the pre-trained model; for The regularity of the three-dimensional structure; , The weighting coefficients are and satisfy the following conditions: ; like Then determine the corresponding point block. corresponding and As valid samples, they are retained, These are preset coefficients; The point mapping unit is used to establish point block mapping associations for frozen sections and surgically removed fresh tissue images, based on effective enhanced point blocks at various scales of paraffin section images. , The multi-scale point mapping topology is obtained based on the intersection operation. ; The second candidate unit is used to construct a second candidate sample based on the mapping topology R, using the point blocks of the effective samples as the reference unit. ; The completion unit is used to include all second candidate samples. The input generative adversarial network model performs a complete cross-domain transformation to obtain cross-domain completion samples. Then, the effective first candidate samples, cross-domain candidate samples, and cross-domain completion samples at all scales are integrated into the sample image set to achieve sample expansion.

6. The lung cancer pleural invasion auxiliary diagnostic system based on multiphoton microscopy as described in claim 5, characterized in that, The second candidate unit includes: The sub-feature extraction subunit is used to divide the point-block feature vector output by the pre-trained histological feature extraction function into several sub-feature vectors related to lung cancer pleural invasion, including the collagen fiber spatial distribution feature sub-vector. Cell boundary structure feature vector Spatial feature vectors of tissue interstitial space ; The weighting subunit is used to assign weight coefficients based on the contribution of each sub-feature to the determination of pleural invasion status. , , The weighted histological feature distances between corresponding points on paraffin-embedded sections and frozen sections, and between paraffin-embedded sections and freshly excised tissue, were calculated for the same lesion site. ; filter The point blocks form a set of associated units; The constraint alignment sub-unit is used to apply spatial topological constraints to the point blocks corresponding to frozen sections and surgically removed fresh tissue, based on the point correspondence relationship of the pathological paraffin section point blocks in the associated unit set as the reference unit, and to obtain the topologically aligned block combination. An initial generation sub-unit is used to perform feature weighted fusion on the topologically aligned block combination to obtain fused features, and to generate an initial second candidate sample based on the fused features. The final sub-unit is used to calculate the spatial relative position deviation between each point block in the initial second candidate sample and the corresponding block in the original image. If the spatial relative position deviation is less than or equal to a preset deviation, the initial second candidate sample is determined to be the final second candidate sample.

7. The lung cancer pleural invasion auxiliary diagnostic system based on multiphoton microscopy as described in claim 5, characterized in that, The location determination unit includes: The encoding subunit is used to obtain the space-spectral fusion feature vector of each pixel in the multiphoton image. and will The input encoding grid is used to perform the first encoding, and the encoding formula is: ,in, Encode grids for pixel features; , These are the weight matrix and bias term of the grid, respectively; Use the Sigmoid activation function; The first encoded feature of the i-th pixel; The deviation vector acquisition subunit is used to obtain the first encoded feature of all pixels. Perform adaptive density clustering to obtain the feature center deviation vector for each pixel. ,in, For the first The feature centers of each cluster are formed by all clusters within that cluster. The average value is obtained; Noise extraction subunit, used for each cluster Extracting basic noise features from all feature center deviation vectors. And generate the initial adjustment vector for the corresponding cluster. ; The influence calculation subunit is used to set the pixel adjustment order based on the spatial location correlation between all initial adjustment vectors and the first encoded feature, and to calculate the influence weight of the image adjustment operation effect set before N1 on N1 for the pixel with the current index N1. ; ,in, These are the spatial coordinates of the t-th pixel and the pixel with the current index N1, respectively, and N1 is greater than 1; Cosine similarity of spatial coordinates; To maintain efficiency for the features of the t-th order adjustment operation, and ,in, The t-th order adjusted coding features; The encoding features before the t-th order adjustment; Parameterized fine-tuning subunit, used to adjust according to the influence weights Perform parameterized fine-tuning on the initial adjustment vector for the current index N1: ,in, For fine-tuning the bias term; This is the fine-tuned vector; Feature modulation subunit, used to... Feature modulation is performed on the image after N1-1 initial adjustment vector operations: ,in, The modulation coefficient; Image processing subunit, used to... The image is processed using the LeakyReLU activation function to obtain the currently adjusted image. This process continues until all pixel adjustments are complete, at which point a self-enhancing subimage is obtained. , This is the LeakyReLU activation function.

8. The method for auxiliary diagnosis of pleural invasion in lung cancer based on multiphoton microscopy according to claim 7, characterized in that, The noise extraction subunit includes: ,in, To adjust the vector generation network; , They are respectively The weight matrix and bias terms; This is the ReLU activation function.

9. A method for auxiliary diagnosis of pleural invasion in lung cancer based on multiphoton microscopy, characterized in that, include: Step 1: Collect multiphoton images of unstained paraffin sections, frozen sections, and freshly excised tissue from lung cancer patients using a multiphoton microscopy system. Select the image at the pleural boundary as the region of interest to construct a multi-source dataset. Step 2: Using existing collagen fiber feature extraction algorithms, analyze each region of interest in the multi-source dataset to obtain 142-dimensional tissue features; Step 3: Using a generative adversarial network model, cross-domain data augmentation is performed between multiphoton images of pathological paraffin sections, frozen sections, and surgically removed fresh tissue to obtain several sample images; Step 4: Fuse the high-dimensional tissue features with the semantic feature vectors of the sample images extracted based on the deep learning pre-trained model to construct a pleural invasion state classifier, and complete supervised training on the labeled dataset; Step 5: During the operation, real-time images are acquired using a portable multiphoton microscope system. The real-time images are then input into the trained classification model to predict the VPI positive or negative status and output the diagnostic results.

Images