A method, system, and computer storage medium for multimodal pathological identification of colorectal cancer.
By combining complex-valued optical convolution accelerators and multi-granularity hierarchical codebooks, the challenges of multimodal feature extraction and multi-label classification in the pathological diagnosis of colorectal cancer have been solved, achieving efficient and accurate pathological image analysis, which is suitable for large-scale screening and clinical auxiliary diagnosis.
Patent Information
- Application Number
- CN202611077459.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-07-20
- Publication Date
- 2026-11-10
AI Technical Summary
Existing technologies for the pathological diagnosis of colorectal cancer suffer from several drawbacks, including reliance on large amounts of anomaly-labeled data, high computational complexity, insufficient accuracy in multi-label classification, inadequate interpretability of feature representations, and inability of single-label classification to handle situations where multiple labels coexist. Consequently, it is difficult to achieve a balance between hardware-level real-time computation and the integrity of multimodal pathological features.
Multimodal preprocessing of pathological images is performed using a complex-valued optical convolution accelerator. Complex-valued feature vectors are extracted by a complex-valued convolution encoder, and multimodal fusion feature vectors are generated by amplitude-phase decoupling transformation. Combined with a multi-granularity hierarchical codebook and a multi-label classification decoder, anomaly assessment indicators are calculated and multi-label diagnostic results are output.
It improves the accuracy of identifying precursor lesions of colorectal cancer, reduces the risk of missed diagnosis and failure, meets the real-time needs of large-scale screening and clinical auxiliary diagnosis, and enhances the ability of feature characterization and the accuracy of multi-label classification.
Smart Images

Figure CN123072007A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of medical image analysis technology, and in particular to a method, system and computer storage medium for multimodal pathological identification of colorectal cancer. Background Technology
[0002] Colorectal cancer is one of the most common malignant tumors of the digestive system worldwide. Its development follows an adenoma-carcinoma sequence: normal colonic mucosa → hyperplastic polyps → adenomatous polyps → dysplasia → carcinoma in situ → invasive carcinoma. This progression typically takes 5 to 15 years, providing a valuable window for early screening and intervention. Studies show that the 5-year survival rate for stage I colorectal cancer patients can reach over 90%, while the 5-year survival rate for stage IV metastatic colorectal cancer patients is only about 14%. Early detection and resection of precancerous adenomatous polyps can reduce the incidence of colorectal cancer by 76% to 90%. Therefore, early screening, early diagnosis, and early treatment of colorectal cancer are of paramount clinical and social value in improving patient prognosis and reducing mortality.
[0003] Histopathological examination is the gold standard for diagnosing colorectal cancer. Pathological analysis of colonoscopy biopsies or resected specimens can clarify the histological type, differentiation degree, and depth of invasion of the lesion, providing a basis for clinical treatment decisions. Precursor lesions for colorectal cancer mainly include adenomas, hyperplastic polyps, and serrated lesions. Adenomas are further classified into tubular adenomas, villous adenomas, and tubulovillous adenomas based on the proportion of villous components and the degree of atypia, as well as low-grade intraepithelial neoplasia and high-grade intraepithelial neoplasia. Accurate pathological diagnosis is crucial for developing a treatment strategy: low-grade adenomas can be resected endoscopically, while high-grade intraepithelial neoplasia and invasive carcinoma require radical surgical resection.
[0004] The pathological features of precursor lesions of colorectal cancer determine their appearance on pathological images, and can mainly include the following types: Adenoma is the most common precursor lesion to colorectal cancer, accounting for approximately 60% to 70% of all colorectal polyps. The histological characteristics of adenoma include abnormal proliferation of glandular structures, atypia of glandular epithelial cells, irregular morphology of glandular lumens, and increased mitotic figures. Based on the proportion of villous component, adenomas are classified into tubular adenomas (villous component <20%), tubulovillous adenomas (villous component 20% to 80%), and villous adenomas (villous component >80%). The degree of atypia in adenomas is classified as low-grade and high-grade: low-grade intraepithelial neoplasia shows less cellular atypia, with elongated oval nuclei and preserved cell polarity; high-grade intraepithelial neoplasia shows significant cellular atypia, with large, deeply stained nuclei, readily visible mitotic figures, and loss of cell polarity.
[0005] Hyperplastic polyps are the most common non-neoplastic polyps, accounting for approximately 75% to 90% of all colorectal polyps. Histologically, hyperplastic polyps are characterized by proliferation of crypt epithelium, regular glandular structure, columnar cells with eosinophilic cytoplasm, nuclei located at the base, and no significant atypia. Hyperplastic polyps are usually benign lesions with a low risk of malignancy; however, some hyperplastic polyps with a serrated structure (such as sessile serrated adenoma) have a higher potential for malignancy.
[0006] Serrated lesions include sessile serrated adenomas / polyps (SSA / P) and conventional serrated adenomas (TSA). SSA / P is characterized by crypt base expansion, horizontal crypt branching, and serrated structure of the crypt epithelium; TSA presents as serrated hyperplasia of the crypt epithelium with atypia. Serrated lesions account for approximately 15% to 25% of all precursor lesions for colorectal cancer, but approximately 20% to 30% of colorectal cancers originate from the serrated lesion pathway, possessing unique molecular characteristics.
[0007] Invasive adenocarcinoma is characterized by malignant invasive growth of glandular structures. The glands are irregular in shape, varying in size, and disordered in arrangement. Significant cellular atypia is observed, with large, deeply stained nuclei and frequent mitotic figures. The stroma exhibits a connective tissue-promoting reaction. Adenocarcinomas are classified according to the proportion of glandular tissue formed: well-differentiated (>95% glandular tissue), moderately differentiated (50% to 95% glandular tissue), and poorly differentiated (25% to 50% glandular tissue). The lower the degree of differentiation, the higher the malignancy. Summary of the Invention
[0008] The purpose of this application is to provide a method, system, and computer storage medium for multimodal pathological identification of colorectal cancer, in order to solve the problem in the prior art of how to balance hardware-level real-time computing efficiency and multimodal integrity of pathological features without relying on a large amount of anomaly labeled data, and to break the logical limitation of single-label output, so as to provide a set of intelligent diagnostic systems for colorectal cancer that has both high sensitivity to anomaly screening capabilities and can accurately output the probability of coexistence of multiple lesions.
[0009] To achieve the above objectives, the first aspect of this application proposes a multimodal pathological identification method for colorectal cancer, the method comprising: The acquired colorectal pathological images to be detected are subjected to multimodal preprocessing, magnification image block extraction and decoupling transformation operations in sequence to obtain the complex value continuous feature vector corresponding to the magnification image block, and the multimodal fusion feature vector corresponding to the complex value continuous feature vector; The multimodal fusion feature vector is used to search for quantization features and quantization indices in a multi-granularity hierarchical codebook. The quantization features corresponding to each codebook are weighted and fused using learnable weights to obtain a comprehensive quantization feature. The comprehensive quantitative features are used to reconstruct the image, and a first anomaly evaluation index is calculated based on the image reconstruction results; The second anomaly evaluation index is calculated based on the quantization index and the activation frequency distribution of the prototype vectors in each codebook. The first abnormality assessment index and the second abnormality assessment index are weighted and fused to obtain a target abnormality score, so as to complete the abnormality screening and judgment of the colorectal pathological image to be detected based on the target abnormality score; The multimodal fusion feature vector is simultaneously input into a multi-label classification decoder, and the multi-label classification result of colorectal cancer precursor lesions is output. The multi-label classification result includes at least the classification probabilities of adenocarcinoma, adenoma, hyperplastic polyp and normal tissue.
[0010] To achieve the above objectives, a second aspect of this application also proposes a multimodal pathological identification system for colorectal cancer, the multimodal pathological identification system for colorectal cancer comprising: The preprocessing module sequentially performs multimodal preprocessing, magnification image block extraction, and decoupling transformation operations on the acquired colorectal pathological image to be detected, to obtain the complex-valued continuous feature vector corresponding to the magnification image block, and the multimodal fusion feature vector corresponding to the complex-valued continuous feature vector; The fusion module searches for quantization features and quantization indices in a multi-granularity hierarchical codebook using the multimodal fusion feature vector, and performs weighted fusion of the quantization features corresponding to each codebook using learnable weights to obtain a comprehensive quantization feature. The reconstruction module performs image reconstruction on the comprehensive quantitative features and calculates the first anomaly evaluation index based on the image reconstruction results. The calculation module calculates a second anomaly evaluation index based on the quantization index and the activation frequency distribution of the prototype vectors in each codebook. The determination module performs weighted fusion of the first abnormality assessment index and the second abnormality assessment index to obtain a target abnormality score, so as to complete the abnormality screening and determination of the colorectal pathological image to be detected based on the target abnormality score. The multi-label classification module simultaneously inputs the multimodal fusion feature vector into the multi-label classification decoder and outputs the multi-label classification result of colorectal cancer precursor lesions. The multi-label classification result includes at least the classification probabilities of adenocarcinoma, adenoma, hyperplastic polyp, and normal tissue.
[0011] To achieve the above objectives, a third aspect of this application also provides a computer storage medium including a program for use in conjunction with an electronic device, the program being executable by a processor to perform the following steps: The acquired colorectal pathological images to be detected are subjected to multimodal preprocessing, magnification image block extraction and decoupling transformation operations in sequence to obtain the complex value continuous feature vector corresponding to the magnification image block, and the multimodal fusion feature vector corresponding to the complex value continuous feature vector; The multimodal fusion feature vector is used to search for quantization features and quantization indices in a multi-granularity hierarchical codebook. The quantization features corresponding to each codebook are weighted and fused using learnable weights to obtain a comprehensive quantization feature. The comprehensive quantitative features are used to reconstruct the image, and a first anomaly evaluation index is calculated based on the image reconstruction results; The second anomaly evaluation index is calculated based on the quantization index and the activation frequency distribution of the prototype vectors in each codebook. The first abnormality assessment index and the second abnormality assessment index are weighted and fused to obtain a target abnormality score, so as to complete the abnormality screening and judgment of the colorectal pathological image to be detected based on the target abnormality score; The multimodal fusion feature vector is simultaneously input into a multi-label classification decoder, and the multi-label classification result of colorectal cancer precursor lesions is output. The multi-label classification result includes at least the classification probabilities of adenocarcinoma, adenoma, hyperplastic polyp and normal tissue.
[0012] Based on the multimodal pathological recognition method, system, and computer storage medium for colorectal cancer described in this application, the input colorectal pathological image is efficiently feature-extracted using a complex-valued optical convolution accelerator and preprocessed using multimodal methods. Complex-valued feature vectors for each modality are extracted using a complex-valued convolution encoder. These complex-valued features are then mapped to the real space using amplitude-phase decoupling transformation to obtain a multimodal fusion feature vector. This fusion feature vector is then quantized using a multi-granularity hierarchical codebook to obtain a comprehensive quantized feature. A first anomaly assessment index and a second anomaly assessment index are calculated, and a target anomaly score is obtained through weighted fusion. Based on this target anomaly score, the colorectal pathological image to be detected is screened for anomalies. Simultaneously, a multi-label classification decoder outputs multi-label diagnostic results, enabling the classification and recognition of adenocarcinoma, adenoma, hyperplastic polyps, and normal tissue.
[0013] In this way, this application combines optical computing with deep learning, and utilizes the feature decoupling capability of the complex value domain to improve the recognition accuracy of colorectal cancer precursor lesions. It has important application value in large-scale screening and clinical auxiliary diagnosis, and effectively reduces the clinical risk caused by the abnormal screening judgment of the colorectal pathological image to be detected or the output of multi-label classification probability, which may result in missed diagnosis or failure. Attached Figure Description
[0014] The accompanying drawings, which are included to provide a further understanding of the embodiments of this specification and form part of the embodiments of this specification, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 A flowchart illustrating a multimodal pathological identification method for colorectal cancer provided in the embodiments of this specification; Figure 2 This is a flowchart illustrating the overall process of a multimodal pathological identification method for colorectal cancer provided in the embodiments of this specification. Figure 3 This is a flowchart illustrating the multimodal preprocessing and three-channel fusion imaging process in a multimodal pathological identification method for colorectal cancer provided in the embodiments of this specification. Figure 4 This is a flowchart illustrating the correspondence between the three-branch feature extraction architecture and the image block space in a multimodal pathological identification method for colorectal cancer provided in the embodiments of this specification. Figure 5 This is a flowchart illustrating the three-branch complex-valued convolutional encoder network structure corresponding to a multimodal pathological identification method for colorectal cancer provided in the embodiments of this specification. Figure 6 This is a flowchart illustrating the multi-granularity hierarchical codebook vector quantization process in a multimodal pathological identification method for colorectal cancer provided in the embodiments of this specification. Figure 7 This is a detailed flowchart illustrating the implementation of a multi-label classification decoder in a multimodal pathological identification method for colorectal cancer provided in the embodiments of this specification. Figure 8 This is a schematic diagram of the auxiliary assessment process for the pathological diagnosis of endoscopically resected polyps, corresponding to a multimodal pathological identification method for colorectal cancer provided in the embodiments of this specification. Figure 9 This is a schematic diagram of the auxiliary diagnosis process for a large-scale colorectal cancer screening project corresponding to a multimodal pathological identification method for colorectal cancer provided in the embodiments of this specification; Figure 10 This is a schematic diagram of the structure of a multimodal pathological recognition system for colorectal cancer provided in the embodiments of this specification; Figure 11 This is a schematic diagram of the structure of a computer storage medium corresponding to a multimodal pathological identification method for colorectal cancer provided in the embodiments of this specification. Detailed Implementation
[0015] In current technology, the diagnosis of precursor lesions for colorectal cancer mainly relies on manual endoscopic examination by pathologists. However, manual endoscopic examination has several limitations: First, excessive workload and poor diagnostic consistency. With the widespread implementation of colorectal endoscopy screening, the number of slides pathologists need to diagnose daily has increased significantly. A routine H&E-stained pathological slide, scanned under a 40x objective lens, has an area of approximately 15mm × 15mm to 25mm × 25mm, containing millions of cellular structures. Pathologists need to complete the diagnosis of a large number of slides within a limited time, which easily leads to visual fatigue and a decline in diagnostic consistency. Studies have shown that the Kappa value of diagnostic consistency for the same lesion among different pathologists is only 0.4 to 0.6, and the consistency is even lower for borderline cases. Borderline cases can include the differentiation between low-grade and high-grade intraepithelial neoplasia, which is not specifically limited here.
[0016] Second, the objectivity of the diagnostic criteria is insufficient. The grading diagnosis of precursor lesions for colorectal cancer relies heavily on the personal experience and subjective judgment of pathologists. Different pathologists have varying understandings and interpretations of descriptive criteria such as marked atypia and loss of arrangement polarity, leading to different diagnostic conclusions for the same case. In addition, the histological characteristics of some lesions fall in the middle of a continuous spectrum, making it difficult to simply classify them into a single category.
[0017] Third, the need for multi-label combined diagnosis remains unmet. In clinical practice, a patient's pathological specimen may simultaneously present multiple lesion types, such as adenomas and hyperplastic polyps, or normal tissue and abnormal lesions within the same field of view. Traditional single-label classification methods struggle to handle such multi-label coexistence situations, necessitating the development of multi-label classification techniques to comprehensively evaluate pathological images.
[0018] Fourth, limitations of auxiliary detection technologies. Traditional immunohistochemistry (IHC) testing requires additional antibody incubation and color development steps, increasing diagnostic costs and time. Molecular pathology testing requires tissue DNA extraction, which is technically demanding and difficult to implement in primary healthcare institutions. Therefore, the development of AI-assisted diagnostic technologies based on routine H&E stained slides is of significant clinical necessity.
[0019] In recent years, deep learning technology has made significant progress in the field of pathological image analysis. Convolutional Neural Networks (CNNs), with their advantages of end-to-end learning and automatic feature extraction, have performed excellently in a variety of pathological analysis tasks, including: tumor detection and classification, such as the classification of breast cancer, skin cancer, and prostate cancer; lymph node metastasis detection, such as the detection of sentinel lymph node metastasis in breast cancer; grade prediction, such as the Gleason score prediction for prostate cancer; and survival analysis.
[0020] However, existing methods still have the following limitations when applied to multi-label classification of colorectal cancer pathology: First, the ability to extract single-modal features is limited. Traditional convolutional networks can only extract morphological and textural features in the real domain, making it difficult to fully utilize multi-modal information in pathological images, such as H&E staining information and PAS staining information. Second, the computational complexity is high. Full-slice analysis of high-resolution pathological images requires processing millions of image patches, resulting in high computational costs and making it difficult to meet the needs of real-time clinical diagnosis. Third, the accuracy of multi-label classification is insufficient. Existing methods are prone to label confusion when dealing with the coexistence of multiple labels. Fourth, the interpretability of feature representations is poor. The "black box" nature of deep learning models limits their application in clinical practice.
[0021] Furthermore, optical computing, as an emerging computing paradigm, leverages the parallel propagation and interference properties of light to achieve efficient computation, demonstrating enormous potential in image processing and pattern recognition. Optical convolution, through the point spread function of a lens system, performs convolution operations between images and filters, possessing inherent advantages in parallelism and low power consumption.
[0022] Complex-valued Neural Networks (CVNNs) are a class of neural network models that process complex-valued data, where the neuron weights and activation functions are all in complex form. Compared to real-valued neural networks, CVNNs have the following advantages: First, the information capacity of the complex domain is greater; a complex number can simultaneously encode both amplitude and phase information, theoretically twice the information capacity of a real number of the same dimension. Second, phase information is crucial for image processing; the phase spectrum carries structural and edge information of the image, while traditional real-valued networks struggle to explicitly model phase features. Third, complex-valued operations are naturally suited for processing data with periodicity or rotation invariance, such as texture images and biomedical images.
[0023] The amplitude and phase components of a complex-valued convolutional kernel can learn different feature patterns: the amplitude component typically learns frequency response characteristics, corresponding to the texture and structural features of an image; the phase component learns nonlinear mappings, corresponding to the contrast and edge features of an image. This feature decoupling capability gives complex-valued neural networks a unique advantage in handling complex visual tasks.
[0024] However, training complex-valued neural networks faces challenges in gradient computation and optimizers. Complex-valued backpropagation requires defining the gradient of the complex-valued loss function. Common strategies include decomposing the complex-valued variables into real and imaginary parts for real-valued optimization, or using complex-valued optimizers, such as Cayley transform and conjugate gradient methods. Among these, phase propagation is an effective method for optimizing complex-valued weights. It achieves joint optimization by parameterizing the convolution kernel as a product of magnitude and phase, utilizing gradient updates of the magnitude component and the chain rule propagation of the phase component.
[0025] Therefore, embodiments of this specification provide a method, system, and computer storage medium for multimodal pathological recognition of colorectal cancer. The method employs a complex-valued optical convolutional accelerator to efficiently extract features from the input colorectal pathological image and performs multimodal preprocessing. A complex-valued convolutional encoder extracts complex-valued feature vectors for each modality. These complex-valued features are then mapped to the real number space through amplitude-phase decoupling transformation to obtain a multimodal fusion feature vector. This fusion feature vector is then quantized using a multi-granularity hierarchical codebook to obtain a comprehensive quantized feature. A first anomaly assessment index and a second anomaly assessment index are calculated, and a target anomaly score is obtained through weighted fusion. Based on the target anomaly score, the anomaly screening and judgment of the colorectal pathological image to be detected are completed. Simultaneously, a multi-label classification decoder outputs multi-label diagnostic results, achieving the classification and recognition of adenocarcinoma, adenoma, hyperplastic polyps, and normal tissue.
[0026] Thus, the embodiments of this specification combine optical computing with deep learning, and utilize the feature decoupling capability of the complex value domain to improve the recognition accuracy of colorectal cancer precursor lesions. This has important application value in large-scale screening and clinical auxiliary diagnosis, and effectively reduces the clinical risks caused by missed diagnoses or failures due to abnormal screening judgments of the colorectal pathological images to be detected or the output of multi-label classification probabilities.
[0027] To make the objectives, technical solutions, and advantages of the embodiments in this specification clearer, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0028] The technical solutions provided in the various embodiments of this specification are described in detail below with reference to the accompanying drawings.
[0029] Please see Figure 1 This is a flowchart illustrating a multimodal pathological identification method for colorectal cancer provided in the embodiments of this specification.
[0030] In the embodiments of this specification, the multimodal pathological identification method for colorectal cancer may specifically include the following steps: S101: The acquired colorectal pathological image to be detected is subjected to multimodal preprocessing, magnification image block extraction and decoupling transformation operations in sequence to obtain the complex value continuous feature vector corresponding to the magnification image block, and the multimodal fusion feature vector corresponding to the complex value continuous feature vector; S103: Search for quantization features and quantization indices in the multi-granularity hierarchical codebook using the multi-modal fusion feature vector, and perform weighted fusion of the quantization features corresponding to each codebook using learnable weights to obtain comprehensive quantization features; S105: Perform image reconstruction on the comprehensive quantization features, and calculate the first anomaly evaluation index based on the image reconstruction results; S107: Calculate the second anomaly evaluation index based on the quantization index and the activation frequency distribution of the prototype vectors in each codebook; S109: The first abnormality assessment index and the second abnormality assessment index are weighted and fused to obtain a target abnormality score, so as to complete the abnormality screening and judgment of the colorectal pathological image to be detected based on the target abnormality score. S111: Simultaneously input the multimodal fusion feature vector into the multi-label classification decoder and output the multi-label classification result of colorectal cancer precursor lesions, wherein the multi-label classification result includes at least the classification probabilities of adenocarcinoma, adenoma, hyperplastic polyp and normal tissue.
[0031] In the embodiments of this specification, a complex-valued optical convolutional accelerator is used to efficiently extract features from the input colorectal pathology image. The amplitude component of the complex-valued convolutional kernel is responsible for extracting the morphological and textural features of the pathology image, while the phase component is responsible for extracting the staining style and color contrast features. These two components complement each other to enhance feature representation capabilities. Furthermore, multimodal preprocessing is performed. A complex-valued feature vector for each modality is extracted using a complex-valued convolutional encoder. This complex-valued feature vector is then mapped to the real number space through an amplitude-phase decoupling transformation to obtain a multimodal fusion feature vector. This fusion feature vector is then vector-quantized using a multi-granularity hierarchical codebook to obtain a comprehensive quantized feature. A first anomaly assessment index and a second anomaly assessment index are calculated, and a target anomaly score is obtained through weighted fusion. Based on the target anomaly score, the anomaly screening and judgment of the colorectal pathology image to be detected is completed. Simultaneously, a multi-label classification decoder outputs multi-label diagnostic results, achieving the classification and identification of adenocarcinoma, adenoma, hyperplastic polyps, and normal tissue.
[0032] Thus, the embodiments in this specification combine optical computing with deep learning, and utilize the feature decoupling capability of the complex value domain to improve the recognition accuracy of colorectal cancer precursor lesions, which has important application value in large-scale screening and clinical auxiliary diagnosis.
[0033] As an application embodiment of this specification, step S101 involves sequentially performing multimodal preprocessing, magnification image block extraction, and decoupling transformation operations on the acquired colorectal pathological image to be detected. Specifically, this may include: Obtain pathological images of the colorectal tissue to be examined; The colorectal pathological images to be detected are subjected to multimodal preprocessing to generate three-channel fused images; The three-channel fused image is subjected to magnification image block extraction; Based on the magnification image patch, a complex-valued optical convolution accelerator is used for efficient feature extraction to obtain a complex-valued continuous feature vector corresponding to the magnification image patch. The complex-valued continuous feature vector is mapped to the real number space through a decoupling transformation to obtain the multimodal fusion feature vector.
[0034] In the embodiments described in this specification, a self-designed multimodal preprocessing and three-channel fusion strategy is used to fully utilize H&E staining information, PAS staining enhancement information, and nuclear morphology feature information, in addition to normalizing reconstruction errors, to enhance the model's ability to perceive precursor lesions of colorectal cancer. H&E staining provides basic morphological information, PAS staining enhances mucosal secretory function, and nuclear morphology features quantify the degree of cellular atypia; the three complement each other to provide comprehensive diagnostic evidence.
[0035] A three-branch complex-valued convolutional encoder extracts image patches at different magnifications to capture multi-scale pathological features. Combined with the hierarchical representation capabilities of a multi-granularity codebook, it can simultaneously cover pathological patterns at the cellular, glandular, and tissue levels, adapting to the cross-scale analysis needs of colorectal cancer precursor lesion diagnosis. The ultra-high magnification branch identifies nuclear atypia, the high magnification branch identifies glandular structural abnormalities, and the low magnification branch identifies changes in tissue architecture. Furthermore, by combining optical computing with deep learning, a complex-valued optical convolutional accelerator is used to achieve efficient feature extraction.
[0036] Furthermore, the colorectal pathology images are subjected to multimodal preprocessing to generate a three-channel fused image, which may specifically include: The original H&E staining image corresponding to the colorectal pathology image to be detected is converted into a grayscale channel. The simulated PAS staining enhancement channel is separated from the RGB channels of the original H&E staining image based on the color unmixing algorithm. The coefficient of the color unmixing is set according to the contrast characteristics of PAS positive substances and cell nuclei. The original H&E staining image was segmented into cell nuclei, and the morphological parameters of the cell nuclei were extracted to generate nuclear morphology feature channels. The grayscale channel, the simulated PAS staining enhancement channel, and the nuclear morphology feature channel are stitched together along the channel dimension to generate the three-channel fused image.
[0037] In the embodiments described in this specification, the three-channel fused image may specifically include an H&E staining channel, a PAS staining enhancement channel, and a nuclear morphology feature channel.
[0038] Among them, the H&E staining channel converts the original H&E staining pathological image into grayscale representation and normalizes it to a grayscale value. The interval preserves the basic morphological and textural information of the H&E stained image. H&E staining is the most commonly used staining method in pathological diagnosis. Hematoxylin makes the cell nucleus appear blue-purple, while eosin makes the cytoplasm and extracellular matrix appear pink. The contrast between the two provides rich diagnostic information.
[0039] PAS staining enhancement channel: A pseudo-color channel corresponding to PAS-positive substances is separated from the original H&E staining image using a color unmixing algorithm to highlight the mucus components secreted by the mucosal epithelium. PAS staining (Periodic Acid-Schiffstain) can specifically stain polysaccharides, glycoproteins, and glycolipids. The mucus secreted by the gland appears purplish-red under PAS staining, which helps to assess the secretory function of the gland and the integrity of the mucosal layer.
[0040] In H&E staining images, characteristic channels of PAS-positive substances can be separated using color unmixing techniques: ; Formula (1) in, , , The original RGB image is in The red, green, and blue channel values at the location. , , The color unmixing coefficient is designed based on the purple-red characteristics of PAS-positive substances and the blue characteristics of cell nuclei.
[0041] Specifically, the purplish-red feature can be a high red or low blue feature, and the blue feature can be a high blue feature, etc., without specific limitations.
[0042] In the embodiments of this specification, the color unmixing coefficient can specifically be used to enhance the contrast between the purplish-red PAS positive substance and the blue cell nucleus.
[0043] The enhanced contrast between the purplish-red PAS-positive material and the blue cell nuclei helps identify abnormalities in glandular structure.
[0044] Nuclear morphology feature channel: Morphological parameters of the cell nucleus are extracted using image processing algorithms, specifically including nuclear area, nuclear perimeter, nuclear roundness, and nuclear chromatin distribution characteristics. These parameters are then normalized to generate a nuclear morphology feature map. Changes in cell nucleus morphology are important indicators of cellular atypia. The nuclei of adenomas and adenocarcinomas typically exhibit: enlarged nuclei, deeply stained nuclei, increased nuclear-cytoplasmic ratio, and increased mitotic figures.
[0045] These changes can be quantified through automated kernel segmentation and morphological measurements: ; Formula (2) in, for The area of the cell nucleus. The perimeter of the cell nucleus. To maximize the area of cell nuclei in the training set, norm For normalization function, This is the roundness coefficient, and its value range is... A circle is 1, and an elongated shape is less than 1.
[0046] In the embodiments of this specification, a heat map reflecting the degree of nuclear morphological abnormality can be generated through normalization processing.
[0047] Furthermore, the three-channel fused image is subjected to magnification image block extraction, which may specifically include: The colorectal pathological image to be detected is subjected to multi-scale sampling, wherein the multi-scale sampling includes at least one of the following methods: Ultra-high magnification sampling is used to extract ultra-high magnification image patches of the first size to capture cellular-level microstructural features; High-magnification sampling is used to extract high-magnification image patches of a second size to capture glandular structural features; Low-magnification sampling is used to extract low-magnification image patches of the third size to capture organizational features; The first dimension is smaller than the second dimension, and the second dimension is smaller than the third dimension.
[0048] In the embodiments of this specification, ultra-high magnification sampling can be used to extract ultra-high magnification image blocks of 32 pixels × 32 pixels with a step size of 16 pixels, to capture microstructural features at the glandular epithelial cell level, including nuclear atypia, chromatin distribution patterns and nuclear-cytoplasmic ratio, etc., which are not specifically limited here. High-magnification sampling can be used to extract 64-pixel × 64-pixel high-magnification image patches with a stride of 32 pixels. It is used to capture glandular structural features, including glandular cavity morphology, glandular arrangement, and stromal infiltration patterns, etc., without specific limitations here. Low-magnification sampling can be used to extract low-magnification image patches of 128 pixels × 128 pixels with a stride of 64 pixels. It is used to capture tissue structure features, including mucosal layer structure, crypt morphology, and blood vessel distribution patterns, etc., without specific limitations.
[0049] Furthermore, as an application embodiment of this specification, based on the magnification image patch, efficient feature extraction is performed using a complex-valued optical convolution accelerator to obtain a complex-valued continuous feature vector corresponding to the magnification image patch. Specifically, this may include: The ultra-high magnification image block, the high magnification image block, and the low magnification image block are respectively input into the three branches of the three-branch complex value convolutional encoder; Efficient feature extraction is performed on the ultra-high magnification image patch, the high magnification image patch, and the low magnification image patch using a feature extraction network to obtain the complex-valued continuous feature vectors corresponding to each of the ultra-high magnification image patch, the high magnification image patch, and the low magnification image patch. The complex-valued continuous feature vectors include amplitude components and phase components. The amplitude components are used to extract the morphological and textural features of the pathological image, and the phase components are used to extract the staining style and color contrast features of the pathological image.
[0050] In the embodiments described in this specification, the feature extraction network may specifically be a feature extraction network based on complex field operations, such as a complex-valued optical convolution accelerator, etc., and is not specifically limited here.
[0051] By obtaining complex-valued continuous feature vectors, the complex-valued convolution kernel achieves feature decoupling enhancement through an amplitude-phase dual-component mechanism. The amplitude component is used to extract the morphological and textural features of the pathological image, while the phase component is used to extract the staining style and color contrast features of the pathological image. The two carry complementary pathological feature information, and their complementarity makes the feature representation more complete.
[0052] The complex-valued convolution operation is performed by the optical computing unit in a parallel optical signal processing mode. In this way, the natural parallelism of optical computing makes the convolution operation speed dozens of times faster than traditional digital computing, while reducing power consumption by more than 90%, providing a hardware foundation for real-time clinical diagnosis.
[0053] Furthermore, as an application embodiment of this specification, mapping the complex-valued continuous feature vector to the real number space through a decoupling transformation to obtain a multimodal fusion feature vector may specifically include: The complex-valued continuous feature vector is mapped to the real space through an amplitude-phase decoupling transformation according to the following formula to obtain the multimodal fusion feature vector: ; Formula (3) in, This is a multimodal fusion feature vector. For complex-valued continuous eigenvectors, specifically, The real space is , For amplitude components, For phase components, This represents a vector concatenation operation. and These represent the real and imaginary parts of a complex number, respectively.
[0054] In the embodiments described in this specification, For amplitude components, representing the magnitude distribution of complex-valued features; The phase component represents the phase distribution of complex-valued features. The amplitude and phase components encode different types of feature information, specifically: the amplitude component mainly reflects the brightness and contrast changes of the image, corresponding to the light and dark changes and texture roughness of glandular structures in pathological images; the phase component mainly reflects the edge and orientation information of the image, corresponding to the boundary clarity of cell nuclei and the morphological features of glandular cavities in pathological images.
[0055] The dimension of the decoupled real-valued feature vector is ,forward The dimension is the encoding of amplitude information, and then... The dimension represents the encoding of phase information. This separate representation allows subsequent multi-label classification decoders to process amplitude and phase features separately, enabling more refined feature representation.
[0056] Furthermore, as an application embodiment of this specification, for step S103, the multimodal fusion feature vector is used to search for quantization features and quantization indices in a multi-granularity hierarchical codebook. The quantization features corresponding to each codebook are then weighted and fused using learnable weights to obtain a comprehensive quantization feature. Specifically, this may include: Obtain multiple discrete codebooks of different capacities, wherein the discrete codebooks contain at least three capacity levels, and each codebook contains a different number of learnable prototype vectors. Find the quantization features corresponding to each of the codebooks; The comprehensive quantization feature is obtained by weighting and fusing the quantization features corresponding to each codebook using learnable weights.
[0057] In the embodiments of this specification, the multi-granularity hierarchical codebook may specifically include multiple discrete codebooks of different capacities.
[0058] Among them, smaller codebooks can be used to characterize coarse-grained tissue distribution features. Higher compression ratios force the model to learn coarse-grained tissue architecture features. Specifically, smaller codebooks can have capacities of 32 or 64. Since the number of prototype vectors is limited, each prototype vector needs to represent a wider range of normal tissue variations, capable of capturing the overall structural pattern of the mucosa and the distribution patterns of crypts. This coarse-grained feature is of great value in identifying large-scale tissue architecture abnormalities, which can specifically include mucosal thickening, crypt atrophy, etc., without specific limitations here.
[0059] Larger codebooks can be used to characterize fine-grained cell morphological features. Due to the larger number of prototype vectors, the feature space can be more finely divided, allowing for the learning of fine-grained cellular-level details. A larger codebook could specifically have a capacity of 512 or 1024, etc. Fine-grained prototypes can distinguish subtle morphological changes in normal cells, such as a slight increase in the nucleocytoplasmic ratio or subtle changes in chromatin. This is crucial for identifying early-stage cancer and precancerous lesions.
[0060] The design of the multi-granularity hierarchical codebook enables the model to understand the abnormal features of colorectal pathology images from both coarse-grained and fine-grained perspectives. By weighted fusion to integrate the advantages of different granularities, the detection capability for various lesions can be effectively improved.
[0061] Furthermore, as an application embodiment of this specification, step S105, which involves image reconstruction of the comprehensive quantization features and calculation of a first anomaly evaluation index based on the image reconstruction results, may specifically include: The comprehensive quantization features are input into the corresponding branch's magnification decoder to reconstruct the ultra-high magnification image block, the high magnification image block, and the low magnification image block, thereby obtaining the image reconstruction result; Based on the image reconstruction results, the pixel-wise mean square reconstruction error between each reconstructed image block and the corresponding original magnified image block is calculated; Calculate the average error value of the pixel-wise mean square reconstruction error for all image blocks under each branch; The weighted arithmetic mean of the average error values corresponding to each branch is used to obtain the original reconstruction score; The original reconstruction score is normalized to percentiles to obtain the first anomaly assessment index.
[0062] In the embodiments of this specification, the abnormal tissue in the colorectal pathological image to be detected does not look like the normal tissue seen during training. Therefore, after compression-quantization-reconstruction, the restored image will be very different from the original image. This difference is the first piece of abnormal evidence, which is also the first abnormality assessment index.
[0063] Since the comprehensive quantization feature is obtained by encoding and quantizing the original magnification image block, it is a discrete feature obtained after "looking up" a multi-granularity hierarchical codebook. It loses some detailed information in the original magnification image block, especially possible abnormal details.
[0064] The integrated quantization features are fed into the corresponding high-magnification (U), high-magnification (H), and low-magnification (L) decoders of the respective branches to reconstruct the magnification image patches, thus obtaining the reconstructed image patches.
[0065] Since the decoder has a symmetrical structure, such as a transposed complex convolutional layer, it upsamples low-dimensional features step by step until it restores the pixel size that is exactly the same as the original input, such as 32×32, 64×64 or 128×128, etc., without making specific limitations here.
[0066] Calculate the pixel-wise mean square reconstruction error between each reconstructed image patch and the corresponding original magnified image patch. If the reconstructed image is blurry, or the glandular edge or nuclear morphology is lost, the pixel-wise mean square reconstruction error will increase sharply.
[0067] Furthermore, since the three branches differ in size and clinical importance, a simple average cannot be taken. In the embodiments of this specification, a weighted arithmetic mean is used to summarize the errors of all image blocks on the entire slice.
[0068] The resulting original reconstruction score is an absolute value, and its magnitude is affected by the staining depth and scanning brightness of different slices. To ensure that this metric has a uniform threshold standard across the entire dataset, the embodiments in this specification use percentile normalization to map it to the [0,1] interval.
[0069] Conventional anomaly detection methods, including traditional AE or VQ-VAE, often directly use the global average reconstruction error. However, the embodiments in this manual employ multi-granularity weighting, forcibly incorporating prior pathological knowledge into the loss calculation, rather than letting the network black box make autonomous decisions. This makes the error score more clinically interpretable. Furthermore, the errors are calculated independently by branch: the errors of the three branches do not interfere with each other, and the ablation experiment can accurately locate whether the problem is with the cell nucleus (high U error) or the glandular structure (high H error). Percentile normalization is also performed: abandoning the traditional Min-Max normalization, which is susceptible to interference from extreme outliers, percentiles are used instead to ensure that even if extremely poor scanning artifacts occasionally occur, the score of normal sections will not be squeezed into a very small range, maintaining the stability of the scoring standard.
[0070] The first anomaly assessment indicator does not directly output a result, but is weighted and fused with the second anomaly assessment indicator. The first anomaly assessment indicator is a pixel-level reconstruction difference, which is responsible for capturing macroscopic morphological distortions, while the second anomaly assessment indicator is a feature-level distribution deviation, which is responsible for capturing microscopic pattern rarity. The two complement each other and together constitute the final judgment basis.
[0071] As an application embodiment of this specification, for step S107, calculating the second anomaly evaluation index based on the quantization index and the activation frequency distribution of the prototype vectors in each codebook may specifically include: Based on the activation frequency distribution and quantization index of the prototype vectors in each codebook, the rarity score corresponding to the prototype vector in the codebook is calculated, wherein the rarity score is negatively correlated with the activation disclosure distribution; The quantized distance score between the multimodal fusion feature vector and the prototype vector is calculated using Euclidean distance. The rarity score and the quantized distance score are weighted and fused to obtain the second anomaly evaluation index.
[0072] In the embodiments described in this specification, the activation frequency distribution reflects the representativeness of the prototype vectors in normal samples. During the training phase, the network is trained using only normal colorectal tissue pathological images, and the number of times each prototype vector is activated in each codebook is recorded. Normal samples tend to activate high-frequency prototype vectors, which represent typical normal tissue patterns; prototype vectors with low activation frequency distributions represent rare normal tissue variations or boundary conditions.
[0073] The rarity score is obtained by taking the negative logarithm of the activation frequency distribution. In this case, the lower the activation frequency distribution, the higher the rarity score. When the activation frequency distribution approaches 0, the rarity score approaches positive infinity, indicating that the feature pattern is extremely rare in normal samples and has a high anomaly confidence level.
[0074] The quantized distance score reflects the Euclidean distance between the input feature and the selected prototype vector. The larger the distance, the further the feature deviates from the normal pattern represented by the prototype vector. By using percentile normalization to map the original distance to the location of the distance distribution in the training set, the differences in the magnitude of distance between different codebooks and different samples can be effectively eliminated.
[0075] The comprehensive MCAR score is obtained by weighted fusion of the rarity scores and quantization distance scores of each codebook. Normal samples tend to activate high-frequency prototype vectors and produce smaller quantization distances, thus obtaining lower MCAR scores. The second anomaly evaluation metric is anomaly samples; anomalous samples are more likely to activate low-frequency prototype vectors or produce larger quantization distances, thus obtaining higher MCAR scores.
[0076] As an application embodiment of this specification, for step S109, the first abnormality assessment index and the second abnormality assessment index are weighted and fused to obtain a target abnormality score, so as to complete the abnormality screening and determination of the colorectal pathology image to be detected based on the target abnormality score. Specifically, this may include: If the target abnormality score exceeds a preset threshold, it is determined that the colorectal pathological image to be detected has abnormal precancerous lesions of colorectal cancer. If the target abnormality score is less than the preset threshold, the colorectal pathological image to be detected is determined to be normal.
[0077] In the embodiments of this specification, the preset threshold can be preset based on actual operating experience. For example, the preset threshold can be set to 0.5 or other values, which are not specifically limited here.
[0078] Abnormal samples are more likely to activate low-frequency prototype vectors or generate larger quantization distances, thus achieving higher MCAR scores.
[0079] As an application embodiment of this specification, for step S111, simultaneously inputting the multimodal fusion feature vector into a multi-label classification decoder and outputting the multi-label classification result of colorectal cancer precursor lesions may specifically include: The multi-label classification decoder adopts a multilayer perceptron structure, and the output layer uses the Sigmoid activation function to achieve multi-label classification.
[0080] In the embodiments of this specification, the characteristic of multi-label classification is that the output of each label is independent, and the same image can belong to multiple categories simultaneously, for example, both adenoma and hyperplastic polyp may exist. The Sigmoid activation function makes the output value of each label... Between these values, a threshold is used to determine whether the label is positive. The threshold can be 0.5 or other values, and no specific limitation is made here.
[0081] Multi-label classification decoders enable multi-label joint diagnosis, simultaneously outputting the classification probabilities of adenocarcinoma, adenoma, hyperplastic polyps, and normal tissue. This solves the problem that traditional single-label classification methods struggle to handle scenarios with multiple coexisting labels. In clinical practice, a patient's specimen may contain multiple lesion types simultaneously; multi-label classification technology can comprehensively assess the lesion composition of pathological images.
[0082] It should be noted that in this embodiment of the specification, step S109 and step S111 are run in parallel. Regardless of the abnormality screening judgment of the colorectal pathological image to be detected, multi-label classification is always running and outputting classification probabilities. This forms a double insurance mechanism: even if the abnormality screening judgment result of the colorectal pathological image to be detected is normal, if step S111 outputs the result of "adenocarcinoma probability 92%", this embodiment of the specification will automatically trigger a screening-diagnosis conflict alarm and force manual review, thereby covering the risk of missed diagnosis in step S109.
[0083] In this way, even if a false negative occurs in step S109, such as an early high-grade adenoma being very similar to normal tissue and causing the abnormal score to not exceed the threshold, step S111 can further determine whether an abnormality has occurred.
[0084] This decoupled design retains the advantages of unsupervised anomaly detection, such as its strong generalization and ability to identify unknown lesions, while also leveraging the strengths of supervised classification and precise naming. It perfectly avoids the clinical risks caused by the failure of either component, which is the core design wisdom that gives this system its high robustness.
[0085] This specification provides a multimodal pathological identification method for colorectal cancer. It employs a complex-valued optical convolutional accelerator to efficiently extract features from the input colorectal pathological image and performs multimodal preprocessing. A complex-valued convolutional encoder extracts complex-valued feature vectors for each modality. These complex-valued features are then mapped to the real space through amplitude-phase decoupling transformation to obtain a multimodal fusion feature vector. This fusion feature vector is then quantized using a multi-granularity hierarchical codebook to obtain a comprehensive quantized feature. A first anomaly assessment index and a second anomaly assessment index are calculated, and a target anomaly score is obtained through weighted fusion. Based on this target anomaly score, the anomaly screening and determination of the colorectal pathological image to be detected is completed. Simultaneously, a multi-label classification decoder outputs multi-label diagnostic results, achieving the classification and identification of adenocarcinoma, adenoma, hyperplastic polyps, and normal tissue.
[0086] Thus, the embodiments of this specification combine optical computing with deep learning, and utilize the feature decoupling capability of the complex value domain to improve the recognition accuracy of colorectal cancer precursor lesions. This has important application value in large-scale screening and clinical auxiliary diagnosis, and effectively reduces the clinical risks caused by missed diagnoses or failures due to abnormal screening judgments of the colorectal pathological images to be detected or the output of multi-label classification probabilities.
[0087] It should be noted that the above-described specific multimodal pathological identification method for colorectal cancer is merely a specific application example and does not limit the scope of the embodiments in this specification. Other specific embodiments may also be included, which will not be elaborated here.
[0088] Based on the same inventive concept, this specification also provides specific application examples of the above-described multimodal pathological identification method for colorectal cancer.
[0089] like Figure 2 The diagram shown is an overall flowchart corresponding to a multimodal pathological identification method for colorectal cancer provided in an embodiment of this specification.
[0090] In the embodiments of this specification, the overall process corresponding to the multimodal pathological identification method for colorectal cancer may specifically include the following steps: Step S1: Obtain the colorectal pathology image to be detected, perform multimodal preprocessing on the colorectal pathology image to be detected, and generate a three-channel fused image including H&E staining channel, PAS staining enhancement channel and nuclear morphology feature channel.
[0091] Step S2: Extract ultra-high magnification image blocks, high magnification image blocks, and low magnification image blocks from the three-channel fused image using a three-branch structure.
[0092] Step S3: Input the ultra-high magnification image block, high magnification image block and low magnification image block into the three branches of the three-branch complex value convolutional encoder respectively, and use the complex value optical convolutional accelerator to perform efficient feature extraction to obtain the complex value continuous feature vector of the corresponding image block.
[0093] Step S4: Map the complex-valued continuous feature vector to the real number space through amplitude-phase decoupling transformation to obtain the multimodal fusion feature vector. Then, search for the quantization features and quantization index in the multi-granularity hierarchical codebook using nearest neighbor search.
[0094] Step S5: The quantization features corresponding to each codebook are weighted and fused using learnable weights to obtain comprehensive quantization features. The comprehensive quantization features are then input into the decoder to reconstruct image blocks and calculate the normalized multimodal reconstruction error.
[0095] Step S6: Calculate the rarity score based on the activation frequency distribution and quantization index of the prototype vector in each codebook, and simultaneously calculate the quantization distance score. Calculate the comprehensive MCAR score using the rarity score and quantization distance score.
[0096] Step S7: The normalized multimodal reconstruction error and the comprehensive MCAR score are weighted and fused to obtain the final anomaly score. The anomaly screening and judgment are completed based on the final anomaly score.
[0097] Step S8: Simultaneously input the multimodal fusion feature vector into the multi-label classification decoder and output the multi-label classification result.
[0098] In the embodiments of this specification, after obtaining the colorectal pathology image to be detected, the original pathological slide scan image is first preprocessed, including: scanning resolution normalization, that is, uniform to 0.5μm / pixel, color space standardization, including white balance correction and color normalization to remove differences in staining batches, etc., and tissue region segmentation to remove blank background.
[0099] Furthermore, the multimodal preprocessing in step S1 may specifically include the following: H&E staining channel, PAS staining enhancement channel, and nuclear morphology feature channel.
[0100] Furthermore, the three-branch structure in step S2 includes: The ultra-high magnification branch, with a size of 32×32 pixels and a stride of 16, extracts 32×32 pixel ultra-high magnification image patches, focusing on capturing the microscopic structural features at the glandular epithelial cell level. Ultra-high magnification branch can distinguish subtle morphological changes in the cell nucleus, including the distribution pattern of nuclear chromatin (fine granular, coarse blocky, etc.), the integrity of the nuclear membrane (smooth, concave, irregular, etc.), and the number and location of nucleoli. In normal glandular epithelium, the cell nucleus is oval, and the chromatin is finely granular; in dysplastic and carcinogenic tissues, the cell nucleus is enlarged and deeply stained, and the chromatin is coarsely blocky. Ultra-high magnification branch is of significant value in identifying early carcinogenesis and high-grade intraepithelial neoplasia.
[0101] High-magnification branch, size 64×64, step size 32: Extracts 64-pixel × 64-pixel high-magnification image blocks to capture glandular structural features. At this magnification, the morphology of the glandular lumens (round, oval, slit-like, or irregular, etc.), the arrangement of the glands (regular, crowded, cribriform, etc.), and the stromal infiltration pattern can be clearly observed. In normal colorectal mucosa, glands are tubular, neatly arranged, and have regular lumens; in adenomas, glands branch and fold, densely arranged but not breaching the basement membrane; in adenocarcinomas, glandular structure is disordered, exhibiting cribriform or solid structures, with an incomplete basement membrane and stromal infiltration.
[0102] Low-magnification branch, size 128×128, stride 64: Extracts a 128×128 pixel low-magnification image patch to capture tissue structure features. At this magnification, the structure of the entire mucosal layer can be observed, including the morphology of crypts (regular tubular, dilated, serrated, etc.), the thickness of the mucosal layer, the degree of inflammatory cell infiltration in the lamina propria, and the vascular distribution in the submucosa. In normal mucosa, crypts are vertically arranged and evenly spaced; in abnormal mucosa, crypts show dilation, atrophy, or abnormal branching, and the mucosal layer may thicken or thin.
[0103] Furthermore, the three-branch complex-valued convolutional encoder in step S3 includes an ultra-high magnification encoder, a high magnification encoder, and a low magnification encoder. Each of the three-branch complex-valued convolutional encoders includes a complex-valued convolutional layer, a complex-valued batch normalization layer, and a complex-valued activation function, which are used to encode the corresponding image block into a complex-valued continuous feature vector.
[0104] Furthermore, the complex-valued convolution kernel of the complex-valued optical convolution accelerator in step S3 is defined as follows: ; Formula (4) in, For complex-valued convolution kernels in Complex weights at the location, For amplitude components, For phase components, It is the imaginary unit.
[0105] Amplitude component The spatial response pattern of the convolution kernel is controlled, similar to the weights of a traditional real-valued convolution kernel; phase components By controlling the rotational properties of the convolution kernel, a spatially varying phase modulation effect can be produced.
[0106] Complex convolution is defined as: ; Formula (5) in, For input real-value image patches, This is the output of a complex convolution.
[0107] When phase component When the phase component is non-zero, the convolution kernel produces a phase rotation effect in space, enhancing the response to image edges and orientations.
[0108] Amplitude-phase separation optimization strategy: amplitude component Through traditional gradient descent optimization learning, it is responsible for extracting the basic texture and structural features of the image; phase component Optimized through a phase propagation mechanism, it is responsible for extracting the contrast and color features of the image.
[0109] During training, the gradient of the amplitude component is: ; Formula (6) The gradient of the phase component is: ; Formula (7) in, To take the real part of a complex number, The conjugate of the input.
[0110] Furthermore, the amplitude-phase decoupling transformation in step S4 will transform the complex-valued eigenvector. Mapping to the real number space The specific steps are as described in formula (3) above, and will not be repeated here.
[0111] The amplitude component and the phase component encode different types of feature information: the amplitude component mainly reflects the changes in brightness and contrast of the image, corresponding to the changes in brightness and darkness and the roughness of texture of glandular structures in pathological images; the phase component mainly reflects the edge and direction information of the image, corresponding to the clarity of cell nuclei boundaries and the morphological features of glandular cavities in pathological images.
[0112] The dimension of the decoupled real-valued feature vector is ,forward The dimension is the encoding of amplitude information, and then... The dimension represents the encoding of phase information. This separate representation allows subsequent multi-label classification decoders to process amplitude and phase features separately, achieving more refined feature representation.
[0113] Furthermore, the multi-granularity hierarchical codebook in step S4 includes Discrete codebooks of different capacities: Small codebooks force the model to learn coarse-grained organizational features through higher compression ratios. Because the number of prototype vectors is limited, each vector needs to represent a wider range of normal tissue variations, capturing the overall structural patterns and crypt distribution regularities of the mucosa. This coarse-grained feature is of significant value in identifying a broad range of organizational anomalies.
[0114] Large-capacity codebooks, due to their large number of prototype vectors, can more finely divide the feature space and learn fine-grained cellular-level details. Fine-grained prototypes can distinguish subtle morphological changes in normal cells, such as a slight increase in the nucleocytoplasmic ratio and subtle changes in chromatin, playing a decisive role in identifying early cancerous changes and precancerous lesions.
[0115] Multi-granularity design enables the model to understand the abnormal features of colorectal pathology images from both coarse-grained and fine-grained perspectives. By weighted fusion, the advantages of different granularities are combined to improve the detection capability of various lesions.
[0116] Furthermore, the activation frequency in step S6 The formula for calculating the distribution is: ; Formula (8) Activation frequency reflects the representativeness of the prototype vectors in normal samples. During the training phase, the network was trained using only normal colorectal tissue pathological images, and the number of times each prototype vector was activated in each codebook was recorded. Normal samples tend to activate high-frequency prototype vectors, which represent typical normal tissue patterns; low-activation-frequency prototype vectors represent rare normal tissue variations or boundary conditions.
[0117] Rarity score calculation: ; Formula (9) The rarity score is obtained by taking the negative logarithm of the activation frequency. The lower the activation frequency, the higher the rarity score. hour, This indicates that the feature pattern is extremely rare in normal samples and has a high degree of anomalous confidence.
[0118] Quantitative distance score calculation: ; Formula (10) The quantized distance score reflects the Euclidean distance between the input feature and the selected prototype vector. A larger distance indicates that the feature deviates further from the normal pattern represented by the prototype vector. Percentile normalization is used to map the original distance to the location of the distance distribution in the training set, eliminating differences in distance magnitude between different codebooks and samples.
[0119] Integrated MCAR score fusion: ; Formula (11) The comprehensive MCAR score is obtained by weighted fusion of the rarity score and quantization distance score of each codebook. Normal samples tend to activate high-frequency prototype vectors and produce small quantization distances, thus obtaining lower MCAR scores; abnormal samples are more likely to activate low-frequency prototype vectors or produce large quantization distances, thus obtaining higher MCAR scores.
[0120] Furthermore, the final anomaly score in step S7 The calculation formula is: ; Formula (12) in, The fusion weighting coefficient has a range of values. .
[0121] when When the value approaches 1, the final score is mainly determined by the MCAR score, which is suitable for scenarios with weak reconstruction error discrimination; when... When the value approaches 0, the final score is mainly determined by the reconstruction error, which is suitable for scenarios where the confidence level of the MCAR score is insufficient. Preferably, The value ranges from 0.85 to 0.95, at which point the MCAR score dominates and can better identify abnormal lesions that are highly similar to the visual features of normal tissue.
[0122] The judgment rule is: if (default If the image shows abnormal signs of a precursor lesion to colorectal cancer, it is considered positive; if If the result is negative, it is considered normal, or negative.
[0123] Furthermore, the multi-label classification decoder in step S8 adopts a multilayer perceptron structure: ; Formula (13) in, For the first The predicted probability of each label. These correspond to adenocarcinoma, adenoma, hyperplastic polyp, and normal tissue, respectively. and The weights and biases for the classification layer.
[0124] The characteristic of multi-label classification is that the output of each label is independent, and the same image can belong to multiple categories simultaneously, such as the coexistence of adenomas and hyperplastic polyps. The Sigmoid activation function makes the output value of each label... Between these values, a threshold is used to determine whether the label is positive. The threshold can be 0.5 or other values, and no specific limitation is made here.
[0125] The multi-label classification loss function uses binary cross-entropy loss: ; Formula (14) in, For the first The true labeling of each tag For the first The predicted probability of each label.
[0126] Furthermore, the training method for the three-branch complex-valued convolutional encoder, multi-granularity hierarchical codebook, decoder, and activation frequency distribution includes: A training set containing only pathological images of normal colorectal tissue is obtained. The selection of the normal tissue training set is the basis of the unsupervised paradigm of this invention. By learning the distribution characteristics of normal tissue, samples deviating from this distribution are identified as potential abnormalities. Normal colorectal tissue includes normal mucosal epithelium, normal crypt structures, and normal vascular tissue, excluding any adenomas, hyperplastic polyps, or cancerous tissue.
[0127] An end-to-end network is constructed, consisting of a three-branch complex-valued convolutional encoder, a multi-granularity hierarchical codebook, and a decoder. The encoder encodes image patches into complex-valued feature vectors, the codebook quantizes continuous feature vectors into discrete prototypes, and the decoder reconstructs image patches based on the quantized features.
[0128] Calculate the total training loss function consisting of the three-branch reconstruction loss and the vector quantization loss. : ; Formula (15) in, The mean square error between the ultra-high magnification image patch and its reconstructed image patch. The mean square error between the high-magnification image patch and its reconstructed image patch. The mean square error between the low-magnification image patch and its reconstructed image patch. This is the vector quantization loss function.
[0129] Specifically, vector quantization loss function The calculation formula is: ; Formula (16) In the formula, This indicates that the gradient operation is stopped. The weighting coefficient for the committed loss is taken as follows: .
[0130] The first term is the codebook loss, which drives the prototype vector to move closer to the encoder output; the second term is the commitment loss, which constrains the encoder output not to deviate too far from the selected prototype vector.
[0131] Stop gradient operation Ensure that the two losses optimize different sets of parameters to avoid gradient conflicts.
[0132] The network is trained end-to-end using the AdamW optimizer, with an initial learning rate of... The weight decays to The learning rate is scheduled using a cosine annealing strategy, with the first 10 cycles being linear warm-up, a batch size of 16, and training for at least 100 cycles.
[0133] After training, fix all network parameters, traverse all normal samples in the training set, count the frequency of each prototype vector in each codebook being selected as the nearest neighbor, obtain the activation frequency distribution and fix it, as the benchmark for calculating the codebook activation sparse score in the inference stage.
[0134] Based on the same inventive concept, embodiments of this specification also provide a multimodal pathological multi-label recognition and diagnosis system for colorectal cancer based on a complex-valued optical convolution accelerator, which may specifically include the following modules: Multimodal preprocessing module: Used to perform multimodal preprocessing and three-branch image block extraction on the colorectal pathology images to be detected, generating a three-channel fused image including H&E staining channel, PAS staining enhancement channel and nuclear morphology feature channel, and extracting image blocks of three resolutions: ultra-high magnification, high magnification and low magnification according to preset size and step size.
[0135] The complex-valued optical convolution acceleration module is used to efficiently extract features from input image patches using complex-valued convolution kernels, outputting complex-valued continuous feature vectors. The amplitude component of the complex-valued convolution kernel extracts morphological and textural features, while the phase component extracts shading style and color contrast features. The complex-valued optical convolution accelerator can be implemented on physical optical systems, such as spatial light modulators and lens systems, utilizing the parallel propagation characteristics of light to achieve high-speed, low-power convolution operations; it can also be simulated on digital systems through complex-valued tensor operations.
[0136] Amplitude-phase decoupling module: This module maps complex-valued feature vectors to the real space through amplitude-phase decoupling transformation, resulting in a multimodal fusion feature vector. The amplitude and phase components encode different types of diagnostic information, and the decoupled representation facilitates processing by subsequent modules.
[0137] Multi-granularity hierarchical codebook module: Used to store pre-built multi-granularity hierarchical codebooks with fixed parameters and pre-statistical activation frequency distributions of each prototype vector in the codebook. The multi-granularity hierarchical codebook contains six discrete codebooks with different capacities of 32, 64, 128, 256, 512 and 1024, respectively, to achieve granular hierarchical representation of pathological features.
[0138] Vector quantization module: Used to perform nearest neighbor search on the multimodal fusion feature vector and to perform weighted fusion of the quantized features of each granularity to obtain the comprehensive quantized feature. .
[0139] Decoder reconstruction module: used to reconstruct based on comprehensive quantization features Reconstruct image patches and calculate normalized multimodal reconstruction error. .
[0140] MCAR scoring module: used for activation frequency distribution-based scoring With quantization index Calculate the rarity score step by step. Quantifying distance scores And the combined MCAR score is obtained by fusion. .
[0141] Multi-label classification module: This module is used to input multimodal fusion feature vectors into a multi-label classification decoder and output the classification probabilities of adenocarcinoma, adenoma, hyperplastic polyps and normal tissues, thereby achieving multi-label joint diagnosis.
[0142] Anomaly fusion determination module: used to determine the normalized multimodal reconstruction error. With the overall MCAR score The weighted fusion yields the final outlier score. It also outputs the results of the abnormality screening.
[0143] The technical solutions provided in the embodiments of this specification have the following advantages compared with the prior art: First, the embodiments in this specification combine optical computing with deep learning, utilizing a complex-valued optical convolution accelerator to achieve efficient feature extraction. The inherent parallelism of optical computing enables convolution operations to be tens of times faster than traditional digital computing, while reducing power consumption by more than 90%, providing a hardware foundation for real-time clinical diagnosis. The complex-valued convolution kernel achieves feature decoupling enhancement through an amplitude-phase dual-component mechanism: the amplitude component extracts morphological and textural features, while the phase component extracts staining style and color contrast features; the two complement each other, resulting in a more complete feature representation.
[0144] Secondly, the embodiments in this specification employ an unsupervised paradigm that uses only normal samples for network training, eliminating the need for scarce anomaly-labeled data and significantly lowering the application threshold for colorectal pathology image analysis. Traditional supervised learning methods require a large amount of labeled data for training, and high-quality pathological annotations need to be completed by experienced pathologists, which is costly and time-consuming. The unsupervised paradigm of this invention only requires the collection of normal colorectal tissue pathological images as training data, making data acquisition relatively easy.
[0145] Third, the embodiments in this specification, through their self-designed multimodal preprocessing and three-channel fusion strategy, fully utilize H&E staining information, PAS staining enhancement information, and nuclear morphology feature information, beyond normalizing reconstruction errors, to enhance the model's ability to perceive precursor lesions of colorectal cancer. H&E staining provides basic morphological information, PAS staining enhancement highlights mucosal secretory function, and nuclear morphology features quantify the degree of cellular atypia; these three complement each other to provide comprehensive diagnostic evidence.
[0146] Fourth, the embodiments in this specification extract image patches at different magnifications using a three-branch complex-valued convolutional encoder to capture multi-scale pathological features. Combined with the hierarchical representation capabilities of a multi-granularity hierarchical codebook, it can simultaneously cover pathological patterns at the cellular, glandular, and tissue levels, adapting to the cross-scale analysis needs for diagnosing precursor lesions of colorectal cancer. The ultra-high magnification branch identifies nuclear atypia, the high magnification branch identifies glandular structural abnormalities, and the low magnification branch identifies changes in tissue architecture.
[0147] Fifth, the embodiments in this specification achieve multi-label joint diagnosis through a multi-label classification decoder, which can simultaneously output the classification probabilities of adenocarcinoma, adenoma, hyperplastic polyps, and normal tissue, solving the problem that traditional single-label classification methods cannot handle scenarios with multiple coexisting labels. In clinical practice, a patient's specimen may contain multiple lesion types simultaneously, and multi-label classification technology can comprehensively assess the lesion composition of pathological images.
[0148] Sixth, the embodiments in this specification employ a multi-granularity hierarchical codebook activation sparseness evaluation mechanism. In addition to normalizing the reconstruction error, this mechanism supplements the evaluation with sparseness assessment based on the activation frequency of prototype vectors statistically derived during the training phase, as well as a quantized distance assessment between features and prototypes. This dual evaluation mechanism compensates for the limitations of pixel-level reconstruction errors at the semantic level, enabling effective identification even when anomalies are highly similar to the visual features of normal tissues.
[0149] Seventh, the design of the embodiments in this specification provides interpretable criteria for anomaly determination. Multi-channel fused images intuitively display various diagnostic information such as H&E staining, PAS staining enhancement, and nuclear morphology features; the three-branch structure corresponds to different levels of pathological features at different magnifications; the MCAR score is decomposed into a rarity score and a quantization distance score, allowing for the tracing of the activated prototype vector and the degree of quantization bias. This interpretable design provides clinicians with an intuitive reference, enhancing their trust in the system's judgments.
[0150] Based on the same inventive concept, this specification also provides a specific application embodiment of the multimodal pathological identification method for colorectal cancer.
[0151] like Figure 3 The diagram shown is a flowchart illustrating the multimodal preprocessing and three-channel fusion image processing in a multimodal pathological identification method for colorectal cancer provided in this embodiment of the specification.
[0152] In the embodiments of this specification, the specific method of multimodal preprocessing is as follows: H&E staining channel generation: The preprocessed H&E-stained pathological image is converted to grayscale and normalized. Interval: ;Formula (17) in, , , The original RGB image is in The red, green, and blue channel values at the location.
[0153] The weighting coefficients are designed based on the human eye's sensitivity to different colors, making the brightness perception of grayscale images more in line with the characteristics of the human eye.
[0154] PAS staining enhancement channel generation: Separation of PAS-positive substances using a color unmixing algorithm: ; Formula (1) in, , , is the color unmixing coefficient.
[0155] PAS-positive substances (such as mucus) appear purplish-red in H&E staining, with a high red channel value and a low blue channel value; cell nuclei appear blue-purple, with a high blue channel value.
[0156] By combining the above coefficients, the contrast between purplish-red and bluish-purple can be amplified.
[0157] Normalization process: ; Formula (18) Nuclear morphology feature channel generation: Extracting nuclear morphology features using a cell nuclear segmentation algorithm. First, a U-Net-structured cell kernel segmentation network is used to identify cell kernel regions in the image, obtaining a binary mask of the cell kernels: ; Formula (19) Then, the morphological parameters of each cell nucleus were calculated: ; ; ; Formula (20) Finally, the nuclear morphology features are mapped back to the original image resolution, assigning a roundness value to the nuclear region and a value of 0 to the non-nuclear region: ; Formula (21) Three-channel fused image: The H&E staining channel, PAS staining enhancement channel, and nuclear morphology feature channel are stitched together along the channel dimension to form a size of [missing information]. The three-channel tensor.
[0158] like Figure 4 The diagram shown is a flowchart illustrating the correspondence between the three-branch feature extraction architecture and the image block space in a multimodal pathological identification method for colorectal cancer provided in this embodiment of the specification.
[0159] In the embodiments of this specification, the pre-processed Extracting three-branch sliding window from three-channel fused images: The ultra-high multiplier branch is [size missing]. Step 16: Extract by sliding across the image in steps of 16 pixels. Image patches, which can be extracted horizontally and vertically respectively. Image patches were extracted in total. Each ultra-high magnification image block covers approximately [number] image patches. μm The actual tissue region, assuming a pixel resolution of 0.5 μm / pixel, is able to distinguish the morphological features of individual cells and is suitable for capturing the aberrations of cell nuclei.
[0160] High-magnification branch, size is Step size 32: Extract by sliding with a step size of 32 pixels. Image patches, which can be extracted horizontally and vertically respectively. Image patches were extracted in total. 1 image block. Each high-magnification image block covers approximately 100 image blocks. μm The actual tissue area is used to capture glandular structural features, including glandular cavity morphology, glandular arrangement, and stromal infiltration pattern.
[0161] Low-magnification branch, size is Step size 64: Extract by sliding with a step size of 64 pixels. Image patches, which can be extracted horizontally and vertically respectively. Image patches were extracted in total. 1 image patch. Each low-magnification image patch covers approximately 100 image patches. μm The actual tissue area is used to assess tissue architecture characteristics, including mucosal layer structure, crypt morphology, and vascular distribution patterns.
[0162] like Figure 5 The diagram shown is a flowchart illustrating the three-branch complex-valued convolutional encoder network structure in a multimodal pathological identification method for colorectal cancer provided in this embodiment of the specification.
[0163] In the embodiments described in this specification, each branch encoder consists of several complex-valued convolutional blocks connected in series. High-magnification branch ( The network structure is explained in detail using the input as an example: The structure of a complex-valued convolutional block: Each complex-valued convolutional block contains a complex-valued convolutional layer, a complex-valued batch normalization layer, and a complex-valued activation function.
[0164] The complex convolution kernel of a complex convolutional layer is defined as the product of magnitude and phase: ; Formula (22) First complex-valued convolutional block: 3 input channels, 32 output channels. Complex-valued convolutional kernel size. padding=1. The convolution operation yields a complex feature map, where the amplitude and phase components are preserved.
[0165] The normalization formula for the complex-valued batch normalization layer is: ; ; Formula (23) The complex-valued activation function uses Complex ReLU (CReLU): ; Formula (24) The second complex-valued convolutional block has 32 input channels and 64 output channels. After the convolution operation, the feature map size is preserved, but the number of channels is doubled.
[0166] The third complex-valued convolutional block has 64 input channels and 128 output channels. It incorporates pooling downsampling with a stride of 2, halving the feature map size.
[0167] Fourth complex-valued convolutional block: 128 input channels, 256 output channels. Feature map size is halved again.
[0168] Amplitude-phase decoupling layer: Maps the complex feature map output from the last complex convolution layer to the real space through an amplitude-phase decoupling transformation. ; Formula (25) The final output real-valued feature vector has a dimension of dimension( ).
[0169] like Figure 6 The diagram shown is a flowchart illustrating the multi-granularity hierarchical codebook vector quantization process in a multimodal pathological identification method for colorectal cancer provided in this embodiment.
[0170] In the embodiments described in this specification, the multi-granularity hierarchical codebook includes Discrete codebooks of different capacities The capacity of each codebook The codebooks contain 32, 64, 128, 256, 512, and 1024 codes respectively. A 256-dimensional learnable prototype vector.
[0171] Vector quantization process: For continuous feature vectors of the input... In each codebook, the nearest prototype vector is found using a nearest neighbor search: ; ; Formula (26) in, For quantization index, For quantification features, Number the prototype vector.
[0172] Specifically, comprehensive quantitative characteristics The calculation formula is: ; ; Formula (27) in, For the softmax function, For the first Each codebook corresponds to a learnable weight. This represents the number of codebooks.
[0173] Weighted fusion of comprehensive quantization features: quantization features from each codebook are weighted by learnable weights. Weighted summation after softmax normalization: ; . Formula (28) Furthermore, the embodiments in this specification also provide a detailed calculation process for the normalized multimodal reconstruction error as follows: Comprehensive quantitative characteristics The image patches are reconstructed by inputting them into the decoder of the corresponding branch. The decoder adopts a structure symmetrical to the encoder, containing transposed complex-valued convolutional layers to progressively upsample the 256-dimensional features to the original image patch size. For each image patch, the pixel-wise mean square reconstruction error between it and the original image patch is calculated: ; Formula (29) in, Three-channel configuration; ultra-high magnification branch: High-magnification branch: Low-magnification branch: .
[0174] The mean square error of all image patches within the same branch is averaged: ; ; ; Formula (30) in, , , The number of image blocks for each of the three branches.
[0175] The weighted arithmetic mean of the average errors of the three branches is taken to obtain the original reconstruction score: ; Formula (31) Weight settings , , The ultra-high magnification branch has the largest weight because abnormalities in cell nuclear morphology are mainly manifested through high-resolution features.
[0176] Finally, percentile normalization method is used: ; ; . Formula (32) like Figure 7 The diagram shown is a detailed flowchart illustrating the implementation of a multi-label classification decoder in a multimodal pathological identification method for colorectal cancer provided in this embodiment of the specification.
[0177] In the embodiments described in this specification, the multi-label classification decoder adopts a multilayer perceptron structure: First layer (input layer): Receives the 512-dimensional feature vector after amplitude-phase decoupling.
[0178] Second layer (hidden layer): 256 neurons, fully connected, activation function is ReLU: ; Formula (33) The third layer (output layer): 4 neurons, corresponding to adenocarcinoma, adenoma, hyperplastic polyp, and normal tissue respectively. Fully connected, with sigmoid activation function. ; Formula (34) Output Indicates the first The probability of a label being positive. The threshold is set to 0.5; a value greater than or equal to 0.5 is considered positive.
[0179] like Figure 8 The diagram shown is a flowchart illustrating the auxiliary assessment of endoscopic polyp pathological diagnosis corresponding to a multimodal pathological identification method for colorectal cancer provided in the embodiments of this specification.
[0180] In an embodiment of this specification, a 58-year-old male patient was admitted to the gastroenterology department of a tertiary hospital. A colonoscopy was performed due to changes in bowel habits, revealing a pedunculated polyp approximately 12 mm in diameter located 8 cm from the anus in the rectum. The polyp had a smooth mucosal surface and was removed via endoscopic mucosal resection (EMR). The resected specimen was routinely stained with H&E for slide preparation and sent for pathological examination.
[0181] Data acquisition and preprocessing: Pathological sections were scanned with a 20x objective lens to generate... Pixel-level digital pathological images. The system automatically performs multimodal preprocessing to generate three-channel fused images.
[0182] Multimodal preprocessing results analysis: The H&E staining channel clearly showed the glandular structure of the polyp, which was arranged in a tubular manner, with columnar epithelial cells and nuclei located at the base; the PAS staining enhancement channel showed that the glandular secretory function of the polyp was normal, the mucus secretion was uniform, and no abnormal secretion deposition was observed; the nuclear morphology characteristic channel showed that the cell nucleus was oval, the nuclear membrane was smooth, the nuclear chromatin was finely granular, and the roundness value was about 0.85.
[0183] Three-branch feature extraction: The ultra-high magnification branch extracts multiple image patches in densely celled areas, with cell nuclear feature vectors showing a regular distribution; the high magnification branch extracts multiple image patches in glandular structure areas, with glands arranged in a tubular shape and evenly spaced; the low magnification branch covers the entire mucosal structure of the polyp, showing an intact mucosal layer and no obvious inflammatory cell infiltration in the lamina propria.
[0184] System detection results: Calculation of normalized multimodal reconstruction error Comprehensive MCAR score Final abnormal score The system determined that the image contained abnormal signs. Multi-label classification results: adenocarcinoma probability 0.12 (low), adenoma probability 0.91 (high), hyperplastic polyp probability 0.08 (low), normal tissue probability 0.03 (low).
[0185] The diagnosis was determined to be an adenoma, such as a tubular adenoma or low-grade intraepithelial neoplasia, consistent with the pathologist's diagnosis.
[0186] Comparative analysis with pathologists: In 100 cases diagnosed as adenomas by pathologists, the system correctly identified 98 cases, with a sensitivity of 98%, specificity of 94%, and accuracy of 96.5%. In 20 cases diagnosed as carcinomas by pathologists, the system correctly identified 19 cases, with a sensitivity of 95%. The Kappa value of the system's diagnosis was 0.91, showing a high degree of consistency with the pathologist's diagnosis.
[0187] like Figure 9 The diagram shown is a flowchart illustrating the auxiliary diagnosis process for a large-scale colorectal cancer screening project corresponding to a multimodal pathological identification method for colorectal cancer provided in the embodiments of this specification.
[0188] In this embodiment of the specification, a provincial cancer hospital, in conjunction with 10 county-level hospitals in the same province, conducted a large-scale colorectal cancer screening program lasting 12 months. The screening target was asymptomatic individuals aged 40 to 70, totaling 25,000 people. The screening used a fecal occult blood test (FIT) as the initial screening method; those with positive FIT results underwent colonoscopy and biopsy. The system of this invention was deployed as a pathological diagnostic aid, performing AI-assisted diagnosis on all colonoscopy biopsy specimens, and pathologists verified and confirmed positive results determined by the system.
[0189] Large-scale screening workflow design: Pathology laboratories in county-level hospitals routinely perform H&E staining and slide preparation on biopsy specimens, and after digital scanning, upload the samples to the regional pathology diagnostic center via the PACS network. The system automatically completes the following processing steps: First, multimodal preprocessing. Digital pathological images are automatically read, and scan resolution normalization, white balance correction, and tissue region segmentation are performed. Three-channel fused images are then generated according to preset parameters.
[0190] Second, three-branch feature extraction. Three-branch image blocks are extracted according to a preset size and stride. Each image block is encoded into a complex-valued feature vector through a complex-valued optical convolution accelerator, and then subjected to an amplitude-phase decoupling transformation to obtain a 512-dimensional real-valued feature vector.
[0191] Third, multi-dimensional anomaly assessment. Normalized multimodal reconstruction error and comprehensive MCAR score are calculated, and weighted fusion is performed to obtain the final anomaly score. Simultaneously, a multi-label classification decoder outputs the classification probabilities of adenocarcinoma, adenoma, hyperplastic polyp, and normal tissue.
[0192] Fourth, automatic triage and report generation.
[0193] Automatic triage based on final anomaly score and multi-label classification results: Highly suspected cancer with an adenocarcinoma probability >0.8 was referred to a provincial cancer hospital for pathological consultation. Suspected adenomas with an adenoma probability >0.7 will be referred to the chief pathologist for review; Suspected hyperplastic polyps with a probability >0.6 will be referred to the attending pathologist for review; Negative reports are generated automatically.
[0194] Screening results statistics: After 12 months of screening and follow-up confirmation, among the 25,000 people screened who completed AI-assisted diagnosis, 3,500 were found to be FIT positive and underwent colonoscopy. Of the 3,500 biopsy specimens, 892 were identified as positive by the system and referred to physicians for review, accounting for 25.5%.
[0195] After review and confirmation by pathologists, the following were diagnosed cases: 1850 cases of adenoma (52.9%), including 1420 cases of low-grade intraepithelial neoplasia and 430 cases of high-grade intraepithelial neoplasia; 128 cases of adenocarcinoma (3.7%), including 67 cases of stage I, 42 cases of stage II, and 19 cases of stage III; 892 cases of hyperplastic polyps (25.5%); and 630 cases of normal tissue (18.0%). There were 186 false positives (5.3%), and approximately 35 missed cases (estimated missed diagnosis rate of 1.0%).
[0196] Adenoma detection rate analysis: The adenoma detection rate in this screening was 1850 / 25000 = 7.4%, significantly higher than the detection rate of adenoma in routine screening, which is approximately 3% to 5%. This indicates that AI-assisted diagnosis significantly improves the detection capability of adenomas. In particular, for small adenomas with a diameter of less than 10 mm, the detection rate using traditional methods is only about 40% to 50%, while this system achieved a detection rate of 82.3%, demonstrating superior detection capabilities for minute lesions.
[0197] Early cancer detection rate analysis: This screening diagnosed 67 cases of stage I colorectal cancer, accounting for 52.3% (67 / 128), significantly higher than the early cancer rate in routine screening, which is approximately 20% to 30%. Early colorectal cancer can be cured through endoscopic resection or minimally invasive surgery, with a 5-year survival rate exceeding 90%. AI-assisted diagnosis enables more early-stage cancer patients to be detected and treated in a timely manner, significantly improving patient prognosis.
[0198] Results of the controlled study: A controlled study was conducted concurrently in three county-level hospitals, involving 500 biopsy specimens, with both systematic auxiliary diagnosis and independent diagnosis by attending pathologists. The results of the controlled study are shown in Table 1 below: Table 1. Results of the comparative study ; Comparative studies have shown that the adenoma detection rate of the system in this specification (96.8%) is significantly higher than that of attending pathologists (89.2%) and resident pathologists (75.4%), and the cancer detection rate (98.4%) is also optimal. The system of this invention diagnoses a case in just 8 seconds, with a daily processing capacity of approximately 1,080 cases, which is 22.5 times that of attending pathologists (approximately 48 cases / day), effectively alleviating the shortage of pathologists in primary healthcare institutions.
[0199] Economic Benefit Analysis: The economic benefit analysis of this large-scale screening project shows that after using the AI-assisted diagnosis system of this invention, the workload of pathologists for review is reduced by approximately 892 / 1620 = 55%. Assuming that all positive cases need to be reviewed, the diagnosis time is shortened by approximately 78%, from 180 seconds / case to 40 seconds / case, including review.
[0200] Based on the average hourly wage of local pathologists, a single screening can save approximately 45 yuan per person in diagnostic costs, resulting in a cumulative saving of approximately 1.125 million yuan for 25,000 people. Furthermore, the cost of treatment after early detection of colorectal cancer is approximately one-quarter to one-third of the cost of treatment for late-stage cancer. Late-stage surgery plus chemotherapy costs approximately 200,000 to 300,000 yuan, while early endoscopic resection costs approximately 10,000 to 20,000 yuan. Early detection can significantly reduce the overall medical burden.
[0201] The specific implementation process of the embodiments in this specification can be referred to the various implementation steps corresponding to the above embodiments, and will not be repeated here.
[0202] Based on the same inventive concept, embodiments of this specification also provide a multimodal pathological identification system for colorectal cancer. For example... Figure 10 The diagram shown is a structural schematic of a multimodal pathological recognition system for colorectal cancer provided in an embodiment of this specification.
[0203] Specifically, the colorectal cancer multimodal pathological recognition system may include: The preprocessing module 1001 performs multimodal preprocessing, magnification image block extraction, and decoupling transformation operations sequentially on the acquired colorectal pathological image to be detected, to obtain the complex value continuous feature vector corresponding to the magnification image block, and the multimodal fusion feature vector corresponding to the complex value continuous feature vector; The fusion module 1002 searches for quantization features and quantization indices in the multi-granularity hierarchical codebooks using the multi-modal fusion feature vector, and performs weighted fusion of the quantization features corresponding to each codebook through learnable weights to obtain comprehensive quantization features. The reconstruction module 1003 performs image reconstruction on the comprehensive quantitative features and calculates the first anomaly evaluation index based on the image reconstruction results. The calculation module 1004 calculates a second anomaly evaluation index based on the quantization index and the activation frequency distribution of the prototype vectors in each codebook. The judgment module 1005 performs weighted fusion of the first abnormality assessment index and the second abnormality assessment index to obtain a target abnormality score, so as to complete the abnormality screening and judgment of the colorectal pathological image to be detected based on the target abnormality score. The multi-label classification module 1006 simultaneously inputs the multimodal fusion feature vector into the multi-label classification decoder and outputs the multi-label classification result of colorectal cancer precursor lesions, wherein the multi-label classification result includes at least the classification probabilities of adenocarcinoma, adenoma, hyperplastic polyp and normal tissue.
[0204] based on Figure 10 The system described in this specification also provides some specific implementation schemes of the system, which will be described below.
[0205] Furthermore, the acquired colorectal pathological images to be detected are subjected to multimodal preprocessing, magnification image patch extraction, and decoupling transformation operations in sequence, including: Obtain pathological images of the colorectal tissue to be examined; The colorectal pathological images to be detected are subjected to multimodal preprocessing to generate three-channel fused images; The three-channel fused image is subjected to magnification image block extraction; Based on the magnification image patch, a complex-valued optical convolution accelerator is used for efficient feature extraction to obtain a complex-valued continuous feature vector corresponding to the magnification image patch. The complex-valued continuous feature vector is mapped to the real number space through a decoupling transformation to obtain the multimodal fusion feature vector.
[0206] Furthermore, the colorectal pathology images are subjected to multimodal preprocessing to generate a three-channel fused image, including: The original H&E staining image corresponding to the colorectal pathology image to be detected is converted into a grayscale channel. The simulated PAS staining enhancement channel is separated from the RGB channels of the original H&E staining image based on the color unmixing algorithm. The coefficient of the color unmixing is set according to the contrast characteristics of PAS positive substances and cell nuclei. The original H&E staining image was segmented into cell nuclei, and the morphological parameters of the cell nuclei were extracted to generate nuclear morphology feature channels. The grayscale channel, the simulated PAS staining enhancement channel, and the nuclear morphology feature channel are stitched together along the channel dimension to generate the three-channel fused image.
[0207] Furthermore, the three-channel fused image is subjected to magnification image block extraction, including: The colorectal pathological image to be detected is subjected to multi-scale sampling, wherein the multi-scale sampling includes at least one of the following methods: Ultra-high magnification sampling is used to extract ultra-high magnification image patches of the first size to capture cellular-level microstructural features; High-magnification sampling is used to extract high-magnification image patches of a second size to capture glandular structural features; Low-magnification sampling is used to extract low-magnification image patches of the third size to capture organizational features; The first dimension is smaller than the second dimension, and the second dimension is smaller than the third dimension.
[0208] Furthermore, based on the magnification image patch, a complex-valued optical convolution accelerator is used for efficient feature extraction to obtain a complex-valued continuous feature vector corresponding to the magnification image patch, including: The ultra-high magnification image block, the high magnification image block, and the low magnification image block are respectively input into the three branches of the three-branch complex value convolutional encoder; Efficient feature extraction is performed on the ultra-high magnification image patch, the high magnification image patch, and the low magnification image patch using a feature extraction network to obtain the complex-valued continuous feature vectors corresponding to each of the ultra-high magnification image patch, the high magnification image patch, and the low magnification image patch. The complex-valued continuous feature vectors include amplitude components and phase components. The amplitude components are used to extract the morphological and textural features of the pathological image, and the phase components are used to extract the staining style and color contrast features of the pathological image.
[0209] Furthermore, the system also includes: The complex-valued continuous feature vector is mapped to the real space through an amplitude-phase decoupling transformation according to the following formula to obtain the multimodal fusion feature vector: ; in, This is a multimodal fusion feature vector. For complex-valued continuous eigenvectors, The real space is , For amplitude components, For phase components, This represents a vector concatenation operation. and These represent the real and imaginary parts of a complex number, respectively.
[0210] Furthermore, the multimodal fusion feature vector is used to search for quantization features and quantization indices in a multi-granularity hierarchical codebook. The quantization features corresponding to each codebook are then weighted and fused using learnable weights to obtain comprehensive quantization features, including: Obtain multiple discrete codebooks of different capacities, wherein the discrete codebooks contain at least three capacity levels, and each codebook contains a different number of learnable prototype vectors. Find the quantization features corresponding to each of the codebooks; The comprehensive quantization feature is obtained by weighting and fusing the quantization features corresponding to each codebook using learnable weights.
[0211] Furthermore, image reconstruction is performed on the comprehensive quantized features, and a first anomaly evaluation index is calculated based on the image reconstruction results, including: The comprehensive quantization features are input into the corresponding branch's magnification decoder to reconstruct the ultra-high magnification image block, the high magnification image block, and the low magnification image block, thereby obtaining the image reconstruction result; Based on the image reconstruction results, the pixel-wise mean square reconstruction error between each reconstructed image block and the corresponding original magnified image block is calculated; Calculate the average error value of the pixel-wise mean square reconstruction error for all image blocks under each branch; The weighted arithmetic mean of the average error values corresponding to each branch is used to obtain the original reconstruction score; The original reconstruction score is normalized to percentiles to obtain the first anomaly assessment index.
[0212] Furthermore, based on the quantization index and the activation frequency distribution of the prototype vectors in each codebook, a second anomaly evaluation index is calculated, including: Based on the activation frequency distribution and quantization index of the prototype vectors in each codebook, the rarity score corresponding to the prototype vector in the codebook is calculated, wherein the rarity score is negatively correlated with the activation disclosure distribution; The quantized distance score between the multimodal fusion feature vector and the prototype vector is calculated using Euclidean distance. The rarity score and the quantized distance score are weighted and fused to obtain the second anomaly evaluation index.
[0213] Furthermore, the first abnormality assessment index and the second abnormality assessment index are weighted and fused to obtain a target abnormality score, so as to complete the abnormality screening and determination of the colorectal pathological image to be detected based on the target abnormality score, including: If the target abnormality score exceeds a preset threshold, it is determined that the colorectal pathological image to be detected has abnormal precancerous lesions of colorectal cancer. If the target abnormality score is less than the preset threshold, the colorectal pathological image to be detected is determined to be normal.
[0214] Furthermore, the multimodal fused feature vector is simultaneously input into a multi-label classification decoder to output a multi-label classification result for colorectal cancer precursor lesions, including: The multi-label classification decoder adopts a multilayer perceptron structure, and the output layer uses the Sigmoid activation function to achieve multi-label classification.
[0215] This specification provides a multimodal pathological recognition system for colorectal cancer. It employs a complex-valued optical convolutional accelerator to efficiently extract features from input colorectal pathological images and performs multimodal preprocessing. A complex-valued convolutional encoder extracts complex-valued feature vectors for each modality. These complex-valued features are then mapped to real space through amplitude-phase decoupling transformation to obtain a multimodal fusion feature vector. This fusion feature vector undergoes vector quantization using a multi-granularity hierarchical codebook to obtain comprehensive quantized features. A first anomaly assessment index and a second anomaly assessment index are calculated, and a target anomaly score is obtained through weighted fusion. Based on this target anomaly score, the system screens and determines anomalies in the colorectal pathological images to be detected. Simultaneously, a multi-label classification decoder outputs multi-label diagnostic results, enabling the classification and recognition of adenocarcinoma, adenoma, hyperplastic polyps, and normal tissue.
[0216] Thus, the embodiments of this specification combine optical computing with deep learning, and utilize the feature decoupling capability of the complex value domain to improve the recognition accuracy of colorectal cancer precursor lesions. This has important application value in large-scale screening and clinical auxiliary diagnosis, and effectively reduces the clinical risks caused by missed diagnoses or failures due to abnormal screening judgments of the colorectal pathological images to be detected or the output of multi-label classification probabilities.
[0217] Based on the same inventive concept, embodiments of this specification also provide an electronic device, including at least one processor and a memory, wherein the memory stores a program and is configured to be executed by the at least one processor in the following steps: The acquired colorectal pathological images to be detected are subjected to multimodal preprocessing, magnification image block extraction and decoupling transformation operations in sequence to obtain the complex value continuous feature vector corresponding to the magnification image block, and the multimodal fusion feature vector corresponding to the complex value continuous feature vector; The multimodal fusion feature vector is used to search for quantization features and quantization indices in a multi-granularity hierarchical codebook. The quantization features corresponding to each codebook are weighted and fused using learnable weights to obtain a comprehensive quantization feature. The comprehensive quantitative features are used to reconstruct the image, and a first anomaly evaluation index is calculated based on the image reconstruction results; The second anomaly evaluation index is calculated based on the quantization index and the activation frequency distribution of the prototype vectors in each codebook. The first abnormality assessment index and the second abnormality assessment index are weighted and fused to obtain a target abnormality score, so as to complete the abnormality screening and judgment of the colorectal pathological image to be detected based on the target abnormality score; The multimodal fusion feature vector is simultaneously input into a multi-label classification decoder, and the multi-label classification result of colorectal cancer precursor lesions is output. The multi-label classification result includes at least the classification probabilities of adenocarcinoma, adenoma, hyperplastic polyp and normal tissue.
[0218] Other functions of the processor can be found in the above embodiments, and will not be repeated here.
[0219] Based on the same inventive concept, embodiments of this specification also provide a computer-readable storage medium, including a program for use in conjunction with an electronic device, the program being executable by a processor to perform the following steps: The acquired colorectal pathological images to be detected are subjected to multimodal preprocessing, magnification image block extraction and decoupling transformation operations in sequence to obtain the complex value continuous feature vector corresponding to the magnification image block, and the multimodal fusion feature vector corresponding to the complex value continuous feature vector; The multimodal fusion feature vector is used to search for quantization features and quantization indices in a multi-granularity hierarchical codebook. The quantization features corresponding to each codebook are weighted and fused using learnable weights to obtain a comprehensive quantization feature. The comprehensive quantitative features are used to reconstruct the image, and a first anomaly evaluation index is calculated based on the image reconstruction results; The second anomaly evaluation index is calculated based on the quantization index and the activation frequency distribution of the prototype vectors in each codebook. The first abnormality assessment index and the second abnormality assessment index are weighted and fused to obtain a target abnormality score, so as to complete the abnormality screening and judgment of the colorectal pathological image to be detected based on the target abnormality score; The multimodal fusion feature vector is simultaneously input into a multi-label classification decoder, and the multi-label classification result of colorectal cancer precursor lesions is output. The multi-label classification result includes at least the classification probabilities of adenocarcinoma, adenoma, hyperplastic polyp and normal tissue.
[0220] Other functions of the processor can be found in the above embodiments, and will not be repeated here.
[0221] like Figure 11 As shown in the figure, this specification also provides a schematic diagram of the structure of a computer storage medium.
[0222] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.
[0223] The embodiments described in this disclosure are for the purpose of more clearly illustrating the technical solutions of this disclosure and do not constitute a limitation on the technical solutions provided by this disclosure. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by this disclosure are also applicable to similar technical problems.
[0224] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this disclosure, and may include more or fewer steps than shown, or combine certain steps, or different steps.
[0225] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.
[0226] The systems, devices, modules, or units described in the embodiments of the above specification can be implemented by computer chips or physical entities, or by products with certain functions. A typical implementation device can be a computer. Specifically, a computer can be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or any combination of these devices.
[0227] For ease of description, the above apparatus is described by dividing it into various functional units. Of course, when implementing the embodiments of this specification, the functions of each unit can be implemented in one or more software and / or hardware.
[0228] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0229] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0230] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0231] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0232] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0233] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.
[0234] Computer-readable media include both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.
[0235] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0236] This application can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.
[0237] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.
[0238] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of the claims of this application.
Claims
1. A multimodal pathological identification method for colorectal cancer, characterized in that, The multimodal pathological identification method for colorectal cancer includes: The acquired colorectal pathological images to be detected are subjected to multimodal preprocessing, magnification image block extraction and decoupling transformation operations in sequence to obtain the complex value continuous feature vector corresponding to the magnification image block, and the multimodal fusion feature vector corresponding to the complex value continuous feature vector; The multimodal fusion feature vector is used to search for quantization features and quantization indices in a multi-granularity hierarchical codebook. The quantization features corresponding to each codebook are weighted and fused using learnable weights to obtain a comprehensive quantization feature. The comprehensive quantitative features are used to reconstruct the image, and a first anomaly evaluation index is calculated based on the image reconstruction results; The second anomaly evaluation index is calculated based on the quantization index and the activation frequency distribution of the prototype vectors in each codebook. The first abnormality assessment index and the second abnormality assessment index are weighted and fused to obtain a target abnormality score, so as to complete the abnormality screening and judgment of the colorectal pathological image to be detected based on the target abnormality score; The multimodal fusion feature vector is simultaneously input into a multi-label classification decoder, and the multi-label classification result of colorectal cancer precursor lesions is output. The multi-label classification result includes at least the classification probabilities of adenocarcinoma, adenoma, hyperplastic polyp and normal tissue.
2. The multimodal pathological identification method for colorectal cancer according to claim 1, characterized in that, The acquired colorectal pathological images to be detected are subjected to multimodal preprocessing, magnification image block extraction, and decoupling transformation operations in sequence, including: Obtain pathological images of the colorectal tissue to be examined; The colorectal pathological images to be detected are subjected to multimodal preprocessing to generate three-channel fused images; The three-channel fused image is subjected to magnification image block extraction; Based on the magnification image patch, a complex-valued optical convolution accelerator is used for efficient feature extraction to obtain a complex-valued continuous feature vector corresponding to the magnification image patch. The complex-valued continuous feature vector is mapped to the real number space through a decoupling transformation to obtain the multimodal fusion feature vector.
3. The multimodal pathological identification method for colorectal cancer as described in claim 2, characterized in that, The colorectal pathology images are subjected to multimodal preprocessing to generate a three-channel fused image, including: The original H&E staining image corresponding to the colorectal pathology image to be detected is converted into a grayscale channel. The simulated PAS staining enhancement channel is separated from the RGB channels of the original H&E staining image based on the color unmixing algorithm. The coefficient of the color unmixing is set according to the contrast characteristics of PAS positive substances and cell nuclei. The original H&E staining image was segmented into cell nuclei, and the morphological parameters of the cell nuclei were extracted to generate nuclear morphology feature channels. The grayscale channel, the simulated PAS staining enhancement channel, and the nuclear morphology feature channel are stitched together along the channel dimension to generate the three-channel fused image.
4. The multimodal pathological identification method for colorectal cancer as described in claim 2, characterized in that, The three-channel fused image is subjected to magnification image patch extraction, including: The colorectal pathological image to be detected is subjected to multi-scale sampling, wherein the multi-scale sampling includes at least one of the following methods: Ultra-high magnification sampling is used to extract ultra-high magnification image patches of the first size to capture cellular-level microstructural features; High-magnification sampling is used to extract high-magnification image patches of a second size to capture glandular structural features; Low-magnification sampling is used to extract low-magnification image patches of the third size to capture organizational features; The first dimension is smaller than the second dimension, and the second dimension is smaller than the third dimension.
5. The multimodal pathological identification method for colorectal cancer as described in claim 4, characterized in that, Based on the magnification image patch, efficient feature extraction is performed using a complex-valued optical convolution accelerator to obtain a complex-valued continuous feature vector corresponding to the magnification image patch, including: The ultra-high magnification image block, the high magnification image block, and the low magnification image block are respectively input into the three branches of the three-branch complex value convolutional encoder; Efficient feature extraction is performed on the ultra-high magnification image patch, the high magnification image patch, and the low magnification image patch using a feature extraction network to obtain the complex-valued continuous feature vectors corresponding to each of the ultra-high magnification image patch, the high magnification image patch, and the low magnification image patch. The complex-valued continuous feature vectors include amplitude components and phase components. The amplitude components are used to extract the morphological and textural features of the pathological image, and the phase components are used to extract the staining style and color contrast features of the pathological image.
6. The method for multimodal pathological identification of colorectal cancer as described in claim 2, characterized in that, The method further includes: The complex-valued continuous feature vector is mapped to the real space through an amplitude-phase decoupling transformation according to the following formula to obtain the multimodal fusion feature vector: ; in, This is a multimodal fusion feature vector. For complex-valued continuous eigenvectors, The real space is , For amplitude components, For phase components, This represents a vector concatenation operation. and These represent the real and imaginary parts of a complex number, respectively.
7. The multimodal pathological identification method for colorectal cancer as described in claim 1, characterized in that, The multimodal fusion feature vector is used to search for quantization features and quantization indices in a multi-granularity hierarchical codebook. The quantization features corresponding to each codebook are then weighted and fused using learnable weights to obtain comprehensive quantization features, including: Obtain multiple discrete codebooks of different capacities, wherein the discrete codebooks contain at least three capacity levels, and each codebook contains a different number of learnable prototype vectors. Find the quantization features corresponding to each of the codebooks; The comprehensive quantization feature is obtained by weighting and fusing the quantization features corresponding to each codebook using learnable weights.
8. The multimodal pathological identification method for colorectal cancer as described in claim 5, characterized in that, Image reconstruction is performed on the comprehensive quantized features, and a first anomaly evaluation index is calculated based on the image reconstruction results, including: The comprehensive quantization features are input into the corresponding branch's magnification decoder to reconstruct the ultra-high magnification image block, the high magnification image block, and the low magnification image block, thereby obtaining the image reconstruction result; Based on the image reconstruction results, the pixel-wise mean square reconstruction error between each reconstructed image block and the corresponding original magnified image block is calculated; Calculate the average error value of the pixel-wise mean square reconstruction error for all image blocks under each branch; The weighted arithmetic mean of the average error values corresponding to each branch is used to obtain the original reconstruction score; The original reconstruction score is normalized to percentiles to obtain the first anomaly assessment index.
9. The multimodal pathological identification method for colorectal cancer as described in claim 7, characterized in that, Based on the quantization index and the activation frequency distribution of the prototype vectors in each codebook, a second anomaly evaluation index is calculated, including: Based on the activation frequency distribution and quantization index of the prototype vectors in each codebook, the rarity score corresponding to the prototype vector in the codebook is calculated, wherein the rarity score is negatively correlated with the activation disclosure distribution; The quantized distance score between the multimodal fusion feature vector and the prototype vector is calculated using Euclidean distance. The rarity score and the quantized distance score are weighted and fused to obtain the second anomaly evaluation index.
10. The multimodal pathological identification method for colorectal cancer as described in claim 1, characterized in that, The first anomaly assessment index and the second anomaly assessment index are weighted and fused to obtain a target anomaly score, which is then used to complete the anomaly screening and determination of the colorectal pathology image to be detected, including: If the target abnormality score exceeds a preset threshold, it is determined that the colorectal pathological image to be detected has abnormal precancerous lesions of colorectal cancer. If the target abnormality score is less than the preset threshold, the colorectal pathological image to be detected is determined to be normal.
11. The multimodal pathological identification method for colorectal cancer as described in claim 1, characterized in that, The multimodal fused feature vectors are simultaneously input into a multi-label classification decoder, which outputs multi-label classification results of colorectal cancer precursor lesions, including: The multi-label classification decoder adopts a multilayer perceptron structure, and the output layer uses the Sigmoid activation function to achieve multi-label classification.
12. A multimodal pathological identification system for colorectal cancer, characterized in that, The colorectal cancer multimodal pathological identification system includes: The preprocessing module sequentially performs multimodal preprocessing, magnification image block extraction, and decoupling transformation operations on the acquired colorectal pathological image to be detected, to obtain the complex-valued continuous feature vector corresponding to the magnification image block, and the multimodal fusion feature vector corresponding to the complex-valued continuous feature vector; The fusion module searches for quantization features and quantization indices in a multi-granularity hierarchical codebook using the multimodal fusion feature vector, and performs weighted fusion of the quantization features corresponding to each codebook using learnable weights to obtain a comprehensive quantization feature. The reconstruction module performs image reconstruction on the comprehensive quantitative features and calculates the first anomaly evaluation index based on the image reconstruction results. The calculation module calculates a second anomaly evaluation index based on the quantization index and the activation frequency distribution of the prototype vectors in each codebook. The determination module performs weighted fusion of the first abnormality assessment index and the second abnormality assessment index to obtain a target abnormality score, so as to complete the abnormality screening and determination of the colorectal pathological image to be detected based on the target abnormality score. The multi-label classification module simultaneously inputs the multimodal fusion feature vector into the multi-label classification decoder and outputs the multi-label classification result of colorectal cancer precursor lesions. The multi-label classification result includes at least the classification probabilities of adenocarcinoma, adenoma, hyperplastic polyp, and normal tissue.
13. A computer-readable storage medium storing a computer program for use in conjunction with an electronic device, characterized in that, The program can be executed by the processor to complete the following steps: The acquired colorectal pathological images to be detected are subjected to multimodal preprocessing, magnification image block extraction and decoupling transformation operations in sequence to obtain the complex value continuous feature vector corresponding to the magnification image block, and the multimodal fusion feature vector corresponding to the complex value continuous feature vector; The multimodal fusion feature vector is used to search for quantization features and quantization indices in a multi-granularity hierarchical codebook. The quantization features corresponding to each codebook are weighted and fused using learnable weights to obtain a comprehensive quantization feature. The comprehensive quantitative features are used to reconstruct the image, and a first anomaly evaluation index is calculated based on the image reconstruction results; The second anomaly evaluation index is calculated based on the quantization index and the activation frequency distribution of the prototype vectors in each codebook. The first abnormality assessment index and the second abnormality assessment index are weighted and fused to obtain a target abnormality score, so as to complete the abnormality screening and judgment of the colorectal pathological image to be detected based on the target abnormality score; The multimodal fusion feature vector is simultaneously input into a multi-label classification decoder, and the multi-label classification result of colorectal cancer precursor lesions is output. The multi-label classification result includes at least the classification probabilities of adenocarcinoma, adenoma, hyperplastic polyp and normal tissue.
