A method and apparatus for grading epithelial tumors

Abstract

The present disclosure provides an epithelial tumor grading method and device. The present disclosure obtains epithelial tumor pathological image data at multiple scales; the number of scales of the multiple scales includes at least two; for the pathological image data of any one scale, a feature extraction mechanism is used to obtain corresponding regional features; all regional features are spliced or fused to obtain fused features; the fused features are sequentially input into a first model and a second model which have been pre-trained to obtain a target grading result; the first model integrates macro-structure information and micro-morphology information of the fused features based on cosine similarity, cross-attention mechanism and gating fusion mechanism; the second model weights, aggregates and grades the information integrated by the first model based on an aggregated constraint attention learning mechanism to obtain the target grading result. In summary, the present disclosure can improve the accuracy and efficiency of epithelial tumor grading and can adapt to various application scenarios.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence medical image analysis and pathological auxiliary diagnosis technology, and in particular to a method and apparatus for grading epithelial tumors. Background Technology

[0002] Currently, the grading of papillary urothelial tumors largely relies on the subjective judgment of pathologists or on single-level features or single supervisory signals in weakly supervised grading models. However, pathologists' subjective judgment is easily influenced by at least one of the following: clinical experience, visual fatigue, and individual cognitive differences, making it difficult to achieve a standardized and accurate distinction in tumor grading. Weakly supervised grading models, limited by single-level feature extraction, cannot simultaneously capture both nuclear-level microscopic morphological features and tissue-level macroscopic features. Furthermore, single supervisory signals are difficult to adapt to the continuous changes in tumor pathological features, making it difficult to construct multi-dimensional, hierarchical feature association expressions. Thus, manual grading by pathologists is inefficient and struggles to meet the diagnostic needs of large volumes of clinical pathological slides, while weakly supervised models, due to incomplete feature capture and single supervisory signals, are prone to grading bias. Therefore, existing methods for grading epithelial tumors have low accuracy and efficiency, making them unsuitable for various application scenarios. Summary of the Invention

[0003] This disclosure provides a method and apparatus for grading epithelial tumors, which to some extent solves the problems of low accuracy and efficiency of existing methods for grading epithelial tumors and their inability to adapt to various application scenarios.

[0004] According to one aspect of this disclosure, a method for grading epithelial tumors is provided. The method includes: acquiring epithelial tumor pathological image data at multiple scales; the number of scales includes at least two; for any one scale of pathological image data, using a feature extraction mechanism to obtain corresponding regional features; concatenating or fusing all regional features to obtain fused features; sequentially inputting the fused features into a pre-trained first model and a second model to obtain a target grading result; the first model integrates the macroscopic structural information and microscopic morphological information of the fused features based on cosine similarity, cross-attention mechanism, and gating fusion mechanism; the second model performs weighted aggregation and grading discrimination on the information integrated by the first model based on an aggregation-constrained attention learning mechanism to obtain the target grading result; both macroscopic structural information and microscopic morphological information are determined based on the fused features.

[0005] Furthermore, according to one aspect of the method of this disclosure, the target classification results include: benign, potentially low-grade malignant, low-grade malignant, and high-grade malignant.

[0006] Furthermore, according to one aspect of the method disclosed herein, macroscopic structural information is the Region of Interest (ROI); microscopic morphological information is the Image Patch; the fused features are sequentially input into a pre-trained first model and a second model to obtain target classification results, including: determining ROIs and Patches based on labeled standard diagnostic regions, sliding window clipping values, and fused features; inputting the fused features into the first model to calculate the cosine similarity between ROIs and Patches, and selecting the ROI with the highest similarity; based on a cross-attention algorithm, alternately determining the selected ROIs and Patches as query vectors and key-value vectors; based on a gated fusion network, weighted integration is performed on the ROIs and Patches before and after the alternation, ROIs determined as query vectors and Patches as key-value vectors, and ROIs determined as key-value vectors and Patches as query vectors to generate feature identifiers; the feature identifiers are input into the second model, and based on preset diagnostic weights, they are distinguished through aggregation constraints to output target classification results.

[0007] Furthermore, according to one aspect of the method disclosed herein, ROI and Patch are determined based on labeled standard diagnostic regions, sliding window cropping values, and fusion features, including: delineating the lesion region range in the multi-scale pathological image corresponding to the fusion features based on labeled standard diagnostic regions to obtain a candidate region set; performing traversal cropping on the candidate region set based on sliding window cropping values ​​to generate an image patch sequence; when the lesion matching degree between the image patch sequence and the fusion features is greater than a preset threshold, the candidate region set is determined as the ROI, and the image patch sequence is determined as the Patch.

[0008] Furthermore, according to one aspect of the method disclosed herein, based on a gated fusion network, the ROI and Patch before and after the alternation, the ROI determined as a query vector and the Patch determined as a key-value vector, and the ROI determined as a key-value vector and the Patch determined as a query vector are weighted and integrated to generate feature identifiers. This includes: determining the first feature vector of the ROI before the alternation and the second feature vector of the Patch; determining the third feature vector of the ROI after the alternation, where the ROI is determined as a query vector and the Patch is determined as a key-value vector; determining the fourth feature vector of the ROI after the alternation, where the ROI is determined as a key-value vector and the Patch is determined as a query vector; inputting the first feature vector, the second feature vector, the third feature vector, and the fourth feature vector into the fully connected layer of the gated fusion network to determine their corresponding weight coefficients; performing a weighted summation of the first feature vector, the second feature vector, the third feature vector, and the fourth feature vector based on the weight coefficients, and performing normalization processing to obtain feature identifiers; the normalization processing includes norm normalization processing and mean centering processing.

[0009] Furthermore, according to one aspect of the method of this disclosure, the diagnostic weights include: structural integrity weights of the ROI and cell morphology abnormality weights of the Patch; the feature identifiers are input into a second model, and based on the preset diagnostic weights, they are distinguished by aggregation constraints to output the target classification result, including: obtaining the preset structural integrity weights of the ROI and cell morphology abnormality weights of the Patch; inputting the feature identifiers into the aggregation constraint attention module of the second model, and performing weighted aggregation on the feature identifiers based on the structural integrity weights of the ROI and the cell morphology abnormality weights of the Patch to obtain aggregated constrained features; inputting the aggregated constrained features into the classification discrimination submodule of the second model to determine the probability distribution of the classification result; and determining the target classification result based on the probability distribution.

[0010] Furthermore, according to one aspect of the method of this disclosure, for pathological image data of any scale, a feature extraction mechanism is used to obtain the corresponding regional features, including: preprocessing the pathological image data; the preprocessing includes at least one of the following: image denoising, color normalization, and image enhancement; inputting the preprocessed pathological image data into a feature extraction convolutional neural network, and outputting the feature vector of the corresponding lesion region as the regional feature; the feature extraction convolutional neural network includes: a deep residual network and a visual geometric group network.

[0011] Furthermore, according to one aspect of the method disclosed herein, all regional features are spliced ​​or fused to obtain fused features, including: performing dimensional alignment processing on all regional features, and concatenating or splicing the dimensionally aligned regional features in series or in parallel based on a preset scale order to obtain fused features; or, inputting all regional features into a feature fusion attention network for fusion weighting to obtain fused features.

[0012] Furthermore, according to one aspect of the method disclosed, the method also includes: presenting the target classification results using a visualized attention heatmap.

[0013] According to another aspect of this disclosure, an epithelial tumor grading device is provided, comprising: an acquisition unit for acquiring epithelial tumor pathological image data at multiple scales; the number of scales includes at least two; an extraction unit for obtaining corresponding regional features for pathological image data at any scale using a feature extraction mechanism; a fusion unit for splicing or fusing all regional features to obtain fused features; and a grading unit for sequentially inputting the fused features into a pre-trained first model and a second model to obtain a target grading result; the first model integrates macroscopic structural information and microscopic morphological information of the fused features based on cosine similarity, cross-attention mechanism, and gating fusion mechanism; the second model performs weighted aggregation and grading discrimination on the information integrated by the first model based on an aggregation-constrained attention learning mechanism to obtain the target grading result; both macroscopic structural information and microscopic morphological information are determined based on the fused features.

[0014] This disclosure provides a method and apparatus for grading epithelial tumors. The method involves acquiring epithelial tumor pathological image data at multiple scales; the number of scales includes at least two; for any one scale of pathological image data, a feature extraction mechanism is used to obtain corresponding regional features; all regional features are concatenated or fused to obtain fused features; the fused features are sequentially input into a pre-trained first model and a second model to obtain the target grading result; the first model integrates the macroscopic structural information and microscopic morphological information of the fused features based on cosine similarity, cross-attention mechanism, and gating fusion mechanism; the second model performs weighted aggregation and grading discrimination on the information integrated by the first model based on an aggregation-constrained attention learning mechanism to obtain the target grading result; both macroscopic structural information and microscopic morphological information are determined based on the fused features. In this way, compared to existing methods that rely on doctors' subjective judgment or weakly supervised models that rely solely on single-level features and single supervisory signals, this disclosure can simultaneously capture information on cell nucleus-level micromorphology and tissue-level macrostructure through multi-scale image data acquisition and feature fusion. Then, by leveraging the multi-mechanism synergistic integration of the first model, a deep correlation between macro and micro features is achieved. Combined with the aggregation-constrained attention learning of the second model, the accuracy of grading is enhanced, fundamentally avoiding individual differences in subjective judgment and the feature capture deficiencies of single-level models. In summary, the technical solution provided by this disclosure can improve the accuracy and efficiency of epithelial tumor grading and can be adapted to various application scenarios.

[0015] It should be understood that both the foregoing general description and the following detailed description are exemplary and intended to provide further illustration of the claimed technology. Attached Figure Description

[0016] The above and other objects, features, and advantages of this disclosure will become more apparent from the more detailed description of the embodiments thereof in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this disclosure and form part of the specification. They are used together with the embodiments of this disclosure to explain the disclosure and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0017] Figure 1 A schematic flowchart illustrating a method for grading epithelial tumors provided in this embodiment of the present disclosure; Figure 2 A comparison chart of different types of target classification results provided in the embodiments of this disclosure; Figure 3 A comparison diagram of another different type of target classification results provided in this embodiment of the disclosure; Figure 4 is a schematic diagram comparing attention heatmaps of different types of grading results provided in the embodiments of this disclosure; Figure 5 A complete flowchart of the method for grading epithelial tumors provided in this disclosure embodiment; Figure 6 ROC curves for different types of grading results provided in the embodiments of this disclosure; Figure 7 Different types of confusion matrix diagrams provided in embodiments of this disclosure; Figure 8 This is a structural block diagram of an epithelial tumor grading device provided in an embodiment of this disclosure. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this disclosure more apparent, exemplary embodiments according to this disclosure will now be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this disclosure, and not all embodiments of this disclosure. It should be understood that this disclosure is not limited to the exemplary embodiments described herein.

[0019] Currently, the grading of papillary urothelial tumors largely relies on the subjective judgment of pathologists or on single-level features or single supervisory signals in weakly supervised grading models. However, pathologists' subjective judgment is easily influenced by at least one of the following: clinical experience, visual fatigue, and individual cognitive differences, making it difficult to achieve a standardized and accurate distinction in tumor grading. Weakly supervised grading models, limited by single-level feature extraction, cannot simultaneously capture both nuclear-level microscopic morphological features and tissue-level macroscopic features. Furthermore, single supervisory signals are difficult to adapt to the continuous changes in tumor pathological features, making it difficult to construct multi-dimensional, hierarchical feature association expressions. Thus, manual grading by pathologists is inefficient and struggles to meet the diagnostic needs of large volumes of clinical pathological slides, while weakly supervised models, due to incomplete feature capture and single supervisory signals, are prone to grading bias. Therefore, existing methods for grading epithelial tumors have low accuracy and efficiency, making them unsuitable for various application scenarios.

[0020] Therefore, to address the aforementioned problems, this disclosure provides a method for grading epithelial tumors. This method simultaneously captures information on cell nuclear-level micromorphology and tissue-level macrostructure through multi-scale image data acquisition and feature fusion. Furthermore, it leverages the multi-mechanism synergistic integration of a first model to achieve deep correlation between macroscopic and microscopic features. Combined with the aggregation-constrained attention learning of a second model, it enhances the accuracy of grading discrimination, fundamentally avoiding individual differences in subjective judgment and the feature capture deficiencies of single-level models. In summary, the technical solution provided by this disclosure can improve the accuracy and efficiency of epithelial tumor grading and can be adapted to various application scenarios.

[0021] Please refer to Figure 1 , Figure 1 This is a schematic flowchart illustrating a method for grading epithelial tumors according to an embodiment of this disclosure. Figure 1 As shown, the method includes: In step S101, epithelial tumor pathological image data at multiple scales are acquired; the number of scales in the multiple scales includes at least two. In step S102, for pathological image data of any scale, the corresponding regional features are obtained using a feature extraction mechanism; In step S103, all regional features are spliced ​​or fused to obtain fused features; In step S104, the fused features are sequentially input into the pre-trained first and second models to obtain the target classification result. The first model integrates the macroscopic structural information and microscopic morphological information of the fused features based on cosine similarity, cross-attention mechanism and gating fusion mechanism. The second model performs weighted aggregation and classification discrimination on the information integrated by the first model based on the aggregation constraint attention learning mechanism to obtain the target classification result. Both macroscopic structural information and microscopic morphological information are determined based on the fused features.

[0022] In this disclosure, epithelial tumor pathological image data can be understood as digital image data obtained by scanning epithelial tumor tissue sections with a digital pathology scanner, covering image information of the tumor lesion area and the surrounding normal tissue area.

[0023] In this disclosure, multi-scale can be understood as the image dimensions obtained by scanning epithelial tumor tissue sections using different magnifications. Different scales correspond to different observation perspectives; low magnification can present the overall tissue structure, while high magnification can clearly display the detailed morphology of cells. For example, the multi-scale of this disclosure is preferably 5X and 20X, with an image patch resolution of 224×224 and an image patch overlap ratio of 50%. The 5X scale is used to capture tissue-level macroscopic structural information, the 20X scale is used to capture cell nucleus-level microscopic morphological information, and the 50% image patch overlap ratio can avoid the omission of key pathological features.

[0024] In this disclosure, the feature extraction mechanism can be understood as a technical means for extracting pathological features with diagnostic value from pathological image data at different scales, including image preprocessing operations and feature extraction networks, which are explained in detail below.

[0025] In this disclosure, regional features can be understood as feature vectors extracted from pathological image data at a single scale through a feature extraction mechanism, which can characterize the core attributes of the tumor lesion region at that scale. Regional features at different scales correspond to different levels of pathological information. Low-scale regional features focus on tissue structure information, while high-scale regional features focus on cell morphology information.

[0026] In this disclosure, the fusion feature can be understood as a comprehensive feature vector obtained by dimensional alignment and weighted concatenation of regional features extracted at various scales. It integrates macroscopic structural information and microscopic morphological information in multi-scale images and can comprehensively characterize the pathological features of tumors.

[0027] In this disclosure, the first model can be understood as a feature fusion model for achieving deep integration of macroscopic structural information and microscopic morphological information. Its core function is to mine the correlation between the two dimensions of information and generate integrated features with strong discriminative power. Specifically, cosine similarity is used to quantify the similarity between macroscopic structure and microscopic morphology; the cross-attention mechanism is used to achieve bidirectional interaction between macroscopic structure and microscopic morphology by alternately using them as query vector and key-value vector, capturing key correlation information between them and strengthening the differentiated expression of features; the gating fusion mechanism is used to adaptively weight and integrate features before and after the interaction, by learning the weight coefficients of each feature, highlighting the contribution of features with high diagnostic value, and suppressing the interference of noisy features, as explained in detail below.

[0028] In this disclosure, the second model can be understood as a classification model that achieves accurate grading of epithelial tumors based on the integrated features (feature identifiers) output by the first model. Its core function is to perform targeted weighted aggregation and category discrimination on the feature identifiers that have integrated macroscopic and microscopic correlation information, and output the final target grading result. Among them, the aggregation-constrained attention learning mechanism is the core mechanism of the second model. This mechanism can differentiate the weighting of different dimensions of features in the feature identifiers through preset diagnostic weights, and at the same time introduce regularization constraints to suppress the interference of irrelevant noise features, so as to strengthen the aggregation of high diagnostic value features and weaken low value features, ensuring that the aggregated features can accurately match the needs of grading discrimination. The macroscopic structural information can be the region of interest (ROI) information, which specifically refers to the feature information corresponding to the core region of suspected lesions screened from low-magnification pathological images, covering the overall structure of tumor tissue, glandular arrangement, lesion boundary and other macroscopic pathological attributes; the microscopic morphological information can be the image patch information, which specifically refers to the feature information corresponding to small-sized image fragments obtained from high-magnification pathological images, focusing on the morphology, size, staining depth and arrangement density of cell nuclei and other microscopic pathological attributes.

[0029] In this disclosure, the target grading result can be understood as the final pathological grading category obtained after analyzing epithelial tumor samples using the methods of this disclosure. The target grading results of this disclosure include: benign, potentially low-grade malignant, low-grade malignant, and high-grade malignant. Specifically, benign corresponds to papillary urothelial tumor, characterized by regular cell morphology, no nuclear atypia, and orderly tissue structure; potentially low-grade malignant corresponds to potentially low-malignant papillary urothelial neoplasm of low malignant potential (PUNLMP), characterized by mild cell morphological abnormalities, weak nuclear atypia, and basically orderly tissue structure; low-grade malignant corresponds to low-grade urothelial carcinoma, characterized by moderate cell morphological abnormalities, moderate nuclear atypia, and some tissue structure disorder; and high-grade malignant corresponds to high-grade urothelial carcinoma, characterized by significantly abnormal cell morphology, obvious nuclear atypia, severely disordered tissue structure, and infiltration.

[0030] For example, Figure 2 A comparison chart of different types of target classification results provided in embodiments of this disclosure. From Figure 2It can be seen that the parameters of benign tumors are the most convergent, mostly concentrated in the inner region of the radar image, corresponding to their regular cell morphology and stable nuclear features. The parameter distribution of potential low-grade malignancy (PUNLMP) and low-grade malignancy is between that of benign and high-grade malignancy. The characteristic region of PUNLMP is closer to that of benign tumors, while that of low-grade malignancy is closer to that of high-grade malignancy. This is consistent with the pathological feature gradient of mild and moderate abnormalities in the two types of tumors. High-grade malignancy has the most prominent characteristics in various nuclear parameters (such as morphological parameters such as nuclear perimeter, mean length of major and minor axes, and nuclear contour eccentricity, as well as texture parameters such as contrast and energy). The parameter values ​​are generally located in the outermost region of the radar image, reflecting significant abnormalities in cell morphology and nuclear atypia.

[0031] For example, Figure 3 A comparison chart showing another different type of target classification results provided in embodiments of this disclosure. From Figure 3 It can be seen that, in terms of regional and nuclear parameters (such as nuclear number density and nuclear area density), the parameter values ​​of high-grade malignancy are significantly higher than those of the other three types, reflecting the characteristics of active tumor cell proliferation and high density; the parameter values ​​of benign malignancy are the lowest, corresponding to the sparse and orderly distribution of tissue cells; while the parameter values ​​of potential low-grade malignancy (PUNLMP) and low-grade malignancy increase in a stepwise manner, further confirming the continuous gradient difference in the pathological characteristics of the two, and also indicating that the method disclosed in this paper can comprehensively distinguish the characteristics of tumors of different grades through multi-dimensional parameters.

[0032] Specifically, the grading of epithelial tumors includes the following steps: First, a digital pathology scanner is used to scan slices of papillary urothelial tumor tissue at magnifications of 5X and 20X to obtain multi-scale pathological image data, and the images are preprocessed, including denoising and color standardization. Second, the preprocessed 5X and 20X scale images are input into a feature extraction convolutional neural network to extract regional features. Third, a feature fusion attention network is used to perform dimensional alignment and weighted fusion of the regional features at the two scales to obtain fused features that integrate multi-scale information. Fourth, the fused features are input into a pre-trained first model. The cosine similarity between the ROI and the Patch in the fused features is calculated and the optimal ROI is selected. Then, a cross-attention mechanism is used to achieve bidirectional feature interaction between the ROI and the Patch. Finally, a gated fusion mechanism is used to weightedly integrate the four types of features to generate feature labels. Fifth, the feature labels are input into a second model. Based on preset weights, the feature labels are weighted and aggregated by an aggregation constraint attention module. The grading discrimination submodule outputs the probability distribution of the four grading results, and the category with the highest probability is selected as the final target grading result.

[0033] The following will explain in detail how to obtain the target classification results, including: Macroscopic structural information is the Region of Interest (ROI); microscopic morphological information is the image patch. The fused features are sequentially input into the pre-trained first and second models to obtain the target classification results, including: Based on the labeled standard diagnostic regions, sliding window clipping values, and fusion features, the ROI and Patch are determined. The fused features are input into the first model, the cosine similarity between the ROI and the Patch is calculated, and the ROI with the highest similarity is selected. Based on the cross-attention algorithm, the filtered ROI and Patch are alternately determined as query vector and key-value vector; Based on the gated fusion network, the ROI and Patch before and after the alternation, the ROI determined as the query vector and the Patch determined as the key-value vector, and the ROI determined as the key-value vector and the Patch determined as the query vector are weighted and integrated to generate feature labels. The feature identifiers are input into the second model, and based on the preset diagnostic weights, they are distinguished through aggregation constraints to output the target classification results.

[0034] In this disclosure, the standard diagnostic region can be understood as a region containing typical tumor pathological features that is manually marked on a pathological image. This region covers the core diagnostic criteria for tumors of different grades and serves as a benchmark reference region for subsequent ROI screening and lesion extent determination.

[0035] In this disclosure, the sliding window clipping value can be understood as a set of key parameters set when performing sliding window traversal clipping on a pathological image. It mainly includes at least one of the following: sliding window size, sliding step size, and overlap ratio. Its core function is to generate a sequence of image blocks with uniform size and complete coverage.

[0036] In this disclosure, the query vector and the key-value vector can be understood as two core vectors used for feature interaction calculation in the cross-attention algorithm. The query vector is a feature vector used to actively retrieve related information, and its core function is to locate the feature dimension that is strongly related to itself. The key-value vector is a feature vector used to provide retrieval basis and matching information. The key vector is used to calculate the correlation degree with the query vector, and the value vector is used to provide specific feature information after the correlation degree is matched. The two work together to achieve accurate interaction between different features.

[0037] In this disclosure, the gated fusion network can be understood as a neural network module with adaptive feature selection and weighted integration capabilities. Its core structure includes fully connected layers and a gating mechanism. The network can automatically learn the importance weights of different input features, assign higher weights to diagnostic features with high value, and assign lower weights to noisy features, thereby achieving efficient fusion of multiple sets of features and generating integrated features with strong discriminative power.

[0038] In this disclosure, the feature identifier can be understood as the final feature vector obtained after processing by the first model, which integrates the information related to macroscopic structure and microscopic morphology. It is the core output generated by the first model after multiple steps of processing such as cosine similarity screening, cross-attention interaction, and gating fusion weighting. This feature vector not only retains the organizational structure information of the ROI and the cell morphology information of the Patch, but also strengthens the correlation features between the two, providing accurate feature support for the hierarchical discrimination of the second model.

[0039] In this disclosure, diagnostic weights can be understood as weight coefficients obtained based on clinical epithelial tumor diagnostic guidelines and statistical analysis of a large number of labeled pathological samples, used to quantify the diagnostic value of different pathological features. This disclosure mainly includes the structural integrity weight of the ROI and the cell morphology abnormality weight of the patch. Among them, the structural integrity weight is used to measure the contribution of the orderliness of the tumor tissue structure to the grade discrimination, and the cell morphology abnormality weight is used to measure the contribution of the degree of nuclear atypia to the grade discrimination. The two types of weights work together to achieve accurate differentiation of tumors of different grades.

[0040] Specifically, when determining the target classification result, firstly, based on the labeled standard diagnostic region, the suspected lesion range is delineated in the multi-scale pathological image corresponding to the fusion feature. Background and normal tissue areas are removed to obtain a candidate region set. Then, based on a preset sliding window cropping value, the candidate region set is traversed and cropped to generate an image patch sequence. By calculating the lesion matching degree between the image patch sequence and the fusion feature, the candidate region set with a matching degree higher than a preset threshold is determined as the ROI, and the image patch sequence is determined as the Patch. Next, the fusion feature is input into the first model, and the ROI with the highest matching degree with the Patch lesion feature is selected through cosine similarity calculation to ensure the targeting of subsequent feature interactions. Subsequently, based on the cross-attention algorithm, the ROI is used as the query vector, and the Patch is used as the query vector. As key-value vectors, and Patch as query vectors and ROI as key-value vectors, two bidirectional feature interactions are completed to obtain two sets of interactive features. Then, the ROI features before interaction, Patch features, and the two sets of features after interaction are input into a gated fusion network. The network learns the weight coefficients of each feature and performs weighted integration to generate feature labels. Finally, the feature labels are input into the second model. Based on the preset ROI structural integrity weight and Patch cell morphology abnormality weight, the expression of high diagnostic value features is enhanced by the aggregation constraint attention module to suppress noise interference. Then, the grading and discrimination submodule calculates the probability distribution of four types of results: benign, potentially low-grade malignant, low-grade malignant, and high-grade malignant. The category with the highest probability is selected as the final target grading result.

[0041] The following will explain in detail how to determine ROI and Patch, including: Based on the labeled standard diagnostic regions, the lesion region range is delineated in the multi-scale pathological image corresponding to the fused features to obtain a candidate region set; Based on the sliding window cropping value, the candidate region set is traversed and cropped to generate an image patch sequence; When the lesion matching degree between the image patch sequence and the fusion feature is greater than a preset threshold, the candidate region set is determined as ROI and the image patch sequence is determined as Patch.

[0042] In this disclosure, the candidate region set can be understood as a set of regions containing suspected tumor lesions, initially delineated through feature matching in multi-scale pathological images corresponding to fused features, using the standard diagnostic regions labeled by pathologists as a benchmark reference. This set has been filtered by lesion features of the fused features, eliminating non-diagnostic interference regions such as image background, normal epithelial tissue, and blood vessels. Its core function is to reduce the lesion retrieval dimension and improve the efficiency and accuracy of ROI localization.

[0043] In this disclosure, the image patch sequence can be understood as an ordered set of small image fragments of uniform size and complete coverage obtained by traversing and cropping the candidate region set row by row and column by column based on preset sliding window cropping values ​​(including sliding window size, sliding step size, overlap ratio, etc.). Each image fragment focuses on local pathological details within the candidate region, and can completely preserve microscopic pathological features such as cell nuclear morphology, local cell density, and micro-glandular structure, providing standardized basic data for subsequent extraction of microscopic morphological information.

[0044] Specifically, determining the ROI and Patch involves the following steps: First, import the standard diagnostic region data labeled by pathologists according to the urothelial tumor classification standards, and perform preliminary matching with the fusion features to obtain the feature dimensions in the fusion features that match the pathological features of the standard diagnostic region. Based on this, delineate the approximate range of suspected lesions in multi-scale pathological images, and after removing irrelevant areas such as background and normal tissue, form a candidate region set; Second, call the preset sliding window cropping parameters to perform full-range, row-by-row, column-by-column traversal cropping of the candidate region set, generating a sequence of image patches of uniform size to ensure that no key pathological features within the candidate regions are missed; Third, ... The first step is to calculate the cosine similarity between the features of each image patch in the image patch sequence and the core lesion features in the fused features. The average similarity of all image patches is used as the lesion matching degree between the image patch sequence and the fused features. The second step is to compare the calculated lesion matching degree with a preset threshold (which can be obtained by statistical calibration based on a large number of historical pathological samples). If the lesion matching degree is greater than the preset threshold, the candidate region set is determined to contain effective lesion features and is identified as the ROI. At the same time, the corresponding image patch sequence is identified as a Patch. If the lesion matching degree does not reach the preset threshold, the first step is returned to readjust the delineation range of the candidate region set until the matching degree requirement is met.

[0045] For example, this disclosure also provides a specific Patch calculation formula that satisfies the following: in, Represented as the first ROI at low magnification Below, the feature set of all patches after processing by the feature encoding network. Represented as the first ROI at high magnification Below, the feature set of all patches after processing by the feature encoding network. Represented as the first The ROI of the th Each patch at low magnification The image data below. Represented as the first ROI at low magnification The number of patches contained below. Represented as the first ROI at high magnification The number of patches contained below.

[0046] in, The calculation can satisfy the following formula: in, An index representing the ROI (used to distinguish different regions of interest). This indicates the patch number within the ROI (used to distinguish different patches within the same ROI). Represents the space of real numbers. Indicates the width of a single patch. This indicates the height of a single patch.

[0047] The following will explain in detail how to determine the feature identifier, including: Determine the first feature vector of the ROI and the second feature vector of the Patch before the alternation; After determining the alternation, the ROI is determined as the query vector and the Patch is determined as the third feature vector of the key-value vector; After determining the alternation, the ROI is determined as the key-value vector and the Patch is determined as the fourth feature vector of the query vector; The first feature vector, the second feature vector, the third feature vector, and the fourth feature vector are all input into the fully connected layer of the gated fusion network to determine their corresponding weight coefficients. The first, second, third, and fourth eigenvectors are weighted and summed based on weight coefficients, and then normalized to obtain the feature identifiers. The normalization process includes norm normalization and mean centering.

[0048] In this disclosure, the weight coefficient can be understood as a numerical coefficient learned by the gated fusion network based on a large amount of labeled pathological data, used to quantify the contribution of different feature vectors to tumor grading. Its value range can be (0,1), and the sum of the weight coefficients corresponding to all input feature vectors satisfies the normalization constraint (i.e., the sum of the weight coefficients is 1). The magnitude of the weight coefficient can be positively correlated with the diagnostic value of the corresponding feature vector. The higher the coefficient, the more critical the tumor pathological information contained in the feature vector.

[0049] In this disclosure, normalization can be understood as aiming to eliminate dimensional differences between different feature dimensions, reduce the interference of data distribution offset on subsequent model calculations, and improve the stability and discriminative power of feature vectors. Specifically, norm normalization preferably employs L2 norm normalization, which calculates the L2 magnitude of the temporary feature vector and divides each element of the vector by this magnitude, ensuring the processed feature vector has a magnitude of 1. This avoids excessive model focus due to some feature dimensions having excessively large values. Mean centering involves calculating the mean of all elements in the temporary feature vector and subtracting this mean from each element of the vector, ensuring the processed feature vector is distributed around the origin, thus eliminating the influence of data baseline offset.

[0050] Specifically, the feature identification process includes the following steps: First, retrieve the macroscopic structural feature vector of the ROI before the alternation from the feature cache unit of the first model and determine it as the first feature vector; simultaneously, retrieve the microscopic morphological feature vector of the Patch before the alternation and determine it as the second feature vector. Second, based on the cross-attention algorithm, use the filtered ROI feature vector as the query vector and the Patch feature vector as the key vector, calculate the attention score, and perform weighted summation to obtain the interaction feature vector, which is determined as the third feature vector. Third, again based on the cross-attention algorithm, use the Patch feature vector as the query vector and the ROI feature vector as the key vector, repeat the attention calculation process to obtain another set of interaction feature vectors, which are determined as the fourth feature vector. Fourth, input the four feature vectors into the fully connected layer of the gated fusion network, learn the importance weights of each feature vector through the Sigmoid activation function, and output the corresponding weight coefficients. Fifth, perform weighted summation to obtain a temporary feature vector. Sixth, perform L2 optimization on the temporary vector. Norm normalization is performed to obtain normalized eigenvectors, which are then used as feature identifiers; or, mean centering is performed on temporary vectors to obtain normalized eigenvectors, which are then used as feature identifiers.

[0051] The following will explain in detail how feature identification and the second model are used to determine the target classification results, including: Diagnostic weights include: the structural integrity weight of the ROI and the cell morphology abnormality weight of the Patch; Obtain the pre-defined structural integrity weights of the ROI and the cell morphology abnormality weights of the Patch; The feature identifiers are input into the aggregation constraint attention module of the second model. The feature identifiers are weighted and aggregated based on the structural integrity weight of the ROI and the cell morphology abnormality weight of the Patch to obtain the aggregated constraint features. The aggregated features are input into the hierarchical discrimination submodule of the second model to determine the probability distribution of the hierarchical results. The target classification result is determined based on the probability distribution.

[0052] In this disclosure, the aggregation constraint attention module can be understood as the core functional module in the second model used for targeted feature enhancement and noise suppression of feature identifiers. It integrates preset diagnostic weights and attention mechanisms, and can dynamically focus on feature dimensions strongly correlated with tumor grade while weighting and aggregating features. The core function of this module is to split macroscopic structural correlation features and microscopic morphological correlation features in feature identifiers based on ROI structural integrity weights and Patch cell morphological abnormality weights, adjust the fusion ratio of the two types of features through attention calculation, and introduce regularization constraints to suppress irrelevant noise features, outputting aggregation constraint features that take into account both diagnostic value and discrimination accuracy.

[0053] In this disclosure, the grading discrimination submodule can be understood as the functional module in the second model used to realize tumor grading classification. It usually adopts a structure of multilayer perceptron combined with Softmax activation function. The core function of this module is to receive the feature vector after aggregation constraint, perform depth mapping and dimension transformation on the feature through multi-level fully connected layers, and finally output the probability value of the feature vector corresponding to different tumor grading categories to complete the grading discrimination task.

[0054] In this disclosure, the probability distribution can be understood as the set of probability values ​​output by the grading discrimination submodule for each tumor grading category corresponding to the feature identifier. It includes the probabilities of four grading results: benign, potentially low-grade malignant, low-grade malignant, and high-grade malignant, and the sum of the probability values ​​of all categories is 1. The magnitude of the probability value represents the confidence that the feature identifier belongs to the corresponding grading category, and the higher the probability value, the stronger the confidence.

[0055] Specifically, determining the target classification result can include the following steps: First, retrieve the preset diagnostic weight parameters; second, input the feature identifiers into the aggregation constraint attention module of the second model, calculate the association weights of features in each dimension through the attention mechanism, perform weighted aggregation of the two types of features, and introduce L1 regularization constraints to suppress noise features, obtaining aggregated and constrained features; third, input the aggregated and constrained features into the classification discrimination submodule of the second model, perform feature transformation through a multilayer perceptron, and then calculate the probability distribution corresponding to the four classification results through the Softmax activation function; fourth, select the classification category corresponding to the largest probability value in the probability distribution P, and determine it as the final target classification result.

[0056] For example, this disclosure also provides a specific formula for calculating cosine similarity, satisfying the following: in, Represented as the first The ROI of the th The eigenvector of the ROI and the _th Cosine similarity between feature vectors of each patch. Represented as the first The ROI of the th The macroscopic structural feature vectors corresponding to each ROI. Represented as the first The ROI of the th The micromorphic feature vector corresponding to each patch.

[0057] Subsequently, within each ROI feature set, similarity scores are calculated. Before selection The patches within the ROIs with the highest relevance form a representative set: in, Represented as the first The representative feature set consists of the feature vectors of the ROIs corresponding to the top k high-matching patches. Represented as the first The cosine similarity score between the ROI features and the Patch features within each ROI. This represents the number of highly matched patches selected.

[0058] For example, this disclosure also provides a specific formula for calculating attention, satisfying the following: in, Represented as the first The weighted enhanced feature vector is obtained by applying attention to the representative feature set of each ROI. This is represented as the weight matrix of the first fully connected layer in the attention calculation. This is represented as the bias term of the first fully connected layer in the attention calculation. This is represented as the weight matrix of the second fully connected layer in the attention calculation. This is represented as the bias term of the second fully connected layer in the attention calculation. This is represented by the Sigmoid activation function. It is represented as the hyperbolic tangent activation function.

[0059] For example, this disclosure also provides a specific formula for calculating cross-attention, satisfying the following: in, It is represented as an interaction feature vector calculated by cross-attention, with Patch features as the key-value vector and ROI enhancement features as the query vector. This is represented as the cross-attention calculation function. Represented as the first The set of micromorphological features corresponding to all patches within a ROI.

[0060] And update the ROI representation after the interaction: in, This is represented as the final ROI interaction feature vector obtained after Patch feature cross-attention enhancement.

[0061] For example, this disclosure also provides a specific calculation formula for gating fusion, which satisfies the following: in, It is represented as a multi-scale heterogeneous feature vector obtained after gated fusion. This represents the weight matrix of the fully connected layer in the post-processing branch of gated fusion. This represents the bias term of the fully connected layer in the post-processing branch of gated fusion. This represents the weight matrix of the fully connected layer in the preprocessing branch of gated fusion. This represents the bias term of the preprocessed branch fully connected layer in gated fusion.

[0062] For example, this disclosure also provides a specific formula for calculating the aggregation hierarchy, satisfying the following: in, It is represented as a feature vector after intermediate mapping of multi-scale heterogeneous features. It is represented as an intermediate feature mapping function. Represented as the first Tumor grade prediction results corresponding to each ROI. It is represented as a hierarchical prediction function.

[0063] The following will explain in detail how to obtain the corresponding regional features, including: Preprocessing of pathological image data includes at least one of the following: image denoising, color normalization, and image enhancement. The preprocessed pathological image data is input into a feature extraction convolutional neural network, and the output feature vector of the corresponding lesion area is determined as the region feature; the feature extraction convolutional neural network includes: deep residual network and visual geometry group network.

[0064] In this disclosure, preprocessing can be understood as a preliminary optimization operation performed on the original pathological image data. The core purpose is to eliminate image noise, unify data distribution, and enhance the recognizability of pathological features, providing high-quality image input for subsequent feature extraction. Among these, image denoising (preferably Gaussian filtering) is used to remove random noise generated during the scanning process; color standardization is used to unify color deviations caused by different slices and different scanners; and image enhancement (preferably histogram equalization) is used to improve the contrast between cells and background, highlighting key pathological structures such as cell nuclei.

[0065] In this disclosure, the feature extraction convolutional neural network can be understood as a deep learning network used to extract high-dimensional pathological features from preprocessed pathological images. It can automatically capture information with diagnostic value, such as texture, morphology, and structure, in the image. Specifically, the deep residual network is used to extract deep semantic features of the image, while the visual geometry network is used to extract local detail features of the image, adapting to the fine structural analysis of pathological images.

[0066] Specifically, the process of determining regional features includes the following steps: First, read the original pathological image data at multiple scales; second, perform Gaussian filtering for noise reduction, color standardization, and histogram equalization enhancement on the original images sequentially to obtain preprocessed images; third, input the preprocessed images into a deep residual network or a visual geometric group network to extract macroscopic structural feature vectors and cellular-level microscopic morphological feature vectors; fourth, determine the output feature vectors at the two scales as the regional features at the corresponding scales.

[0067] The following will explain in detail how to obtain the fusion features, including: All regional features are dimensionally aligned, and based on a preset scale order, the dimensionally aligned regional features are concatenated or concatenated to obtain fused features. or, All regional features are input into a feature fusion attention network for fusion and weighting to obtain fused features.

[0068] In this disclosure, scale order can be understood as the priority order for splicing features of different scale regions. In this disclosure, the order from low scale to high scale is preferred to ensure that the fused features first retain organizational structure information and then supplement cell detail information.

[0069] In this disclosure, the fusion attention network can be understood as a fusion module with dynamic feature weighting capability. It learns the importance weights of features at different scales through the attention mechanism, assigns higher weights to features with high diagnostic value, and achieves adaptive fusion of multi-scale features.

[0070] Specifically, obtaining the fused features involves the following steps (taking a feature fusion attention network as an example): First, retrieve the region features corresponding to the 5× and 20× scales; second, align the dimensions of the two region features through a fully connected layer to unify them into feature vectors of the same dimension; third, input the aligned features into the fusion attention network and calculate the attention weight of each feature; fourth, perform a weighted summation of the two features based on the attention weights; fifth, determine the weighted feature vector as the final fused feature.

[0071] The following details the epithelial tumor grading method disclosed herein, which also includes: presenting the target grading results using a visual attention heatmap.

[0072] In this disclosure, the visual attention heatmap can be understood as a visualization tool that overlays the key pathological regions of interest during the model grading process onto the original pathological image in the form of a heatmap. Its core function is to intuitively display the model's discrimination criteria and improve the interpretability of the grading results.

[0073] Specifically, the steps for generating and presenting a visual attention heatmap include: First, during the second model's grading and discrimination process, recording the attention weight distribution of the aggregation constraint attention module; second, mapping the attention weights to the corresponding regions of the original pathological image, using color depth to represent the model's attention level (the darker the color, the greater the contribution of the region to the grading result); third, overlaying the attention heatmap with the original pathological image and simultaneously labeling the target grading result; and fourth, outputting a visual report containing the heatmap and grading result.

[0074] For example, Figure 4 is a schematic diagram comparing attention heatmaps of different types of grading results provided in this embodiment of the present disclosure. As can be seen from Figure 4, the color distribution of the attention heatmap is highly matched with the pathological annotation area, indicating that the model can accurately focus on the real tumor lesion area, verifying the reliability of the grading results; the focus of the heatmaps corresponding to different grades is different: the heatmap of low-grade lesions is concentrated in the local tissue area, while the heatmap of high-grade lesions has a wider coverage and darker color, corresponding to the pathological features of more significant atypia of high-grade tumor cells and a larger lesion area; combined with low-magnification and high-magnification views, it can be seen that the heatmap covers both the tissue structure area under low magnification and focuses on the cell nucleus details under high magnification, which can reflect the model's discrimination logic of simultaneously paying attention to macroscopic structure and microscopic morphology.

[0075] For example, Figure 5 This is a schematic flowchart illustrating a complete method for grading epithelial tumors as provided in this disclosure. Figure 5As can be seen, the epithelial tumor grading process disclosed herein consists of three core steps: 1. Image preprocessing and feature encoding: First, 5× and 20× image blocks of the lesion area are cropped, and features at different scales are extracted through a feature encoder, followed by multi-scale fusion to obtain initial features; 2. Bi-branch feature interaction and fusion: Through modules such as linear modules, sampling, and cross-attention, the interaction enhancement of multi-scale features is achieved, and then key features are integrated through a gating fusion module; 3. Grading prediction: Finally, the classification module outputs four grading results: benign, PUNLMP, low-grade, and high-grade. The process covers the complete link from image to feature to interaction to classification, and at the same time, it combines multi-scale information and attention mechanisms to improve grading accuracy.

[0076] For example, this disclosure also provides specific experimental procedures for the classification of epithelial tumors and comparisons of different models, as follows: Step 1: Sample collection and grading labeling: Full-field sections of papillary urothelial tumors were collected from the pathology center. All samples were stained with hematoxylin and eosin (HE), and the pathological diagnoses were reviewed and agreed upon by three experienced urological pathologists. The samples covered four categories: benign papillary urothelial tumors, low-potential papillary urothelial tumors (PUNLMP), low-grade papillary urothelial carcinoma, and high-grade papillary urothelial carcinoma. The specific sample numbers are shown in Table 1 below. Classification benign Low-grade malignant potential papillary urothelial carcinoma (PUNLMP) Low-grade malignancy High-grade malignancy total Area of ​​Interest 77 71 360 322 830 Table 1 As shown in Table 1, the samples used in this experiment cover all four grades of papillary urothelial carcinoma, and the sample types are complete and representative. In terms of the number distribution of samples in each grade, the number of regions of interest corresponding to low-grade malignancy and high-grade malignancy is relatively large, reaching 360 and 322 respectively, accounting for 82.17% of the total sample size, which can provide sufficient malignant lesion feature data for model training. The number of regions of interest corresponding to benign and low-grade malignant potential papillary urothelial carcinoma (PUNLMP) is relatively small, with 77 and 71 respectively. The number of samples in the two types is basically equal, which can meet the model's learning needs for early lesion features. The total number of samples is 830 regions of interest, which is moderate in size, ensuring the effectiveness of model training while avoiding the problem of reduced training efficiency due to excessive sample size.

[0077] Step 2: Conduct model experiments according to this disclosure and compare the experimental results: Model Accuracy (%) Area under the curve (%) The combined precision and recall metric (F1 score) (%) The Kappa coefficient (%) measures the consistency between model predictions and ground truth labels. ViT 73.87±4.91 91.18±0.23 70.68±3.6 67.65±2.30 CONCH+CLAM 76.88±2.69 92.11±0.93 63.95±5.65 69.88±2.35 UNI 80.62±2.20 95.86±0.15 79.31±0.7 77.11±3.20 UNI+RRT-MIL 75.03±2.91 91.91±2.84 74.10±6.88 69.92±2.79 UNI+FR-MIL 77.11±4.67 93.22±1.31 71.95±7.03 72.62±4.57 UNI+TransMIL 80.36±0.94 94.14±0.70 79.01±3.96 74.91±4.61 UNI+DFTD 80.15±4.30 94.44±1.93 73.87±9.94 75.01±6.24 This public model 85.57±3.48 95.27±0.88 84.51±4.41 82.48±0.05 Table 2 As shown in Table 2, the experimental results demonstrate that the proposed method outperforms existing multi-instance learning models in terms of overall performance, achieving an average accuracy of 85.57%, an AUC of 95.27%, a macro-average F1 score of 84.51%, and a Kappa coefficient of 82.48%, while maintaining stable performance in four-class classification tasks.

[0078] To further analyze the model's performance under different pathological grades, evaluation indicators were calculated for the four categories (benign, PUNLMP, low grade, and high grade), and receiver operating characteristic (ROC) curves were plotted. Figure 6 Figure 6 shows the ROC curves for different types of grading results provided in the embodiments of this disclosure. As can be seen from Figure 6, the ROC curves of the model of this disclosure perform well in all four pathological grading categories: the area under the ROC curve (AUC) for the benign category reaches 0.99, with the curve almost touching the upper left corner of the figure, indicating that the model has a very strong ability to identify and distinguish benign lesions; the AUCs for the PUNLMP category and the high-grade category are both 0.95, with steep curves and a rapid increase in the true positive rate, reflecting the high accuracy of the model in distinguishing these two types of lesions; the AUC for the low-grade category is 0.91, slightly lower than the other categories, but still at a high level, indicating that the model also has good distinguishing ability for low-grade lesions; overall, the AUCs for all four categories exceed 0.9, proving that the classification performance of the model of this disclosure is stable and excellent under different pathological grades.

[0079] Furthermore, this disclosure also provides performance comparison results for different grading types determined using the method of this disclosure, as shown in Table 3 below: index Classification benign Low-grade malignant potential papillary urothelial carcinoma (PUNLMP) Recall rate (%) Recall(%) 92.33±8.8 71.10±3.54 Accuracy (%) ACC(%) 97.95±1.00 96.51±1.14 F1 score (%) F1(%) 94.13±2.90 87.38±3.79 Table 3 As shown in Table 3, the performance of the proposed method varies across different classification categories, but remains at a high level overall. For the benign category, all indicators are high, with a recall of 92.33%, accuracy close to 98%, and an F1 score of 94.13%, indicating that the model has extremely strong accuracy and recall in identifying benign lesions. The accuracy (96.51%) for the PUNLMP category is also high, but the recall (71.10%) is slightly lower than that of the benign category, presumably due to the relatively small sample size of this type of lesion. However, the F1 score remains at a good level of 87.38%, proving that the model's classification performance for early lesions like PUNLMP is still reliable. The data differences between the two categories also reflect the model's good discriminative ability between benign and early lesions, effectively identifying lesion characteristics at different stages.

[0080] To verify the contributions of each module, this invention also discloses systematic ablation experiments, including verifying the optimization effects of modules such as addition, cross-attention mechanism, cosine similarity sampling, and gated feature fusion on overall performance by calculating evaluation metrics, and drawing confusion matrices, as detailed in Table 4 below: Stage / Module Type method Accuracy (%) Area under the curve (%) F1 score (%) Kappa coefficient (%) UNI 80.6±2.20 95.8±0.15 79.3±0.70 77.1±3.20 WSL (Weakly Supervised Learning) Sorted 81.9±0.86 94.7±1.13 81.0±1.35 77.9±1.40 Multi-scale 82.6±1.76 94.9±1.03 82.0±3.35 79.4±2.31 MSL (Multi-Scale Learning) Crossattn 82.5±3.57 95.1±0.66 82.3±1.14 79.0±3.53 Sampling 83.7±5.55 95.1±2.62 82.1±5.98 79.1±6.06 This method is publicly available. 85.5±3.48 95.2±0.88 84.5±4.41 82.4±0.05 Table 4 As shown in Table 4 above, the experimental results fully demonstrate the synergistic effect of each module in the system of this invention. Among them, the cross-attention mechanism and the gating feature fusion make the most significant contributions, proving that the comprehensive method proposed in this invention effectively improves the model's structural recognition ability and hierarchical robustness. Figure 7 Different types of confusion matrix diagrams are provided for embodiments of this disclosure. From Figure 7 It can be seen that, from left to right, the confusion matrices correspond to the hierarchical interaction based on the cross-attention mechanism directly, the hierarchical interaction based on the cross-attention mechanism with the addition of cosine similarity sampling, and the hierarchical interaction based on the cross-attention mechanism with the addition of cosine similarity sampling and gating fusion modules. It can be seen that the proposed method has the best recognition performance in all four categories, and the accuracy is improved.

[0081] Based on the above experimental results, the multi-level hybrid supervised feature interaction model proposed in this disclosure achieves a balance between performance and interpretability in the grading task of papillary urothelial tumors, and is significantly better than existing pathological grading methods based on weak supervision.

[0082] To further evaluate the generality of the proposed framework, we collected a separate test cohort of papillary urothelial tumor cases. This cohort contained images with expert-annotated regions of interest (ROIs), totaling 79 ROIs for evaluation. Since our framework operates at the ROI level, each image underwent a preprocessing pipeline to locate the ROI and extract image patches for subsequent inference. Specifically, we used a combination of color space transformation and morphological operations to segment the tissue foreground from the slice thumbnails. Then, connected component analysis and area thresholding were applied to generate foreground masks from which ROIs were identified and their coordinates recorded. Based on these coordinates, 224×224 pixel image patches were extracted at 5x and 20x magnification for subsequent feature fusion, interaction, and classification, as detailed in Tables 5 and 6 below. Classification benign Low-grade malignant potential papillary urothelial carcinoma (PUNLMP) Low-grade malignancy High-grade malignancy total Area of ​​Interest 25 5 24 25 79

[0083] Table 5 index Classification benign Low-grade malignant potential papillary urothelial carcinoma (PUNLMP) Low-grade malignancy High-grade malignancy Recall rate (%) Recall(%) 40.00 20.00 70.83 92.00 Accuracy (%) ACC(%) 81.01 88.61 71.15 87.34 F1 score (%) F1(%) 72.47 56.03 69.57 86.17 Table 6 As shown in Tables 5 and 6, the evaluation by category reveals that the model performs differently across different tumor grades. The model demonstrates strong sensitivity in detecting high-grade cases, with a recall of 92.00% and an F1 score of 86.17%, highlighting its potential clinical value in identifying invasive diseases. The recall for low-grade cases is 70.83%, and the F1 score is 69.57%, which is moderate. The F1 score for benign lesions is 72.47%, but the recall is only 40.00%, indicating that some benign ROIs are misclassified as other subtypes. In contrast, PUNLMP remains the most challenging subtype, with a recall decreasing to 20.00% and an F1 score decreasing to 56.03%, indicating that PUNLMP is frequently confused with adjacent categories due to its subtle histological differences and limited training samples.

[0084] Through this step, the present invention achieves a complete closed loop from feature learning to grade prediction, verifying the superiority and practical value of the hybrid supervision and multi-scale fusion strategy in the grading of papillary urothelial tumors.

[0085] This disclosure also provides a grading device for epithelial tumors. Figure 8 This disclosure provides a structural block diagram of an epithelial tumor grading device, as shown in the embodiments below. Figure 8 The epithelial tumor grading device 800 includes: Acquisition unit 801 is used to acquire epithelial tumor pathological image data at multiple scales; the number of scales includes at least two. Extraction unit 802 is used to obtain the corresponding regional features for pathological image data of any scale using a feature extraction mechanism; The fusion unit 803 is used to stitch or fuse all regional features to obtain fused features; The hierarchical unit 804 is used to sequentially input the fused features into the pre-trained first and second models to obtain the target hierarchical result. The first model integrates the macroscopic structural information and microscopic morphological information of the fused features based on cosine similarity, cross-attention mechanism and gating fusion mechanism. The second model performs weighted aggregation and hierarchical discrimination on the information integrated by the first model based on the aggregation constraint attention learning mechanism to obtain the target hierarchical result. Both macroscopic structural information and microscopic morphological information are determined based on the fused features.

[0086] In one exemplary embodiment, the grading unit 804 is specifically used to: include the target grading result as: benign, potentially low-grade malignant, low-grade malignant, and high-grade malignant.

[0087] In one exemplary embodiment, the grading unit 804 is specifically used for: determining ROIs and patches based on labeled standard diagnostic regions, sliding window clipping values, and fusion features; inputting the fusion features into a first model, calculating the cosine similarity between ROIs and patches, and selecting the ROI with the highest similarity; using a cross-attention algorithm, alternately determining the selected ROIs and patches as query vectors and key-value vectors; using a gated fusion network, weightedly integrating the ROIs and patches before and after the alternation, the ROI determined as a query vector and the patch determined as a key-value vector, and the ROI determined as a key-value vector and the patch determined as a query vector, to generate feature identifiers; inputting the feature identifiers into a second model, and distinguishing them based on preset diagnostic weights and aggregation constraints, outputting the target grading result.

[0088] In one exemplary embodiment, the grading unit 804 is specifically used to: delineate the lesion region range in the multi-scale pathological image corresponding to the fusion feature based on the labeled standard diagnostic region to obtain a candidate region set; perform traversal cropping on the candidate region set based on the sliding window cropping value to generate an image patch sequence; when the lesion matching degree between the image patch sequence and the fusion feature is greater than a preset threshold, the candidate region set is determined as ROI and the image patch sequence is determined as Patch.

[0089] In one exemplary embodiment, the hierarchical unit 804 is specifically used to: determine the first feature vector of the ROI before the alternation and the second feature vector of the Patch; determine the third feature vector after the alternation where the ROI is determined to be a query vector and the Patch is determined to be a key-value vector; determine the fourth feature vector after the alternation where the ROI is determined to be a key-value vector and the Patch is determined to be a query vector; input the first feature vector, the second feature vector, the third feature vector and the fourth feature vector into the fully connected layer of the gated fusion network to determine their corresponding weight coefficients; perform a weighted summation of the first feature vector, the second feature vector, the third feature vector and the fourth feature vector based on the weight coefficients, and perform normalization processing to obtain feature identifiers; the normalization processing includes: norm normalization processing and mean centering processing.

[0090] In one exemplary embodiment, the grading unit 804 is specifically used to: obtain the pre-set structural integrity weight of the ROI and the cell morphology abnormality weight of the Patch; input the feature identifier into the aggregation constraint attention module of the second model, and perform weighted aggregation on the feature identifier based on the structural integrity weight of the ROI and the cell morphology abnormality weight of the Patch to obtain the aggregated constraint features; input the aggregated constraint features into the grading discrimination submodule of the second model to determine the probability distribution of the grading results; and determine the target grading result based on the probability distribution.

[0091] In one exemplary embodiment, the extraction unit 802 is specifically used to: preprocess pathological image data; the preprocessing includes at least one of the following: image denoising, color normalization, and image enhancement; input the preprocessed pathological image data into a feature extraction convolutional neural network, and output the feature vector of the corresponding lesion area to determine the region feature; the feature extraction convolutional neural network includes: a deep residual network and a visual geometric group network.

[0092] In one exemplary embodiment, the fusion unit 803 is specifically used to: perform dimensional alignment processing on all regional features, and concatenate or concatenate the dimensionally aligned regional features based on a preset scale order to obtain fused features; or, input all regional features into a feature fusion attention network for fusion weighting to obtain fused features.

[0093] In one exemplary embodiment, the grading unit 804 is further configured to: present the target grading results using a visual attention heatmap.

[0094] In summary, this disclosure provides a method and apparatus for grading epithelial tumors. This disclosure acquires epithelial tumor pathological image data at multiple scales; the number of scales includes at least two; for any one scale of pathological image data, a feature extraction mechanism is used to obtain the corresponding regional features; all regional features are concatenated or fused to obtain fused features; the fused features are sequentially input into a pre-trained first model and a second model to obtain the target grading result; the first model integrates the macroscopic structural information and microscopic morphological information of the fused features based on cosine similarity, cross-attention mechanism, and gating fusion mechanism; the second model performs weighted aggregation and grading discrimination on the information integrated by the first model based on an aggregation-constrained attention learning mechanism to obtain the target grading result; both macroscopic structural information and microscopic morphological information are determined based on the fused features. In this way, compared to existing methods that rely on doctors' subjective judgment or weakly supervised models that rely solely on single-level features and single supervisory signals, this disclosure can simultaneously capture information on cell nucleus-level micromorphology and tissue-level macrostructure through multi-scale image data acquisition and feature fusion. Then, by leveraging the multi-mechanism synergistic integration of the first model, a deep correlation between macro and micro features is achieved. Combined with the aggregation-constrained attention learning of the second model, the accuracy of grading is enhanced, fundamentally avoiding individual differences in subjective judgment and the feature capture deficiencies of single-level models. In summary, the technical solution provided by this disclosure can improve the accuracy and efficiency of epithelial tumor grading and can be adapted to various application scenarios.

[0095] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this disclosure.

[0096] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0097] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0098] Additionally, as used herein, the "or" used in a list of items beginning with "at least one" indicates a separate list, such that a list of, for example, "at least one of A, B, or C" means A or B or C, or AB or AC or BC, or ABC (i.e., A and B and C). Furthermore, the word "exemplary" does not imply that the described example is preferred or better than other examples.

[0099] It should also be noted that in the systems and methods of this disclosure, the components or steps can be decomposed and / or recombined. These decompositions and / or recombinations should be considered as equivalent solutions to this disclosure.

[0100] Various changes, substitutions, and modifications can be made to the technology described herein without departing from the teachings defined by the appended claims. Furthermore, the scope of the claims of this disclosure is not limited to the specific aspects of the processes, machines, manufactures, events, means, methods, and actions described above. Currently existing or later-developed processes, machines, manufactures, events, means, methods, or actions that perform substantially the same function or achieve substantially the same result as the corresponding aspects described herein can be utilized. Therefore, the appended claims include such processes, machines, manufactures, events, means, methods, or actions within their scope.

[0101] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0102] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

Claims

1. A method for grading epithelial tumors, characterized in that, The method includes: Acquire pathological image data of epithelial tumors at multiple scales; the number of scales in the multiple scales includes at least two. For any scale of pathological image data, the corresponding regional features are obtained using a feature extraction mechanism; All the region features are spliced ​​or fused to obtain fused features; The fused features are sequentially input into the pre-trained first and second models to obtain the target classification result. The first model integrates the macroscopic structural information and microscopic morphological information of the fused features based on cosine similarity, cross-attention mechanism and gating fusion mechanism. The second model performs weighted aggregation and classification discrimination on the information integrated by the first model based on aggregation constraint attention learning mechanism to obtain the target classification result. The macroscopic structural information and the microscopic morphological information are both determined based on the fused features.

2. The method according to claim 1, characterized in that, The target classification results include: benign, potentially low-grade malignant, low-grade malignant, and high-grade malignant.

3. The method according to claim 1, characterized in that, The macroscopic structural information is the Region of Interest (ROI); the microscopic morphological information is the image patch; the step of sequentially inputting the fused features into the pre-trained first and second models to obtain the target classification result includes: Based on the labeled standard diagnostic region, sliding window clipping value, and the fusion feature, the ROI and the Patch are determined. The fused features are input into the first model, the cosine similarity between the ROI and the Patch is calculated, and the ROI with the highest similarity is selected. Based on the cross-attention algorithm, the filtered ROI and Patch are alternately determined as query vector and key-value vector; Based on the gated fusion network, the ROI and Patch before and after the alternation, the ROI determined as the query vector and the Patch determined as the key-value vector, and the ROI determined as the key-value vector and the Patch determined as the query vector are weighted and integrated to generate feature identifiers; The feature identifier is input into the second model, and based on the preset diagnostic weights, it is distinguished by aggregation constraints to output the target classification result.

4. The method according to claim 3, characterized in that, The determination of the ROI and the Patch based on the labeled standard diagnostic region, sliding window clipping value, and the fusion feature includes: Based on the standard diagnostic region labeled, the lesion region range is delineated in the multi-scale pathological image corresponding to the fusion feature to obtain a candidate region set. Based on the sliding window cropping value, the candidate region set is traversed and cropped to generate an image patch sequence; When the lesion matching degree between the image patch sequence and the fusion feature is greater than a preset threshold, the candidate region set is determined as the ROI, and the image patch sequence is determined as the Patch.

5. The method according to claim 3, characterized in that, The method based on the gated fusion network performs weighted integration on the ROI and Patch before and after the alternation, the ROI determined as the query vector and the Patch determined as the key-value vector, and the ROI determined as the key-value vector and the Patch determined as the query vector, to generate feature identifiers, including: Determine the first feature vector of the ROI and the second feature vector of the Patch before the alternation; After the alternation is determined, the ROI is determined as the query vector and the Patch is determined as the third feature vector of the key-value vector; After the alternation is determined, the ROI is determined as the key-value vector and the Patch is determined as the fourth feature vector of the query vector; The first feature vector, the second feature vector, the third feature vector, and the fourth feature vector are all input into the fully connected layer of the gated fusion network to determine their corresponding weight coefficients; The first feature vector, the second feature vector, the third feature vector, and the fourth feature vector are weighted and summed based on the weight coefficients, and then normalized to obtain the feature identifier; the normalization process includes norm normalization and mean centering.

6. The method according to claim 3, characterized in that, The diagnostic weights include: the structural integrity weight of the ROI and the cell morphology abnormality weight of the Patch; the step of inputting the feature identifier into the second model, distinguishing based on the preset diagnostic weights through aggregation constraints, and outputting the target classification result includes: Obtain the preset structural integrity weight of the ROI and the cell morphology abnormality weight of the Patch; The feature identifier is input into the aggregation constraint attention module of the second model. The feature identifier is weighted and aggregated based on the structural integrity weight of the ROI and the cell morphology abnormality weight of the Patch to obtain the aggregated constraint features. The aggregated and constrained features are input into the hierarchical discrimination submodule of the second model to determine the probability distribution of the hierarchical results; The target classification result is determined based on the probability distribution.

7. The method according to claim 1, characterized in that, For the pathological image data at any scale, the corresponding regional features are obtained using a feature extraction mechanism, including: The pathological image data is preprocessed; the preprocessing includes at least one of the following: image denoising, color normalization, and image enhancement; The preprocessed pathological image data is input into a feature extraction convolutional neural network, and the output feature vector of the corresponding lesion area is determined as the region feature; the feature extraction convolutional neural network includes: a deep residual network and a visual geometric group network.

8. The method according to claim 1, characterized in that, The step of concatenating or fusing all the region features to obtain fused features includes: All the region features are dimensionally aligned, and the dimensionally aligned region features are concatenated or concatenated in parallel based on a preset scale order to obtain the fused features. or, All the region features are input into a feature fusion attention network for fusion and weighting to obtain the fused features.

9. The method according to claim 1, characterized in that, The method further includes: The target classification results are presented using a visual attention heatmap.

10. A grading device for epithelial tumors, characterized in that, The device includes: An acquisition unit is used to acquire epithelial tumor pathological image data at multiple scales; the number of scales in the multiple scales includes at least two. The extraction unit is used to obtain the corresponding regional features for the pathological image data at any scale using a feature extraction mechanism; The fusion unit is used to perform feature splicing or fusion of all the region features to obtain fused features; The hierarchical unit is used to sequentially input the fused features into the pre-trained first model and second model to obtain the target hierarchical result. The first model integrates the macroscopic structural information and microscopic morphological information of the fused features based on cosine similarity, cross-attention mechanism and gating fusion mechanism. The second model performs weighted aggregation and hierarchical discrimination on the information integrated by the first model based on aggregation constraint attention learning mechanism to obtain the target hierarchical result. The macroscopic structural information and the microscopic morphological information are both determined based on the fused features.

Images