Intestinal cancer immune response graph neural network prediction system, medium, and device

Abstract

The application discloses an intestinal cancer immune response graph neural network prediction system, medium and equipment, through collecting pathological image information, immune detection information and basic clinical information, using a deep convolution network to extract a tissue spatial distribution feature map, combining an immune marker expression characteristic matrix to construct a graph neural network model, adopting a graph attention mechanism to model spatial interaction characteristics of a tumor microenvironment, finally predicting a treatment response probability, an optimal treatment time and an adverse reaction risk through a multi-task learning framework, and outputting a clinical decision report containing a predicted response curve, a risk early warning threshold and a treatment time window suggestion. Through multi-modal data fusion and spatial interaction modeling, the application realizes accurate prediction of the intestinal cancer immunotherapy response, and provides a more comprehensive reference basis for clinical decision-making.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of medical image technology, in particular to an intestinal cancer immune response graph neural network prediction system, medium and device. BACKGROUND

[0002] Intestinal cancer immunotherapy is an important breakthrough in the field of tumor treatment, and its efficacy is closely related to the immune status of the tumor microenvironment. At present, in clinical practice, the expression level of PD-L1 or the microsatellite instability (MSI) state and other indicators are mainly detected by tissue biopsy to evaluate the potential response of patients to immunotherapy. However, the tumor microenvironment is a highly heterogeneous dynamic system, and factors such as the distribution of immune cells, the spatial relationship between tumor tissue and interstitial structure, and other factors jointly affect the treatment effect. The existing evaluation methods mainly rely on single-dimensional biomarkers or static pathological analysis, which is difficult to fully reflect the complex spatial characteristics of the tumor microenvironment and its dynamic correlation with immunotherapy. In addition, clinical decision-making is usually based on limited detection data, and lacks integrated analysis of the individualized treatment history of patients, resulting in limited adaptability and accuracy of the prediction model. How to establish a more comprehensive tumor microenvironment characterization method and combine it with clinical treatment data to achieve dynamic prediction is a key problem to be solved in the current field of intestinal cancer immunotherapy. SUMMARY

[0003] In view of the above problems, the present application provides an intestinal cancer immune response graph neural network prediction system, medium and device, which realizes accurate prediction of immunotherapy response by fusing multi-modal data and modeling the spatial interaction of the tumor microenvironment, and solves the problem that the existing evaluation system cannot fully reflect the heterogeneity of the tumor microenvironment.

[0004] To achieve the above purpose, in a first aspect, the present application provides an intestinal cancer immune response graph neural network prediction system, comprising:

[0005] Collecting pathological image information and immune detection information of intestinal cancer patients, and collecting basic clinical information of intestinal cancer patients as auxiliary features;

[0006] Performing feature extraction on the pathological image information, using a deep convolutional neural network to perform multi-level feature extraction on the pathological image information to obtain cell morphological features and spatial distribution features, the spatial distribution features including the spatial distribution of tumor regions, interstitial regions and normal tissues, and generating a spatial distribution feature map containing tissue heterogeneity information;

[0007] Performing quantitative analysis on the immune detection information, spatially registering the immune detection information with the pathological image information to obtain a registration result, and using a density clustering algorithm to identify the spatial aggregation characteristics of immune cells in the immune detection information based on the registration result;

[0008] Meanwhile, the gradient change rate of the PD-L1 expression level is calculated, and an immune marker expression feature matrix reflecting the tumor immune microenvironment state is generated by feature fusion combining the spatial clustering characteristics;

[0009] Based on the tissue spatial distribution feature map and the immune marker expression feature matrix, a graph neural network model is constructed, the spatial interaction between different functional regions is modeled through a graph attention mechanism, and the spatial interaction features in the tumor microenvironment are extracted;

[0010] In combination with clinical treatment information, a treatment response prediction model is established, the extracted spatial interaction features are time-aligned with the patient's treatment history, medication regimen and follow-up results, a multi-task learning framework is used to simultaneously predict treatment response probability, optimal treatment time and adverse reaction risk, and a clinical decision report containing immune therapy response prediction results is output. The clinical decision report includes a predicted response curve, a risk warning threshold and a treatment time window suggestion.

[0011] Further, feature extraction is performed on the pathological image information, a deep convolutional neural network is used to extract multi-level features from the pathological image information, and cell morphological features and spatial distribution features are obtained, including:

[0012] The pathological image information is sequentially standardized and regionally enhanced to obtain processed pathological image information;

[0013] The processed pathological image information is used to extract basic features using a pre-trained ResNet50 network, cell nucleus morphology, chromatin distribution and cell arrangement features are captured through shallow convolutional layers, and cell morphological features are generated;

[0014] Based on the cell morphological features, semantic segmentation is performed through the encoder-decoder structure of the U-Net network architecture, global context information of tumor regions, interstitial regions and normal tissues is extracted by gradually downsampling in the encoding stage, and spatial distribution features containing accurate information of tissue boundaries are generated by combining jump connection to restore spatial details in the decoding stage.

[0015] Further, a spatial distribution feature map containing tissue heterogeneity information is generated, including:

[0016] The spatial distribution features are regionally quantified, and the cell atypia index in the tumor region, the collagen fiber orientation distribution in the interstitial region, and the structural integrity parameter of the normal tissue are calculated respectively;

[0017] The cell atypia index, collagen fiber orientation distribution and structural integrity parameter are integrated through a feature fusion layer, multi-scale tissue distribution features are extracted using spatial pyramid pooling, and the quantized regional features are obtained;

[0018] The quantified regional features are concatenated with the corresponding cell morphology features to obtain tissue heterogeneity information, which includes microscopic cell features and macroscopic tissue distribution features.

[0019] Spatial distribution feature maps are obtained based on organizational heterogeneity information.

[0020] Furthermore, quantitative analysis of the immunoassay information is performed, and spatial registration is conducted between the immunoassay information and pathological image information to obtain the registration results, including:

[0021] The fluorescence signal of the immune detection information was quantified, and the positive cell density, staining intensity spatial gradient and immune cell spatial aggregation coefficient corresponding to the preset labels were calculated respectively. The preset labels included CD3+ label, CD8+ label and PD-L1 label.

[0022] The multimodal registration module integrates the spatial gradient of positive cell density, staining intensity, and spatial aggregation coefficient of immune cells, and uses a feature point matching algorithm to establish the spatial correspondence between immune detection information and pathological image information to obtain initial registration parameters.

[0023] The initial registration parameters are optimized by elastic transformation with the tissue structure features in the pathological image to obtain the optimized spatial mapping relationship, which includes global affine transformation and local deformation compensation.

[0024] Based on the spatial mapping relationship, the immune detection information is spatially transformed to generate an immune marker distribution map that is spatially aligned with the pathological image, which is denoted as the registration result.

[0025] Furthermore, density clustering algorithms are used to identify the spatial aggregation characteristics of immune cells in the immune detection information based on the registration results, including:

[0026] Based on the density clustering algorithm, the immune cells in the immune marker distribution map are spatially clustered to obtain high-density clustered regions and discrete distribution regions;

[0027] Calculate the immune cell density gradient, spatial distribution entropy, and contact index with adjacent tissues in high-density clustered regions and discretely distributed regions;

[0028] By correlating immune cell density gradient, spatial distribution entropy, and contact index with adjacent tissues with the tumor microenvironment characteristics of the corresponding region, spatial aggregation characteristics of immune cells are generated.

[0029] Furthermore, the gradient rate of change of PD-L1 expression level is calculated, and combined with spatial clustering features, an immune marker expression feature matrix reflecting the state of the tumor immune microenvironment is generated through feature fusion, including:

[0030] The spatial gradient rate of change of PD-L1 expression level was calculated using the Sobel operator;

[0031] Based on the spatial aggregation feature map of immune cells, the spatial colocalization coefficient of CD8+ T cells and PD-L1 expression regions was extracted, and the immunosuppressive microenvironment index was calculated.

[0032] By integrating spatial gradient change rate, spatial colocalization coefficient and immunosuppressive microenvironment index through feature fusion layer, and using attention mechanism to weight the contribution of immune features in different spatial regions, weighted multidimensional immune features are obtained.

[0033] The weighted multidimensional immune features are mapped to a three-dimensional feature tensor according to spatial coordinates. The first dimension represents the expression intensity of PD-L1, the second dimension represents the spatial distribution of immune cells, and the third dimension records the microenvironment state parameters, thus generating an immune marker expression feature matrix.

[0034] Furthermore, based on tissue spatial distribution feature maps and immune marker expression feature matrices, a graph neural network model is constructed to extract spatial interaction features within the tumor microenvironment, including:

[0035] The regional features in the tissue spatial distribution feature map are quantified into node attributes of graph nodes. The node attributes include the cellular atypia index of the tumor region, the collagen fiber orientation distribution of the stroma region, and the structural integrity parameters of normal tissue.

[0036] A topological connectivity graph is constructed based on the spatial coordinates of pathological image information, with the center point of the tissue region as the graph node and the spatial distance and structural continuity between adjacent regions as the edge weights.

[0037] The expression feature matrix of immune markers was used as an additional node feature and fused with the regional features in the tissue spatial distribution feature map across modalities. The expression intensity of PD-L1 was associated with the graph nodes of the tumor region, and the distribution features of immune cells were associated with the graph nodes of the stromal region.

[0038] A multi-layer graph attention network is used for message passing. By calculating the spatial interaction strength between graph nodes, spatial interaction features are generated. These features include the spatial dependence of the tumor-immune-mesenchymal ternary microenvironment.

[0039] Furthermore, by combining clinical treatment information, a treatment response prediction model is established. The extracted spatial interaction features are temporally aligned with the patient's treatment history, medication regimen, and follow-up results. A multi-task learning framework is used to simultaneously predict the probability of treatment response, optimal treatment timing, and adverse reaction risk. The output is a clinical decision report containing the immunotherapy response prediction results, including:

[0040] Spatial interaction features are concatenated with clinical treatment information, including the patient's past treatment history, medication regimen, and follow-up results. A spatiotemporal feature matrix is ​​constructed by aligning the feature representations of different time points through a time sequence encoder.

[0041] A multi-task prediction network was constructed, with the spatiotemporal feature matrix as input. The network outputs the treatment response probability, the optimal treatment timing, and the adverse reaction risk score through three prediction branches. The treatment response probability branch uses the spatial dependence feature of the tumor-immune-mesenchyma ternary microenvironment in the spatial interaction features as the key input.

[0042] A cross-modal attention module is added to the prediction network to dynamically calculate the correlation weights between spatial interaction features and clinical treatment features, automatically identify the feature combinations that have the greatest impact on the prediction results, and obtain multi-task prediction results.

[0043] The system integrates multi-task prediction results to generate a clinical decision report, which includes a predicted response curve based on spatial interaction characteristics, medication dosage adjustment recommendations combined with adverse reaction risk scores, risk warning thresholds, and treatment time window recommendations generated based on the prediction of optimal treatment timing.

[0044] In a second aspect, the present invention also provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the system described in the first aspect.

[0045] In a third aspect, the present invention also provides an electronic device including a memory and a processor, the memory being used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the system described in the first aspect.

[0046] Unlike existing technologies, the above-mentioned technical solution provides a graph neural network prediction system, medium, and device for colorectal cancer immune response. The system includes: collecting pathological image information, immune detection information, and clinical information from patients; extracting cell morphology and spatial distribution features from pathological image information using a deep convolutional neural network to generate a map containing tissue spatial distribution features; quantitatively analyzing immune detection information and registering it with pathological image information to identify spatial aggregation features of immune cells, and generating an immune marker expression feature matrix by combining PD-L1 gradient changes; constructing a graph neural network model based on the tissue spatial distribution feature map and the immune marker expression feature matrix, and using a graph attention mechanism to model the spatial interactions between different functional regions; finally, combining clinical treatment information, predicting the probability of treatment response, optimal treatment timing, and adverse reaction risks through a multi-task learning framework, and outputting a clinical decision report including a predicted response curve, risk warning threshold, and treatment time window suggestions. This invention effectively solves the problem of insufficient characterization of tumor microenvironment heterogeneity, significantly improves prediction accuracy through multimodal data fusion, provides more interpretable decision-making basis for clinicians through spatial interaction modeling, and enables dynamic monitoring of the treatment process through a time-aligned prediction framework, providing reliable support for the formulation of personalized immunotherapy plans.

[0047] The above description of the invention is merely an overview of the technical solution of this application. In order to enable those skilled in the art to better understand the technical solution of this application and to implement it based on the description and drawings, and to make the above-mentioned objectives and other objectives, features and advantages of this application easier to understand, the following description is provided in conjunction with the specific embodiments and drawings of this application. Attached Figure Description

[0048] The accompanying drawings are only used to illustrate the principles, implementation methods, applications, features, and effects of specific embodiments of the present invention and other related contents, and should not be considered as limitations on this application.

[0049] In the accompanying drawings of the instruction manual:

[0050] Figure 1 A system step diagram of steps S101 to S106 of the prediction system described in the specific implementation embodiment;

[0051] Figure 2 A system step diagram of steps S201 to S203 of the prediction system described in the specific implementation embodiment;

[0052] Figure 3 A system step diagram of steps S301 to S304 of the prediction system described in the specific implementation embodiment;

[0053] Figure 4A system step diagram of steps S401 to S404 of the prediction system described in the specific implementation embodiment;

[0054] Figure 5 This is a schematic diagram of the structure of the electronic device described in a specific embodiment.

[0055] The reference numerals used in the above figures are explained as follows:

[0056] 1. Electronic equipment;

[0057] 11. Memory;

[0058] 12. Processor. Detailed Implementation

[0059] To illustrate the possible application scenarios, technical principles, implementable specific solutions, and achievable objectives and effects of this application in detail, the following description, in conjunction with the listed specific embodiments and accompanying drawings, provides a detailed explanation. The embodiments described herein are merely illustrative of the technical solutions of this application and are therefore intended to limit the scope of protection of this application.

[0060] In this document, the term "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The term "embodiment" appearing in various places throughout the specification does not necessarily refer to the same embodiment, nor does it specifically limit its independence or connection with other embodiments. In principle, in this application, as long as there are no technical contradictions or conflicts, the technical features mentioned in each embodiment can be combined in any way to form corresponding implementable technical solutions.

[0061] Unless otherwise defined, the technical terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the use of related terms herein is merely for the purpose of describing particular embodiments and is not intended to limit this application.

[0062] In the description of this application, the term "and / or" is used to describe the logical relationship between objects, indicating that three relationships can exist. For example, A and / or B means: A exists, B exists, and A and B exist simultaneously. Additionally, the character " / " in this document generally indicates that the preceding and following objects have an "or" logical relationship.

[0063] In this application, terms such as “first” and “second” are used only to distinguish one entity or operation from another, and do not necessarily require or imply any actual quantity, hierarchy or order relationship between these entities or operations.

[0064] Without further limitations, the use of terms such as “comprising,” “including,” “having,” or other similar open-ended expressions in this application is intended to cover non-exclusive inclusion, which does not exclude the presence of additional elements in a process, system, or product that includes the stated elements, such that a process, system, or product that includes a list of elements may include not only those defined elements but also other elements not expressly listed, or elements inherent to such a process, system, or product.

[0065] As understood in the Examination Guidelines, in this application, expressions such as "greater than," "less than," and "exceeding" are understood to exclude the stated number; expressions such as "above," "below," and "within" are understood to include the stated number. Furthermore, in the description of the embodiments in this application, "multiple" means two or more (including two), and similar expressions related to "multiple" are also understood in this way, such as "multiple groups" and "multiple times," unless otherwise explicitly specified.

[0066] In the description of the embodiments of this application, the space-related expressions used, such as "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "vertical," "top," "bottom," "inner," "outer," "clockwise," "counterclockwise," "axial," "radial," and "circumferential," indicate the orientation or positional relationship based on the orientation or positional relationship shown in the specific embodiments or drawings. They are only for the purpose of describing the specific embodiments of this application or for the reader's understanding, and do not indicate or imply that the device or component referred to must have a specific position, a specific orientation, or be constructed or operated in a specific orientation. Therefore, they should not be construed as limitations on the embodiments of this application.

[0067] Please see Figure 1 In a first aspect, this embodiment provides a neural network prediction system for colorectal cancer immune response maps, comprising:

[0068] S101. Collect pathological image information and immune detection information of colorectal cancer patients, and at the same time collect basic clinical information of colorectal cancer patients as auxiliary features;

[0069] S102. Extract features from pathological image information. Use a deep convolutional neural network to extract features from pathological image information at multiple levels to obtain cell morphology features and spatial distribution features. Spatial distribution features include the spatial distribution of tumor area, stroma area and normal tissue. Generate a spatial distribution feature map containing tissue heterogeneity information, which is the tissue spatial distribution feature map.

[0070] S103. Perform quantitative analysis on the immune detection information, spatially register the immune detection information with the pathological image information to obtain the registration result, and use the density clustering algorithm to identify the spatial aggregation characteristics of immune cells in the immune detection information.

[0071] S104. Simultaneously calculate the gradient change rate of PD-L1 expression level, and combine spatial aggregation features to generate an immune marker expression feature matrix that can reflect the state of the tumor immune microenvironment through feature fusion.

[0072] S105. Based on the spatial distribution feature map of tissues and the expression feature matrix of immune markers, a graph neural network model is constructed. The spatial interaction between different functional regions is modeled through the graph attention mechanism, and the spatial interaction features of the tumor microenvironment are extracted.

[0073] S106. Combine clinical treatment information to establish a treatment response prediction model. Align the extracted spatial interaction features with the patient's treatment history, medication regimen and follow-up results in time. Use a multi-task learning framework to simultaneously predict the probability of treatment response, the optimal treatment timing and the risk of adverse reactions. Output a clinical decision report containing the prediction results of immunotherapy response. The clinical decision report includes the predicted response curve, risk warning threshold and treatment time window suggestions.

[0074] In step S101, pathological image information of colorectal cancer patients is acquired through a digital pathology scanning system, including whole-section images of tumor tissue, used to characterize the cell arrangement morphology and tissue structure features of the tumor region; immune detection information is obtained through immunohistochemistry or flow cytometry, recording the expression levels of immune checkpoint proteins such as PD-L1 and the infiltration of immune cells; basic clinical information includes structured data such as patient age, tumor stage, and treatment history, used as auxiliary features for feature fusion in subsequent models. This multi-source data is stored and managed in a standardized manner through a hospital information system.

[0075] In step S102, preferably, the deep convolutional neural network adopts a pre-trained ResNet architecture, extracting local texture features and global structural features of the pathological image step by step through convolutional layers. Among them, cell morphological features include microscopic morphological parameters such as nucleocytoplasmic ratio and cell polarity; spatial distribution features are generated by identifying the spatial boundaries of tumor regions, stroma regions and normal tissues through a semantic segmentation network, generating a spatial distribution feature map containing tissue heterogeneity information. This map quantifies the distribution density and spatial relationship of different functional regions in the form of a heatmap.

[0076] In step S103, spatial registration employs a feature-point-based elastic registration algorithm to align the cell location coordinates in the immune detection information with the coordinate system of the pathological image information. The preferred density clustering algorithm is the DBSCAN algorithm, which analyzes the spatial coordinate distribution of immune cells to identify clinically significant spatial clustering patterns, such as clusters of tumor-infiltrating lymphocytes. These clustering patterns reflect the functional distribution characteristics of immune cells in the tumor microenvironment.

[0077] In step S104, the gradient change rate of PD-L1 expression level is obtained by calculating the expression difference between adjacent tissue regions. Combined with the aforementioned spatial clustering features, an immune marker expression feature matrix is ​​generated by feature splicing. The row vectors of this matrix correspond to spatial positions, and the column vectors contain features such as immune marker expression intensity and cell density gradient, which fully characterize the heterogeneity of the tumor immune microenvironment.

[0078] In the graph neural network model constructed in step S105, nodes are jointly defined by the tissue spatial distribution feature map and the immune marker expression feature matrix, and edge weights are calculated through spatial distance and feature similarity. The graph attention mechanism pays special attention to the spatial interactions of the tumor-immune interface region, and captures the directionality and intensity features of the immune regulatory relationship between different functional regions through a multi-head attention layer.

[0079] In step S106, temporal alignment is achieved through a dynamic temporal warping algorithm, matching spatial interaction features with the time series of clinical treatment records. The multi-task learning framework includes a shared feature extraction layer and three task-specific heads: a response prediction head outputs a treatment response probability curve that changes over time; a timing prediction head determines the optimal intervention window based on the cumulative effect of treatment efficacy; and a risk prediction head generates a risk warning threshold by monitoring the dynamic changes of immune-related adverse reaction biomarkers. The final output clinical decision report presents the prediction results through a visualization interface, where the recommended treatment time window comprehensively considers the synergistic effect of tumor burden change rate and immune activation status.

[0080] This embodiment extracts multi-level tissue spatial distribution feature maps from pathological images using a deep convolutional neural network. Combined with quantitative analysis and spatial registration of immune detection information, it generates an immune marker expression feature matrix. A graph attention mechanism is used to model the spatial interaction characteristics of different functional regions within the tumor microenvironment. Finally, a multi-task learning framework is employed to collaboratively predict treatment response probability, optimal treatment timing, and adverse reaction risk. This embodiment integrates tissue heterogeneity information and dynamic features of the immune microenvironment, overcoming the shortcomings of insufficient spatial interaction modeling. The generated clinical decision report can intuitively reflect the temporal effect characteristics and risk evolution trends of immunotherapy, providing decision support with both spatial resolution and temporal prediction capabilities for the formulation of individualized treatment plans.

[0081] Please see Figure 2 In some embodiments, feature extraction is performed on pathological image information. A deep convolutional neural network is used to perform multi-level feature extraction on the pathological image information to obtain cell morphology features and spatial distribution features, including:

[0082] S201. The pathological image information is standardized and region-enhanced sequentially to obtain the processed pathological image information.

[0083] S202. The pre-trained ResNet50 network is used to extract basic features from the processed pathological image information. The cell nuclear morphology, chromatin distribution and cell arrangement features are captured through shallow convolutional layers to generate cell morphology features.

[0084] S203. Based on cell morphology features, semantic segmentation is performed through the encoder-decoder structure of the U-Net network architecture. In the encoding stage, global contextual information of tumor region, stroma region and normal tissue is extracted by progressive downsampling. In the decoding stage, skip connections are combined to restore spatial details and generate spatial distribution features containing accurate information of tissue boundaries.

[0085] In step S201, preferably, the standardization process is achieved through histogram equalization and Z-score normalization to eliminate brightness differences and staining deviations between different scanning devices; the region enhancement employs adaptive gamma correction combined with a locally contrast-limited histogram equalization algorithm to focus on improving the texture visibility of the tumor-stromal junction region. The processed pathological image information retains the biological authenticity of the tissue structure while optimizing the input quality for subsequent feature extraction.

[0086] In step S202, the pre-trained ResNet50 network is initialized with ImageNet weights. Its shallow convolutional layers are specialized for capturing cell nuclear morphological features, including microscopic parameters such as nuclear membrane irregularity and chromatin condensation patterns. The middle convolutional layers extract cell polarity features to characterize the invasive arrangement patterns of tumor cells. The cell morphological features are output as a 256-dimensional feature vector through a global average pooling layer, which retains the cell-level discriminative information while meeting the input dimension requirements of the downstream semantic segmentation network.

[0087] In step S203, preferably, the encoder of the U-Net network uses a VGG16 skeleton and gradually abstracts the global context information of the tumor region through five downsampling stages; the decoder restores the spatial resolution of the feature map through transposed convolution and skip connections, where the skip connections fuse the high-resolution details of each stage of the encoder with the semantic information of the decoder. The spatial distribution features are finally output as a three-channel probability map, with each channel corresponding to the pixel-level classification results of the tumor region, stroma region, and normal tissue, respectively. Preferably, its boundary accuracy reaches a resolution of 5μm, meeting the needs of spatial analysis of the tumor microenvironment.

[0088] This embodiment optimizes the quality of pathological image information through standardization and region enhancement. A pre-trained ResNet50 network is used to extract cellular morphological features such as nuclear morphology and chromatin distribution from shallow convolutional layers. Then, a multi-scale semantic segmentation is achieved through the encoder-decoder structure of the U-Net network. Combined with a skip connection mechanism, tissue boundary details are accurately restored while preserving global contextual information, ultimately generating high-resolution spatial distribution features. This cascaded feature extraction architecture effectively integrates microscopic morphological features at the cellular level with macroscopic spatial distribution information at the tissue level, solving the problem of insufficient feature extraction in heterogeneous tumor regions. This provides a feature expression foundation with both cell discrimination power and spatial accuracy for subsequent immune microenvironment analysis.

[0089] Please see Figure 3 In some embodiments, generating a spatial distribution feature map containing tissue heterogeneity information includes:

[0090] S301. Quantify the regional characteristics of spatial distribution, and calculate the cell atypia index, collagen fiber orientation distribution in the stroma region, and structural integrity parameters of normal tissue within the tumor region.

[0091] S302. By integrating cell atypia index, collagen fiber orientation distribution and structural integrity parameters through feature fusion layer, multi-scale tissue distribution features are extracted by spatial pyramid pooling to obtain quantified regional features.

[0092] S303. The quantified regional features are spliced ​​with the corresponding cell morphology features to obtain tissue heterogeneity information, which includes microscopic cell features and macroscopic tissue distribution features.

[0093] S304. Obtain spatial distribution feature map based on tissue heterogeneity information.

[0094] In step S301, regional feature quantification assesses malignant characteristics such as abnormal nuclear-cytoplasmic ratio and mitotic figures by calculating the cellular atypia index of the tumor region, uses Fourier transform to analyze the collagen fiber orientation distribution in the stroma region to reflect the degree of stroma remodeling, and measures the glandular structure integrity parameters of normal tissue through morphological opening operations. These quantitative indicators characterize tissue heterogeneity from three dimensions: cellular abnormality, matrix arrangement, and tissue architecture.

[0095] In step S302, the feature fusion layer integrates cellular atypia index, collagen fiber orientation distribution, and structural integrity parameters through spatial pyramid pooling. Preferably, a multi-level grid division strategy is adopted, including three levels: single-unit division, four-unit division, and sixteen-unit division, to capture local subtle differences, medium-range distribution patterns, and overall regional features, respectively. Single-unit division retains the complete details of the original quantified features, four-unit division reflects the structural correlation of tissue subregions, and sixteen-unit division characterizes the tissue distribution trend across the entire field of view. Through this hierarchical aggregation mechanism, the quantified regional features include both cellular-level heterogeneity measurements and tissue-level spatial distribution patterns.

[0096] In step S303, the channel splicing operation integrates the quantified regional features and cell morphology features along the feature dimension. Preferably, the cell morphology features include microscopic attributes such as nuclear morphology and chromatin distribution, while the quantified regional features characterize the macroscopic distribution characteristics of tumor regions, stromal regions, and normal tissues. During the splicing process, zero-padding is used at the edges of the feature maps to maintain spatial alignment, ensuring that the microscopic features corresponding to each spatial location are precisely matched with the macroscopic features. The resulting tissue heterogeneity information forms a unified multidimensional feature expression, where microscopic cell features provide the basis for identification, and macroscopic tissue distribution features reflect spatial contextual relationships.

[0097] In step S304, the generation of the spatial distribution feature map is achieved by upsampling the feature map through deconvolution operation. Preferably, the deconvolution operation forms a symmetrical mapping relationship with the encoder-decoder structure of the U-Net network architecture in the aforementioned embodiment: the feature extraction stage compresses the spatial dimension and extracts deep semantic features through downsampling operation, while this step gradually restores the feature map resolution through the corresponding upsampling operation, so that each pixel accurately corresponds to the tissue region in the original pathological image.

[0098] Optionally, in the visualization, tumor atypia regions are represented by a warm color gradient to indicate malignancy, stromal fiber orientation is indicated by a linear texture to show the direction of arrangement, and normal tissue regions are marked with cool colors to indicate structural integrity. This atlas transforms quantified regional features into anatomically meaningful spatial distribution expressions, providing a pathological analysis tool that combines quantitative accuracy with intuitive visualization, ultimately completing a multi-scale analysis loop from microscopic cellular features to macroscopic tissue distribution.

[0099] This embodiment constructs a multi-scale tissue heterogeneity characterization system by quantifying the cellular atypia index of tumor regions, the collagen fiber orientation distribution of stroma regions, and the structural integrity parameters of normal tissues from multiple dimensions. A spatial pyramid pooling layered fusion of microscopic cellular features and macroscopic tissue distribution characteristics is employed. Channel splicing achieves precise alignment of cellular morphological details with regional spatial distribution, and finally, deconvolution upsampling generates an intuitive and visual spatial distribution feature atlas. This embodiment integrates cellular-level pathological features with tissue-level spatial topological associations into a unified framework, preserving distinguishing features such as cellular atypia while revealing the three-dimensional spatial interaction patterns of cells, stroma, and normal tissues in the tumor microenvironment through multi-scale feature fusion. This provides a tool for pathological diagnosis that simultaneously possesses quantitative analytical precision and spatial visualization capabilities, significantly improving the comprehensiveness and clinical applicability of tissue heterogeneity analysis.

[0100] Please see Figure 4 In some embodiments, quantitative analysis of the immune detection information is performed, and spatial registration is performed between the immune detection information and pathological image information to obtain the registration result, including:

[0101] S401. Quantify the fluorescence signal of the immune detection information, and calculate the positive cell density, staining intensity spatial gradient and immune cell spatial aggregation coefficient corresponding to the preset labels. The preset labels include CD3+ label, CD8+ label and PD-L1 label.

[0102] S402. The positive cell density, staining intensity spatial gradient and immune cell spatial aggregation coefficient are integrated through the multimodal registration module. The feature point matching algorithm is used to establish the spatial correspondence between immune detection information and pathological image information to obtain the initial registration parameters.

[0103] S403. The initial registration parameters are optimized by elastic transformation with the tissue structure features in the pathological image to obtain the optimized spatial mapping relationship, which includes global affine transformation and local deformation compensation.

[0104] S404. Perform spatial transformation on the immune detection information according to the spatial mapping relationship to generate an immune marker distribution map that is spatially aligned with the pathological image, which is recorded as the registration result.

[0105] In step S401, fluorescence signal quantification is achieved through high-resolution fluorescence microscopy image processing. Positive cell density refers to the statistical number of CD3+, CD8+, or PD-L1-labeled positive cells per unit area, reflecting the degree of immune cell infiltration. The spatial gradient of staining intensity characterizes the regional differences in marker expression levels, calculated using pixel-level fluorescence intensity change rate. The spatial aggregation coefficient of immune cells quantifies the aggregation characteristics of immune cell distribution using a spatial autocorrelation algorithm. These parameters collectively constitute a multi-dimensional quantitative indicator of the immune microenvironment, where CD3+ labeling reflects the total T cell distribution, CD8+ labeling characterizes the localization of cytotoxic T cells, and PD-L1 labeling indicates the expression status of immune checkpoints.

[0106] In step S402, the multimodal registration module uses a scale-invariant feature transform algorithm to extract H&E-stained tissue structure feature points from the pathological image and immunolabeled feature points from the fluorescence image, establishing an initial correspondence through bidirectional nearest neighbor matching. Preferably, the feature point matching process incorporates the RANSAC algorithm to eliminate abnormal matching pairs, ensuring registration robustness. The initial registration parameters include a rotation matrix and a translation vector, achieving preliminary alignment of the immunodetection information and the pathological image in the global coordinate system.

[0107] In step S403, the elastic transformation optimization is implemented based on a thin-plate spline interpolation algorithm. Global affine transformation corrects overall displacement and rotational deviations, while local deformation compensation addresses non-rigid deformations generated during tissue section preparation. The optimization process uses tissue structural features as constraints, including anatomical landmarks such as glandular contours and blood vessel orientations, and iteratively optimizes the spatial mapping relationship by minimizing the reprojection error of feature points. This step effectively solves the problem of local misalignment in multimodal images caused by differences in sample processing.

[0108] In step S404, the spatial transformation uses bicubic interpolation to maintain the continuity of the fluorescence signal, ensuring that each pixel in the generated immunomarker distribution map precisely corresponds to the tissue structure of the pathological image. Preferably, spatial correlation features such as the overlap between the PD-L1 expression region and the tumor parenchyma, and the distribution pattern of CD8+ cells at the tumor front, can be directly quantified and analyzed using this map. This registration result achieves accurate mapping of immunophenotypic data to the histological background, providing a reliable basis for analyzing the spatial interaction between immune cells and tumor cells.

[0109] This embodiment employs a multi-parameter fluorescence signal quantification and multi-stage registration strategy to precisely anchor molecular-level immunodetection information to a histological structural framework. The spatial clustering coefficient reveals the distribution patterns of immune cell communities, while elastic transformation optimization addresses the nonlinear deformation problem of multimodal images. The resulting immunomarker distribution map not only retains the quantitative information of the original fluorescence signal but also establishes a topological correlation between immunomarker expression and histological features through spatial registration, creating a standardized data foundation for the quantitative spatial analysis of the tumor immune microenvironment.

[0110] In some embodiments, a density clustering algorithm is used to identify the spatial clustering characteristics of immune cells in the immune detection information based on the registration results, including:

[0111] Based on the density clustering algorithm, the immune cells in the immune marker distribution map are spatially clustered to obtain high-density clustered regions and discrete distribution regions;

[0112] Calculate the immune cell density gradient, spatial distribution entropy, and contact index with adjacent tissues in high-density clustered regions and discretely distributed regions;

[0113] By correlating immune cell density gradient, spatial distribution entropy, and contact index with adjacent tissues with the tumor microenvironment characteristics of the corresponding region, spatial aggregation characteristics of immune cells are generated.

[0114] In this embodiment, the density clustering algorithm is implemented using the DBSCAN algorithm. By setting a neighborhood radius and a minimum sample number threshold, it identifies the spatial distribution patterns of immune cells. High-density clustering regions refer to continuous cell communities that meet the core point condition, while discrete distribution regions are isolated cells that have not formed significant clusters. This algorithm can effectively distinguish tumor-infiltrating lymphocyte colonies from background noise, preserving biologically significant spatial distribution features.

[0115] The immune cell density gradient is obtained by calculating the rate of change in cell number per unit distance, reflecting the attenuation trend of immune cells from the aggregation center to the periphery. Spatial distribution entropy, using the Shannon entropy formula, quantifies the disorder of cell distribution and characterizes the orderliness of immune cell infiltration. The adjacent tissue contact index is calculated by dividing the contact boundary length between immune cells and tumor / stromal tissue, assessing the strength of spatial interaction between immune cells and specific tissue components. The immune cell density gradient, spatial distribution entropy, and adjacent tissue contact index quantify the spatial characteristics of the immune microenvironment from three dimensions: cell distribution dynamics, tissue arrangement regularity, and interfacial contact characteristics, respectively.

[0116] Preferably, the correlation analysis employs a spatial regression model to establish the mapping relationship between immune characteristics and tumor microenvironment parameters. These tumor microenvironment characteristics include quantitative indicators such as cell atypia index and collagen fiber orientation distribution. By calculating the Spearman correlation coefficient and spatial autocorrelation index, the spatial co-localization patterns between immune cell aggregation patterns and tumor malignancy and stromal remodeling status can be identified. The final generated spatial aggregation features include clinically significant topological parameters such as the hotspot coordinates of immune cell distribution, dominant invasion direction, and spatial association strength with the tumor boundary.

[0117] This embodiment transforms discrete immune cell localization data into spatial distribution features with a histological background through density clustering algorithms and multi-parameter spatial analysis. The immune cell density gradient reflects the dynamic process of immune infiltration, spatial distribution entropy reveals the orderliness of cell migration, and the neighboring tissue contact index quantifies the immune-tumor interface interaction. This embodiment not only preserves spatial information at the single-cell level but also establishes a spatial correspondence between immune distribution and histopathological changes by associating tumor microenvironment features, providing quantifiable spatial biomarkers for assessing immunotherapy response.

[0118] In some embodiments, the gradient rate of change of PD-L1 expression level is calculated, and an immune marker expression feature matrix reflecting the state of the tumor immune microenvironment is generated by feature fusion in combination with spatial clustering features, including:

[0119] The spatial gradient rate of change of PD-L1 expression level was calculated using the Sobel operator;

[0120] Based on the spatial aggregation feature map of immune cells, the spatial colocalization coefficient of CD8+ T cells and PD-L1 expression regions was extracted, and the immunosuppressive microenvironment index was calculated.

[0121] By integrating spatial gradient change rate, spatial colocalization coefficient and immunosuppressive microenvironment index through feature fusion layer, and using attention mechanism to weight the contribution of immune features in different spatial regions, weighted multidimensional immune features are obtained.

[0122] The weighted multidimensional immune features are mapped to a three-dimensional feature tensor according to spatial coordinates. The first dimension represents the expression intensity of PD-L1, the second dimension represents the spatial distribution of immune cells, and the third dimension records the microenvironment state parameters, thus generating an immune marker expression feature matrix.

[0123] In this embodiment, the Sobel operator calculates the first derivatives of the PD-L1 expression image in the x and y directions using a convolution kernel. The generated spatial gradient rate of change reflects the spatial variation characteristics of PD-L1 expression intensity, where the gradient magnitude characterizes the severity of expression level changes, and the gradient direction indicates the potential diffusion path of the immunosuppressive microenvironment. This calculation process is implemented using a 3×3 convolution template, which can effectively capture the expression mutation characteristics of the tumor-stromal junction.

[0124] Preferably, the spatial colocalization coefficient is obtained by calculating the percentage overlap between CD8+ T cell aggregation regions and PD-L1 highly expressed regions, and the Jaccard similarity coefficient is used to quantify the spatial correlation between the two. The immunosuppressive microenvironment index integrates the weighted sum of the colocalization coefficient and the PD-L1 gradient amplitude to reflect the degree of inhibition of effector T cell function by immune checkpoint molecules. The spatial colocalization coefficient and the immunosuppressive microenvironment index together constitute a quantitative indicator for assessing the risk of immune escape. The spatial colocalization coefficient reveals the spatial interaction between "immune cells and target molecules," while the immunosuppressive microenvironment index characterizes the biological effect of this interaction.

[0125] The feature fusion layer is implemented using a multi-head attention mechanism. It dynamically weights the spatial gradient change rate, spatial colocalization coefficient, and immunosuppressive microenvironment index through a learnable weight matrix. The attention weights are automatically adjusted according to the discriminative power of each feature in the tumor region, invasion front, and normal tissue. This can enhance the feature contribution of important microenvironment regions (such as immunosuppressive hotspots at the tumor edge) while suppressing interference from irrelevant background regions.

[0126] The resulting three-dimensional feature tensor unifies multi-dimensional immune features to a single reference frame through spatial coordinate alignment. The first dimension records gradient-weighted molecular expression levels, the second dimension encodes density gradient parameters in spatial clustering features, and the third dimension integrates comprehensive indicators such as inhibition index and distribution entropy. The immune marker expression feature matrix can be decomposed into tensor components to extract potential factors reflecting immunotherapy sensitivity, providing a spatial multi-omics basis for predicting immunotherapy response.

[0127] This embodiment quantifies the spatial gradient change rate of PD-L1 expression using the Sobel operator, calculates the immunosuppressive microenvironment index by combining the spatial colocalization coefficients of CD8+ T cells and PD-L1 expression regions, and dynamically weights and fuses multi-dimensional spatial features using an attention mechanism. Finally, it generates a three-dimensional feature tensor containing parameters of PD-L1 expression intensity, immune cell distribution, and microenvironment state. This breaks through the limitation of traditional immunohistochemistry that only focuses on expression intensity. Through spatial gradient analysis and multi-feature fusion, it reveals the heterogeneity characteristics of the tumor immune microenvironment and the distribution of immunosuppressive hotspots, providing a more comprehensive spatial multi-omics basis for evaluating immunotherapy response and significantly improving the accuracy of predicting immunotherapy efficacy.

[0128] In some embodiments, a graph neural network model is constructed based on tissue spatial distribution feature maps and immune marker expression feature matrices to extract spatial interaction features in the tumor microenvironment, including:

[0129] The regional features in the tissue spatial distribution feature map are quantified into node attributes of graph nodes. The node attributes include the cellular atypia index of the tumor region, the collagen fiber orientation distribution of the stroma region, and the structural integrity parameters of normal tissue.

[0130] A topological connectivity graph is constructed based on the spatial coordinates of pathological image information, with the center point of the tissue region as the graph node and the spatial distance and structural continuity between adjacent regions as the edge weights.

[0131] The expression feature matrix of immune markers was used as an additional node feature and fused with the regional features in the tissue spatial distribution feature map across modalities. The expression intensity of PD-L1 was associated with the graph nodes of the tumor region, and the distribution features of immune cells were associated with the graph nodes of the stromal region.

[0132] A multi-layer graph attention network is used for message passing. By calculating the spatial interaction strength between graph nodes, spatial interaction features are generated. These features include the spatial dependence of the tumor-immune-mesenchymal ternary microenvironment.

[0133] In this embodiment, the cell atypia index is calculated using the morphological characteristics of cell nuclei in pathological images (such as nuclear-cytoplasmic ratio and nuclear irregularity) to quantify the malignancy of the tumor region. The nuclear-cytoplasmic ratio is quantified by the ratio of the nuclear region area to the cytoplasmic area, and the nuclear irregularity is analyzed by Fourier descriptor harmonic component analysis of the nuclear membrane contour. The collagen fiber orientation distribution is analyzed by performing a two-dimensional Fourier transform on Masson-stained sections of the stroma region, and the standard deviation of the fiber orientation angle is calculated by extracting the main frequency domain component, reflecting the changes in the microenvironment of tumor invasiveness. The structural integrity parameter is analyzed using gray-level co-occurrence matrix analysis of H&E-stained images, and the preservation status of normal tissue is comprehensively evaluated by texture features such as contrast, correlation, and energy.

[0134] The construction of the topological connectivity graph uses the coordinates of the center point of each tissue region in the pathological image as the location of the graph nodes. The connection strength between nodes is determined by two factors: first, spatial distance, where closer regions have higher connection weights; and second, tissue structural similarity, which is determined by comparing image features such as cell arrangement patterns and staining depth between adjacent regions. Furthermore, smoother and more natural boundary lines at tissue boundaries indicate better structural continuity between the two sides, resulting in stronger connection weights. Finally, the weight value of each connection is a weighted combination of standardized spatial distance and structural similarity.

[0135] In the cross-modal feature fusion process, the expression feature matrix of immune markers and tissue features are associated through feature splicing. Among them, the docking of PD-L1 expression intensity with tumor region nodes reveals the distribution pattern of immune checkpoint molecules in malignant cells, while the association between immune cell distribution features and stromal region nodes reflects the infiltration status of effector cells in the periphery of the tumor.

[0136] The multi-layer graph attention network achieves message passing through a multi-head attention mechanism. Each attention head learns interaction patterns at different spatial scales, and the resulting spatial interaction features capture multi-dimensional interactions between tumor cells, such as PD-L1-mediated immune escape and the physical barrier of mesenchymal collagen remodeling against immune infiltration. In particular, the spatially dependent features of the ternary microenvironment quantify the dynamic equilibrium state among tumor, immune system, and mesenchymal matrix through cross-regional information aggregation of graph nodes.

[0137] This embodiment quantifies regional features in the tissue spatial distribution feature map into graph node attributes, combines spatial relationship modeling of topological connectivity graphs and cross-modal fusion of immune marker expression feature matrices, and uses graph attention networks to capture multi-level spatial interactions between tumor regions, stromal regions and immune features. This enables the quantitative characterization of complex interactions such as PD-L1-mediated immune escape and stromal barrier effects in the tumor microenvironment, providing a computable spatial feature framework for analyzing the ternary dynamic balance of tumor-immune-stromal.

[0138] In some embodiments, a treatment response prediction model is established by combining clinical treatment information. The extracted spatial interaction features are temporally aligned with the patient's treatment history, medication regimen, and follow-up results. A multi-task learning framework is used to simultaneously predict the probability of treatment response, optimal treatment timing, and adverse reaction risk. The output is a clinical decision report containing the immunotherapy response prediction results, including:

[0139] Spatial interaction features are concatenated with clinical treatment information, including the patient's past treatment history, medication regimen, and follow-up results. A spatiotemporal feature matrix is ​​constructed by aligning the feature representations of different time points through a time sequence encoder.

[0140] A multi-task prediction network was constructed, with the spatiotemporal feature matrix as input. The network outputs the treatment response probability, the optimal treatment timing, and the adverse reaction risk score through three prediction branches. The treatment response probability branch uses the spatial dependence feature of the tumor-immune-mesenchyma ternary microenvironment in the spatial interaction features as the key input.

[0141] A cross-modal attention module is added to the prediction network to dynamically calculate the correlation weights between spatial interaction features and clinical treatment features, automatically identify the feature combinations that have the greatest impact on the prediction results, and obtain multi-task prediction results.

[0142] The system integrates multi-task prediction results to generate a clinical decision report, which includes a predicted response curve based on spatial interaction characteristics, medication dosage adjustment recommendations combined with adverse reaction risk scores, risk warning thresholds, and treatment time window recommendations generated based on the prediction of optimal treatment timing.

[0143] In this embodiment, the key to constructing the treatment response prediction model lies in the deep integration of the spatial interaction characteristics of the tumor microenvironment with clinical treatment information. The temporal encoder processes time-series data such as patient treatment history, medication regimens, and follow-up results using a Long Short-Term Memory (LSTM) network structure, aligning them with the spatial interaction characteristics in the time dimension to form a spatiotemporal feature matrix. The treatment history includes the type, duration, and response status of previous treatment regimens; the medication regimen records the specific drug types, dosages, and administration methods; and the follow-up results integrate dynamic monitoring data such as imaging assessments and laboratory tests.

[0144] The multi-task prediction network employs a structural design that combines shared underlying feature extraction with task-specific branches. The treatment response probability branch focuses on the spatially dependent features of the tumor-immune-stromal ternary microenvironment. These features, extracted through a graph attention network using cross-regional interaction patterns, effectively reflect the microenvironmental basis of immunotherapy sensitivity. The optimal treatment timing branch analyzes the correlation patterns between spatial features and treatment time series. The adverse reaction risk branch focuses on the synergistic early warning signals of clinical and microenvironmental features. The cross-modal attention module automatically identifies key feature combinations by calculating the cross-attention weights between spatial and clinical features; for example, it may discover the synergistic predictive value of PD-L1 spatial distribution features and specific drug regimens.

[0145] The generation process of clinical decision reports is dynamically adaptive. The predicted response curve reflects changes in treatment sensitivity through the temporal evolution trend of spatial interaction characteristics; the medication dosage adjustment recommendations comprehensively consider the balance between adverse reaction risk scores and treatment response probabilities; the risk warning threshold is set based on safety boundary parameters in clinical treatment information; and the treatment time window recommendations are dynamically generated by combining the prediction of optimal treatment timing with the patient's current status.

[0146] This embodiment aligns spatial interaction features with clinical treatment information in a temporal sequence and constructs a spatiotemporal feature matrix. It utilizes a multi-task prediction network to synchronously analyze treatment response probability, optimal treatment timing, and adverse reaction risks. Combined with a cross-modal attention module, it dynamically filters key feature combinations and ultimately generates a clinical decision report containing predicted response curves, dose adjustment suggestions, and treatment time windows. This achieves deep integration of tumor microenvironment spatial features and clinical temporal data, enabling more accurate prediction of immunotherapy response and optimization of individualized treatment plans, providing multi-dimensional quantitative support for clinical decision-making.

[0147] In a second aspect, this embodiment also provides a computer-readable storage medium having stored computer program instructions thereon, which, when executed by a processor, implement the system described in the first aspect.

[0148] The computer program involved in this embodiment can be stored in a computer device readable storage medium, which includes, but is not limited to, disks, magnetic tapes, magnetic cards, floppy disks, flash memory, optical disks, optical cards, read-only memory (ROM), random access memory (RAM), erasable programmable ROM (EPROM), and electrically erasable programmable ROM (EEPROM), etc. It also includes other biological, physical, or chemical structures capable of performing similar or equivalent functions to the storage media listed above, such as DNA, RNA, proteins, and other units with information storage capabilities. In specific embodiments, the storage medium involved can be one of the above-mentioned media types or a combination of the above media types. In different embodiments, the computer program involved in the embodiment can be centrally stored in a single medium or distributed across multiple media. The memory containing the computer device readable storage medium can be non-volatile memory or random access memory. These computer device readable storage media can be built into the device or connected to the device involved in the embodiment as an external device or part of an external device. In some embodiments, the memory having a computer device readable storage medium is deployed locally; in other embodiments, the memory may be deployed remotely from the processor, for example, as a network-attached memory accessed via RF circuitry or an external port and a communication network, wherein the communication network may be the Internet, one or more intranets, a local area network (LAN), a wide area network (WLAN), a storage area network (SAN), or a suitable combination thereof, as long as computer device access to the memory is enabled. Furthermore, the computer program involved in the embodiments may be stored in plaintext / ciphertext form, or it may be designed as training data, integrated and recombined through model training and implicitly stored in the parameter states of a deep neural network or other machine learning model.

[0149] Please see Figure 5 In a third aspect, this embodiment also provides an electronic device 1, including a memory 11 and a processor 12, wherein the memory 11 is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor 12 to implement the system described in the first aspect.

[0150] The processor described in this embodiment can be implemented by hardware, firmware, software, or a combination thereof. It can be a circuit, one or more of an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field-programmable gate array (FPGA), a central processing unit (CPU), a controller, a microcontroller, or a microprocessor. It also includes other physical, biological, or chemical structures that can implement the same or equivalent functions as the processors listed above, such as biological neurons, quantum computing units, DNA computing units, etc., so that the processor can execute some or all of the steps or any combination of the steps mentioned therein in the computer program or system involved in the various embodiments of this application.

[0151] By adopting the above technical solutions, this invention differs from existing technologies and has the following beneficial effects: It extracts cellular morphological features and spatial distribution feature maps from pathological images using deep convolutional neural networks, combines spatial registration and density clustering analysis of immune detection information to obtain spatial aggregation features of immune cells, and integrates the gradient change rate of PD-L1 expression levels to generate an immune marker expression feature matrix; it constructs a graph neural network model based on tissue spatial distribution feature maps and immune marker expression feature matrices, and uses graph attention mechanisms to extract the spatial dependence features of the tumor-immune-mesenchymal ternary microenvironment; by aligning spatial interaction features with the temporal sequence of clinical treatment information, it uses a multi-task learning framework to simultaneously predict the probability of treatment response, optimal treatment timing, and adverse reaction risks, ultimately generating a clinical decision report containing predicted response curves, risk warning thresholds, and treatment time window suggestions.

[0152] The above technical solutions achieve deep integration of the spatial heterogeneity characteristics of the tumor microenvironment with clinical time-series data, overcoming the problems of insufficient modeling of spatial interactions in the tumor microenvironment and single prediction dimension. The generated clinical decision reports can simultaneously reflect the temporal dynamic characteristics and spatial distribution patterns of immunotherapy response, providing accurate decision support for colorectal cancer immunotherapy with both spatial resolution and temporal prediction capabilities.

[0153] Finally, it should be noted that although the above embodiments have been described in the text and drawings of this application, this should not limit the scope of patent protection of this application. Any technical solutions that are based on the essential concept of this application and utilize the content described in the text and drawings of this application, resulting in equivalent structural or procedural substitutions or modifications, as well as the direct or indirect application of the technical solutions of the above embodiments to other related technical fields, are all included within the scope of patent protection of this application.

Claims

1. A neural network prediction system for colorectal cancer immune response maps, characterized in that, include: We collected pathological image information and immune test information from colorectal cancer patients, and also collected basic clinical information from colorectal cancer patients as auxiliary features. Feature extraction is performed on the pathological image information. A deep convolutional neural network is used to perform multi-level feature extraction on the pathological image information to obtain cell morphology features and spatial distribution features. The spatial distribution features include the spatial distribution of tumor regions, stroma regions, and normal tissues, generating a spatial distribution feature atlas containing tissue heterogeneity information, including: The pathological image information is then subjected to standardization and region enhancement processes in sequence to obtain the processed pathological image information. The pre-trained ResNet50 network was used to extract basic features from the processed pathological image information. Cell nuclear morphology, chromatin distribution and cell arrangement features were captured through shallow convolutional layers to generate cell morphology features. Based on the cell morphology features, semantic segmentation is performed using the encoder-decoder structure of the U-Net network architecture. In the encoding stage, global contextual information of tumor region, stroma region and normal tissue is extracted by progressive downsampling. In the decoding stage, skip connections are combined to restore spatial details and generate spatial distribution features containing accurate information of tissue boundaries. The spatial distribution characteristics are quantified by region features, and the cell atypia index, collagen fiber orientation distribution in the stroma region, and structural integrity parameters of normal tissue are calculated respectively within the tumor region. By integrating the cell atypia index, collagen fiber orientation distribution and structural integrity parameters through the feature fusion layer, multi-scale tissue distribution features are extracted by spatial pyramid pooling to obtain quantified regional features. The quantified regional features are concatenated with the corresponding cell morphology features to obtain tissue heterogeneity information, which includes microscopic cell features and macroscopic tissue distribution features. The spatial distribution feature map is obtained based on the tissue heterogeneity information; Quantitative analysis is performed on the immune detection information, and spatial registration is performed between the immune detection information and pathological image information to obtain the registration result. A density clustering algorithm is then used to identify the spatial aggregation characteristics of immune cells in the immune detection information based on the registration result, including: The immunodetection information is quantified by fluorescence signal, and the positive cell density, staining intensity spatial gradient and immune cell spatial aggregation coefficient corresponding to the preset labels are calculated respectively. The preset labels include CD3+ label, CD8+ label and PD-L1 label. The positive cell density, staining intensity spatial gradient, and immune cell spatial aggregation coefficient are integrated by the multimodal registration module. The spatial correspondence between immune detection information and pathological image information is established by the feature point matching algorithm to obtain the initial registration parameters. The initial registration parameters are optimized by elastic transformation with the tissue structure features in the pathological image to obtain the optimized spatial mapping relationship, which includes global affine transformation and local deformation compensation. Based on the spatial mapping relationship, the immune detection information is spatially transformed to generate an immune marker distribution map that is spatially aligned with the pathological image, which is denoted as the registration result. Simultaneously, the gradient change rate of PD-L1 expression level is calculated, and an immune marker expression feature matrix that reflects the state of the tumor immune microenvironment is generated by feature fusion in combination with spatial aggregation characteristics. Based on the spatial distribution feature map of tissues and the expression feature matrix of immune markers, a graph neural network model is constructed. The spatial interaction between different functional regions is modeled through the graph attention mechanism, and the spatial interaction features of the tumor microenvironment are extracted. By combining clinical treatment information, a treatment response prediction model is established. The extracted spatial interaction features are temporally aligned with the patient's treatment history, medication regimen, and follow-up results. A multi-task learning framework is used to simultaneously predict the probability of treatment response, the optimal timing of treatment, and the risk of adverse reactions. The model outputs a clinical decision report containing the predicted results of immunotherapy response, which includes the predicted response curve, risk warning threshold, and treatment time window recommendations.

2. The colorectal cancer immune response map neural network prediction system according to claim 1, characterized in that, The registration results are analyzed using a density clustering algorithm to identify spatial clustering features of immune cells in the immune detection information, including: Based on the density clustering algorithm, the immune cells in the immune marker distribution map are spatially clustered to obtain high-density clustered regions and discrete distribution regions; Calculate the immune cell density gradient, spatial distribution entropy, and contact index with adjacent tissues in high-density clustered regions and discretely distributed regions; The spatial aggregation characteristics of immune cells are generated by correlating the immune cell density gradient, spatial distribution entropy, and contact index with the tumor microenvironment characteristics of the corresponding region.

3. The colorectal cancer immune response map neural network prediction system according to claim 1, characterized in that, The gradient rate of change of PD-L1 expression levels was calculated, and an immune marker expression feature matrix reflecting the state of the tumor immune microenvironment was generated through feature fusion by combining spatial clustering features. This matrix includes: The spatial gradient rate of change of PD-L1 expression level was calculated using the Sobel operator; Based on the spatial aggregation feature map of immune cells, the spatial colocalization coefficient of CD8+ T cells and PD-L1 expression regions was extracted, and the immunosuppressive microenvironment index was calculated. By integrating spatial gradient change rate, spatial colocalization coefficient and immunosuppressive microenvironment index through feature fusion layer, and using attention mechanism to weight the contribution of immune features in different spatial regions, weighted multidimensional immune features are obtained. The weighted multidimensional immune features are mapped to a three-dimensional feature tensor according to spatial coordinates. The first dimension represents the PD-L1 expression intensity, the second dimension represents the spatial distribution of immune cells, and the third dimension records the microenvironment state parameters, thereby generating the immune marker expression feature matrix.

4. The colorectal cancer immune response map neural network prediction system according to claim 1, characterized in that, Based on the aforementioned tissue spatial distribution feature map and immune marker expression feature matrix, a graph neural network model is constructed to extract spatial interaction features within the tumor microenvironment, including: The regional features in the tissue spatial distribution feature map are quantified into node attributes of graph nodes. The node attributes include the cell atypia index of the tumor region, the collagen fiber orientation distribution of the stroma region, and the structural integrity parameters of normal tissue. A topological connectivity graph is constructed based on the spatial coordinates of pathological image information, with the center point of the tissue region as the graph node and the spatial distance and structural continuity between adjacent regions as the edge weights. The expression feature matrix of the immune markers is used as an additional node feature and fused with the regional features in the tissue spatial distribution feature map across modalities. The PD-L1 expression intensity is associated with the graph nodes of the tumor region, and the immune cell distribution features are associated with the graph nodes of the stromal region. A multi-layer graph attention network is used for message passing. By calculating the spatial interaction strength between graph nodes, spatial interaction features are generated. These spatial interaction features include the spatial dependence features of the tumor-immune-mesenchymal ternary microenvironment.

5. The colorectal cancer immune response map neural network prediction system according to claim 1, characterized in that, By combining clinical treatment information, a treatment response prediction model is established. The extracted spatial interaction features are temporally aligned with the patient's treatment history, medication regimen, and follow-up results. A multi-task learning framework is used to simultaneously predict the probability of treatment response, optimal treatment timing, and adverse reaction risk. The output is a clinical decision report containing the immunotherapy response prediction results, including: The spatial interaction features are concatenated with clinical treatment information, which includes the patient's past treatment history, medication regimen, and follow-up results. The feature representations of different time points are aligned using a time encoder to construct a spatiotemporal feature matrix. A multi-task prediction network was constructed, with the spatiotemporal feature matrix as input. The network outputs the treatment response probability, the optimal treatment timing, and the adverse reaction risk score through three prediction branches. The treatment response probability branch uses the spatial dependence feature of the tumor-immune-mesenchyma ternary microenvironment in the spatial interaction features as the key input. A cross-modal attention module is added to the prediction network to dynamically calculate the correlation weights between spatial interaction features and clinical treatment features, automatically identify the feature combinations that have the greatest impact on the prediction results, and obtain multi-task prediction results. The clinical decision report is generated by integrating the prediction results of multiple tasks. The clinical decision report includes a prediction response curve based on spatial interaction characteristics, medication dosage adjustment suggestions combined with adverse reaction risk scores and risk warning thresholds, and treatment time window suggestions generated based on the prediction of the optimal treatment timing.

6. A computer-readable storage medium storing computer program instructions thereon, characterized in that, The computer program instructions, when executed by a processor, implement the system as described in any one of claims 1 to 5.

7. An electronic device comprising a memory and a processor, characterized in that, The memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the system as described in any one of claims 1 to 5.

Images