Tumor pathology progression prediction system based on multi-modal perception

Abstract

This invention relates to a tumor pathological progression prediction system based on multimodal perception. The system includes a multi-source data acquisition module, a preliminary optimization processing module, a progression prediction model construction module, and a tumor pathological progression prediction module. The invention obtains raw data through data acquisition; employs preliminary optimization processing methods including multi-source spatiotemporal alignment, anomaly detection and repair, feature engineering and enhancement, and dataset construction; utilizes a pathological feature primitive interpretation method, maintaining a learnable pathological feature primitive library to reveal potential progression drivers from high-dimensional heterogeneous signals, while introducing biological constraints to ensure that the proportions of the resolved components conform to the actual tumor evolution patterns; and employs a counterfactual baseline generation method, modeling and reasoning about the normal ideal control state without tumor progression interference to remove individual background noise, while adversarial learning ensures that the baseline state is not contaminated by tumor information, improving the scientific validity and credibility of progression attribution.

Description

Technical Field

[0001] This invention relates to the field of intelligent tumor pathological progression prediction technology, specifically to a tumor pathological progression prediction system based on multimodal perception. Background Technology

[0002] Tumor pathological progression prediction refers to the process of dynamically assessing and predicting the growth, invasion, metastasis, and treatment response of tumors by integrating multimodal medical data such as medical imaging, pathological slides, genome sequencing, clinical testing, and medical history. It can assist clinicians in identifying high-risk patients in advance, optimizing individualized treatment plans, improving patient survival outcomes, and reducing unnecessary consumption of medical resources.

[0003] However, traditional tumor pathology progression prediction systems suffer from several limitations. When faced with multimodal heterogeneous data, they can only provide a rough assessment of a single modality, failing to effectively integrate and analyze the synergistic features contained in multi-source information such as imaging, molecular, and pathology. This makes it difficult to accurately quantify the contribution weight of different modalities to progression and to reveal the multi-factor driving mechanisms behind progression. Furthermore, traditional tumor pathology progression prediction systems fail to construct an analytical framework that isolates individual baseline differences and identifies the effects of tumor evolution itself. They cannot distinguish whether a change in imaging is caused by actual tumor progression or by differences in treatment response, inflammation, or scanning conditions, resulting in prediction results that are consistently decoupled from the intrinsic biological behavior of the tumor. Summary of the Invention

[0004] To address the above issues and overcome the shortcomings of existing technologies, this invention provides a tumor pathological progression prediction system based on multimodal perception. Traditional tumor pathological progression prediction systems, when faced with multimodal heterogeneous data, can only provide a rough assessment of a single modality, failing to effectively integrate and analyze the synergistic features inherent in multi-source information such as imaging, molecular, and pathological data. This makes it difficult to accurately quantify the contribution weights of different modalities to progression and to reveal the multi-factor driving mechanisms behind progression. This solution creatively employs a pathological feature primitive interpretation method. By maintaining a learnable pathological feature primitive library, it analyzes the dynamically changing tumor phenotypic components and their proportions over time in multimodal data, revealing potential or complex progression-driving factors from high-dimensional heterogeneous signals, while simultaneously introducing biological constraints. This approach ensures that the analyzed component proportions conform to the actual tumor evolution patterns. Addressing the technical limitations of traditional tumor pathological progression prediction systems—namely, their inability to construct an analytical framework that isolates individual baseline differences and identifies the tumor's own evolutionary effects, thus failing to distinguish whether a change in imaging is caused by actual tumor progression or by differences in treatment response, inflammation, or scanning conditions, and resulting in predictions consistently being decoupled from the tumor's intrinsic biological behavior—this solution creatively employs a counterfactual baseline generation method. By modeling and inferring the normal ideal control state without tumor progression interference, it isolates the pure driving effect of the tumor on multimodal phenotypes. This method can isolate individual background noise from a causal perspective, while adversarial learning ensures that the baseline state is not contaminated by tumor information, improving the scientific validity and credibility of progression attribution.

[0005] The technical solution adopted by the present invention is as follows: The tumor pathological progression prediction system based on multimodal perception provided by the present invention includes a multi-source data acquisition module, a preliminary optimization processing module, a progression prediction model construction module, and a tumor pathological progression prediction module;

[0006] The multi-source data acquisition module obtains the original dataset for progress prediction through data collection and sends the original dataset for progress prediction to the preliminary optimization processing module.

[0007] The preliminary optimization processing module employs multi-source spatiotemporal alignment, anomaly detection and repair, feature engineering and enhancement, and dataset construction as preliminary optimization processing methods to obtain a dataset to be processed, a prediction training set, and a prediction test set. The dataset to be processed is then sent to the tumor pathology progression prediction module, and the prediction training set and the prediction test set are sent to the progression prediction model construction module.

[0008] The progression prediction model construction module obtains a progression prediction model by constructing an improved deep state space model, and sends the progression prediction model to the tumor pathology progression prediction module.

[0009] The tumor pathology progression prediction module performs real-time tumor pathology progression prediction by inputting the dataset to be processed into the progression prediction model, and obtains the final progression prediction result.

[0010] Furthermore, in the multi-source data acquisition module, the original dataset for progress prediction specifically includes a historical original dataset for progress prediction and a current original dataset for progress prediction. Both the historical original dataset for progress prediction and the current original dataset for progress prediction contain tumor multimodal monitoring data and individual baseline reference data. The historical original dataset for progress prediction also includes tumor pathological progression risk level labels.

[0011] Furthermore, in the preliminary optimization processing module, the multi-source spatiotemporal alignment is used to unify the time and space reference of all data to eliminate misalignment. Specifically, it aligns the timestamps of different devices to millisecond-level precision and registers data from different spatial locations to the same anatomical coordinate system to obtain strictly aligned data.

[0012] The anomaly detection and repair is used to identify and correct noise and missing data. Specifically, it detects anomalies using an adaptive threshold algorithm and repairs them using temporal prediction and spatial interpolation methods to obtain anomaly-repaired data.

[0013] The feature engineering and enhancement are used to extract features from the original signal, specifically by performing three-dimensional wavelet transform on image data to extract texture features, constructing statistical and hysteresis features on time-series data, and selecting features based on mutual information methods.

[0014] The dataset construction is used to organize the processed data into a sample set. Specifically, it involves extracting data from consecutive time periods using a sliding window to form samples, and using oversampling techniques to balance the categories. Finally, the dataset is divided into a training data set and a test data set.

[0015] The original dataset for current progress prediction is initially optimized through multi-source spatiotemporal alignment, anomaly detection and repair, and feature engineering and enhancement to obtain a dataset to be processed. The original dataset for historical progress prediction is initially optimized through multi-source spatiotemporal alignment, anomaly detection and repair, feature engineering and enhancement, and dataset construction to obtain a prediction training set and a prediction test set.

[0016] Furthermore, in the progression prediction model building module, the model required to automatically learn the progression-driving components and source causal relationships from the data is specifically constructed by building an improved deep state space model as the progression prediction model. The improved deep state space model maintains a learnable pathological feature primitive library and combines it with a counterfactual baseline generation design to output the model prediction component-source matrix.

[0017] The progression prediction model construction module specifically includes state-space encoding, pathological feature primitive interpretation, counterfactual baseline generation, fusion prediction output, and model construction and training.

[0018] The state-space encoding is used to extract a unified joint state representation from multi-source data using a state-space model, and includes:

[0019] Input data organization and embedding are used to normalize data of different physical meanings and scales into tensor forms that the model can process. Specifically, by defining the specific data dimensions and time lengths of micro-flow input and macro-flow input, micro-flow input features and macro-flow input features are obtained.

[0020] The micro-flow input features are obtained from high-resolution image data, digital image data of pathological sections, genomic mutation spectrum data, and protein biomarker data. The time length is determined by sampling at equal intervals within the treatment cycle, and the feature dimension is the sum of the number of image voxels and the number of biomarker types. The macro-flow input features are obtained from clinical test data. The time length is aligned with the micro-flow input features, and the feature dimension is the number of clinical indicator types.

[0021] Selective state space layer forward propagation is used to encode information with long-range dependency modeling capability for each data stream. Specifically, it dynamically generates state transition parameters that depend on the input through a selective scanning mechanism and performs recursive calculations to obtain micro-flow state features and macro-flow state features of the same dimension.

[0022] Multi-scale dynamic fusion is used to adaptively integrate the features of micro-flow and macro-flow. Specifically, it uses a learnable weight calculation module to dynamically determine the contribution ratio of micro-flow state features and macro-flow state features at each time step and perform weighted fusion to obtain joint state features.

[0023] The pathological feature primitive interpretation is used to analyze the dynamically changing tumor phenotypic components and their proportions over time in multimodal data by maintaining a learnable library of pathological feature primitives. The content includes:

[0024] The initialization and updating of the pathological feature primitive library is used to establish and maintain an internal knowledge base representing different tumor evolution patterns. Specifically, it defines multiple learnable pathological feature primitives, each containing a primitive state vector and a primitive evolution matrix. The initialization of the primitive state vector follows a normal distribution. The primitive evolution matrix is ​​a learnable diagonal matrix whose elements take values ​​in the range of (0,1);

[0025] Attention component parsing is used to calculate the contribution weight of each pathological feature primitive to the overall tumor state at the current time. Specifically, attention is calculated by using the joint state feature as the query and the primitive state vector as the key and value, and sparsity constraints are added to obtain the pathological component weight vector.

[0026] Biological reconstruction constraints are used to ensure that the component analysis process conforms to the laws of tumor biology. Specifically, the input multimodal features are reconstructed using learnable standard phenotypic vectors of pathological component weights and pathological feature primitives, and the reconstruction error is calculated as a constraint.

[0027] The counterfactual baseline generation, used to infer the patient's normal state in the absence of tumor progression, to quantify the net driving effect of tumor, includes:

[0028] Counterfactual baseline prediction is used to generate a sequence of tumor states based solely on individual baseline reference data. Specifically, it encodes baseline features through a recurrent neural network and predicts future states autoregressively to obtain counterfactual baseline state features that represent the absence of tumor progression interference.

[0029] Causal effect calculation is used to calculate the net impact of tumor on multimodal phenotypes. Specifically, it is obtained by subtracting the counterfactual baseline state features from the true joint state features element by element and then normalizing them.

[0030] Adversarial training is introduced to force the counterfactual baseline state features to completely ignore tumor information during generation, ensuring the purity of the baseline. Specifically, an adversarial discriminator is introduced to distinguish whether the generated counterfactual baseline state features are contaminated by tumor information, and adversarial loss is calculated for adversarial training.

[0031] The fusion prediction output is used to fuse pathological components with tumor-driving effects to predict tumor pathological progression, and includes:

[0032] Dimension alignment and feature enhancement are used to map different features to the same semantic space and extract higher-order information. Specifically, through two independent linear transformations and nonlinear activation functions, the pathological component weight features and tumor-driven influence features are mapped to the same semantic space to obtain pathological component enhancement features and tumor-driven enhancement features.

[0033] The outer product fusion is used to establish the correlation between pathological components and tumor driving sources in two dimensions. Specifically, it involves performing an outer product operation on the enhanced features of pathological components and the enhanced features of tumor driving sources, and then normalizing the result to obtain a two-dimensional matrix sequence. The elements of the two-dimensional matrix at each time step represent the proportion of a specific pathological phenotype component from a specific tumor driving factor at that time step, thus obtaining the model prediction component-source matrix.

[0034] The model is constructed and trained by integrating the state space encoding, pathological feature primitive interpretation, counterfactual baseline generation, and fusion prediction output to build the improved deep state space model. The model is then trained and its performance is verified based on the prediction training set and prediction test set to obtain the improved deep state space model, which serves as the progression prediction model.

[0035] Furthermore, in the tumor pathology progression prediction module, the dataset to be processed is used as the input of the progression prediction model to predict the risk of tumor pathology progression. The model prediction component-source matrix output by the model is used as the final progression prediction result, and the tumor pathology progression risk level is determined based on the final progression prediction result.

[0036] The beneficial effects achieved by the present invention using the above solution are as follows:

[0037] (1) In view of the technical problems of traditional tumor pathological progression prediction systems, which can only provide a rough assessment of a single modality when faced with multimodal heterogeneous data, and cannot effectively integrate and analyze the synergistic features contained in multi-source information such as imaging, molecular, and pathology, it is difficult to accurately quantify the contribution weight of different modalities to progression and cannot reveal the multi-factor driving mechanism behind progression, this solution creatively adopts the pathological feature primitive interpretation method. By maintaining a learnable pathological feature primitive library, it analyzes the tumor phenotypic components and their proportions that change dynamically over time in multimodal data, which can reveal potential or complex progression driving factors from high-dimensional heterogeneous signals. At the same time, biological constraints are introduced to ensure that the proportions of the analyzed components conform to the real tumor evolution law.

[0038] (2) In response to the technical problems of traditional tumor pathological progression prediction systems, which fail to construct an analytical framework that separates individual baseline differences and identifies the tumor's own evolutionary effects, and cannot distinguish whether a change in imaging is caused by actual tumor progression or by differences in treatment response, inflammation or scanning conditions, and whose prediction results are always decoupled from the intrinsic biological behavior of the tumor, this solution creatively adopts a counterfactual baseline generation method. By modeling and reasoning about the normal ideal control state without tumor progression interference, the pure driving effect of the tumor on multimodal phenotypes can be separated. This method can separate individual background noise from the causal level. At the same time, adversarial learning is used to ensure that the baseline state is not contaminated by tumor information, thereby improving the scientificity and credibility of progression attribution. Attached Figure Description

[0039] Figure 1 A schematic diagram of the modules of the tumor pathological progression prediction system based on multimodal perception provided by the present invention;

[0040] Figure 2 A flowchart illustrating the initial optimization of the processing module;

[0041] Figure 3 A flowchart illustrating the process of building a module for the progress prediction model;

[0042] Figure 4 A flowchart illustrating the process of interpreting pathological feature primitives in the module for constructing a progression prediction model.

[0043] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof. Detailed Implementation

[0044] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0045] In the description of this invention, it should be understood that the terms "upper", "lower", "front", "rear", "left", "right", "top", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0046] Example 1, see Figure 1 The tumor pathology progression prediction system based on multimodal perception provided by the present invention includes a multi-source data acquisition module, a preliminary optimization processing module, a progression prediction model construction module, and a tumor pathology progression prediction module.

[0047] The multi-source data acquisition module obtains the original dataset for progress prediction through data collection and sends the original dataset for progress prediction to the preliminary optimization processing module.

[0048] The preliminary optimization processing module employs multi-source spatiotemporal alignment, anomaly detection and repair, feature engineering and enhancement, and dataset construction as preliminary optimization processing methods to obtain a dataset to be processed, a prediction training set, and a prediction test set. The dataset to be processed is then sent to the tumor pathology progression prediction module, and the prediction training set and the prediction test set are sent to the progression prediction model construction module.

[0049] The progression prediction model construction module obtains a progression prediction model by constructing an improved deep state space model, and sends the progression prediction model to the tumor pathology progression prediction module.

[0050] The tumor pathology progression prediction module performs real-time tumor pathology progression prediction by inputting the dataset to be processed into the progression prediction model, and obtains the final progression prediction result.

[0051] Example 2, see Figure 1 This embodiment is based on the above embodiment. In the multi-source data acquisition module, the original dataset for progress prediction specifically includes a historical original dataset for progress prediction and a current original dataset for progress prediction. Both the historical original dataset for progress prediction and the current original dataset for progress prediction contain tumor multimodal monitoring data and individual baseline reference data. The historical original dataset for progress prediction also includes tumor pathological progression risk level labels.

[0052] The tumor multimodal monitoring data specifically includes high-resolution imaging data, digital image data of pathological sections, genomic mutation spectrum data, protein biomarker data, and clinical test data. The high-resolution imaging data specifically includes MRI imaging data and CT imaging data. The individual baseline reference data specifically includes patient age data, patient gender data, patient past medical history data, patient treatment history data, and non-tumor tissue multimodal monitoring data of the same dimension as the tumor multimodal monitoring data.

[0053] Example 3, see Figure 1 and Figure 2 This embodiment is based on the above embodiment. In the preliminary optimization processing module, the multi-source spatiotemporal alignment is used to unify the time and space reference of all data to eliminate misalignment. Specifically, it aligns the timestamps of different devices to millisecond-level precision and registers data from different spatial locations to the same anatomical coordinate system to obtain strictly aligned data.

[0054] The anomaly detection and repair is used to identify and correct noise and missing data. Specifically, it detects anomalies using an adaptive threshold algorithm and repairs them using temporal prediction and spatial interpolation methods to obtain anomaly-repaired data.

[0055] The feature engineering and enhancement are used to extract features from the original signal, specifically by performing three-dimensional wavelet transform on image data to extract texture features, constructing statistical and hysteresis features on time-series data, and selecting features based on mutual information methods.

[0056] The dataset construction is used to organize the processed data into a sample set. Specifically, it involves extracting data from consecutive time periods using a sliding window to form samples, and using oversampling techniques to balance the categories. Finally, the dataset is divided into a training data set and a test data set.

[0057] The original dataset for current progress prediction is initially optimized through multi-source spatiotemporal alignment, anomaly detection and repair, and feature engineering and enhancement to obtain a dataset to be processed. The original dataset for historical progress prediction is initially optimized through multi-source spatiotemporal alignment, anomaly detection and repair, feature engineering and enhancement, and dataset construction to obtain a prediction training set and a prediction test set.

[0058] Example 4, see Figure 1 , Figure 3 and Figure 4 This embodiment is based on the above embodiment. In the progression prediction model construction module, a model is used to automatically learn the progression-driving components and source causal relationships from the data. Specifically, an improved deep state space model is constructed as the progression prediction model. The improved deep state space model maintains a learnable pathological feature primitive library and combines it with a counterfactual baseline generation design to output the model prediction component-source matrix.

[0059] The progression prediction model construction module specifically includes state-space encoding, pathological feature primitive interpretation, counterfactual baseline generation, fusion prediction output, and model construction and training.

[0060] The state-space encoding is used to extract a unified joint state representation from multi-source data using a state-space model, and includes:

[0061] Input data organization and embedding are used to normalize data of different physical meanings and scales into tensor forms that the model can process. Specifically, by defining the specific data dimensions and time lengths of micro-flow input and macro-flow input, micro-flow input features and macro-flow input features are obtained.

[0062] The micro-flow input features are obtained from high-resolution image data, digital image data of pathological sections, genomic mutation spectrum data, and protein biomarker data. The time length is determined by sampling at equal intervals within the treatment cycle, and the feature dimension is the sum of the number of image voxels and the number of biomarker types. The macro-flow input features are obtained from clinical test data. The time length is aligned with the micro-flow input features, and the feature dimension is the number of clinical indicator types.

[0063] Selective state space layer forward propagation is used to encode information with long-range dependency modeling capability for each data stream. Specifically, it dynamically generates state transition parameters that depend on the input through a selective scanning mechanism and performs recursive calculations to obtain micro-flow state features and macro-flow state features of the same dimension.

[0064] Multi-scale dynamic fusion is used to adaptively integrate the features of micro-flows and macro-flows. Specifically, it uses a learnable weight calculation module to dynamically determine the contribution ratio of micro-flow state features and macro-flow state features at each time step and performs weighted fusion to obtain joint state features. The formula used is as follows:

[0065] ;

[0066] In the formula, Indicates the fusion weights at time step t. This represents the learnable weights used to calculate the fusion weights. This represents the learnable bias term used to calculate the fusion weights. This indicates a splicing operation. The t-th row represents the characteristics of the micro-flow state. The t-th row represents the macroscopic flow state characteristics. The t-th row represents the joint state features. This represents element-wise multiplication. Indicates the characteristics of the joint state. Indicates the transpose operation;

[0067] The pathological feature primitive interpretation is used to analyze the dynamically changing tumor phenotypic components and their proportions over time in multimodal data by maintaining a learnable library of pathological feature primitives. The content includes:

[0068] The initialization and updating of the pathological feature primitive library is used to establish and maintain an internal knowledge base representing different tumor evolution patterns. Specifically, it defines multiple learnable pathological feature primitives, each containing a primitive state vector and a primitive evolution matrix. The initialization of the primitive state vector follows a normal distribution. The primitive evolution matrix is ​​a learnable diagonal matrix with elements ranging from (0,1). The primitive state vector is updated at each time step using the following formula:

[0069] ;

[0070] In the formula, Let represent the primitive state vector of the a-th pathological feature primitive at time step t. This represents the primitive evolution matrix of the a-th pathological feature primitive. Let I represent the primitive state vector of the a-th pathological feature primitive at time step t-1, and let I represent the identity matrix of the same size as the primitive evolution matrix. The weights are used to calculate the primitive state update values. This indicates the bias term used to calculate the primitive state update amount;

[0071] Attention component analysis is used to calculate the contribution weights of each pathological feature primitive to the overall tumor state at the current time. Specifically, it involves performing attention calculations by using the joint state features as queries and the primitive state vectors as keys and values, and incorporating sparsity constraints to obtain the pathological component weight vector. The formula used is as follows:

[0072] ;

[0073] In the formula, This indicates an attention query at time step t. This represents the attention key of the a-th pathological feature primitive at time step t. This represents the attention value of the a-th pathological feature primitive at time step t. Represents the attention query mapping matrix. Represents the attention key mapping matrix. This represents the attention value mapping matrix. This represents the attention weight of the a-th pathological feature primitive at time step t. The dimension representing the attention key. Indicates the sparsity coefficient of pathological components. This represents the L1 norm operation, where Num represents the total number of pathological feature primitives. This represents the attention key of the b-th pathological feature primitive at time step t. This represents the pathological component weight vector at time step t, and the pathological component weight vectors at each time step constitute the pathological component weight feature.

[0074] Biological reconstruction constraints are used to ensure that the component analysis process conforms to the laws of tumor biology. Specifically, it uses learnable standard phenotypic vectors of pathological component weights and pathological feature primitives to reconstruct the input multimodal features, and calculates the reconstruction error as a constraint. The formula used is as follows:

[0075] ;

[0076] In the formula, This represents the t-th row of the reconstructed input features. This represents the standard phenotypic vector of the a-th pathological feature primitive. This represents the reconstruction loss value. The t-th row represents the micro-flow input features. Represents L2 norm operations;

[0077] By performing the above operations, this solution addresses the technical problems of traditional tumor pathological progression prediction systems, which can only provide a rough assessment of a single modality when faced with multimodal heterogeneous data. These systems cannot effectively integrate and analyze the synergistic features contained in multi-source information such as imaging, molecular, and pathology, making it difficult to accurately quantify the contribution weight of different modalities to progression and reveal the multi-factor driving mechanism behind progression. This solution creatively adopts a pathological feature primitive interpretation method. By maintaining a learnable pathological feature primitive library, it analyzes the dynamic changes in tumor phenotypic components and their proportions over time in multimodal data. This can reveal potential or complex progression driving factors from high-dimensional heterogeneous signals. At the same time, biological constraints are introduced to ensure that the proportions of the analyzed components conform to the actual tumor evolution law.

[0078] The counterfactual baseline generation, used to infer the patient's normal state in the absence of tumor progression, to quantify the net driving effect of tumor, includes:

[0079] Counterfactual baseline prediction is used to generate a sequence of tumor states based solely on individual baseline reference data. Specifically, it encodes baseline features through a recurrent neural network and predicts future states autoregressively to obtain counterfactual baseline state features that represent the absence of tumor progression interference.

[0080] Causal effect calculation, used to calculate the net impact of tumor on multimodal phenotypes, is specifically performed by subtracting the counterfactual baseline state features element-by-element from the true joint state features and then normalizing the results to obtain the tumor-driven impact features. The formula used is as follows:

[0081] ;

[0082] In the formula, The t-th row represents the tumor-driven influence characteristics. This represents the normalized calculation function. The t-th row represents the counterfactual baseline state characteristics;

[0083] Adversarial training is introduced to force the counterfactual baseline state features to completely ignore tumor information during generation, ensuring the purity of the baseline. Specifically, an adversarial discriminator is introduced to distinguish whether the generated counterfactual baseline state features are contaminated by tumor information, and adversarial training is performed by calculating an adversarial loss. The adversarial discriminator is specifically a multilayer perceptron, and the adversarial loss is calculated using the following formula:

[0084] ;

[0085] In the formula, Indicates the adversarial loss value. This label indicates baseline purity; its value is 1 when the counterfactual baseline state characteristic is pure, and 0 otherwise. This represents the prediction result of the adversarial discriminator;

[0086] By performing the above operations, this solution addresses the technical problems of traditional tumor pathological progression prediction systems, which fail to construct an analytical framework that isolates individual baseline differences and identifies the effects of tumor evolution itself. It cannot distinguish whether a change in imaging is caused by actual tumor progression or by differences in treatment response, inflammation, or scanning conditions, and the prediction results are always decoupled from the intrinsic biological behavior of the tumor. This solution creatively employs a counterfactual baseline generation method. By modeling and reasoning about the normal ideal control state without tumor progression interference, it separates the pure driving effect of the tumor on multimodal phenotypes. This method can isolate individual background noise from a causal perspective. Furthermore, adversarial learning ensures that the baseline state is not contaminated by tumor information, improving the scientific validity and credibility of progression attribution.

[0087] The fusion prediction output is used to fuse pathological components with tumor-driving effects to predict tumor pathological progression, and includes:

[0088] Dimension alignment and feature enhancement are used to map different features to the same semantic space and extract higher-order information. Specifically, through two independent linear transformations and nonlinear activation functions, the pathological component weight features and tumor-driven influence features are mapped to the same semantic space to obtain pathological component enhancement features and tumor-driven enhancement features.

[0089] The outer product fusion is used to establish the correlation between pathological components and tumor driving sources in two dimensions. Specifically, it involves performing an outer product operation on the enhanced features of pathological components and the enhanced features of tumor driving sources, and then normalizing the result to obtain a two-dimensional matrix sequence. The elements of the two-dimensional matrix at each time step represent the proportion of a specific pathological phenotype component from a specific tumor driving factor at that time step, thus obtaining the model prediction component-source matrix.

[0090] The model is constructed and trained by integrating the state space encoding, pathological feature primitive interpretation, counterfactual baseline generation, and fusion prediction output to build the improved deep state space model. The model is then trained and its performance is verified based on the prediction training set and prediction test set to obtain the improved deep state space model, which serves as the progression prediction model.

[0091] Example 5, see Figure 1 This embodiment is based on the above embodiment. In the tumor pathology progression prediction module, the dataset to be processed is used as the input of the progression prediction model to predict the risk of tumor pathology progression. The model prediction component-source matrix output by the model is used as the final progression prediction result. The tumor pathology progression risk level is determined based on the final progression prediction result.

[0092] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0093] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention.

[0094] The present invention and its embodiments have been described above. This description is not restrictive, and the accompanying drawings are only one embodiment of the present invention; the actual structure is not limited thereto. In conclusion, if those skilled in the art are inspired by this description and design similar structures and embodiments without departing from the spirit of the invention, such designs should fall within the protection scope of the present invention.

Claims

1. A tumor pathological progression prediction system based on multimodal perception, characterized in that: The system includes a multi-source data acquisition module, a preliminary optimization processing module, a progression prediction model construction module, and a tumor pathology progression prediction module; The multi-source data acquisition module obtains the original dataset for progress prediction through data collection. The original dataset for progress prediction specifically includes the historical original dataset for progress prediction and the current original dataset for progress prediction. The preliminary optimization processing module employs multi-source spatiotemporal alignment, anomaly detection and repair, feature engineering and enhancement, and dataset construction as preliminary optimization processing methods to obtain the dataset to be processed, the prediction training set, and the prediction test set. The progression prediction model building module is used to build a model that automatically learns the progression-driving components and source causal relationships from the data. Specifically, it builds an improved deep state space model as the progression prediction model. The improved deep state space model maintains a learnable pathological feature primitive library and combines it with a counterfactual baseline generation design to output a model prediction component-source matrix. The tumor pathology progression prediction module specifically uses the dataset to be processed as input to the progression prediction model to predict the risk of tumor pathology progression. The model outputs the model prediction component-source matrix as the final progression prediction result, and determines the tumor pathology progression risk level based on the final progression prediction result.

2. The tumor pathological progression prediction system based on multimodal perception according to claim 1, characterized in that: The progression prediction model construction module specifically includes state space encoding, pathological feature primitive interpretation, counterfactual baseline generation, fusion prediction output, and model construction and training.

3. The tumor pathological progression prediction system based on multimodal perception according to claim 2, characterized in that: The pathological feature primitive interpretation is used to analyze the dynamically changing tumor phenotypic components and their proportions over time in multimodal data by maintaining a learnable library of pathological feature primitives. The content includes: The initialization and updating of the pathological feature primitive library is used to establish and maintain an internal knowledge base representing different tumor evolution patterns. Specifically, it defines multiple learnable pathological feature primitives, each containing a primitive state vector and a primitive evolution matrix. The initialization of the primitive state vector follows a normal distribution. The primitive evolution matrix is ​​a learnable diagonal matrix whose elements take values ​​in the range of (0,1); Attention component parsing is used to calculate the contribution weight of each pathological feature primitive to the overall tumor state at the current time. Specifically, attention is calculated by using the joint state feature as the query and the primitive state vector as the key and value, and sparsity constraints are added to obtain the pathological component weight vector. Biological reconstruction constraints are used to ensure that the component analysis process conforms to the laws of tumor biology. Specifically, the input multimodal features are reconstructed using learnable standard phenotypic vectors of pathological component weights and pathological feature primitives, and the reconstruction error is calculated as a constraint. The counterfactual baseline generation, used to infer the patient's normal state in the absence of tumor progression, quantifies the net driving effect of tumor, and includes: Counterfactual baseline prediction is used to generate a sequence of tumor states based solely on individual baseline reference data. Specifically, it encodes baseline features through a recurrent neural network and predicts future states autoregressively to obtain counterfactual baseline state features that represent the absence of tumor progression interference. Causal effect calculation is used to calculate the net impact of tumor on multimodal phenotypes. Specifically, it is obtained by subtracting the counterfactual baseline state features from the true joint state features element by element and then normalizing them. Adversarial training is introduced to force the counterfactual baseline state features to completely ignore tumor information during generation, ensuring the purity of the baseline. Specifically, an adversarial discriminator is introduced to distinguish whether the generated counterfactual baseline state features are contaminated by tumor information, and adversarial loss is calculated for adversarial training.

4. The tumor pathological progression prediction system based on multimodal perception according to claim 2, characterized in that: The state-space encoding is used to extract a unified joint state representation from multi-source data using a state-space model, and includes: Input data organization and embedding are used to normalize data of different physical meanings and scales into tensor forms that the model can process. Specifically, by defining the specific data dimensions and time lengths of micro-flow input and macro-flow input, micro-flow input features and macro-flow input features are obtained. The micro-flow input features are obtained from high-resolution image data, digital image data of pathological sections, genomic mutation spectrum data, and protein biomarker data. The time length is determined by sampling at equal intervals within the treatment cycle, and the feature dimension is the sum of the number of image voxels and the number of biomarker types. The macro-flow input features are obtained from clinical test data. The time length is aligned with the micro-flow input features, and the feature dimension is the number of clinical indicator types. Selective state space layer forward propagation is used to encode information with long-range dependency modeling capability for each data stream. Specifically, it dynamically generates state transition parameters that depend on the input through a selective scanning mechanism and performs recursive calculations to obtain micro-flow state features and macro-flow state features of the same dimension. Multi-scale dynamic fusion is used to adaptively integrate the features of micro-flow and macro-flow. Specifically, it uses a learnable weight calculation module to dynamically determine the contribution ratio of micro-flow state features and macro-flow state features at each time step and perform weighted fusion to obtain joint state features. The fusion prediction output is used to fuse pathological components with tumor-driving effects to predict tumor pathological progression, and includes: Dimension alignment and feature enhancement are used to map different features to the same semantic space and extract higher-order information. Specifically, through two independent linear transformations and nonlinear activation functions, the pathological component weight features and tumor-driven influence features are mapped to the same semantic space to obtain pathological component enhancement features and tumor-driven enhancement features. The outer product fusion is used to establish the correlation between pathological components and tumor driving sources in two dimensions. Specifically, it involves performing an outer product operation on the enhanced features of pathological components and the enhanced features of tumor driving sources, and then normalizing the result to obtain a two-dimensional matrix sequence. The elements of the two-dimensional matrix at each time step represent the proportion of a specific pathological phenotype component from a specific tumor driving factor at that time step, thus obtaining the model prediction component-source matrix. The model is constructed and trained by integrating the state space encoding, pathological feature primitive interpretation, counterfactual baseline generation, and fusion prediction output to build the improved deep state space model. The model is then trained and its performance is verified based on the prediction training set and prediction test set to obtain the improved deep state space model, which serves as the progression prediction model.

5. The tumor pathological progression prediction system based on multimodal perception according to claim 1, characterized in that: Both the historical progression prediction dataset and the current progression prediction dataset contain tumor multimodal monitoring data and individual baseline reference data. The historical progression prediction dataset also includes tumor pathological progression risk level labels.

6. The tumor pathological progression prediction system based on multimodal perception according to claim 1, characterized in that: In the preliminary optimization processing module, the multi-source spatiotemporal alignment is used to unify the time and space reference of all data to eliminate misalignment. Specifically, it aligns the timestamps of different devices to millisecond-level precision and registers data from different spatial locations to the same anatomical coordinate system to obtain strictly aligned data. The anomaly detection and repair is used to identify and correct noise and missing data. Specifically, it detects anomalies using an adaptive threshold algorithm and repairs them using temporal prediction and spatial interpolation methods to obtain anomaly-repaired data. The feature engineering and enhancement are used to extract features from the original signal, specifically by performing three-dimensional wavelet transform on image data to extract texture features, constructing statistical and hysteresis features on time-series data, and selecting features based on mutual information methods. The dataset construction is used to organize the processed data into a sample set. Specifically, it involves extracting data from consecutive time periods using a sliding window to form samples, and using oversampling techniques to balance the categories. Finally, the dataset is divided into a training data set and a test data set. The original dataset for current progress prediction is initially optimized through multi-source spatiotemporal alignment, anomaly detection and repair, and feature engineering and enhancement to obtain a dataset to be processed. The original dataset for historical progress prediction is initially optimized through multi-source spatiotemporal alignment, anomaly detection and repair, feature engineering and enhancement, and dataset construction to obtain a prediction training set and a prediction test set.

Images