A method and system for predicting the timing of multimodal neoadjuvant efficacy in esophageal cancer

Abstract

This invention discloses a time-series multimodal prediction method and system for neoadjuvant therapy efficacy in esophageal cancer, comprising: acquiring imaging and molecular data at multiple time points before and during treatment; calculating imaging change direction features based on radiomics characteristics at adjacent time points, and calculating molecular change direction features based on circulating tumor DNA methylation levels at adjacent time points; pairing and combining the two directional features to define cross-modal contradictory states, including a first contradictory state where the imaging change direction is increasing and the molecular change direction is decreasing; modeling the transition path of the cross-modal contradictory state with the treatment progress based on a time-series state transition model, and learning the probability of transitioning from the first contradictory state to a consistent remission state; generating a false progression confidence score as a dynamic prediction result based on the state transition probability output by the model and outputting it. This invention can accurately identify false progression in neoadjuvant therapy for esophageal cancer and improve the accuracy of efficacy prediction.

Description

Technical Field

[0001] This invention relates to the field of medical information technology, specifically to a time-series multimodal prediction method and system for neoadjuvant therapy efficacy in esophageal cancer. Background Technology

[0002] Esophageal cancer is a highly prevalent malignant tumor worldwide, and neoadjuvant therapy (especially immunotherapy combined with chemotherapy) has become the standard treatment for patients with locally advanced disease. However, during immunotherapy, approximately 10-15% of patients experience a unique phenomenon called "pseudoprogression," where early imaging assessments show tumor enlargement, but continued treatment leads to significant tumor shrinkage or even pathological complete remission. How to non-invasively and dynamically differentiate between pseudoprogression and true progression remains a core challenge in current clinical decision-making. While existing multimodal fusion methods attempt to combine imaging and molecular data for efficacy prediction, these methods typically assume that the data changes in different modalities are consistent or complementary, performing end-to-end classification and prediction through feature splicing or attention weighting. These methods cannot effectively handle contradictory signals such as "increased imaging lesions" and "decreased molecular marker levels," leading to insufficient ability to identify pseudoprogression, misjudgment, delays in optimal treatment, or unnecessary treatment interruptions. Summary of the Invention

[0003] In view of the above problems, the present invention provides a time-series multimodal prediction method and system for neoadjuvant therapy in esophageal cancer. By identifying contradictory signals where the changes in imaging and molecular markers are contrary to each other and modeling their dynamic evolution, the method can accurately identify pseudoprogression.

[0004] To achieve the above objectives, in a first aspect, this application provides a time-series multimodal prediction method for neoadjuvant therapy efficacy in esophageal cancer, comprising:

[0005] Acquire imaging and molecular data from patients at multiple time points before and during treatment. The imaging data includes radiomics features of esophageal lesions at at least two time points, and the molecular data includes circulating tumor DNA methylation levels at at least three time points.

[0006] Image change direction features are calculated based on radiomics features at adjacent time points in image acquisition data, and molecular change direction features are calculated based on circulating tumor DNA methylation levels at adjacent time points in molecular acquisition data.

[0007] By pairing and combining image change direction features with molecular change direction features, a cross-modal contradictory state is defined. The cross-modal contradictory state includes the first contradictory state where the image change direction is increasing and the molecular change direction is decreasing.

[0008] The temporal state transition model is used to model the transition path of cross-modal contradictory states as the treatment progresses. The temporal state transition model learns the probability of transitioning from the first contradictory state to the consistent remission state.

[0009] Based on the state transition probabilities output by the temporal state transition model, the false progression confidence score of the current treatment stage is generated and output as a dynamic prediction result of neoadjuvant efficacy.

[0010] In some embodiments, calculating the image change direction features based on the radiomics features of adjacent time points in the image acquisition data includes:

[0011] Extract the first radiomics feature vector at the first time point and the second radiomics feature vector at the second time point from the image acquisition data. Both the first radiomics feature vector and the second radiomics feature vector contain tumor volume features and tumor gray-level co-occurrence matrix contrast features.

[0012] Calculate the difference between the feature values ​​of each dimension in the second radiomics feature vector and the corresponding feature values ​​in the first radiomics feature vector. Divide the difference by the sum of the corresponding feature values ​​in the first radiomics feature vector and the preset minimum constant to generate the image feature change rate of each dimension. The image feature change rate vector is composed of the image feature change rate of each dimension.

[0013] The image feature change rates of each dimension in the image feature change rate vector are sequentially input into the sign discriminant function. The sign discriminant function maps the image feature change rates of each dimension into discrete directional labels, which include increase labels, decrease labels and stable labels.

[0014] The frequency of increasing labels in discrete direction labels is counted, and the ratio of the frequency of increasing labels to the total number of discrete direction labels is calculated. If the ratio exceeds a preset majority threshold, the image change direction feature is determined to be increasing direction.

[0015] In some embodiments, the molecular change direction characteristics are calculated based on the circulating tumor DNA methylation levels at adjacent time points in the molecular acquisition data, including:

[0016] The first methylation level value at the third time point and the second methylation level value at the fourth time point were extracted from the molecular acquisition data. The methylation level value is the quantitative detection result of a preset single methylation site.

[0017] Calculate the difference between the second methylation level value and the first methylation level value, divide the difference by the sum of the first methylation level value and the preset minimum constant, and generate a scalar of the rate of change of molecular characteristics.

[0018] The molecular characteristic rate of change scalar is input into the sign discrimination function, which maps the continuous value of the molecular characteristic rate of change scalar to discrete directional labels. The discrete directional labels include descending labels, ascending labels and stable labels.

[0019] If the discrete direction label is a descending label, then the molecular change direction characteristic is determined to be a descending direction.

[0020] In some embodiments, image change direction features and molecular change direction features are paired and combined to define cross-modal contradictory states, including:

[0021] The image change direction features and molecular change direction features calculated within the same treatment cycle are combined to form a direction feature pair, which includes an image direction label and a molecular direction label.

[0022] Establish a state mapping rule. The state mapping rule maps the combination of directional feature alignment image directional label as increasing label and molecular directional label as decreasing label as the first contradictory state. The combination of directional feature alignment image directional label as decreasing label and molecular directional label as decreasing label as the consistent relief state.

[0023] According to the state mapping rule, the directional features of each treatment cycle are mapped one by one to the corresponding cross-modal contradictory state labels. The cross-modal contradictory state labels include at least the first contradictory state, the consistent remission state, and the consistent progression state.

[0024] In some embodiments, establishing state mapping rules includes:

[0025] Define all possible combinations of orientation labels and molecular orientation labels in the image centering feature. All possible combinations include nine combinations, which are formed by pairing up the labels, shrinking labels and stable labels.

[0026] From the nine combinations, the combinations in which the image orientation label and the molecular orientation label change in opposite directions are identified. The combinations in which the change in opposite directions change are classified into a candidate set of contradictory states. The candidate set of contradictory states includes at least the first combination in which the image orientation label is an increasing label and the molecular orientation label is a decreasing label, and the second combination in which the image orientation label is a decreasing label and the molecular orientation label is an increasing label.

[0027] The first combination is selected from the candidate set of contradictory states and mapped to the first contradictory state. The first contradictory state corresponds to the clinical scenario of pseudoprogression in neoadjuvant therapy of esophageal cancer, where the imaging manifestation is progression but the molecular manifestation is remission.

[0028] The second combination in the candidate set of contradictory states is mapped to the second contradictory state, which corresponds to the clinical scenario of minimal residual disease in neoadjuvant therapy for esophageal cancer, where imaging shows remission but molecular manifestations show progression.

[0029] In some embodiments, a temporal state transition model is used to model the transition path of cross-modal contradictory states as the treatment progresses. The temporal state transition model learns the probability of transitioning from a first contradictory state to a consistent remission state, including:

[0030] Arrange the cross-modal contradictory state labels corresponding to each treatment cycle in chronological order to generate a state sequence. The state sequence contains cross-modal contradictory state labels for at least two consecutive treatment cycles.

[0031] A hidden Markov model is constructed as a temporal state transition model. The hidden Markov model includes a set of hidden states and a state transition probability matrix. The set of hidden states includes at least the first contradictory state, the consistent relief state, and the consistent progress state.

[0032] The state sequence is input into the Hidden Markov Model (HMM), which uses the expectation-maximization algorithm to estimate the parameters of the state transition probability matrix and calculates the first transition probability from the first contradictory state to the consistent relief state and the second transition probability from the first contradictory state to the consistent progress state.

[0033] The first and second transition probabilities are output as a quantitative representation of the transition path of the cross-modal contradictory state as the treatment progresses.

[0034] In some embodiments, constructing a hidden Markov model as a temporal state transition model includes:

[0035] Define the set of hidden states of the Hidden Markov Model. The set of hidden states includes the first contradictory state, the second contradictory state, the uniformly relieved state, the uniformly progressive state, and the image-stabilized molecular stable state. The second contradictory state corresponds to the combination where the image change direction is shrinking and the molecular change direction is increasing. The image-stabilized molecular stable state corresponds to the combination where the image change direction is stable and the molecular change direction is stable.

[0036] Define the set of observation symbols for the Hidden Markov Model, which contains the labels of cross-modal contradictory states actually observed in each treatment cycle;

[0037] Initialize the state transition probability matrix of the hidden Markov model. Each row of the state transition probability matrix corresponds to a current hidden state, and each column corresponds to a next hidden state. The initial transition probability from the first contradictory state to the consistent relief state in the state transition probability matrix is ​​set to be higher than the initial transition probability from the first contradictory state to the consistent progression state, so as to reflect the prior knowledge of the pseudo-progression clinical scenario.

[0038] Initialize the observation probability matrix of the Hidden Markov Model. Each row of the observation probability matrix corresponds to a hidden state, and each column corresponds to an observation sign. The initial values ​​of the diagonal elements in the observation probability matrix are set higher than the initial values ​​of the off-diagonal elements to reflect the prior assumption that the hidden states and observation signs are consistent.

[0039] In some embodiments, the state sequence is input into a Hidden Markov Model (HMM), and the HMM estimates the parameters of the state transition probability matrix using an expectation-maximization algorithm, including:

[0040] The state sequence is input into the hidden Markov model as the observation sequence. The state sequence contains continuous cross-modal contradictory state labels from the first treatment cycle to the Nth treatment cycle.

[0041] The expectation step of the expectation maximization algorithm is to calculate the posterior probability distribution of the hidden state sequence corresponding to the current parameters of the hidden Markov model. The posterior probability distribution includes the probability of each treatment cycle being in each hidden state and the probability of state transition between adjacent treatment cycles.

[0042] The maximization step of the expectation maximization algorithm is executed. Based on the posterior probability distribution, the values ​​of each element in the state transition probability matrix are re-estimated. The re-estimated frequency includes the statistical calculation of the expected frequency of transition from the first contradictory state to the consistent relief state. The expected frequency is divided by the sum of the expected frequencies of transition from the first contradictory state to all hidden states to generate the updated first transition probability.

[0043] Iteratively execute the expected step and the maximization step until the update magnitude of the state transition probability matrix is ​​lower than the preset convergence threshold, and output the converged state transition probability matrix.

[0044] In some embodiments, a false progression confidence score for the current treatment stage is generated based on the state transition probabilities output by the temporal state transition model, including:

[0045] Obtain the first transition probability and the second transition probability output by the temporal state transition model. The first transition probability is the probability of transitioning from the first contradictory state to the consistent relief state, and the second transition probability is the probability of transitioning from the first contradictory state to the consistent progress state.

[0046] The ratio of the first metastasis probability to the sum of the first and second metastasis probabilities is calculated to generate a pseudoprogression confidence scalar. The pseudoprogression confidence scalar represents the probability that a patient currently in the first contradictory state will subsequently enter a consistent remission state.

[0047] If the confidence scalar of pseudoprogression exceeds the preset confidence threshold, the current treatment stage is determined to be a pseudoprogression state, and a pseudoprogression warning signal is output.

[0048] If the confidence scalar of pseudoprogression does not exceed the preset confidence threshold, the current treatment stage is determined to be a true progression state, and a true progression warning signal is output.

[0049] In a second aspect, the present invention also provides a time-series multimodal prediction system for neoadjuvant therapy efficacy in esophageal cancer, applicable to the method described in the first aspect. The system includes a data acquisition module, a directional feature calculation module, a contradictory state definition module, a time-series modeling module, and a confidence generation module. The data acquisition module acquires image and molecular data from multiple time points before and during treatment. The image acquisition data includes radiomics features of esophageal lesions at at least two time points, and the molecular acquisition data includes circulating tumor DNA methylation levels at at least three time points. The directional feature calculation module is connected to the data acquisition module and is used to calculate the directional features of image changes based on the radiomics features of adjacent time points in the image acquisition data, and to calculate the directional features of circulating tumor DNA methylation levels based on the radiomics features of adjacent time points in the molecular acquisition data. The system calculates the molecular change direction characteristics at the chemical level. A contradictory state definition module, connected to the direction characteristic calculation module, is used to pair and combine image change direction characteristics with molecular change direction characteristics to define cross-modal contradictory states. Cross-modal contradictory states include the first contradictory state where the image change direction is increasing and the molecular change direction is decreasing. A temporal modeling module, connected to the contradictory state definition module, is used to model the transition path of cross-modal contradictory states as treatment progresses based on a temporal state transition model. The temporal state transition model learns the probability of transitioning from the first contradictory state to a consistent remission state. A confidence generation module, connected to the temporal modeling module, generates a false progression confidence score for the current treatment stage based on the state transition probability output by the temporal state transition model. This confidence score serves as a dynamic prediction result of neoadjuvant therapy efficacy and is then output.

[0050] Unlike existing technologies, the above-mentioned technical solution provides a time-series multimodal prediction method and system for the neoadjuvant therapy efficacy of esophageal cancer, including: acquiring imaging and molecular data at multiple time points before and during treatment; calculating the directional features of image changes based on radiomics characteristics at adjacent time points, and calculating the directional features of molecular changes based on the circulating tumor DNA methylation levels at adjacent time points; pairing and combining the two directional features to define cross-modal contradictory states, including a first contradictory state where the image change direction is increasing and the molecular change direction is decreasing; modeling the transition path of the cross-modal contradictory state with the treatment process based on a time-series state transition model, and learning the probability of transitioning from the first contradictory state to a consistent remission state; generating a false progression confidence score as a dynamic prediction result based on the state transition probability output by the model and outputting it. This invention can accurately identify false progression in neoadjuvant therapy for esophageal cancer and improve the accuracy of efficacy prediction.

[0051] The above description of the invention is merely an overview of the technical solution of this application. In order to enable those skilled in the art to better understand the technical solution of this application and to implement it based on the description and drawings, and to make the above-mentioned objectives and other objectives, features and advantages of this application easier to understand, the following description is provided in conjunction with the specific embodiments and drawings of this application. Attached Figure Description

[0052] The accompanying drawings are only used to illustrate the principles, implementation methods, applications, features, and effects of specific embodiments of the present invention and other related contents, and should not be considered as limitations on this application.

[0053] In the accompanying drawings of the instruction manual:

[0054] Figure 1 This is a schematic diagram illustrating steps S101 to S105 of the method described in a specific embodiment.

[0055] Figure 2 This is a schematic diagram illustrating steps S201 to S203 of the method described in a specific implementation.

[0056] Figure 3 This is a schematic diagram of the structure of the time-series multimodal prediction system described in a specific implementation.

[0057] The reference numerals used in the above figures are explained as follows:

[0058] 1. Temporal multimodal prediction system; 11. Data acquisition module; 12. Directional feature calculation module; 13. Contradictory state definition module; 14. Temporal modeling module; 15. Confidence generation module. Detailed Implementation

[0059] To illustrate the possible application scenarios, technical principles, implementable specific solutions, and achievable objectives and effects of this application in detail, the following description, in conjunction with the listed specific embodiments and accompanying drawings, provides a detailed explanation. The embodiments described herein are merely illustrative of the technical solutions of this application and are therefore intended to limit the scope of protection of this application.

[0060] In this document, the term "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The term "embodiment" appearing in various places throughout the specification does not necessarily refer to the same embodiment, nor does it specifically limit its independence or connection with other embodiments. In principle, in this application, as long as there are no technical contradictions or conflicts, the technical features mentioned in each embodiment can be combined in any way to form corresponding implementable technical solutions.

[0061] Unless otherwise defined, the technical terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the use of related terms herein is merely for the purpose of describing particular embodiments and is not intended to limit this application.

[0062] In the description of this application, the term "and / or" is used to describe the logical relationship between objects, indicating that three relationships can exist. For example, A and / or B means: A exists, B exists, and A and B exist simultaneously. Additionally, the character " / " in this document generally indicates that the preceding and following objects have an "or" logical relationship.

[0063] In this application, terms such as “first” and “second” are used only to distinguish one entity or operation from another, and do not necessarily require or imply any actual quantity, hierarchy or order relationship between these entities or operations.

[0064] Without further limitations, the use of terms such as “comprising,” “including,” “having,” or other similar open-ended expressions in this application is intended to cover non-exclusive inclusion, which does not exclude the presence of additional elements in a process, method, or product that includes the stated elements, such that a process, method, or product that includes a list of elements may include not only those defined elements but also other elements not expressly listed, or elements inherent to such a process, method, or product.

[0065] As understood in the Examination Guidelines, in this application, expressions such as "greater than," "less than," and "exceeding" are understood to exclude the stated number; expressions such as "above," "below," and "within" are understood to include the stated number. Furthermore, in the description of the embodiments in this application, "multiple" means two or more (including two), and similar expressions related to "multiple" are also understood in this way, such as "multiple groups" and "multiple times," unless otherwise explicitly specified.

[0066] In the description of the embodiments of this application, the space-related expressions used, such as "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "vertical," "top," "bottom," "inner," "outer," "clockwise," "counterclockwise," "axial," "radial," and "circumferential," indicate the orientation or positional relationship based on the orientation or positional relationship shown in the specific embodiments or drawings. They are only for the purpose of describing the specific embodiments of this application or for the reader's understanding, and do not indicate or imply that the device or component referred to must have a specific position, a specific orientation, or be constructed or operated in a specific orientation. Therefore, they should not be construed as limitations on the embodiments of this application.

[0067] The processor described in the embodiments of this application can be implemented by hardware, firmware, software, or a combination thereof. It can be a circuit, one or more of an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field-programmable gate array (FPGA), a central processing unit (CPU), a controller, a microcontroller, or a microprocessor. It also includes other physical, biological, or chemical structures that can implement the same or equivalent functions as the processors listed above, such as biological neurons, quantum computing units, DNA computing units, etc., so that the processor can execute some or all of the steps in the computer program or method involved in the various embodiments of this application, or any combination of the steps mentioned therein.

[0068] The computer program involved in the embodiments can be stored in a computer device readable storage medium, which includes, but is not limited to, disks, magnetic tapes, magnetic cards, floppy disks, flash memory, optical disks, optical cards, read-only memory (ROM), random access memory (RAM), erasable programmable ROM (EPROM), and electrically erasable programmable ROM (EEPROM), etc., and also includes other biological, physical, or chemical structures that can achieve the same or equivalent functions as the storage media listed above, such as DNA, RNA, proteins, and other units with information storage capabilities. In specific embodiments, the storage medium involved can be one of the above-mentioned media types, or a combination of the above-mentioned media types. In different embodiments, the computer program involved in the embodiments can be centrally stored in a single medium, or distributed and stored in multiple media. The memory containing the computer device readable storage medium can be non-volatile memory or random access memory. These computer device readable storage media can be built into the device, or can be connected to the device involved in the embodiments as an external device or part of an external device. In some embodiments, the memory having a computer device readable storage medium is deployed locally; in other embodiments, the memory may be deployed remotely from the processor, for example, as a network-attached memory accessed via RF circuitry or an external port and a communication network, wherein the communication network may be the Internet, one or more intranets, a local area network (LAN), a wide area network (WLAN), a storage area network (SAN), or a suitable combination thereof, as long as computer device access to the memory is enabled. Furthermore, the computer program involved in the embodiments may be stored in plaintext / ciphertext form, or it may be designed as training data, integrated and recombined through model training and implicitly stored in the parameter states of a deep neural network or other machine learning model.

[0069] Please see Figure 1 In a first aspect, this embodiment provides a time-series multimodal prediction method for neoadjuvant therapy efficacy in esophageal cancer, comprising:

[0070] S101. Acquire imaging and molecular data of the patient at multiple time points before and during treatment. The imaging data includes radiomics features of esophageal lesions at at least two time points, and the molecular data includes circulating tumor DNA methylation levels at at least three time points.

[0071] S102. Calculate the image change direction characteristics based on the radiomics characteristics of adjacent time points in the image acquisition data, and calculate the molecular change direction characteristics based on the circulating tumor DNA methylation level of adjacent time points in the molecular acquisition data.

[0072] S103. Pair and combine the image change direction features and the molecular change direction features to define a cross-modal contradictory state. The cross-modal contradictory state includes the first contradictory state where the image change direction is increasing and the molecular change direction is decreasing.

[0073] S104. Based on the temporal state transition model, the transition path of cross-modal contradictory states with the treatment process is modeled, and the temporal state transition model learns the probability of transitioning from the first contradictory state to the consistent remission state.

[0074] S105. Based on the state transition probability output by the time-series state transition model, generate the false progression confidence score of the current treatment stage, and output it as a dynamic prediction result of neoadjuvant efficacy.

[0075] In step S101, image acquisition data is obtained through multimodal imaging examination of esophageal lesions, including but not limited to high-resolution radiomics features acquired by CT scans, MRI, and multiphoton microscopy. These features quantitatively describe the imaging manifestations of the lesions from dimensions such as tumor morphology, texture, and boundaries. Molecular acquisition data is obtained by extracting circulating tumor DNA from the patient's peripheral blood using liquid biopsy technology and quantitatively detecting the methylation level at specific sites. The image data and molecular data acquired before treatment constitute a baseline reference. During treatment, data is repeatedly acquired at a preset cycle (e.g., every two treatment cycles) to form image data sequences at least two time points and molecular data sequences at least three time points. The resulting time-series data sequences provide multi-time-section observation samples for subsequent directional analysis.

[0076] In step S102, the generation of image change direction features involves comparing the radiomics feature vectors of adjacent time points dimension by dimension. The overall evolution direction of the image features is determined based on the changing trends of the feature values ​​in each dimension. The output discrete direction labels are used to characterize the increase, decrease, or stabilization of tumor imaging manifestations under treatment intervention. The generation of molecular change direction features involves comparing the quantitative values ​​of circulating tumor DNA methylation levels of adjacent time points. The molecular-level response direction is determined based on the rising and falling trends of methylation levels. The output discrete direction labels are used to characterize the increase, decrease, or stabilization of tumor molecular activity under treatment intervention. The introduction of directional features unifies the original data of different dimensions and absolute levels into comparable change direction information, eliminating the interference of baseline differences between patients on the analysis results.

[0077] In step S103, the cross-modal contradictory state refers to a combination of states where the direction of image change and the direction of molecular change are inconsistent, used to identify special scenarios where the directions of change of the two are contradictory. The first contradictory state corresponds in clinical practice to the pseudo-progression phenomenon unique to neoadjuvant immunotherapy for esophageal cancer, i.e., the tumor volume increases on imaging, but the molecular level shows a decrease in tumor activity. This step breaks through the implicit assumption of existing multimodal fusion methods that assume the direction of change of each modality is consistent, explicitly modeling the inconsistency of cross-modal signals as a computable feature, providing a key criterion for subsequent identification of pseudo-progression.

[0078] In step S104, the temporal state transition model is used to learn the dynamic evolution of cross-modal contradictory states as treatment progresses. A consistent remission state refers to an ideal treatment response state where both the direction of image changes and the direction of molecular changes show remission. The temporal state transition model analyzes the patient's state sequence across multiple treatment cycles to quantify the probability of transitioning from a contradictory state to a remission state, thereby revealing whether the current image progression is a temporary contradictory phenomenon or a persistent worsening trend. This step elevates the identification of contradictory states from static discrimination to dynamic prediction, achieving a quantifiable description of the evolutionary path of pseudo-progression by modeling state transition probabilities.

[0079] In step S105, the false progression confidence score is calculated by combining the probability values ​​of transition from the first contradictory state to the consistent remission state output by the time-series state transition model with the probability values ​​of transition from the first contradictory state to other states. The generated confidence score scalar ranges from 0 to 1, with a higher value indicating a greater likelihood that the currently observed image progression is false progression. The false progression confidence score serves as a dynamic prediction of neoadjuvant therapy efficacy and can be used by clinicians to make subsequent treatment decisions. For example, a higher confidence score may prompt consideration of continuing the current treatment plan rather than immediate modification.

[0080] This embodiment achieves dynamic identification of pseudoprogression in neoadjuvant therapy for esophageal cancer by constructing a prediction process that obtains pseudoprogression confidence output from multimodal time-series data. Directional features are used to eliminate the influence of patient baseline differences; the definition of cross-modal contradictory states overcomes the limitation of existing multimodal fusion methods in handling signal inconsistencies; and the introduction of a time-series state transition model extends the identification of contradictory states from static discrimination to dynamic evolution prediction. The final output of pseudoprogression confidence provides a quantifiable basis for clinical decision-making. This approach addresses the unique clinical challenges in neoadjuvant immunotherapy for esophageal cancer by effectively identifying pseudoprogression through a contradictory perception mechanism.

[0081] In some embodiments, calculating the image change direction features based on the radiomics features of adjacent time points in the image acquisition data includes:

[0082] Extract the first radiomics feature vector at the first time point and the second radiomics feature vector at the second time point from the image acquisition data. Both the first radiomics feature vector and the second radiomics feature vector contain tumor volume features and tumor gray-level co-occurrence matrix contrast features.

[0083] Calculate the difference between the feature values ​​of each dimension in the second radiomics feature vector and the corresponding feature values ​​in the first radiomics feature vector. Divide the difference by the sum of the corresponding feature values ​​in the first radiomics feature vector and the preset minimum constant to generate the image feature change rate of each dimension. The image feature change rate vector is composed of the image feature change rate of each dimension.

[0084] The image feature change rates of each dimension in the image feature change rate vector are sequentially input into the sign discriminant function. The sign discriminant function maps the image feature change rates of each dimension into discrete directional labels, which include increase labels, decrease labels and stable labels.

[0085] The frequency of increasing labels in discrete direction labels is counted, and the ratio of the frequency of increasing labels to the total number of discrete direction labels is calculated. If the ratio exceeds a preset majority threshold, the image change direction feature is determined to be increasing direction.

[0086] In this embodiment, the first and second radiomics feature vectors are derived from radiomics analysis of high-resolution images of esophageal lesions acquired by multiphoton microscopy. Specifically, the region of interest is delineated using a preset tumor segmentation algorithm, and then calculated by radiomics feature extraction software. Tumor volume features are obtained by voxel accumulation of the segmented region using 3D reconstruction technology. The tumor gray-level co-occurrence matrix contrast feature is calculated based on the contrast statistics of the gray-level co-occurrence matrix of the tumor region, which reflects the intensity of local changes in image texture. The extracted feature vectors include multiple dimensions such as tumor volume features and tumor gray-level co-occurrence matrix contrast features, each corresponding to a specific radiomics quantitative indicator.

[0087] By introducing a preset minimum constant to avoid division-by-zero errors when a feature value in a certain dimension of the first radiomics feature vector is zero, its value is set to be much smaller than the normal range of feature values, usually determined based on the typical magnitude of radiomics feature values. This ensures that adding this constant does not affect the accuracy of the rate of change calculation. The rate of change of image features calculated from each dimension together constitutes the image feature rate of change vector, which retains the relative change magnitude information of each dimension feature between adjacent time points.

[0088] The mapping rule used in the sign discrimination function is based on the comparison between the absolute value of the rate of change and a set threshold. The absolute values ​​of the positive and negative thresholds are determined according to empirical standards for judging significant changes in tumors in clinical practice, such as the threshold values ​​for judging changes in tumor diameter in the evaluation criteria for solid tumor treatment. The output discrete directional labels represent the trend type of changes in each dimension between adjacent time points.

[0089] The preset majority threshold can be determined based on clinical experience or historical data statistical analysis, and is usually set to a value greater than half to ensure that the judgment results have statistically significant majority consistency. When the proportion of the increasing label exceeds the preset majority threshold, it indicates that most radiomics feature dimensions show an increasing trend, and the image change direction feature is judged to be an increasing direction. If the proportion of the increasing label does not exceed the preset majority threshold, the proportions of the decreasing label and the stable label are further compared to determine whether the image change direction feature is a decreasing direction or a stable direction. This majority voting mechanism can integrate the change information of multi-dimensional features and avoid the interference of abnormal fluctuations in a single dimension on the overall direction judgment.

[0090] This embodiment extracts radiomics feature vectors containing tumor volume characteristics and tumor gray-level co-occurrence matrix contrast characteristics from multiphoton microscopy images. It calculates the rate of change of image features at adjacent time points, discretizes these features into directional labels using a sign-discriminant function, and then uses a majority voting mechanism to comprehensively determine the directional features of image changes. This embodiment transforms high-dimensional continuous radiomics features into discrete directional labels with clear clinical semantics, providing standardized image direction input for subsequent identification of cross-modal contradictory states.

[0091] In some embodiments, the molecular change direction characteristics are calculated based on the circulating tumor DNA methylation levels at adjacent time points in the molecular acquisition data, including:

[0092] The first methylation level value at the third time point and the second methylation level value at the fourth time point were extracted from the molecular acquisition data. The methylation level value is the quantitative detection result of a preset single methylation site.

[0093] Calculate the difference between the second methylation level value and the first methylation level value, divide the difference by the sum of the first methylation level value and the preset minimum constant, and generate a scalar of the rate of change of molecular characteristics.

[0094] The molecular characteristic rate of change scalar is input into the sign discrimination function, which maps the continuous value of the molecular characteristic rate of change scalar to discrete directional labels. The discrete directional labels include descending labels, ascending labels and stable labels.

[0095] If the discrete direction label is a descending label, then the molecular change direction characteristic is determined to be a descending direction.

[0096] In this embodiment, the first and second methylation levels are derived from targeted methylation analysis of circulating tumor DNA in peripheral blood samples from patients. Specifically, peripheral blood is collected, plasma is separated, cell-free DNA is extracted, and bisulfite conversion is performed. Then, methylation-specific PCR or targeted methylation sequencing technology is used to quantitatively detect a pre-defined single methylation site. The obtained detection result is the methylation level value of that site. The pre-defined single methylation site is determined based on literature reports of esophageal cancer-related methylation markers or screening of previous clinical samples. Typically, gene sites that exhibit high or low methylation in esophageal cancer tissue and have stable methylation levels in normal tissue are selected to ensure that the site can effectively reflect changes in tumor burden.

[0097] By introducing a preset minimum constant to avoid division-by-zero errors when the first methylation level is zero, its value is set to be much smaller than the normal methylation level range, typically determined based on the detection limit of the methylation detection technology and the typical magnitude of the background methylation level in normal tissue. Since the molecular side is calculated only for a single methylation site, the resulting molecular characteristic change rate scalar is a scalar value rather than a vector form. This scalar value directly characterizes the relative change magnitude of the methylation site between adjacent time points.

[0098] The mapping rule used in the sign discrimination function is based on the comparison between the absolute value of the molecular characteristic change rate scalar and a set threshold. The absolute values ​​of the positive and negative thresholds are determined according to the coefficient of variation of circulating tumor DNA methylation detection technology and empirical standards for determining significant changes in methylation levels in clinical practice. These thresholds are typically set with reference to the recognized significant change thresholds in the field of liquid biopsy. The output discrete directional labels characterize the change trend of the methylation site between adjacent time points.

[0099] When the discrete direction label is a decreasing label, it indicates that the methylation level of the methylation site shows a decreasing trend between adjacent time points, and the molecular change direction feature is determined to be a decreasing direction. If the discrete direction label is an increasing label or a stable label, the molecular change direction feature is determined to be an increasing direction or a stable direction, respectively. Since the molecular side only involves the scalar value of a single methylation site, its direction determination process is simpler than the multi-dimensional majority voting mechanism on the image side.

[0100] This embodiment extracts circulating tumor DNA from the patient's peripheral blood, quantitatively detects a preset single methylation site to obtain methylation level values ​​at adjacent time points, calculates a scalar of molecular feature change rate, discretizes it into directional labels using a sign discriminant function, and finally determines the molecular change direction feature. This embodiment transforms the continuous quantitative detection results of circulating tumor DNA methylation levels into discrete directional labels with clear clinical semantics, providing a standardized molecular direction input for subsequent identification of cross-modal contradictory states.

[0101] Please see Figure 2 In some embodiments, image change direction features and molecular change direction features are paired and combined to define cross-modal contradictory states, including:

[0102] S201. Combine the image change direction features and molecular change direction features calculated within the same treatment cycle into a direction feature pair. The direction feature pair includes an image direction label and a molecular direction label.

[0103] S202. Establish state mapping rules. The state mapping rules map the combination of directional feature alignment image directional label as increasing label and molecular directional label as decreasing label as the first contradictory state, and the combination of directional feature alignment image directional label as decreasing label and molecular directional label as decreasing label as the consistent relief state.

[0104] S203. According to the state mapping rule, the directional feature pairs of each treatment cycle are mapped one by one to the corresponding cross-modal contradictory state labels. The cross-modal contradictory state labels include at least the first contradictory state, the consistent remission state, and the consistent progression state.

[0105] In step S201, the image change direction feature in the directional feature pair is derived from the majority vote result of the radiomics feature vectors at adjacent time points, and the molecular change direction feature is derived from the sign discrimination result of the circulating tumor DNA methylation level scalar at adjacent time points. The definition of the same treatment cycle is based on whether the acquisition time points of the image data and molecular data belong to the same treatment stage, usually aligned using the chemotherapy or immunotherapy administration cycle as the time unit. The data structure of the directional feature pair can be in the form of a binary tuple, with the first element being the image direction label and the second element being the molecular direction label, each label being one of three discrete values: increase, decrease, or stability.

[0106] In step S202, the state mapping rules are established based on the statistical analysis of the correlation between imaging and molecular changes in the clinical cohort of neoadjuvant esophageal cancer patients. The first contradictory state corresponds to a unique pattern in clinical data where early immunotherapy leads to increased imaging volume due to tumor necrosis and release, while circulating tumor DNA methylation levels decrease. This pattern is statistically correlated with subsequent pathological complete remission. The directional feature pairs corresponding to the consistent remission and consistent progression states correspond to conventional treatment response patterns where the tumor exhibits co-directional remission or co-directional progression at both the imaging and molecular levels. These three states constitute a cross-modal contradictory state classification system, providing a state space with clear clinical semantics for subsequent temporal modeling.

[0107] In step S203, the generation of cross-modal contradictory state labels is achieved by aligning and matching the directional feature pairs of each treatment cycle with a state mapping rule table one by one. The rule table stores the mapping relationship between the combination of directional feature pairs and the state labels in key-value pairs. The data format of the cross-modal contradictory state labels is a discrete category label, used to identify the degree of consistency and contradiction type between the image and molecular change directions within the treatment cycle. After the cross-modal contradictory state labels of each treatment cycle are arranged in chronological order, they constitute the state sequence required for subsequent temporal modeling.

[0108] This embodiment pairs image orientation labels with molecular orientation labels within the same treatment cycle to form orientation feature pairs. A mapping rule based on clinical cohort statistical analysis is established, mapping each orientation feature pair from each treatment cycle to a cross-modal contradictory state label with clear clinical semantics. This embodiment integrates the dynamic change information of both image and molecular modalities into a unified state representation, providing standardized input data for subsequent modeling of the evolution of contradictory states throughout the treatment process.

[0109] In some embodiments, establishing state mapping rules includes:

[0110] Define all possible combinations of orientation labels and molecular orientation labels in the image centering feature. All possible combinations include nine combinations, which are formed by pairing up the labels, shrinking labels and stable labels.

[0111] From the nine combinations, the combinations in which the image orientation label and the molecular orientation label change in opposite directions are identified. The combinations in which the change in opposite directions change are classified into a candidate set of contradictory states. The candidate set of contradictory states includes at least the first combination in which the image orientation label is an increasing label and the molecular orientation label is a decreasing label, and the second combination in which the image orientation label is a decreasing label and the molecular orientation label is an increasing label.

[0112] The first combination is selected from the candidate set of contradictory states and mapped to the first contradictory state. The first contradictory state corresponds to the clinical scenario of pseudoprogression in neoadjuvant therapy of esophageal cancer, where the imaging manifestation is progression but the molecular manifestation is remission.

[0113] The second combination in the candidate set of contradictory states is mapped to the second contradictory state, which corresponds to the clinical scenario of minimal residual disease in neoadjuvant therapy for esophageal cancer, where imaging shows remission but molecular manifestations show progression.

[0114] In this embodiment, all possible combinations are generated by a Cartesian product operation of the three discrete values ​​of the image orientation label and the three discrete values ​​of the molecular orientation label. The three values ​​of the image orientation label are increasing, decreasing, and stable, and the three values ​​of the molecular orientation label are increasing, decreasing, and stable. These are paired to form nine combinations. Enumerating all possible combinations ensures that subsequent state mapping rules cover all possible directional feature pairs, avoiding incomplete mapping rules due to omissions of certain combinations. The enumeration results of the nine combinations are stored in list form, with each combination corresponding to a unique index. This step compresses continuous variation information into category labels with clear clinical semantics, enabling subsequent state mapping rules to be executed efficiently using a lookup table, while avoiding subjective bias caused by setting continuous value thresholds.

[0115] The candidate set of contradictory states is identified based on whether the image orientation label and the molecular orientation label change in opposite directions. The first combination (image orientation label is increasing and molecular orientation label is decreasing) and the second combination (image orientation label is decreasing and molecular orientation label is increasing) are the only two combinations with completely opposite directions. This screening logic quantifies the inconsistency of cross-modal signals into enumerable discrete states, providing a clear input range for subsequent targeted modeling of the dynamic evolution of contradictory signals.

[0116] The basis for mapping the first combination to the first contradictory state is that the first combination is statistically correlated with pseudoprogression in the clinical cohort of neoadjuvant immunotherapy for esophageal cancer. That is, the contradictory pattern of imaging showing progression while molecular manifestations show remission is a key indicator of pseudoprogression. The first contradictory state is assigned the highest clinical priority in the state mapping rule to avoid unnecessary treatment interruptions due to misjudgment of pseudoprogression.

[0117] The second contradictory state corresponds to the minimal residual disease (MRD) scenario in neoadjuvant therapy for esophageal cancer, where imaging shows remission but molecular patterns indicate progression. In this case, the tumor volume shrinks on imaging, but the circulating tumor DNA methylation level remains elevated, suggesting that imaging remission may mask residual activity at the molecular level. This state is associated with the risk of recurrence after treatment. The second contradictory state serves as an auxiliary discriminant state in the state mapping rule, enabling the state space to cover two main clinical scenarios where imaging and molecular changes contradict each other.

[0118] This embodiment exhaustively enumerates all nine combinations of image orientation labels and molecular orientation labels, identifies combinations with opposite directions of change to form a candidate set of contradictory states, and then maps the first combination and the second combination to the first contradictory state and the second contradictory state, respectively. This improves the classification system of cross-modal contradictory states and provides a complete state space covering two clinical scenarios, pseudo-progression and minimal residual disease, for subsequent time series modeling.

[0119] In some embodiments, a temporal state transition model is used to model the transition path of cross-modal contradictory states as the treatment progresses. The temporal state transition model learns the probability of transitioning from a first contradictory state to a consistent remission state, including:

[0120] Arrange the cross-modal contradictory state labels corresponding to each treatment cycle in chronological order to generate a state sequence. The state sequence contains cross-modal contradictory state labels for at least two consecutive treatment cycles.

[0121] A hidden Markov model is constructed as a temporal state transition model. The hidden Markov model includes a set of hidden states and a state transition probability matrix. The set of hidden states includes at least the first contradictory state, the consistent relief state, and the consistent progress state.

[0122] The state sequence is input into the Hidden Markov Model (HMM), which uses the expectation-maximization algorithm to estimate the parameters of the state transition probability matrix and calculates the first transition probability from the first contradictory state to the consistent relief state and the second transition probability from the first contradictory state to the consistent progress state.

[0123] The first and second transition probabilities are output as a quantitative representation of the transition path of the cross-modal contradictory state as the treatment progresses.

[0124] In this embodiment, the state sequence is generated using treatment cycles as the time unit, and the cross-modal contradictory state labels corresponding to each treatment cycle are arranged in ascending order by cycle number. The cross-modal contradictory state labels are stored in integer encoding form, with each state corresponding to a unique integer value, facilitating subsequent model processing. The length of the state sequence depends on the number of treatment cycles available to the patient. For patients who have completed only two treatment cycles, the state sequence length is two, exactly containing one state transition event; for patients who have completed more treatment cycles, the state sequence length increases accordingly, providing richer transition samples for parameter estimation. If the acquisition time points of image data and molecular data are not perfectly aligned, window alignment is performed based on the start or end time of the treatment cycle to ensure that image orientation labels and molecular orientation labels within the same treatment cycle can be correctly paired.

[0125] The set of hidden states in the Hidden Markov Model (HMM) is determined based on the clinical classification system of cross-modal contradictory states in neoadjuvant therapy for esophageal cancer. This limits the state space to core state types directly related to pseudoprogression identification, avoiding data sparsity issues in subsequent parameter estimation due to an excessively large state space. The initialization of the state transition probability matrix employs a priori approach based on historical clinical cohort statistics. The frequency of transitions between states is statistically analyzed from past treatment data, and the initial transition probabilities are calculated as the starting point for model training. This initialization method accelerates the convergence process of the EM algorithm. The size of the hidden state set is directly related to the dimension of the state transition probability matrix; the number of rows and columns in the matrix are equal to the number of hidden states.

[0126] The Expectation-Maximization algorithm estimates the parameters of the state transition probability matrix by iteratively executing expectation and maximization steps. Preferably, in the expectation step, the forward-backward algorithm is used to calculate the posterior probability of being in each hidden state at each time point given the current model parameters and observation sequence, as well as the posterior probability of each state transition between adjacent time points. In the maximization step, based on the posterior probabilities calculated in the expectation step, the expected frequency of transition from the first contradictory state to each hidden state is counted, and the expected frequency is divided by the total expected frequency starting from the first contradictory state to obtain the updated transition probability value. The iterative process continues until the update magnitude of each element in the state transition probability matrix is ​​lower than the preset convergence threshold, at which point the model parameters tend to stabilize.

[0127] The first and second transition probabilities are output as floating-point numbers, ranging from 0 to 1. The first transition probability represents the estimated probability of transitioning from the first contradictory state to a consistent remission state, while the second transition probability represents the estimated probability of transitioning from the first contradictory state to a consistent progression state. The relative magnitudes of the two probability values ​​reflect the model's inference about the subsequent evolutionary trend of patients currently in the first contradictory state, providing direct numerical input for calculating the confidence level of subsequent pseudoprogression.

[0128] This embodiment generates a state sequence by arranging cross-modal contradictory state labels in the order of treatment cycles. A Hidden Markov Model (HMM) is constructed, and the expectation-maximization (EM) algorithm is used to estimate the parameters of the state transition probability matrix, outputting the probability values ​​of transitions from the first contradictory state to the consistent remission state and the consistent progression state. The required length of the state sequence ensures that at least one observable transition event is included. The initialization strategy based on clinical priors accelerates model convergence, and the EMM algorithm achieves efficient estimation of model parameters even when the observed sequence is known but the hidden state sequence is unknown. This embodiment models the dynamic evolution of cross-modal contradictory states as computable transition probabilities, providing a quantitative basis for generating subsequent false progression confidence scores.

[0129] In some embodiments, constructing a hidden Markov model as a temporal state transition model includes:

[0130] Define the set of hidden states of the Hidden Markov Model. The set of hidden states includes the first contradictory state, the second contradictory state, the uniformly relieved state, the uniformly progressive state, and the image-stabilized molecular stable state. The second contradictory state corresponds to the combination where the image change direction is shrinking and the molecular change direction is increasing. The image-stabilized molecular stable state corresponds to the combination where the image change direction is stable and the molecular change direction is stable.

[0131] Define the set of observation symbols for the Hidden Markov Model, which contains the labels of cross-modal contradictory states actually observed in each treatment cycle;

[0132] Initialize the state transition probability matrix of the hidden Markov model. Each row of the state transition probability matrix corresponds to a current hidden state, and each column corresponds to a next hidden state. The initial transition probability from the first contradictory state to the consistent relief state in the state transition probability matrix is ​​set to be higher than the initial transition probability from the first contradictory state to the consistent progression state, so as to reflect the prior knowledge of the pseudo-progression clinical scenario.

[0133] Initialize the observation probability matrix of the Hidden Markov Model. Each row of the observation probability matrix corresponds to a hidden state, and each column corresponds to an observation sign. The initial values ​​of the diagonal elements in the observation probability matrix are set higher than the initial values ​​of the off-diagonal elements to reflect the prior assumption that the hidden states and observation signs are consistent.

[0134] In this embodiment, the latent state set expands the state space from three to five types. The newly added second contradictory state corresponds to the combination where the imaging change direction is shrinking and the molecular change direction is increasing. The image-stable and molecularly stable state corresponds to the combination where the imaging change direction is stable and the molecular change direction is stable. The introduction of the second contradictory state covers the clinical scenario of minimal residual disease, i.e., the situation where the tumor volume shrinks on imaging but residual tumor activity is observed at the molecular level. This state is associated with the risk of recurrence after treatment. The introduction of the image-stable and molecularly stable state covers the situation where the treatment response is in a plateau phase or the efficacy has not yet been shown. At this time, there are no significant changes in imaging or molecular dynamics. This state is more common in the early stages of treatment. The five latent states together constitute a complete cross-modal contradictory state space, covering various clinical scenarios such as the imaging and molecular change directions being consistent, opposite, and one side being stable, providing a comprehensive state classification system for subsequent temporal modeling.

[0135] The set of observation symbols corresponds to the set of hidden states. Each element in the set of observation symbols corresponds to a label of a cross-modal contradictory state actually observed in a treatment cycle. The difference between observation symbols and hidden states is that hidden states are the patient's true state inferred by the model, while observation symbols are observation results obtained based on the mapping of directional features. The two may differ due to detection errors or incomplete accuracy of mapping rules. The size of the set of observation symbols is the same as that of the set of hidden states, both containing five state types, ensuring that each hidden state has a corresponding observation symbol.

[0136] The state transition probability matrix is ​​initialized using a non-uniform initialization strategy based on clinical prior knowledge. The row indices of the matrix correspond to the current hidden state, the column indices correspond to the next hidden state, and the values ​​of the matrix elements represent the probability of transitioning from the current hidden state to the next hidden state. The initial transition probability from the first contradictory state to the consistent remission state is set higher than the initial transition probability from the first contradictory state to the consistent progression state. This setting is based on the clinical prior knowledge of pseudoprogression in neoadjuvant immunotherapy for esophageal cancer, i.e., the probability of pseudoprogression patients subsequently achieving remission is higher than the probability of progression. The initial transition probabilities between other states can be set to a uniform distribution or a non-uniform distribution based on prior knowledge, depending on clinical experience or historical data statistics.

[0137] The initialization of the observation probability matrix also employs a non-uniform initialization strategy based on prior assumptions. Row indices correspond to hidden states, column indices to observation symbols, and the values ​​of matrix elements represent the probability of observing a specific observation symbol given a hidden state. The initial values ​​of diagonal elements are set higher than those of off-diagonal elements. This setting is based on the prior assumption of consistency between hidden states and observation symbols, meaning that in most cases, the observed cross-modal contradictory state labels are consistent with the patient's true hidden state. The initial probability values ​​of diagonal elements are set to be one order of magnitude higher than those of off-diagonal elements. The initial probability values ​​of off-diagonal elements are uniformly distributed among the remaining probabilities, and their specific values ​​can be adjusted according to the level of observational noise in the clinical data.

[0138] This embodiment defines a set of hidden states comprising five states and a corresponding set of observation symbols. A non-uniform initialization strategy based on clinical prior knowledge is used to initialize the state transition probability matrix and the observation probability matrix. The expanded set of hidden states covers various clinical scenarios, including pseudo-progression, minimal residual disease, uniform remission, uniform progression, and plateau phase. The non-uniform initialization strategy incorporates clinical prior knowledge into the model parameters, providing a reasonable starting point for subsequent parameter estimation using the expectation-maximization algorithm, accelerating the model convergence process and improving the accuracy of parameter estimation.

[0139] In some embodiments, the state sequence is input into a Hidden Markov Model (HMM), and the HMM estimates the parameters of the state transition probability matrix using an expectation-maximization algorithm, including:

[0140] The state sequence is input into the hidden Markov model as the observation sequence. The state sequence contains continuous cross-modal contradictory state labels from the first treatment cycle to the Nth treatment cycle.

[0141] The expectation step of the expectation maximization algorithm is to calculate the posterior probability distribution of the hidden state sequence corresponding to the current parameters of the hidden Markov model. The posterior probability distribution includes the probability of each treatment cycle being in each hidden state and the probability of state transition between adjacent treatment cycles.

[0142] The maximization step of the expectation maximization algorithm is executed. Based on the posterior probability distribution, the values ​​of each element in the state transition probability matrix are re-estimated. The re-estimated frequency includes the statistical calculation of the expected frequency of transition from the first contradictory state to the consistent relief state. The expected frequency is divided by the sum of the expected frequencies of transition from the first contradictory state to all hidden states to generate the updated first transition probability.

[0143] Iteratively execute the expected step and the maximization step until the update magnitude of the state transition probability matrix is ​​lower than the preset convergence threshold, and output the converged state transition probability matrix.

[0144] In this embodiment, the state sequence stores the cross-modal contradictory state labels of each treatment cycle in integer encoding form, with each state corresponding to a unique integer value. The value of N depends on the number of treatment cycles actually completed by the patient. The sequence length directly affects the sample size of transition events that can be used for parameter estimation; the longer the sequence, the higher the statistical reliability of the parameter estimation.

[0145] The expected step is executed based on a forward-backward algorithm. Forward recursion calculates partial probabilities from the start of the sequence to the current time point, and backward recursion calculates partial probabilities from the current time point to the end of the sequence. Combining these two probabilities yields the posterior probability of each time point being in each hidden state given the observed sequence. The posterior probability distribution includes the probability of each treatment cycle being in each hidden state and the probability of state transitions between adjacent treatment cycles. These two types of probabilities are used to estimate the most likely hidden state sequence and the expected frequency of state transitions, respectively.

[0146] In the maximization step, the expected frequency of transition from the first contradictory state to the consistent remission state is obtained by summing the posterior probabilities of this transition path in all adjacent treatment cycles. The sum of the expected frequencies of transition from the first contradictory state to all hidden states is obtained by summing the posterior probabilities of all transition paths originating from the first contradictory state in all adjacent treatment cycles. The updated first transition probability is obtained by dividing the former by the latter. The updates of other elements in the state transition probability matrix all follow the same expected frequency statistics and normalization logic.

[0147] The preset convergence threshold can be set to a small positive number based on model training experience to determine whether the model parameters tend to stabilize. When the update magnitude of each element in the state transition probability matrix is ​​lower than this threshold, the iteration terminates and the converged state transition probability matrix is ​​output. If the convergence condition is not met even after the preset maximum number of iterations is reached, the current parameters are output as an approximate solution.

[0148] This embodiment achieves efficient calculation of posterior probability in the expectation step through a forward-backward algorithm, and achieves maximum likelihood estimation of parameters in the maximization step through expectation frequency statistics and normalization. Combined with a convergence threshold, it ensures reasonable termination of the iteration process, and concretizes the complete execution process of the expectation maximization algorithm into an operable technical solution, providing an implementation path for parameter estimation of hidden Markov models.

[0149] In some embodiments, a false progression confidence score for the current treatment stage is generated based on the state transition probabilities output by the temporal state transition model, including:

[0150] Obtain the first transition probability and the second transition probability output by the temporal state transition model. The first transition probability is the probability of transitioning from the first contradictory state to the consistent relief state, and the second transition probability is the probability of transitioning from the first contradictory state to the consistent progress state.

[0151] The ratio of the first metastasis probability to the sum of the first and second metastasis probabilities is calculated to generate a pseudoprogression confidence scalar. The pseudoprogression confidence scalar represents the probability that a patient currently in the first contradictory state will subsequently enter a consistent remission state.

[0152] If the confidence scalar of pseudoprogression exceeds the preset confidence threshold, the current treatment stage is determined to be a pseudoprogression state, and a pseudoprogression warning signal is output.

[0153] If the confidence scalar of pseudoprogression does not exceed the preset confidence threshold, the current treatment stage is determined to be a true progression state, and a true progression warning signal is output.

[0154] In this embodiment, the first transition probability and the second transition probability are derived from the state transition probability matrix output by the Hidden Markov Model after parameter estimation using the Expectation-Maximization algorithm. Specifically, they are the element values ​​in the matrix that point from the row containing the first contradictory state to the columns containing the consistent relief state and the consistent progress state, respectively. Both transition probabilities are stored in floating-point form, with values ​​ranging from 0 to 1, representing the estimated probability of transitioning to relief or progress after starting from the first contradictory state.

[0155] The confidence scalar for pseudoprogression is calculated by dividing the first metastasis probability by the sum of the first and second metastasis probabilities, with this ratio continuously ranging from 0 to 1. When the first metastasis probability is significantly greater than the second metastasis probability, the ratio approaches 1, indicating that the patient currently in a paradoxical state is much more likely to enter a consistent remission state than a consistent progression state, meaning the current imaging progression is more likely to be pseudoprogression. When the first and second metastasis probabilities are close, the ratio approaches 0.5, indicating that the two metastasis pathways are equally likely, and the determination of pseudoprogression in this case requires comprehensive judgment based on other clinical information. When the first metastasis probability is significantly less than the second metastasis probability, the ratio approaches 0, indicating that the current imaging progression is more likely to be true progression.

[0156] The preset reliability threshold is determined based on receiver operating characteristic (ROC) curve analysis of historical clinical cohorts. By iterating through the true positive and false positive rates at different thresholds, the value corresponding to the maximum Youden index or the balance point between sensitivity and specificity specified by clinical needs is selected as the threshold. When the false progression confidence scalar exceeds the preset reliability threshold, the current treatment stage is determined to be in a false progression state. At this time, the output false progression warning signal includes the patient identifier, the current treatment cycle number, the false progression confidence scalar value, and the determination result, for clinicians to refer to when making subsequent treatment decisions.

[0157] When the confidence scalar of false progression does not exceed the preset confidence threshold, the current treatment stage is determined to be true progression. At this time, the output true progression warning signal also includes the patient identifier, the current treatment cycle number, the false progression confidence scalar value, and the judgment result, indicating to the clinician that the current imaging progression is more likely to be true progression, and it is necessary to consider adjusting the treatment plan or performing a biopsy for confirmation.

[0158] This embodiment generates a false progression confidence scalar by calculating the ratio of the first transition probability to the sum of the first and second transition probabilities. Combined with a pre-set reliability threshold determined based on receiver operating characteristic (ROC) curve analysis of historical clinical cohorts, this transforms continuous probability values ​​into classification decision results with clear clinical meaning. The output of false progression warning signals and true progression warning signals provides clinicians with intuitive decision support information, helping to avoid unnecessary treatment interruptions due to misjudgments of false progression.

[0159] Please see Figure 3 In a second aspect, this embodiment also provides a time-series multimodal prediction system 1 for neoadjuvant therapy efficacy in esophageal cancer, applicable to the method described in the first aspect. The system includes a data acquisition module 11, a directional feature calculation module 12, a contradictory state definition module 13, a time-series modeling module 14, and a confidence generation module 15. The data acquisition module 11 is used to acquire image acquisition data and molecular acquisition data of the patient at multiple time points before and during treatment. The image acquisition data includes radiomics features of esophageal lesions at at least two time points, and the molecular acquisition data includes circulating tumor DNA methylation levels at at least three time points. The directional feature calculation module 12 is connected to the data acquisition module 11 and is used to calculate the directional features of image changes based on the radiomics features of adjacent time points in the image acquisition data, and calculate the directional features of circulating tumor DNA methylation levels based on the directional features of adjacent time points in the molecular acquisition data. The module calculates the molecular change direction characteristics of NA methylation level; the contradictory state definition module 13 is connected to the direction feature calculation module 12, and is used to pair and combine the image change direction characteristics and the molecular change direction characteristics to define cross-modal contradictory states. The cross-modal contradictory states include the first contradictory state where the image change direction is increasing and the molecular change direction is decreasing; the temporal modeling module 14 is connected to the contradictory state definition module 13, and is used to model the transition path of cross-modal contradictory states with the treatment process based on the temporal state transition model. The temporal state transition model learns the probability of transitioning from the first contradictory state to the consistent remission state; the confidence generation module 15 is connected to the temporal modeling module 14, and is used to generate the false progression confidence of the current treatment stage based on the state transition probability output by the temporal state transition model, as a dynamic prediction result of neoadjuvant efficacy and output it.

[0160] In this embodiment, the data acquisition module 11, the orientation feature calculation module 12, the contradictory state definition module 13, the temporal modeling module 14, and the confidence generation module 15 are sequentially connected to form a complete processing link from data acquisition to prediction result output. The data acquisition module 11 receives multimodal raw data from imaging equipment and liquid biopsy equipment. The orientation feature calculation module 12 performs directional transformation on the received data and then transmits it to the contradictory state definition module 13. The contradictory state definition module 13 inputs the identified contradictory state labels into the temporal modeling module 14. The temporal modeling module 14 calculates the state transition probability using a hidden Markov model and then passes it to the confidence generation module 15 to calculate the false progression confidence score and output it. This system solidifies the temporal multimodal prediction method into a deployable hardware and software system through a modular architecture. The data flow between modules achieves automated processing from raw data to clinical decision support, providing a feasible engineering implementation solution for identifying false progression in neoadjuvant therapy for esophageal cancer.

[0161] By adopting the above technical solutions, this invention differs from existing technologies and possesses the following beneficial effects: By extracting the image change direction features and molecular change direction features at adjacent time points, and combining them in pairs, a cross-modal contradictory state of image enlargement and molecular decrease is defined. This transforms the inconsistency between image and molecular signals into calculable feature labels, overcoming the limitation of existing multimodal fusion methods that assume consistent change directions across modalities. Furthermore, a temporal state transition model is constructed to model the transition path of the contradictory state as the treatment progresses, learning the probability of transitioning from the contradictory state to a consistent remission state, thus elevating the identification of contradictory states from static discrimination to dynamic evolution prediction. Finally, a false progression confidence scalar is generated by calculating the ratio of state transition probabilities, providing a quantitative basis for clinical decision-making. This invention achieves dynamic identification of false progression in neoadjuvant therapy for esophageal cancer, avoiding unnecessary treatment interruptions caused by misjudgment of false progression, and improving the accuracy and clinical operability of efficacy prediction.

[0162] Finally, it should be noted that although the above embodiments have been described in the text and drawings of this application, this should not limit the scope of patent protection of this application. Any technical solutions that are based on the essential concept of this application and utilize the content described in the text and drawings of this application, resulting in equivalent structural or procedural substitutions or modifications, as well as the direct or indirect application of the technical solutions of the above embodiments to other related technical fields, are all included within the scope of patent protection of this application.

Claims

1. A time-series multi-modal prediction method for neoadjuvant efficacy of esophageal cancer, characterized in that, include: The patient's imaging and molecular data were acquired at multiple time points before and during treatment. The imaging data included radiomics features of esophageal lesions at at least two time points, and the molecular data included circulating tumor DNA methylation levels at at least three time points. The image change direction features are calculated based on the radiomics features of adjacent time points in the image acquisition data, and the molecular change direction features are calculated based on the circulating tumor DNA methylation level of adjacent time points in the molecular acquisition data. The image change direction feature and the molecular change direction feature are paired and combined to define a cross-modal contradictory state. The cross-modal contradictory state includes a first contradictory state in which the image change direction is increasing and the molecular change direction is decreasing. The transition path of the cross-modal contradictory state as the treatment progress is modeled based on the temporal state transition model, and the temporal state transition model learns the probability of transitioning from the first contradictory state to the consistent remission state. Based on the state transition probabilities output by the time-series state transition model, a false progression confidence score for the current treatment stage is generated and output as a dynamic prediction result of neoadjuvant efficacy.

2. The method of Claim 1, wherein the method is used for predicting the neoadjuvant treatment efficacy of esophageal cancer. The image change direction features are calculated based on the radiomics features of adjacent time points in the image acquisition data, including: Extract the first radiomics feature vector at the first time point and the second radiomics feature vector at the second time point from the image acquisition data. Both the first radiomics feature vector and the second radiomics feature vector contain tumor volume features and tumor gray-level co-occurrence matrix contrast features. Calculate the difference between the feature values ​​of each dimension in the second radiomics feature vector and the corresponding feature values ​​in the first radiomics feature vector, divide the difference by the sum of the corresponding feature values ​​in the first radiomics feature vector and a preset minimum constant, generate the image feature change rate of each dimension, and construct the image feature change rate vector from the image feature change rates of each dimension. The image feature change rates of each dimension in the image feature change rate vector are sequentially input into the sign discrimination function, and the sign discrimination function maps the image feature change rates of each dimension into discrete directional labels, which include increase labels, decrease labels and stable labels. The frequency of occurrence of the increasing label in the discrete direction labels is counted, and the ratio of the frequency of occurrence of the increasing label to the total number of discrete direction labels is calculated. If the ratio exceeds a preset majority threshold, the image change direction feature is determined to be an increasing direction. 3.The method of claim 1, wherein, The molecular change direction characteristics are calculated based on the circulating tumor DNA methylation levels at adjacent time points in the molecular acquisition data, including: Extract the first methylation level value at the third time point and the second methylation level value at the fourth time point from the molecular acquisition data. The methylation level value is the quantitative detection result of a preset single methylation site. Calculate the difference between the second methylation level value and the first methylation level value, divide the difference by the sum of the first methylation level value and a preset minimum constant, and generate a scalar of the rate of change of molecular characteristics. The molecular characteristic change rate scalar is input into the sign discrimination function, and the sign discrimination function maps the continuous value of the molecular characteristic change rate scalar to discrete direction labels, the discrete direction labels including descent labels, ascending labels and stable labels; If the discrete direction label is a descending label, then the molecular change direction feature is determined to be a descending direction.

4. The method of Claim 3, wherein the method is used for predicting the neoadjuvant treatment efficacy of esophageal cancer. The image change direction features and the molecular change direction features are paired and combined to define cross-modal contradictory states, including: The image change direction features and the molecular change direction features calculated within the same treatment cycle are combined to form a direction feature pair, and the direction feature pair includes an image direction label and a molecular direction label. A state mapping rule is established, which maps the combination of the directional feature alignment image directional label being an increasing label and the molecular direction label being a decreasing label to a first contradictory state, maps the combination of the directional feature alignment image directional label being a decreasing label and the molecular direction label being a decreasing label to a consistent relief state, and maps the combination of the directional feature alignment image directional label being an increasing label and the molecular direction label being an increasing label to a consistent progress state. According to the state mapping rule, the directional feature pairs of each treatment cycle are mapped one by one to corresponding cross-modal contradictory state labels, and the cross-modal contradictory state labels include at least the first contradictory state, the consistent remission state, and the consistent progress state.

5. The method of Claim 4, wherein the method is used for predicting the neoadjuvant treatment efficacy of esophageal cancer. Establish state mapping rules, including: Define all possible combinations of the directional feature-aligned image directional label and molecular directional label. The total possible combinations include nine combinations, which are formed by pairing the enlargement label, the shrinkage label and the stability label. From the nine combinations, identify combinations where the image orientation label and the molecular orientation label change in opposite directions, and classify the combinations with opposite change directions into a candidate set of contradictory states. The candidate set of contradictory states includes at least a first combination where the image orientation label is an increasing label and the molecular orientation label is a decreasing label, and a second combination where the image orientation label is a decreasing label and the molecular orientation label is an increasing label. The first combination is selected from the candidate set of contradictory states, and the first combination is mapped to the first contradictory state. The first contradictory state corresponds to the clinical scenario of pseudoprogression in neoadjuvant therapy for esophageal cancer, where imaging shows progression but molecular manifestations show remission. The second combination in the candidate set of contradictory states is mapped to the second contradictory state, which corresponds to the clinical scenario of minimal residual disease in neoadjuvant therapy for esophageal cancer, where imaging shows remission but molecular manifestations show progression. 6.The method of claim 4, wherein, The transition path of the cross-modal contradictory state during treatment is modeled based on a temporal state transition model. The temporal state transition model learns the probability of transitioning from the first contradictory state to a consistent remission state, including: Arrange the cross-modal contradictory state labels corresponding to each treatment cycle in chronological order to generate a state sequence, wherein the state sequence contains cross-modal contradictory state labels for at least two consecutive treatment cycles. A hidden Markov model is constructed as the temporal state transition model. The hidden Markov model includes a set of hidden states and a state transition probability matrix. The set of hidden states includes at least the first contradictory state, the consistent relief state, and the consistent progress state. The state sequence is input into the Hidden Markov Model, and the Hidden Markov Model estimates the parameters of the state transition probability matrix using the expectation-maximization algorithm to calculate the first transition probability from the first contradictory state to the consistent relief state, and the second transition probability from the first contradictory state to the consistent progress state. The first transition probability and the second transition probability are output as a quantitative representation of the transition path of the cross-modal contradictory state as the treatment progresses.

7. The method of Claim 6, wherein the method is for predicting the neoadjuvant treatment efficacy of esophageal cancer. Constructing a Hidden Markov Model as the temporal state transition model includes: Define the set of hidden states of the Hidden Markov Model, which includes a first contradictory state, a second contradictory state, a uniformly relieved state, a uniformly progressive state, and an image-stabilized molecular stable state. The second contradictory state corresponds to the combination where the image change direction is shrinking and the molecular change direction is increasing. The image-stabilized molecular stable state corresponds to the combination where the image change direction is stable and the molecular change direction is stable. Define the set of observation symbols for the Hidden Markov Model, which includes the labels of the cross-modal contradictory states actually observed in each treatment cycle; The state transition probability matrix of the hidden Markov model is initialized. Each row of the state transition probability matrix corresponds to a current hidden state, and each column corresponds to a next hidden state. The initial transition probability from the first contradictory state to the consistent relief state in the state transition probability matrix is ​​set to be higher than the initial transition probability from the first contradictory state to the consistent progress state, so as to reflect the prior knowledge of the pseudo-progression clinical scenario. The observation probability matrix of the hidden Markov model is initialized. Each row of the observation probability matrix corresponds to a hidden state, and each column corresponds to an observation symbol. The initial values ​​of the diagonal elements in the observation probability matrix are set higher than the initial values ​​of the off-diagonal elements to reflect the prior assumption that the hidden states and observation symbols are consistent. 8.The method of claim 6, wherein, The state sequence is input into the Hidden Markov Model (HMM), and the HMM estimates the parameters of the state transition probability matrix using the Expectation-Maximization (EM) algorithm, including: The state sequence is input into the hidden Markov model as an observation sequence, and the state sequence contains continuous cross-modal contradictory state labels from the first treatment cycle to the Nth treatment cycle. The expectation step of the expectation maximization algorithm is executed, and the posterior probability distribution of the hidden state sequence corresponding to the state sequence is calculated based on the current parameters of the hidden Markov model. The posterior probability distribution includes the probability of each treatment cycle being in each hidden state and the probability of state transition between adjacent treatment cycles. The maximization step of the expectation maximization algorithm is executed, and the values ​​of each element in the state transition probability matrix are re-estimated according to the posterior probability distribution. The re-estimated value includes counting the expected frequency of the transition from the first contradictory state to the consistent relief state, dividing the expected frequency by the sum of the expected frequencies of the transition from the first contradictory state to all hidden states, and generating the updated first transition probability. The desired step and the maximization step are executed iteratively until the update magnitude of the state transition probability matrix is ​​lower than the preset convergence threshold, and the converged state transition probability matrix is ​​output. 9.The method of claim 6, wherein, Based on the state transition probabilities output by the temporal state transition model, a false progression confidence score for the current treatment stage is generated, including: Obtain the first transition probability and the second transition probability output by the time-series state transition model, wherein the first transition probability is the probability of transitioning from the first contradictory state to the consistent relief state, and the second transition probability is the probability of transitioning from the first contradictory state to the consistent progress state; Calculate the ratio of the first transition probability to the sum of the first transition probability and the second transition probability to generate a false progression confidence scalar. The false progression confidence scalar represents the probability that a patient currently in the first contradictory state will subsequently enter the consistent remission state. If the confidence scalar of the false progression exceeds the preset confidence threshold, the current treatment stage is determined to be a false progression state, and a false progression warning signal is output. If the confidence scalar of the pseudoprogression does not exceed the preset confidence threshold, the current treatment stage is determined to be a true progression state, and a true progression warning signal is output.

10. An esophageal cancer neoadjuvant therapy efficacy time-series multi-modal prediction system, characterized in that, The system applicable to the method of any one of claims 1 to 9 comprises: The data acquisition module is used to acquire imaging and molecular data of the patient at multiple time points before and during treatment. The imaging data includes radiomics features of esophageal lesions at at least two time points, and the molecular data includes circulating tumor DNA methylation levels at at least three time points. The directional feature calculation module, connected to the data acquisition module, is used to calculate the directional features of image changes based on the radiomics features of adjacent time points in the image acquisition data, and to calculate the directional features of molecular changes based on the circulating tumor DNA methylation levels of adjacent time points in the molecular acquisition data. The contradictory state definition module, connected to the direction feature calculation module, is used to pair and combine the image change direction feature and the molecular change direction feature to define a cross-modal contradictory state. The cross-modal contradictory state includes a first contradictory state in which the image change direction is increasing and the molecular change direction is decreasing. The temporal modeling module, connected to the contradictory state definition module, is used to model the transition path of the cross-modal contradictory state as the treatment progress based on the temporal state transition model, wherein the temporal state transition model learns the probability of transitioning from the first contradictory state to the consistent remission state. A confidence generation module, connected with the timing modeling module, is configured to generate a false progression confidence of a current treatment stage according to a state transition probability output by the timing state transition model, as a dynamic prediction result of neoadjuvant efficacy and output.

Images