Method and system for preoperative staging of esophageal cancer

Abstract

This invention discloses a method and system for preoperative staging prediction of esophageal cancer, belonging to the field of image processing technology. The method includes: acquiring multimodal feature data; the multimodal feature data includes CT feature vectors, MRI feature vectors, and clinical feature vectors; using the clinical feature vectors as contextual information, fusing the CT and MRI feature vectors to obtain a fused feature vector; and inputting the fused feature vector into a prediction network for prediction to obtain the predicted staging result of esophageal cancer. This approach utilizes clinical feature vectors as contextual information and fuses CT and MRI feature vectors based on the spatial information of the tumor in both datasets, improving the accuracy of the esophageal cancer staging prediction result obtained by inputting the fused feature vector into the prediction network.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method and system for predicting the preoperative staging of esophageal cancer. Background Technology

[0002] Esophageal cancer is a prevalent malignant tumor worldwide, and its treatment strategies and prognosis heavily rely on accurate preoperative staging. Currently, commonly used preoperative staging methods are typically based on computed tomography (CT) scans. Physicians can use CT results to assess tumor extent, lymph node involvement, and distant metastases, thereby achieving staging. Recent studies have also utilized deep learning for esophageal cancer staging; for example, the ResoLSTM-Depth model was developed to differentiate between T1-T2 and T3-T4 esophageal cancer using arterial phase CT images.

[0003] However, the aforementioned methods, which rely on doctors for staging, place extremely high demands on their CT scan interpretation skills and medical experience. While deep learning-based esophageal cancer staging can only differentiate tumor size, it cannot determine lymph node involvement or distant metastasis. Therefore, we will first explore a technical approach that can accurately predict esophageal cancer staging preoperatively. Summary of the Invention

[0004] Therefore, it is necessary to provide a method and system for preoperative staging prediction of esophageal cancer to address the above problems and improve the accuracy of staging prediction.

[0005] This application provides a method for preoperative staging prediction of esophageal cancer, the method comprising: Acquire multimodal feature data; the multimodal feature data includes CT feature vectors, MRI feature vectors, and clinical feature vectors; the CT feature vectors are obtained by feature extraction from the CT images of the target object; the MRI feature vectors are obtained by feature extraction from the MRI data of the target object; the clinical feature vectors are obtained by encoding and standardizing the clinical data of the target object; the clinical data includes the spatial information of the tumor in the CT images and MRI data; Using the clinical feature vector as contextual information, the CT feature vector and MRI feature vector are fused to obtain a fused feature vector; The fused feature vector is input into the prediction network for prediction to obtain the prediction result of esophageal cancer staging.

[0006] Furthermore, this application also provides a preoperative staging prediction system for esophageal cancer, the system comprising: The system includes: The data acquisition module is used to acquire multimodal feature data, including CT feature vectors, MRI feature vectors, and clinical feature vectors. The CT feature vectors are obtained by extracting features from the CT images of the target object. The MRI feature vectors are obtained by extracting features from the MRI data of the target object. The clinical feature vectors are obtained by encoding and standardizing the clinical data of the target object. The clinical data includes the spatial information of the tumor in the CT images and MRI data. The vector fusion module is used to fuse the CT feature vector and the MRI feature vector using the clinical feature vector as context information to obtain a fused feature vector. The staging prediction module is used to input the fused feature vector into the prediction network for prediction, and obtain the prediction result of esophageal cancer staging.

[0007] Compared with the prior art, the technical solution provided in this application has the following advantages: In preoperative staging of target patients, computer equipment can extract features from CT images to obtain CT feature vectors, extract features from MRI data to obtain MRI feature vectors, and encode and standardize clinical data to obtain clinical feature vectors. Then, the computer equipment can use the clinical feature vectors as context to fuse the CT and MRI feature vectors, obtaining a fused feature vector. Finally, the fused feature vector is input into a prediction network for prediction, obtaining the predicted staging result for esophageal cancer, which can be used by doctors to assess the patient's condition. This approach utilizes clinical feature vectors as contextual information and fuses CT and MRI feature vectors based on the spatial information of the tumor in both datasets, improving the accuracy of the esophageal cancer staging prediction result obtained by inputting the fused feature vector into the prediction network. Attached Figure Description

[0008] Figure 1 A flowchart of a method for preoperative staging prediction of esophageal cancer according to an embodiment of the present invention is shown.

[0009] Figure 2 A flowchart of a method for preoperative staging prediction of esophageal cancer according to an embodiment of the present invention is shown.

[0010] Figure 3 This is a schematic diagram of the structure of a preoperative staging prediction system for esophageal cancer provided in an embodiment of this application.

[0011] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an optional embodiment of the present invention. Detailed Implementation

[0012] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention can be practiced in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0013] Esophageal cancer is a highly prevalent malignant tumor worldwide, and its treatment strategies and prognosis heavily depend on accurate preoperative staging. Currently, commonly used preoperative staging methods include: Computed tomography (CT): It is widely used and is mainly used to assess the extent of tumors, lymph nodes and distant metastases. However, it has limited resolution for different layers of the esophageal wall. The distinction between T1-T2 and T3-T4 mainly relies on indirect signs such as wall thickness. It is highly subjective and its accuracy is unstable.

[0014] Endoscopic ultrasound (EUS): It is one of the "gold standards" for T staging and can clearly show the esophageal wall structure. However, it is an invasive procedure with risks of bleeding and perforation, and it cannot pass through in about 30% of cases of luminal stenosis.

[0015] Magnetic resonance imaging (MRI): It has excellent soft tissue contrast. MRI has a sensitivity and specificity of 86% in differentiating between ≤T2 and ≥T3 stage esophageal cancer, and a sensitivity of 71% and a specificity of 72% in detecting lymph node metastasis (N+), showing potential, but the specificity still needs to be improved.

[0016] Artificial intelligence-assisted staging: Recent studies have explored the use of deep learning for esophageal cancer staging. For example, the ResoLSTM-Depth model uses arterial phase CT images to distinguish between T1-T2 and T3-T4. However, this model only predicts T stage and does not cover N stage prediction, which is crucial for treatment decisions. Furthermore, this model only uses a single CT modality. While the CT modality has high accuracy for T stage, its accuracy drops sharply when N stage prediction is involved.

[0017] Therefore, to ensure the accuracy of esophageal cancer staging prediction, this application provides a preoperative staging prediction method for esophageal cancer. Figure 1 A flowchart of a method for preoperative staging prediction of esophageal cancer according to an embodiment of the present invention is shown, as follows: Figure 1 As shown, the method flow includes: Step 101: Obtain multimodal feature data.

[0018] The multimodal feature data includes CT feature vectors, MRI feature vectors, and clinical feature vectors. The CT feature vectors are obtained by extracting features from the CT images of the target object. The MRI feature vectors are obtained by extracting features from the MRI data of the target object. The clinical feature vectors are obtained by encoding and standardizing the clinical data of the target object. The clinical data includes the spatial information of the tumor in the CT images and MRI data.

[0019] Furthermore, this clinical data can include basic patient information, physiological indicators, and tumor spatial information extracted from CT images and MRI data, such as tumor location, size, morphological parameters, and its relative position to adjacent tissues. By numerically encoding and normalizing the above information, data from different sources can be made comparable in a unified feature space, thereby improving the stability of the subsequent fusion process.

[0020] Step 102: Using the clinical feature vector as contextual information, the CT feature vector and the MRI feature vector are fused to obtain the fused feature vector.

[0021] Computer devices construct a cross-modal feature interaction model based on CT feature vectors, MRI feature vectors, and clinical feature vectors, thereby fusing CT feature vectors, MRI feature vectors, and clinical feature vectors.

[0022] In one alternative implementation, a cross-attention mechanism can be used to achieve information interaction between CT feature vectors and MRI feature vectors. In this mechanism, one modality feature is used as the query vector, and the other modality feature is used as the key vector and value vector. Cross-modal related features are obtained through attention calculation.

[0023] Furthermore, in the process of cross-attention calculation, clinical feature vectors are introduced as contextual information to modulate the query vector, key vector, or attention weights, so that the distribution of attention weights can be adaptively adjusted according to the individual clinical characteristics of the target object.

[0024] Specifically, modulation parameters can be generated based on clinical feature vectors, and feature transformations can be performed on query vectors or key vectors, or a bias term generated from clinical feature vectors can be introduced during the attention weight calculation process, thereby constraining the cross-modal feature interaction process with clinical information.

[0025] Step 103: Input the fused feature vector into the prediction network for prediction to obtain the prediction results of esophageal cancer staging.

[0026] Specifically, the fused feature vector is input into a pre-trained prediction network, which can be a multilayer perceptron network, an attention-based deep neural network, or other model structures suitable for classification tasks. The prediction network performs a non-linear mapping on the fused feature vector and outputs the corresponding esophageal cancer staging prediction result. The staging prediction result can include tumor staging information and lymph node metastasis status information, thereby providing auxiliary decision-making basis for clinicians to formulate treatment plans.

[0027] Furthermore, during model training, the prediction network can be supervised learning based on training data with real stage labels. By minimizing the loss function between the prediction result and the real label, the model can gradually learn the mapping relationship between different modal features and tumor stages, thereby improving the prediction accuracy and generalization ability of the model in practical applications.

[0028] In summary, when staging a target patient preoperatively, the computer system can extract features from the target patient's CT images to obtain CT feature vectors, extract features from the target patient's MRI data to obtain MRI feature vectors, and simultaneously encode and standardize clinical data to obtain clinical feature vectors. Then, the computer system can use the clinical feature vectors as context to fuse the CT and MRI feature vectors, obtaining a fused feature vector. Finally, the fused feature vector is input into a prediction network for prediction, obtaining the predicted staging result for esophageal cancer, which can be used by doctors to assess the patient's condition. This approach utilizes clinical feature vectors as contextual information and fuses CT and MRI feature vectors based on the spatial information of the tumor in both datasets, improving the accuracy of the esophageal cancer staging prediction result obtained by inputting the fused feature vector into the prediction network.

[0029] Figure 2 A flowchart of a method for preoperative staging prediction of esophageal cancer according to an embodiment of the present invention is shown, as follows: Figure 2 As shown, the method flow includes: Step 201: Acquire CT images and MRI data; the CT images and MRI data are marked with the primary tumor area and the suspicious lymph node area.

[0030] In this embodiment of the application, a computer device acquires computed tomography (CT) images and magnetic resonance imaging (MRI) data of a target object, wherein the MRI data includes diffusion-weighted imaging (DWI) sequence data and apparent diffusion coefficient (ADC) image data calculated based on the DWI sequence data.

[0031] Furthermore, in this embodiment, the CT images and MRI data also need to be annotated to determine the location and extent of the primary tumor area and suspicious lymph node areas. The annotation can be performed by an experienced radiologist, or automatically generated based on a pre-trained segmentation model and manually corrected, thereby ensuring the accuracy of the annotated areas.

[0032] Step 202: Extract features from the CT image using the first feature extraction network to obtain the CT feature vector.

[0033] In this embodiment, the CT image can be preprocessed, including image resampling, grayscale normalization, and cropping based on the labeled region, to obtain standardized input data. Subsequently, the preprocessed CT image is input into a first feature extraction network.

[0034] The first feature extraction network can be a convolutional neural network structure, such as a 3D ResNet model structure, used to extract the morphological and structural features, boundary features, and spatial relationship information with surrounding tissues of the tumor region, thereby generating CT feature vectors.

[0035] Step 203: Extract features from the MRI data using the second feature extraction network to obtain MRI feature vectors.

[0036] In one optional implementation, the second feature extraction network is a dual-path network; in this case, the dual-path network can be used to extract features from the MRI data to obtain MRI feature vectors; the MRI data includes diffusion-weighted imaging sequence data and apparent diffusion coefficient image data calculated based on the diffusion-weighted imaging sequence data; and the dual-path network includes a first path and a second path; the first path is used to extract the signal and texture features of the solid part of the tumor in the diffusion-weighted imaging sequence data; the second path is used to extract the cell density heterogeneity within the tumor.

[0037] Specifically, the input data for the first path is a high-b-value image from the DWI sequence data (high-b-value images are more sensitive to water molecule diffusion restriction and can clearly show the boundary, morphology, and signal intensity distribution of the solid tumor region). In the diffusion-weighted imaging sequence of this application embodiment, there are multiple images with different b-values, among which high-b-value images (e.g., images with b-values ​​greater than a threshold) are used to enhance the signal contrast of diffusion-restricted regions, thereby highlighting the solid part of the tumor, and are used to extract the signal intensity and texture features of the tumor.

[0038] The first path employs a lightweight 3D convolutional neural network submodule (which is derived from the 3D ResNet used for CT feature extraction but is adapted to be lightweight and suited to the soft tissue characteristics of MRI images). It includes convolutional layers, pooling layers, and batch normalization layers, which are used to focus on capturing local signal features (such as signal intensity and edge blurring in the solid area of ​​the tumor) and texture features (such as signal uniformity and texture coarseness within the tumor) during the feature extraction process.

[0039] The input data for the second path is apparent diffusion coefficient (ADC) image data. The ADC value directly reflects the diffusion capacity of water molecules within the tissue. The higher the tumor cell density, the more restricted the diffusion of water molecules, and the lower the ADC value. Cell density heterogeneity is reflected in the numerical fluctuations and regional differences on the ADC map.

[0040] The second path employs a model structure consisting of convolutional layers, global pooling layers, and feature quantization layers. The kernel size of the convolutional kernels in the second path is larger than that in the first path, focusing on capturing the global numerical distribution features of the ADC map rather than local signals. The feature quantization layer is used to numerically quantify the ADC features extracted by convolution, outputting quantitative features such as "ADC minimum / maximum / variance / coefficient of variation," which directly characterize cell density heterogeneity (the larger the variance / coefficient of variation, the greater the difference in cell density in different regions within the tumor, and the stronger the invasiveness).

[0041] After the first path outputs the texture feature vector and the second path outputs the cell density heterogeneity quantization feature vector, the texture feature vector (denoted as F_mri1) and the cell density heterogeneity quantization feature vector (denoted as F_mri2) can be fused.

[0042] For example, F_mri1 and F_mri2 can be concatenated along the channel dimension to obtain a joint feature vector that integrates the features of the two types, and the joint feature vector can be used as the final MRI feature vector.

[0043] Optionally, since the sub-vectors of the two paths may have different dimensions (e.g., the semantic feature dimension of path 1 is higher and the quantization feature dimension of path 2 is lower), the concatenated initial vectors can be mapped to a unified dimension through a fully connected layer and batch normalization to eliminate the feature weight imbalance problem caused by the difference in dimensions. Optionally, in this embodiment, a self-attention mechanism can be introduced to assign feature weights to the normalized joint feature vector—automatically increasing the weights of features strongly correlated with esophageal cancer staging (T / N staging) (such as the minimum value of ADC and the fuzziness of tumor edge texture), and reducing the weights of irrelevant features to achieve refined feature fusion.

[0044] Step 204: Standardize and encode the spatial information of the primary tumor area and suspicious lymph node area on the CT image and MRI data to obtain a clinical data vector.

[0045] In this embodiment, spatial information of the primary tumor region and suspicious lymph node region in CT images and MRI data can be extracted, including but not limited to the distance from the upper and lower edges of the tumor to the incisors (for localization), tumor length, regional location coordinates, shape parameters, and relative spatial relationships. Furthermore, the clinical data vector may also include the encoded results of the biopsy pathology type and other pathological features of the target object.

[0046] Subsequently, in this embodiment, the aforementioned spatial information can be numerically encoded (for example, different types of parameters can be directly used as data of different dimensions; non-numerical parameters, such as biopsy pathology types, can be numbered according to type and thus numerically encoded), and then normalized to map them to a unified numerical range to obtain a clinical data vector. At this point, the spatial structural information in the original image is transformed into structured features, enabling it to be fused with deep learning features in the same feature space, thus improving the stability and effectiveness of the fusion process.

[0047] Step 205: Based on the clinical feature vector, the CT feature vector, and the MRI feature vector, generate the corresponding query vector, key vector, and value vector.

[0048] Optionally, in this embodiment, the query vector includes a first query vector and a second query vector; the key vector includes a first key vector and a second key vector; and the value vector includes a first value vector and a second value vector. At this point, the computer device can generate a first query vector and a first key vector based on the clinical feature vector and the CT feature vector; generate a second query vector and a second key vector based on the clinical feature vector and the MRI feature vector; and generate a first value vector and a second value vector based on the correspondence between the CT feature vector and the MRI feature vector.

[0049] Step 206: Calculate the attention weight between the CT feature vector and the MRI feature vector based on the query vector and the key vector to obtain the fused feature vector.

[0050] Optionally, in this embodiment, a first attention weight is obtained based on the first query vector and the second key vector, and the second value vector corresponding to the MRI feature vector is weighted by the first attention weight to obtain a first multimodal vector; then, a second attention weight is obtained based on the second query vector and the first key vector, and the first value vector corresponding to the CT feature vector is weighted by the second attention weight to obtain a second multimodal vector; the first multimodal vector and the second multimodal vector are fused to obtain the fused feature vector.

[0051] The above scheme employs a cross-attention mechanism, enabling CT and MRI features to mutually query and complement each other. This allows for the determination of whether DWI signals indicate high cell density in suspected invasive areas identified by CT features, while MRI features can obtain more precise anatomical localization information from CT features. Clinical features, acting as contextual information, participate in the interaction, allowing the query process between CT and DWI features to consider the clinical characteristics of the target patient, selecting more important and relevant features for querying and fusion.

[0052] In one alternative implementation, a gating vector can be generated based on the first multimodal feature, the second multimodal feature, and the clinical feature vector; then, the gating vector is used to perform element-wise weighted fusion of the first multimodal feature and the second multimodal feature to obtain a fused feature vector.

[0053] When the first multimodal feature and the second multimodal feature are fused element-wise using gating vectors, the neural network that generates the gating vectors is trained. Therefore, the trained neural network can automatically select feature information that is useful for the current task based on the current vector features and assign it a higher weight. This makes the fused feature vector obtained by weighted fusion more accurately express the physiological state of the target object.

[0054] Step 207: Input the fused feature vector into the prediction network for prediction to obtain the prediction result of esophageal cancer staging.

[0055] Optionally, the prediction network in this embodiment is a multi-branch prediction network; the multi-branch prediction network includes a first prediction head and a second prediction head; the first prediction head is used to output the prediction result for stage T; the second prediction head is used to output the prediction result for stage N. That is, by setting multiple prediction heads, the model can simultaneously complete different stage tasks based on the same feature representation, thereby achieving multi-task joint learning and improving the model's comprehensive modeling ability for different stage information.

[0056] Furthermore, the first prediction head and the second prediction head are cascaded; the first prediction head takes the fused feature vector as input and the prediction result of T-stage as output; the second prediction head takes the prediction result of T-stage and the fused feature vector as input and the prediction result of N-stage as output.

[0057] At this point, the T-staging result serves as prior information in the N-staging prediction process, thereby using the degree of local tumor invasion to assist in the judgment of lymph node metastasis risk, enhancing the model's ability to model the intrinsic correlation between stages, and thus improving the accuracy of N-staging prediction.

[0058] In one alternative implementation, the second prediction head can encode the T-stage prediction results, for example, by vectorization or embedding mapping, converting discrete stage labels into continuous feature representations, and then concatenating or weighting them with the fused feature vectors, thus using them as input for subsequent prediction calculations. This approach avoids the information loss caused by directly using discrete labels, allowing the model to more fully utilize the latent semantic features in the T-stage information.

[0059] Furthermore, the multi-branch prediction network also includes a third prediction head; this third prediction head takes the fused feature vector, the prediction result of T stage, and the prediction result of N stage as input; this third prediction head is used to output the treatment suggestion category.

[0060] The above scheme, by introducing a third prediction head, can achieve end-to-end prediction from imaging features to clinical decision-making, enabling the model not only to complete staging tasks, but also to provide auxiliary references for subsequent treatment.

[0061] Furthermore, the third prediction head can employ a multilayer perceptron structure or an attention-based network structure to weight the input features, thereby enhancing the influence of key staging information in the decision-making process. In addition, during model training, a multi-task joint optimization strategy can be adopted, constructing loss functions for T-staging, N-staging, and treatment suggestion tasks respectively, and jointly optimizing them through weighted summation. This allows the model to achieve collaborative learning across multiple tasks, improving overall prediction performance and generalization ability.

[0062] In summary, when staging a target patient preoperatively, the computer system can extract features from the target patient's CT images to obtain CT feature vectors, extract features from the target patient's MRI data to obtain MRI feature vectors, and simultaneously encode and standardize clinical data to obtain clinical feature vectors. Then, the computer system can use the clinical feature vectors as context to fuse the CT and MRI feature vectors, obtaining a fused feature vector. Finally, the fused feature vector is input into a prediction network for prediction, obtaining the predicted staging result for esophageal cancer, which can be used by doctors to assess the patient's condition. This approach utilizes clinical feature vectors as contextual information and fuses CT and MRI feature vectors based on the spatial information of the tumor in both datasets, improving the accuracy of the esophageal cancer staging prediction result obtained by inputting the fused feature vector into the prediction network.

[0063] This application also provides a preoperative staging prediction system for esophageal cancer, which is used to implement the above embodiments and preferred embodiments, and will not be repeated hereafter. As used below, the term "module" can be a combination of software and / or hardware that performs a predetermined function. Although the system described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0064] This application provides a preoperative staging prediction system for esophageal cancer. Figure 3 This is a schematic diagram of the structure of a preoperative staging prediction system for esophageal cancer provided in an embodiment of this application. The device includes: The data acquisition module 301 is used to acquire multimodal feature data; the multimodal feature data includes CT feature vectors, MRI feature vectors, and clinical feature vectors; the CT feature vectors are obtained by feature extraction from the CT images of the target object; the MRI feature vectors are obtained by feature extraction from the MRI data of the target object; the clinical feature vectors are obtained by encoding and standardizing the clinical data of the target object; the clinical data includes the spatial information of the tumor in the CT images and MRI data. The vector fusion module 302 is used to fuse the CT feature vector and the MRI feature vector using the clinical feature vector as context information to obtain a fused feature vector. The staging prediction module 303 is used to input the fused feature vector into the prediction network for prediction to obtain the prediction result of esophageal cancer staging.

[0065] In summary, when staging a target patient preoperatively, the computer system can extract features from the target patient's CT images to obtain CT feature vectors, extract features from the target patient's MRI data to obtain MRI feature vectors, and simultaneously encode and standardize clinical data to obtain clinical feature vectors. Then, the computer system can use the clinical feature vectors as context to fuse the CT and MRI feature vectors, obtaining a fused feature vector. Finally, the fused feature vector is input into a prediction network for prediction, obtaining the predicted staging result for esophageal cancer, which can be used by doctors to assess the patient's condition. This approach utilizes clinical feature vectors as contextual information and fuses CT and MRI feature vectors based on the spatial information of the tumor in both datasets, improving the accuracy of the esophageal cancer staging prediction result obtained by inputting the fused feature vector into the prediction network.

[0066] Please see Figure 4 , Figure 4 This is a schematic diagram of the structure of an electronic device provided in an optional embodiment of the present invention. This electronic device can be a computer device used to execute the above-described method. Figure 4 As shown, the electronic device includes one or more processors 10, a memory 20, and interfaces for connecting the various components, including high-speed interfaces and low-speed interfaces. The various components communicate with each other via different buses and can be mounted on a common motherboard or otherwise installed as needed. The processor can process instructions executed within the electronic device, including instructions stored in or on memory to display graphical information of a GUI on external input / output devices (such as display devices coupled to the interfaces).

[0067] The processor 10 may further include a hardware chip. This hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The programmable logic device may be a complex programmable logic device (CAMP), a field-programmable gate array (FPGA), a general-purpose array logic (GDA), or any combination thereof.

[0068] The memory 20 stores instructions executable by at least one processor 10 to cause the at least one processor 10 to perform the method shown in the above embodiments.

[0069] The memory 20 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created by the use of the electronic device based on the display of a mini-program landing page. Furthermore, the memory 20 may include high-speed random access memory (RAM), and may also include non-transient memory, such as at least one disk storage device, flash memory device, or other non-transient solid-state storage device. The memory 20 may include volatile memory, such as RAM; the memory may also include non-volatile memory, such as flash memory, hard disk, or solid-state drive; the memory 20 may also include combinations of the above types of memory.

[0070] The electronic device also includes a communication interface 30 for communicating with other devices or communication networks.

[0071] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as computer code that can be recorded on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods shown in the above embodiments.

[0072] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0073] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this invention patent should be determined by the appended claims.

Claims

1. A method for preoperative staging prediction of esophageal cancer, characterized in that, The method includes: Acquire multimodal feature data; the multimodal feature data includes CT feature vectors, MRI feature vectors, and clinical feature vectors; the CT feature vectors are obtained by feature extraction from the CT images of the target object; the MRI feature vectors are obtained by feature extraction from the MRI data of the target object; the clinical feature vectors are obtained by encoding and standardizing the clinical data of the target object; the clinical data includes the spatial information of the tumor in the CT images and MRI data; Using the clinical feature vector as contextual information, the CT feature vector and MRI feature vector are fused to obtain a fused feature vector; The fused feature vector is input into the prediction network for prediction to obtain the prediction result of esophageal cancer staging.

2. The method according to claim 1, characterized in that, The step of fusing the CT feature vector and the MRI feature vector using the clinical feature vector as context information to obtain a fused feature vector includes: Based on the clinical feature vector, the CT feature vector, and the MRI feature vector, generate the corresponding query vector, key vector, and value vector; Based on the query vector and the key vector, the attention weight between the CT feature vector and the MRI feature vector is calculated to obtain the fused feature vector.

3. The method according to claim 2, characterized in that, The step of generating corresponding query vectors, key vectors, and value vectors based on the clinical feature vectors, CT feature vectors, and MRI feature vectors includes: Based on the clinical feature vector and the CT feature vector, a first query vector and a first key vector are generated; Based on the clinical feature vector and the MRI feature vector, a second query vector and a second key vector are generated; Generate a first value vector and a second value vector based on the correspondence between CT feature vectors and MRI feature vectors; The step of calculating the attention weight between the CT feature vector and the MRI feature vector based on the query vector and the key vector to obtain the fused feature vector includes: Based on the first query vector and the second key vector, a first attention weight is obtained, and the second value vector corresponding to the MRI feature vector is weighted by the first attention weight to obtain a first multimodal vector; Based on the second query vector and the first key vector, a second attention weight is obtained, and the first value vector corresponding to the CT feature vector is weighted by the second attention weight to obtain a second multimodal vector; The first multimodal vector and the second multimodal vector are fused to obtain the fused feature vector.

4. The method according to claim 3, characterized in that, The step of fusing the first multimodal vector and the second multimodal vector to obtain the fused feature vector includes: Based on the first multimodal feature, the second multimodal feature, and the clinical feature vector, a gating vector is generated; The first multimodal feature and the second multimodal feature are fused element-wise using the gating vector to obtain the fused feature vector.

5. The method according to any one of claims 1 to 4, characterized in that, The acquisition of multimodal feature data includes: Acquire CT images and MRI data; the CT images and MRI data are marked with the primary tumor area and suspicious lymph node areas; The first feature extraction network is used to extract features from the CT image to obtain the CT feature vector; The MRI data is used to extract features through a second feature extraction network to obtain MRI feature vectors; The spatial information of the primary tumor area and suspicious lymph node area on the CT images and MRI data is standardized and encoded to obtain a clinical data vector.

6. The method according to claim 5, characterized in that, The second feature extraction network is a dual-path network; The step of extracting features from MRI data using a second feature extraction network to obtain MRI feature vectors includes: MRI feature vectors are obtained by extracting features from MRI data using a dual-path network; the MRI data includes diffusion-weighted imaging sequence data and apparent diffusion coefficient image data calculated based on the diffusion-weighted imaging sequence data. The dual-path network includes a first path and a second path; the first path is used to extract signal and texture features of the solid portion of the tumor in diffusion-weighted imaging sequence data; the second path is used to extract intratumoral cell density heterogeneity.

7. The method according to any one of claims 1 to 4, characterized in that, The prediction network is a multi-branch prediction network; the multi-branch prediction network includes a first prediction head and a second prediction head; The first prediction head is used to output the prediction result for period T; the second prediction head is used to output the prediction result for period N.

8. The method according to claim 7, characterized in that, The first prediction head and the second prediction head are cascaded; the first prediction head takes the fused feature vector as input and the prediction result of the T-stage as output; The second prediction head takes the prediction result of the T-stage and the fused feature vector as input, and the prediction result of the N-stage as output.

9. The method according to claim 7, characterized in that, The multi-branch prediction network also includes a third prediction head; the third prediction head takes the fused feature vector, the prediction result of T stage, and the prediction result of N stage as input; the third prediction head is used to output the treatment suggestion category.

10. A preoperative staging prediction system for esophageal cancer, characterized in that, The system includes: The data acquisition module is used to acquire multimodal feature data, including CT feature vectors, MRI feature vectors, and clinical feature vectors. The CT feature vectors are obtained by extracting features from the CT images of the target object. The MRI feature vectors are obtained by extracting features from the MRI data of the target object. The clinical feature vectors are obtained by encoding and standardizing the clinical data of the target object. The clinical data includes the spatial information of the tumor in the CT images and MRI data. The vector fusion module is used to fuse the CT feature vector and the MRI feature vector using the clinical feature vector as context information to obtain a fused feature vector. The staging prediction module is used to input the fused feature vector into the prediction network for prediction, and obtain the prediction result of esophageal cancer staging.

Images