Gastric cancer prediction system based on fusion of ultrasonic image and molecular omics

Abstract

The invention relates to the technical field of gastric cancer risk prediction, in particular to a gastric cancer prediction system based on fusion of ultrasonic images and molecular omics, which comprises a data input device, a data output device, a processor and a memory, when the program instruction is executed by the processor, the processor executes the following steps: acquiring a transabdominal two-dimensional ultrasonic image of a patient, and acquiring an image block token; the method comprises the following steps: acquiring an RNA sample of a patient, and acquiring a pathway token; performing cross-modal alignment and fusion on the image block token and the path token to obtain fusion features; and constructing a prediction model, and inputting the fusion features into the prediction model to obtain probability distribution of a prognosis risk value, lymph node metastasis and tumor staging. According to the invention, prognosis risk prediction, lymph node metastasis discrimination and staging prediction can be completed at the same time based on the transabdominal two-dimensional ultrasonic image and the RNA sample of the patient, mechanism-level interpretation and non-invasive risk tracking can be realized, and the method is suitable for intelligent prognosis evaluation of gastric cancer and other solid tumors.

Description

Technical Field

[0001] This invention relates to the field of gastric cancer risk prediction technology, and in particular to a gastric cancer prediction system based on the fusion of ultrasound imaging and molecular omics. Background Technology

[0002] Gastric cancer is one of the most common malignant tumors worldwide, and early detection and molecular subtyping are crucial for precision treatment and prognostic assessment. Traditional imaging prediction mainly relies on morphological information such as ultrasound, CT, or MRI, but cannot reflect the molecular pathway activity of the tumor. While molecular omics testing (such as RNA sequencing) can reveal changes in biological pathways, it requires invasive sampling and is costly. Existing prediction methods are mostly based on a single modality, such as using convolutional neural networks (CNNs) to analyze images or fully connected networks to analyze omics features, lacking cross-modal correlation modeling. This leads to a "semantic gap" between imaging and molecular information, limiting predictive performance and interpretability. Therefore, there is an urgent need for a multimodal fusion method that can establish an interpretable mapping relationship between spatial features of imaging and molecular pathway features to achieve non-invasive, personalized gastric cancer risk assessment and prognostic prediction. Summary of the Invention

[0003] To address the shortcomings of existing technologies, this invention provides a gastric cancer prediction system based on the fusion of ultrasound imaging and molecular omics.

[0004] To achieve the above objectives, this invention provides a gastric cancer prediction system based on the fusion of ultrasound imaging and molecular omics, comprising a data input device, a data output device, a processor, and a memory. The memory includes a computer-readable storage medium storing a computer program. The computer program includes program instructions, which, when executed by the processor, cause the processor to perform the following steps: acquiring a transabdominal two-dimensional ultrasound image of the patient and obtaining an image block token; acquiring an RNA sample of the patient and obtaining a pathway token; performing cross-modal alignment and fusion of the image block token and the pathway token to obtain a fusion feature; constructing a prediction model and inputting the fusion feature into the prediction model to obtain a prognostic risk value, as well as the probability distribution of lymph node metastasis and tumor stage. This invention, based on a patient's transabdominal two-dimensional ultrasound image and RNA sample, can simultaneously complete prognostic risk prediction, lymph node metastasis identification, and stage prediction, and can achieve mechanism-level interpretation and non-invasive risk tracking, making it suitable for intelligent prognostic assessment of gastric cancer and other solid tumors.

[0005] Optionally, acquiring the patient's transabdominal two-dimensional ultrasound image and obtaining the image block token includes the following steps: Two-dimensional transabdominal ultrasound images of the patient were acquired and preprocessed to obtain high-quality ultrasound images. The high-quality ultrasound images are divided into multiple image blocks of the same size, and an embedded representation of each image block is extracted using a self-supervised visual Transformer. The embedded representation of all the image blocks is input to the image branch Transformer encoder, and the image block token is output.

[0006] Optionally, obtaining the patient's RNA sample and acquiring the pathway token includes the following steps: RNA samples from patients were obtained and sequenced and quality controlled to obtain gene expression matrices. The gene expression matrix was aggregated pathway by pathway based on a biological pathway database, and the initial pathway token sequence was obtained by dimensionality reduction using a multilayer perceptron. The initial path token sequence is input into the path branch Transformer encoder, and the path token is output.

[0007] Optionally, the transabdominal two-dimensional ultrasound image and the RNA sample are time-registered using the patient's unique identification number, and the sampling time difference between the transabdominal two-dimensional ultrasound image and the RNA sample is less than 14 days.

[0008] Optionally, the image branch Transformer encoder and the path branch Transformer encoder share the Transformer backbone structure.

[0009] Optionally, the step of performing cross-modal alignment and fusion of the image block token and the path token to obtain fused features includes the following steps: Calculate the semantic similarity between the image block token and the path token to obtain the similarity matrix of image features and path features, and combine contrastive learning to achieve modality alignment; After modality alignment, the image features and the pathway features are fused through a cross-attention mechanism to obtain the fused features.

[0010] Optionally, the semantic similarity satisfies the following relationship: in, The semantic similarity between the p-th path token and the i-th image block token is used. For the p-th path token, Let be the token of the i-th image block.

[0011] Optionally, the fusion features satisfy the following relationship: Where Z is the fusion feature, Q is the query vector, K is the key vector, V is the value vector, d is the vector dimension, and T represents the transpose operation; the path token is used as Q, and the image block token is used as K and V.

[0012] Optionally, it also includes: mapping the attention weights of the cross-attention mechanism back to the original image space of the transabdominal two-dimensional ultrasound image to generate a pathway image heatmap.

[0013] Optionally, the step of constructing a prediction model and inputting the fused features into the prediction model to obtain prognostic risk values, as well as the probability distributions of lymph node metastasis and tumor stage, includes the following steps: The prediction model is built based on the Transformer architecture and includes three task heads: prognostic risk prediction, lymph node metastasis prediction, and tumor staging prediction. The fusion features are input into the prediction model to obtain the prognostic risk value, as well as the probability distribution of lymph node metastasis and tumor stage.

[0014] The present invention has at least the following beneficial effects: 1. This method constructs pathway-level molecular expressions as pathway tokens, segments ultrasound images into image block tokens, achieves cross-modal alignment through contrastive learning, and establishes semantic guidance between pathways and images through cross-attention mechanism, forming interpretable multimodal representations. Thus, the prediction model can simultaneously complete prognostic risk prediction, lymph node metastasis identification, and staging prediction.

[0015] 2. This method maps the attention weights of the cross-attention mechanism back to the original image space to generate a pathway image heatmap, thereby achieving mechanism-level interpretation.

[0016] 3. This method can update pathway tokens through plasma testing during the follow-up phase, enabling non-invasive risk tracking and expanding the application of the model in efficacy evaluation and long-term monitoring. It is suitable for intelligent prognostic assessment of gastric cancer and other solid tumors. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic diagram of the framework of a gastric cancer prediction system based on the fusion of ultrasound imaging and molecular omics, according to an embodiment of the present invention. Figure 2 This is a schematic diagram of a gastric cancer prediction process based on the fusion of ultrasound imaging and molecular omics, according to an embodiment of the present invention. Detailed Implementation

[0019] Specific embodiments of the present invention will now be described in detail. It should be noted that the embodiments described herein are for illustrative purposes only and are not intended to limit the invention. In the following description, numerous specific details are set forth in order to provide a thorough understanding of the invention. However, it will be apparent to those skilled in the art that these specific details are not necessary to practice the invention. In other instances, well-known circuits, software, or methods have not been specifically described to avoid obscuring the invention.

[0020] Throughout this specification, references to "an embodiment," "an embodiment," "an example," or "an example" mean that a particular feature, structure, or characteristic described in connection with that embodiment or example is included in at least one embodiment of the invention. Therefore, the phrases "in an embodiment," "in an embodiment," "an example," or "an example" appearing in various places throughout the specification do not necessarily refer to the same embodiment or example. Furthermore, specific features, structures, or characteristics can be combined in one or more embodiments or examples in any suitable combination and / or sub-combination. Moreover, those skilled in the art will understand that the illustrations provided herein are for illustrative purposes and are not necessarily drawn to scale.

[0021] It should be noted in advance that, in one alternative embodiment, apart from being described independently, the same symbols or letters appearing in all formulas have the same meaning and value.

[0022] In one optional embodiment, please refer to Figure 1 The present invention provides a gastric cancer prediction system based on the fusion of ultrasound imaging and molecular omics, including a data input device 1, a data output device 2, a processor 3 and a memory 4.

[0023] Data input device 1 can include common human-computer interaction interface devices such as keyboards, mice, and touch screens. Through data input device 1, medical staff can input data including, but not limited to, transabdominal two-dimensional ultrasound images of patients, gene expression matrices, biological pathway data, and patient inpatient medical record numbers, which are stored in memory 4 for easy retrieval by processor 3. Data output device 2 is used to present the results of system processing and analysis; it can be a monitor, printer, charting device, etc. Through data output device 2, the system can display predicted results for reference by medical staff or patients to assist in decision-making and communication. Processor 3 is the core component of the system, responsible for executing computer program instructions and performing data processing and analysis. Processor 3 analyzes the data input from data input device 1 or stored in memory 4 by running pre-programmed algorithms and models. Processor 3 can be a central processing unit (CPU), graphics processing unit (GPU), or other dedicated processing unit. Memory 4 is used to store the computer programs, data, and parameters required by the system; it can include random access memory (RAM) for temporary data storage and processing, and persistent storage (such as hard disks or solid-state drives) for long-term data storage and preservation.

[0024] The memory 4 includes a computer-readable storage medium storing a computer program. The computer program includes program instructions that, when executed by the processor 3, cause the processor 3 to perform the following steps: S1. Obtain the patient's transabdominal two-dimensional ultrasound image and obtain the image block token.

[0025] Step S1 specifically includes the following steps: S11. Obtain the patient's transabdominal two-dimensional ultrasound image and perform preprocessing to obtain high-quality ultrasound images.

[0026] Specifically, in this embodiment, an ultrasound device is used to acquire a transabdominal two-dimensional ultrasound image of the patient, and the data is input to the data input device 1 via a USB interface. After receiving the transabdominal two-dimensional ultrasound image, the system stores it in the memory 4 for easy access by the processor 3. The transabdominal two-dimensional ultrasound image can be any one of B-mode, CDFI (Color Doppler Flow Imaging) image, or CEUS (Contrast-Enhanced Ultrasound) image, and is stored in DICOM (Digital Imaging and Communications in Medicine) format. Among them, B-mode, i.e., brightness modulation mode, is the most basic and commonly used mode in ultrasound imaging. It displays the morphology and structure of tissues in the form of grayscale images by using the difference in echo intensity of different tissues; CDFI, i.e., color Doppler flow imaging, can display the direction, velocity, and distribution of blood flow using color coding on the basis of two-dimensional ultrasound images, helping to determine the blood perfusion status of organs or tissues; CEUS, i.e., contrast-enhanced ultrasound, enhances the ultrasound signal by injecting ultrasound contrast agent into the patient's body, more clearly displaying the microvascular perfusion and hemodynamic characteristics of tissues, and improving the detection rate and diagnostic accuracy of lesions.

[0027] Furthermore, preprocessing operations are performed on the transabdominal two-dimensional ultrasound images to improve image quality and obtain high-quality ultrasound images. These preprocessing operations include resolution adjustment, brightness adjustment, contrast adjustment, pseudo-color processing, and noise suppression. Resolution adjustment unifies the images to a uniform resolution; brightness adjustment alters the overall brightness of the image, making details more clearly visible; contrast adjustment enhances the grayscale differences between different tissues or structures, improving image contrast and facilitating the differentiation of different tissue components; pseudo-color processing converts grayscale images to color images, using different colors to represent different grayscale values, making the images more intuitive; and noise suppression removes noise interference from the image, improving the signal-to-noise ratio and making the image clearer and smoother. These preprocessing operations are all existing technologies and will not be described in further detail here.

[0028] S12. Divide the high-quality ultrasound image into multiple image blocks of the same size, and use a self-supervised visual Transformer to extract the embedding representation of each image block.

[0029] Specifically, in this embodiment, the high-quality ultrasound image is divided into image blocks of size 16×16 pixels. The obtained image blocks are then input into a pre-trained self-supervised vision Transformer, specifically DINO-ViT (DIstillation with NO labels for Vision Transformer), to extract the embedding representation of each image block, i.e., the embedding vector v, with an embedding dimension of 384.

[0030] S13. Input the embedding representation of all the image blocks into the image branch Transformer encoder, and output the image block token.

[0031] Specifically, in this embodiment, the embedded representation obtained in step S12 is input to the Transformer encoder branch of the image to learn global spatial dependencies and output image patch tokens, i.e., image patch features. Then, batch normalization or adaptive instance normalization can be used to process the image patch features to ensure consistent feature distribution.

[0032] Tumor regions exhibit heterogeneity, with variations in cell morphology and tissue structure across different areas. In other alternative embodiments, clustering can characterize this heterogeneity. The cluster center features represent typical characteristics of different sub-regions. To enhance the characterization of tumor heterogeneity and aid in understanding it, K-means clustering (K=50) can be used to aggregate image patch features from the same case. The cluster center features are defined as follows: in, For the features of the k-th cluster center, Let be the set of all image patch features in the k-th cluster. for The number of image patch features in the image. Let be the feature of the i-th image patch in the k-th cluster. By clustering the image patch features and extracting the cluster center features, we can highlight the representative features of different tumor sub-regions while reducing image feature redundancy, providing a more stable and biologically meaningful input representation for subsequent cross-modal alignment and fusion of image features and pathway features.

[0033] S2. Obtain the patient's RNA sample and obtain the pathway token.

[0034] Step S2 specifically includes the following steps: S21. Obtain the patient's RNA sample and perform sequencing and quality control to obtain the gene expression matrix.

[0035] Specifically, in this embodiment, the patient's RNA sample includes preoperative tissue ribonucleic acid (RNA) samples or plasma circulating cell-free ribonucleic acid (cfRNA) samples. First, RNA or cfRNA is extracted from the patient's tissue samples (such as surgically removed tissue) or plasma samples to obtain RNA or cfRNA samples. Then, an RNA-seq (RNA sequencing) process, including FastQC quality control, Trimmomatic shearing, STAR alignment, and FeatureCounts quantification, is used to obtain the raw gene expression matrix. Each element in the matrix represents the expression level of the corresponding gene in the corresponding sample. Finally, the raw gene expression matrix is ​​input into data input device 1. After receiving the raw gene expression matrix, the system stores it in memory 4 for easy access by processor 3. When using the gene expression matrix, processor 3 will sequentially process the raw gene expression matrix... Transformation and Z-score normalization are performed to ensure consistent distribution across different samples, resulting in the final gene expression matrix. Here, z represents the gene expression count in the original gene expression matrix.

[0036] Furthermore, by using the patient's unique identification number, such as the patient's inpatient medical record number, time registration between transabdominal two-dimensional ultrasound images and RNA samples can be achieved, ensuring that the sampling time difference between transabdominal two-dimensional ultrasound images and RNA samples is less than 14 days. A database is constructed to store high-quality ultrasound images and gene expression matrices, and an image-molecule pairing index table is established in the database, using the patient's inpatient medical record number as the primary key, to achieve precise correspondence between images and gene samples.

[0037] S22. The gene expression matrix is ​​aggregated pathway by pathway based on the biological pathway database, and the initial pathway token sequence is obtained by dimensionality reduction through a multilayer perceptron.

[0038] Specifically, in this embodiment, a pathway-gene mask matrix is ​​constructed based on either the KEGG (Kyoto Encyclopedia of Genes and Genomes) or Reactome biological pathway database. If genes If it belongs to pathway p, then Otherwise, it is 0. For each pathway, based on the pathway-gene mask matrix, genes belonging to that pathway are selected from the gene expression matrix, and the expression levels of these genes in each sample are aggregated to obtain the expression vector of that pathway in each sample. This expression vector reflects the overall expression level of that pathway in the corresponding sample. The expression vector of each pathway satisfies the following relationship: in, Let p be the expression vector of pathway p. The set of genes for pathway p. for The number of genes in For genes The gene expression count value, where P is the total number of pathways.

[0039] Using a multilayer perceptron (MLP) Dimensional compression and nonlinear mapping are performed to obtain the initial pathway token. This process can be represented by the following relation: in, Let p be the initial pathway token, and ReLU be the activation function. For the Sigmoid function, This is the weight matrix of the first layer in an MLP. This is the weight matrix of the second layer in an MLP. This is the bias vector of the first layer in the MLP. This is the bias vector for the second layer in the MLP.

[0040] After obtaining the initial channel tokens for each channel, the initial channel token sequence can be obtained. , .

[0041] S23. Input the initial path token sequence into the path branch Transformer encoder and output the path token.

[0042] Specifically, in this embodiment, the initial path token sequence is input into the path branch Transformer encoder. A multi-head self-attention mechanism is used to model the functional dependencies between paths, outputting a path token sequence, thereby obtaining the path tokens for each path, i.e., the path features. This process can be represented by the following relationship: in, For the path token sequence, Q is the query vector, K is the key vector, V is the value vector, d is the vector dimension, and T represents the transpose operation.

[0043] Furthermore, the path branch Transformer encoder adopts a 2-4 layer stacked structure, with each layer containing a multi-head attention network, a feedforward network, and a layer normalization module. The number of attention heads is set to 8, and the hidden layer dimension is 256. In addition, the image branch Transformer encoder shares the Transformer backbone structure with the path branch Transformer encoder.

[0044] S3. Perform cross-modal alignment and fusion on the image block token and the path token to obtain fused features.

[0045] Step S3 specifically includes the following steps: S31. Calculate the semantic similarity between the image block token and the path token, thereby obtaining the similarity matrix of image features and path features, and combine contrastive learning to achieve modality alignment.

[0046] Specifically, in this embodiment, the semantic similarity between the image block token and the path token satisfies the following relationship: in, Let p be the semantic similarity between the p-th path token and the i-th image block token. For the p-th path token, Let be the token of the i-th image block. The pair of image block tokens with the highest similarity to the pathway token is denoted as a positive sample, and the semantic similarity of the positive samples is denoted as . , Let p be the most relevant image block token; all other pairs of image block tokens and the path token are denoted as negative samples, and the semantic similarity of the negative samples is denoted as . , The image block token for the j-th negative sample.

[0047] Calculated Then, firstly, using the path as the row index and the image block as the column index, use... A similarity matrix is ​​constructed between image features and pathway features. Then, contrastive learning is used to bring semantically related pathway-image features closer together and suppress irrelevant pairings, thereby optimizing the similarity matrix and achieving modality alignment. The contrastive learning loss function is: in, This is a temperature parameter, and its value ranges from 0.05 to 0.2.

[0048] S32. After modal alignment, the image features and the pathway features are fused through a cross-attention mechanism to obtain the fused features.

[0049] Specifically, in this embodiment, Cross-Attention is used to achieve semantic guidance from the path to the image, resulting in fused features. The fused features Z satisfy the following relationship: Furthermore, the attention weights of the cross-attention mechanism are mapped back to the original image space of the transabdominal 2D ultrasound image. Different pathways generate heatmaps of different colors, which are superimposed on the transabdominal 2D ultrasound image to generate pathway image heatmaps, which are then displayed on data output device 2 for doctors to view. By generating pathway image heatmaps, an interpretable mapping from image space to molecular mechanisms is achieved, providing doctors with mechanism-level auxiliary judgment. The pathway image heatmap can be represented by the following relationship: in, For pathway image heatmap, The Cross-Attention weights reflect the degree of attention that pathway p pays to image patch i. This represents the original pixel value or feature response of image block i at coordinates (x, y).

[0050] In other alternative embodiments, bidirectional Cross-Attention can also be used, and KL divergence constraints can be employed to force the feature distribution of the pathway and the image to remain consistent, thereby improving the stability and generalization of feature alignment.

[0051] S4. Construct a prediction model and input the fusion features into the prediction model to obtain the prognostic risk value, as well as the probability distribution of lymph node metastasis and tumor stage.

[0052] Step S4 specifically includes the following steps: S41. Construct the prediction model based on the Transformer architecture. The prediction model includes three task heads: prognostic risk prediction, lymph node metastasis prediction, and tumor staging prediction.

[0053] Specifically, in this embodiment, transabdominal two-dimensional ultrasound images and RNA samples are collected from multiple patients. The ultrasound image data and RNA samples are paired using the patient's unique identification number, and the sampling time difference is controlled to be less than 14 days. Then, the corresponding fusion features are obtained according to the methods described in steps S1 to S3. Simultaneously, clinical label data for each patient is collected, including patient survival outcome information, lymph node metastasis status, and tumor stage information. Survival outcome information includes the patient's survival time and survival status, used to enable the model to output a risk score consistent with survival data patterns; tumor stage can be obtained based on clinical pathological examination results. The fusion features, survival outcome information, lymph node metastasis status, and tumor stage information of the same patient are treated as a single data set. Multiple sets of data are used to form a dataset, which is pre-stored in memory 4.

[0054] Furthermore, a prediction network is constructed based on the Transformer architecture. The input to the prediction network is the fused features, which are encoded by an encoder to obtain a high-level semantic feature representation. The encoder is formed by stacking multiple Transformer encoding layers. Each Transformer encoding layer includes a multi-head self-attention module, a feedforward neural network module, and a layer normalization module. Typically, the encoder can have 2 to 4 Transformer encoding layers, with 4 to 8 attention heads in the multi-head self-attention module, and the hidden layer feature dimension is 128 to 512. The high-level semantic feature representation output by the encoder is pooled before entering the output layer.

[0055] The output layer of the prediction network includes three task heads: prognostic risk prediction, lymph node metastasis prediction, and tumor stage prediction. These heads output the prognostic risk value, the probability distribution of lymph node metastasis, and the tumor stage, respectively. The prediction results of the three task heads are displayed through data output device 2. Specifically, the prognostic risk prediction task head uses a fully connected layer and linear activation to directly output the prognostic risk value, and the loss function is negative log-likelihood loss. The task head for lymph node metastasis prediction uses a fully connected layer and a sigmoid activation function to output the probability distribution of lymph node metastasis, specifically including probability distributions for metastatic and non-metastatic categories. The loss function is binary cross-entropy. Tumor staging prediction uses a fully connected layer and a softmax activation function to output the probability distribution of tumor staging, and the loss function is cross-entropy. Specifically, it predicts the probability distribution of tumor stages (T, primary tumor size), (N, lymph node metastasis), and (M, distant metastasis), using cross-entropy as the loss function. .

[0056] The loss function of the prediction model satisfies the following equation: Where L is the total loss, for The weight, for The weight, for The weight.

[0057] Temperature scaling can be used for probability calibration of the predicted probability distributions of lymph node metastasis and tumor stage. The predicted prognostic risk values ​​can be calibrated using a confidence network. Since these calibration methods are existing technologies, the specific calibration process will not be described in detail here.

[0058] Furthermore, the dataset stored in memory 4 beforehand is divided into a training set and a validation set in a 7:3 ratio to complete the training and validation of the prediction network and obtain the prediction model.

[0059] S42. Input the fusion features into the prediction model to obtain the prognostic risk value, as well as the probability distribution of lymph node metastasis and tumor stage.

[0060] During the follow-up period, patients only need to provide a plasma cfRNA sample to update the pathway token sequence, which enables personalized efficacy monitoring and recurrence risk prediction without repeat tissue biopsies.

[0061] Furthermore, after obtaining the prediction results, time-risk trend graphs and time-probability trend graphs can be plotted based on the obtained prognostic risk values, the probability distribution of lymph node metastasis, and the probability distribution of tumor stage, respectively, for postoperative efficacy assessment and recurrence risk monitoring.

[0062] It should be noted that in some cases, the actions described in the specification can be performed in different orders and still achieve the desired results. In this embodiment, the order of steps is given only to make the embodiment clearer and easier to explain, and not to limit it.

[0063] In summary, this invention has at least the following beneficial effects: This method constructs pathway-level molecular expressions into pathway tokens, segments ultrasound images into image block tokens, achieves cross-modal alignment through contrastive learning, and establishes semantic guidance between pathways and images using a cross-attention mechanism, forming interpretable multimodal representations. This allows for simultaneous prognostic risk prediction, lymph node metastasis identification, and staging prediction through a predictive model. Furthermore, this method maps the attention weights of the cross-attention mechanism back to the original image space to generate pathway image heatmaps, thereby achieving mechanism-level interpretation. Finally, this method allows for updating pathway tokens through plasma testing during the follow-up phase, enabling non-invasive risk tracking and expanding the model's application in efficacy evaluation and long-term monitoring. It is suitable for intelligent prognostic assessment of gastric cancer and other solid tumors.

[0064] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can understand its implications. Those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be covered within the scope of the claims and specification of the present invention.

Claims

1. A gastric cancer prediction system based on the fusion of ultrasound imaging and molecular omics, characterized in that, The system includes a data input device, a data output device, a processor, and a memory. The memory includes a computer-readable storage medium storing a computer program. The computer program includes program instructions that, when executed by the processor, cause the processor to perform the following steps: Acquire two-dimensional transabdominal ultrasound images of the patient and obtain image block tokens; Obtain RNA samples from patients and acquire pathway tokens; Cross-modal alignment and fusion are performed on the image block token and the path token to obtain fused features; A prediction model is constructed, and the fused features are input into the prediction model to obtain the prognostic risk value, as well as the probability distribution of lymph node metastasis and tumor stage.

2. The gastric cancer prediction system based on the fusion of ultrasound imaging and molecular omics according to claim 1, characterized in that, The process of acquiring the patient's transabdominal two-dimensional ultrasound image and obtaining the image block token includes the following steps: Two-dimensional transabdominal ultrasound images of the patient were acquired and preprocessed to obtain high-quality ultrasound images. The high-quality ultrasound images are divided into multiple image blocks of the same size, and an embedded representation of each image block is extracted using a self-supervised visual Transformer. The embedded representation of all the image blocks is input to the image branch Transformer encoder, and the image block token is output.

3. The gastric cancer prediction system based on the fusion of ultrasound imaging and molecular omics according to claim 2, characterized in that, Obtaining the patient's RNA sample and acquiring the pathway token includes the following steps: RNA samples from patients were obtained and sequenced and quality controlled to obtain gene expression matrices. The gene expression matrix was aggregated pathway by pathway based on a biological pathway database, and the initial pathway token sequence was obtained by dimensionality reduction using a multilayer perceptron. The initial path token sequence is input into the path branch Transformer encoder, and the path token is output.

4. The gastric cancer prediction system based on the fusion of ultrasound imaging and molecular omics according to claim 1, characterized in that: The transabdominal two-dimensional ultrasound image and the RNA sample are time-registered using the patient's unique identification number, and the sampling time difference between the transabdominal two-dimensional ultrasound image and the RNA sample is ensured to be less than 14 days.

5. A gastric cancer prediction system based on the fusion of ultrasound imaging and molecular omics according to claim 3, characterized in that: The image branch Transformer encoder and the path branch Transformer encoder share the Transformer backbone structure.

6. The gastric cancer prediction system based on the fusion of ultrasound imaging and molecular omics according to claim 1, characterized in that, The process of performing cross-modal alignment and fusion of the image block token and the path token to obtain fused features includes the following steps: Calculate the semantic similarity between the image block token and the path token to obtain the similarity matrix of image features and path features, and combine contrastive learning to achieve modality alignment; After modality alignment, the image features and the pathway features are fused through a cross-attention mechanism to obtain the fused features.

7. A gastric cancer prediction system based on the fusion of ultrasound imaging and molecular omics according to claim 6, characterized in that, The semantic similarity satisfies the following relationship: in, The semantic similarity between the p-th path token and the i-th image block token is used. For the p-th path token, Let be the token of the i-th image block.

8. A gastric cancer prediction system based on the fusion of ultrasound imaging and molecular omics according to claim 7, characterized in that, The fusion features satisfy the following relationship: Where Z is the fusion feature, Q is the query vector, K is the key vector, V is the value vector, d is the vector dimension, and T represents the transpose operation; the path token is used as Q, and the image block token is used as K and V.

9. A gastric cancer prediction system based on the fusion of ultrasound imaging and molecular omics according to claim 6, characterized in that, Also includes: The attention weights of the cross-attention mechanism are mapped back to the original image space of the transabdominal two-dimensional ultrasound image to generate a pathway image heatmap.

10. A gastric cancer prediction system based on the fusion of ultrasound imaging and molecular omics according to claim 1, characterized in that, The process of constructing a prediction model and inputting the fused features into the prediction model to obtain prognostic risk values, as well as the probability distributions of lymph node metastasis and tumor stage, includes the following steps: The prediction model is built based on the Transformer architecture and includes three task heads: prognostic risk prediction, lymph node metastasis prediction, and tumor staging prediction. The fusion features are input into the prediction model to obtain the prognostic risk value, as well as the probability distribution of lymph node metastasis and tumor stage.

Images