A gastric cancer distal metastasis prediction method and system based on multiple examinations

Abstract

The present application belongs to the technical field of intelligent medical treatment, and discloses a stomach cancer distal metastasis prediction method and system based on multiple examinations, which comprises the following steps: first, acquiring the imaging, endoscopy and blood examination data of the same patient; then, mapping each examination original feature to a pre-defined uniform semantic concept set, and converting it into a uniform dimension semantic embedding vector through a modal encoder adapted to the data type; subsequently, constructing a cross-modal semantic alignment module, narrowing the distance between vectors of the same semantics and different sources based on contrast learning; further, detecting semantic conflicts and dynamically allocating the credibility weight of each source, weighting and fusing the multi-source vectors of the same semantics according to the weight to obtain a fused feature vector; finally, inputting the downstream prediction model to output the metastasis prediction result. The present application solves the semantic conflict problem of multi-modal examination data, improves the prediction accuracy and stability, and provides a reliable basis for the clinical treatment decision of stomach cancer.

Description

Technical Field

[0001] This invention relates to the field of intelligent medical technology, specifically to a method and system for predicting distant metastasis of gastric cancer based on multiple examinations. Background Technology

[0002] Gastric cancer, as one of the most common malignant tumors worldwide, has a high incidence of distant metastasis, which is a key factor leading to poor prognosis and treatment failure. Accurate prediction of distant metastasis risk is of significant clinical importance for developing individualized treatment plans and improving patient survival rates. Furthermore, research and application of such predictive technologies can be used in teaching, providing practical cases and technical references for medical education. Currently, clinical practice mainly relies on multimodal data, including imaging examinations, endoscopy, and blood tests, to assess metastasis risk. These examinations provide tumor-related information from different dimensions; for example, imaging examinations can reflect tumor size and distant organ involvement, endoscopy can clarify the depth of tumor invasion and surface morphology, and blood tests can help determine disease progression through tumor markers and inflammatory indicators.

[0003] However, existing technologies for predicting distant metastasis of gastric cancer have significant shortcomings. For example, multimodal examination data often exhibit semantic inconsistencies for the same clinical feature due to differences in detection principles, equipment accuracy, and operational procedures. This includes discrepancies in tumor size measurements between CT and endoscopy, and contradictory descriptions of lymph node metastasis status from different examinations. Traditional prediction methods do not effectively address this issue, simply concatenating the original features before inputting them into the model. This leads to data conflicts, causing structural errors in the prediction results, and resulting in insufficient model stability and cross-center generalization ability.

[0004] Meanwhile, existing technologies lack in-depth mining of the semantic associations of multimodal data, fail to clarify the consistency of clinical significance among different examination features, and cannot dynamically assess the credibility of data from various sources. This makes it difficult for models to distinguish whether data discrepancies represent actual changes in the patient's condition or detection errors, further reducing prediction accuracy and limiting their application value as standardized cases in teaching. Summary of the Invention

[0005] In response, this invention provides a method and system for predicting distant metastasis of gastric cancer based on multiple examinations, so as to at least partially solve the above-mentioned technical problems.

[0006] This invention provides a method for predicting distant metastasis of gastric cancer based on multiple examinations, comprising the following steps:

[0007] S1, acquire multiple medical examination data of the same patient, wherein the medical examination data includes at least imaging examination data, endoscopic examination data and blood examination data;

[0008] S2, map the various original features in the multiple medical examination data to a predefined unified semantic concept set, which contains multiple semantic domains for describing the clinical status of gastric cancer;

[0009] S3 utilizes a modal encoder corresponding to the data type being inspected to convert the original features under each semantic concept after mapping into semantic embedding vectors of a unified dimension.

[0010] S4. Construct a cross-modal semantic alignment module. This module is based on a contrastive learning mechanism to calculate the similarity between semantic embedding vectors that belong to the same semantic concept but come from different medical examination data. It also uses a loss function to shorten the distance between the semantic embedding vectors in the embedding space to achieve cross-modal semantic alignment.

[0011] S5. Based on the output of the cross-modal semantic alignment module, perform semantic conflict detection and credibility estimation; the semantic conflict detection is used to identify whether there is a significant inconsistency between semantic embedding vectors belonging to the same semantic concept but from different sources; the credibility estimation is used to assign a dynamic credibility weight to the semantic embedding vector from each source.

[0012] S6. For the same semantic concept with multiple sources, the corresponding semantic embedding vector is weighted and fused according to the credibility weight to obtain the fused feature vector of the semantic concept.

[0013] S7, the fused feature vector is input into the downstream prediction model, and the prediction result of distant metastasis of gastric cancer is output.

[0014] In one possible embodiment, the construction of the cross-modal semantic alignment module specifically includes:

[0015] Construct positive sample pairs by constructing semantic embedding vectors of the same patient and the same semantic concept from different medical examination data;

[0016] Construct negative sample pairs from semantic embedding vectors of different semantic concepts, or semantic embedding vectors of the same semantic concept from different patients;

[0017] By using a contrastive learning loss function based on cosine similarity, the vectors in positive sample pairs are made closer to each other in the embedding space, while the vectors in negative sample pairs are made farther apart in the embedding space.

[0018] The contrastive learning loss function includes a temperature parameter to adjust the degree of attention given to difficult negative samples.

[0019] In one possible embodiment, the semantic conflict detection is achieved by calculating the cosine distance between semantic embedding vectors from different sources under the same semantic concept. When the cosine distance exceeds a preset conflict threshold, a semantic conflict is determined to exist.

[0020] The credibility estimate is calculated based on one or more of the following factors through a learnable weighted composition function: the mean cosine similarity between the semantic embedding vector and other source vectors, the historical reliability statistic of the source field in the training set, and the quality factor indicated by the predefined metadata of the field.

[0021] The credibility weight is constructed such that when a conflict detection determines that a semantic conflict exists, the credibility weight corresponding to the source of inconsistency will be automatically reduced.

[0022] In one possible embodiment, a correction constant is learned for a specific field of each source, the correction constant being used to characterize the systematic measurement bias of that source on the semantic concept;

[0023] Before weighted fusion, the clinical values ​​mapped by the semantic embedding vectors from the corresponding sources or the semantic embedding vectors themselves are first corrected using the correction constant.

[0024] The fusion process uses the aforementioned confidence weights to perform a weighted average on the corrected values ​​or vectors to obtain the final fused feature vector.

[0025] In one possible embodiment, the training process employs a two-stage training method, specifically as follows:

[0026] In the first stage, the modality encoder and the credibility estimation module are trained using the contrastive learning loss function; in the second stage, the trained modality encoder is frozen, and the downstream prediction model is trained using the fused feature vector as input, and the weighted fusion layer and the correction constant are jointly optimized.

[0027] In one possible embodiment, the predefined unified semantic concept set is achieved by constructing a feature semantic graph that categorizes raw feature fields from different medical examination data into a limited, predefined semantic domain based on their clinical significance.

[0028] In another aspect, this application also provides a gastric cancer distant metastasis prediction system based on multiple examinations, including:

[0029] The data acquisition module is used to acquire multiple medical examination data of the same patient, including at least imaging examination data, endoscopic examination data, and blood examination data.

[0030] The semantic mapping module is used to map the various original features in the multiple medical examination data to a predefined unified semantic concept set, which contains multiple semantic domains for describing the clinical status of gastric cancer.

[0031] The semantic transformation module is used to convert the original features under each semantic concept after mapping into semantic embedding vectors of a unified dimension using a modal encoder corresponding to the data type being inspected.

[0032] The semantic alignment module is used to construct a cross-modal semantic alignment module. This module is based on a contrastive learning mechanism to calculate the similarity between semantic embedding vectors that belong to the same semantic concept but come from different medical examination data, and uses a loss function to shorten the distance between the semantic embedding vectors in the embedding space to achieve cross-modal semantic alignment.

[0033] The semantic conflict detection module is used to perform semantic conflict detection and credibility estimation based on the output of the cross-modal semantic alignment module; the semantic conflict detection is used to identify whether there is a significant inconsistency between semantic embedding vectors belonging to the same semantic concept but from different sources; the credibility estimation is used to assign a dynamic credibility weight to the semantic embedding vector from each source.

[0034] The feature vector fusion module is used to perform weighted fusion of the corresponding semantic embedding vectors of the same semantic concept with multiple sources according to the credibility weight, so as to obtain the fused feature vector of the semantic concept.

[0035] The prediction module is used to input the fused feature vector into the downstream prediction model and output the prediction result of distant metastasis of gastric cancer.

[0036] In another aspect, this application also provides an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the gastric cancer distant metastasis prediction method based on multiple examinations as described above.

[0037] In another aspect, this application provides a computer-readable storage medium having stored thereon computer program instructions that can be executed by a processor to implement the gastric cancer distant metastasis prediction method based on multiple examinations as described above.

[0038] Another aspect of this application provides a computer program product, including a computer program that, when executed by a processor, implements the method for predicting distant metastasis of gastric cancer based on multiple examinations as described above.

[0039] This application constructs a feature semantic map to classify heterogeneous examination features into a unified semantic domain. Combined with contrastive learning, it achieves cross-modal semantic alignment, enabling the model to accurately identify synonymous clinical features across different examinations, overcoming the limitations of traditional techniques that ignore semantic associations. Simultaneously, by designing a dynamic conflict detection and credibility estimation mechanism, it quantifies conflicts based on cosine distance, statistical bias, and time correction. Credibility weights are dynamically allocated through multi-factor weighted synthesis, avoiding misjudgments based on single thresholds and human experience bias. Furthermore, by introducing a source correction constant and a weighted fusion strategy, it eliminates systematic detection errors, outputting a unified and reliable feature vector, significantly improving the predicted AUC value and data stability. The two-stage training strategy balances semantic alignment and prediction performance, enhancing the model's cross-hospital generalization ability, providing more reliable metastasis risk assessment data for clinical practice, and supporting individualized treatment decisions. Attached Figure Description

[0040] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0041] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0042] Figure 1 This is a schematic diagram of a method for predicting distant metastasis of gastric cancer based on multiple examinations, provided in an embodiment of the present invention.

[0043] Figure 2 This is a schematic diagram of the cross-modal semantic alignment process provided in an embodiment of the present invention.

[0044] Figure 3 This is a schematic diagram of the semantic conflict detection process provided in an embodiment of the present invention.

[0045] Figure 4 This is a schematic diagram of the structure of a gastric cancer distant metastasis prediction system based on multiple examinations provided in an embodiment of the present invention.

[0046] Figure 5 This is a schematic diagram of the structure of a device provided in an embodiment of the present invention.

[0047] Figure 6 A comparison diagram of vector distribution before and after cross-modal semantic alignment provided in this application embodiment.

[0048] Figure 7 This is a schematic diagram of a liver slice with metastatic gastric cancer provided in an embodiment of this application. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0050] It should be noted that all user information (including but not limited to user device information, user personal information, object information corresponding to device usage data, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, device usage data, etc.) involved in all embodiments of this disclosure are information and data authorized by the user or fully authorized by all parties.

[0051] The following detailed description, in conjunction with specific embodiments, illustrates the implementation process of the gastric cancer distant metastasis prediction method based on multiple examinations described in this invention. It should be noted that these embodiments are merely illustrative of the invention and not intended to limit its scope. Any conventional adjustments or substitutions made by those skilled in the art to the steps without departing from the inventive concept should be included within the scope of protection of this invention.

[0052] like Figure 1 As shown in the diagram, this invention discloses a method for predicting distant metastasis of gastric cancer based on multiple examinations, including the following steps:

[0053] S1, acquire multiple medical examination data of the same patient, wherein the medical examination data includes at least imaging examination data, endoscopic examination data and blood examination data;

[0054] S2, map the various original features in the multiple medical examination data to a predefined unified semantic concept set, which contains multiple semantic domains for describing the clinical status of gastric cancer;

[0055] S3 utilizes a modal encoder corresponding to the data type being inspected to convert the original features under each semantic concept after mapping into semantic embedding vectors of a unified dimension.

[0056] S4. Construct a cross-modal semantic alignment module. This module is based on a contrastive learning mechanism to calculate the similarity between semantic embedding vectors that belong to the same semantic concept but come from different medical examination data. It also uses a loss function to shorten the distance between the semantic embedding vectors in the embedding space to achieve cross-modal semantic alignment.

[0057] S5. Based on the output of the cross-modal semantic alignment module, perform semantic conflict detection and credibility estimation; the semantic conflict detection is used to identify whether there is a significant inconsistency between semantic embedding vectors belonging to the same semantic concept but from different sources; the credibility estimation is used to assign a dynamic credibility weight to the semantic embedding vector from each source.

[0058] S6. For the same semantic concept with multiple sources, the corresponding semantic embedding vector is weighted and fused according to the credibility weight to obtain the fused feature vector of the semantic concept.

[0059] S7, the fused feature vector is input into the downstream prediction model, and the prediction result of distant metastasis of gastric cancer is output.

[0060] In some embodiments, for step S1, specifically, the core of step S1 is to collect medical examination data covering different detection dimensions generated during the clinical diagnosis and treatment of the same gastric cancer patient, so as to ensure that the data can comprehensively reflect the patient's condition and provide a basis for subsequent semantic processing and prediction.

[0061] For example, the medical examination data includes at least three core data categories: imaging examination data, endoscopic examination data, and blood examination data. The specific acquisition methods and contents of each type of data are as follows:

[0062] Imaging examination data, preferably computed tomography (CT) examination data, is obtained by scanning the patient's stomach and surrounding tissues with a clinical CT device. After generating CT images, the radiologist issues a structured CT report based on the images. The report should include quantitative or qualitative indicators such as tumor size, such as maximum diameter, volume, tumor boundary clarity, number of enlarged lymph nodes, and whether there are suspected metastases in distant organs. Optionally, the imaging examination data may also include magnetic resonance imaging (MRI) examination data, which is obtained in the same way as CT data, only needing to supplement the tissue signal characteristics indicators specific to MRI, such as the tumor signal intensity under T1-weighted and T2-weighted images.

[0063] Endoscopic examination data, such as gastroscopy data, is obtained by endoscopists using gastroscopy equipment to directly observe the gastric mucosa of patients and record the location and surface morphology of tumors, such as the presence of ulcers, erosions, estimated diameter, and depth of invasion, such as whether they have invaded the mucosal layer or muscular layer, to form an endoscopic examination report. This report must include both structured fields, such as "tumor estimated diameter: 4cm" and text summaries, such as "an irregular ulcerative mass was seen in the antrum of the stomach, which is hard in texture and bleeds easily upon touch," so that numerical and textual features can be extracted simultaneously later.

[0064] Blood test data, for example, involves collecting peripheral venous blood from patients using clinical blood collection equipment and sending it to the laboratory for biochemical and hematologic analysis to obtain a blood test report. The report must include tumor markers such as carcinoembryonic antigen (CEA) and carbohydrate antigen (CA19-9), nutritional and inflammatory indicators such as prognostic nutritional index (PNI) and neutrophil-to-lymphocyte ratio (NLR), and liver and kidney function indicators such as alanine aminotransferase (ALT) and creatinine (Cr). It should be noted that the blood test data must record the blood collection time to ensure temporal correlation with imaging and endoscopic examination data, providing a basis for subsequent time correction.

[0065] Understandably, when acquiring the above data, it is necessary to ensure that all data corresponds to the same patient and that the time interval between examinations does not exceed a preset time window, such as 30 days, in order to avoid data loss of relevance due to dynamic changes in the condition. At the same time, it is necessary to conduct preliminary legality verification on the data and remove data with obvious missing information, such as key tumor indicators not being recorded or logical errors, such as data with "tumor size: -5cm", to ensure the validity of the original data.

[0066] In some embodiments, for step S2, according to the present invention, the purpose of step S2 is to classify heterogeneous original features from different examination data into a unified semantic domain by constructing a feature semantic map, clarifying which features should maintain semantic consistency in clinical sense, and laying the foundation for subsequent cross-modal alignment. The implementation process of this step is divided into two parts: "constructing a feature semantic map" and "original feature mapping".

[0067] Specifically, for the feature semantic map, this map is the core carrier of a predefined unified set of semantic concepts. Its construction logic can be based on clinical diagnostic guidelines for gastric cancer, such as the "Chinese Guidelines for the Diagnosis and Treatment of Gastric Cancer (2023 Edition)," dividing all clinical features related to distant metastasis of gastric cancer into multiple semantic domains, or semantic concepts, according to their described "clinical significance." Each semantic domain corresponds to a specific clinical topic. For example, the set of semantic concepts includes the following semantic domains:

[0068] "Tumor spatial scale semantic domain (TumorSize)": used to describe the physical size characteristics of the tumor, clinically reflecting the scale of tumor growth and directly related to the risk of metastasis;

[0069] "Infiltration Depth" semantic domain: used to describe the extent to which the tumor invades the gastric wall tissue. Its clinical significance is to reflect the local progression stage of the tumor and is a key indicator for judging the possibility of metastasis.

[0070] "LymphNodeStatus": This domain describes whether regional lymph nodes are enlarged and metastatic, and its clinical significance is to reflect the spread of tumors in the region.

[0071] "Tumor Marker Composite": This domain describes the levels of blood markers associated with gastric cancer, and its clinical significance is to reflect the biological activity of the tumor.

[0072] "Nutrition Inflammation Semantic Domain": Used to describe the patient's overall nutritional status and inflammation level, with clinical significance reflecting the host's tolerance and resistance to tumors;

[0073] "Distant Metastasis Sign" is used to describe suspected features of distant organ (such as liver or lung) metastasis, and its clinical significance is that it is directly related to the judgment of distant metastasis.

[0074] For the original feature mapping, based on the aforementioned feature semantic map, each original feature obtained in step S1 is mapped to its corresponding semantic domain. The mapping rule uses "consistency between the clinical significance of the original feature and the theme of the semantic domain" as the core criterion. A specific mapping example is as follows:

[0075] The "maximum tumor diameter (3.2cm)" in the CT report and the "estimated tumor diameter (4cm)" in the endoscopy report both describe the physical size of the tumor, and therefore are mapped to the "tumor spatial scale semantic domain".

[0076] Both the "T stage (T3 stage)" in the CT report and the "depth of invasion (invasion of the muscle layer)" in the endoscopy report reflect the degree of tumor invasion of the gastric wall, and therefore are mapped to the "tumor invasion depth semantic domain".

[0077] The "number of enlarged lymph nodes (1)" in the CT report and the "no clearly enlarged lymph nodes" in the endoscopy report both describe the state of lymph nodes, so they are both mapped to the "lymph node state semantic domain".

[0078] For “CEA (5.8ng / mL)” and “CA19-9 (35U / mL)” in the blood test report, both are tumor markers related to gastric cancer, so they are both mapped to the “tumor marker semantic domain”.

[0079] For “PNI (45)” and “NLR (2.8)” in the blood test report, both reflect nutritional and inflammatory status, so they are mapped to the “nutritional inflammatory response semantic domain”.

[0080] The CT report states "A low-density lesion with a diameter of 1 cm is seen in the right lobe of the liver, which is considered to be a possible metastasis." This directly describes the suspected features of distant metastasis, so it is mapped to the "semantic domain of signs of distant metastasis."

[0081] Optionally, during the mapping process, if a certain original feature is associated with multiple semantic domains, such as "tumor surface ulcer" which is related to both tumor morphology and indirectly related to invasion depth, the "primary semantic priority" principle is adopted to map it to the semantic domain that best reflects the core clinical significance of the feature. For example, "tumor surface ulcer" is preferentially mapped to the "tumor spatial scale semantic domain". For features that cannot be clearly classified, such as "patient's complaint of stomach pain", they can be directly removed because they have a weak correlation with distant metastasis prediction and do not participate in subsequent mapping.

[0082] In some embodiments, for step S3, a dedicated modality encoder is designed for different types of original features, such as numerical, textual, and hierarchical features, to ensure that all types of features can be converted into semantic embedding vectors with uniform dimensions, thereby realizing the numericalization and standardization of features and providing a computable vector form for subsequent comparative learning.

[0083] According to an embodiment of the present invention, the design of the modal encoder follows the principle of "feature type adaptation". The encoder structure and workflow corresponding to different inspection data types are as follows:

[0084] 1. Numerical feature encoder, suitable for CT structured indicators and blood test indicators.

[0085] Specifically, this includes the preprocessing stage: First, the original numerical features are normalized, preferably using the z-score normalization method, with the following calculation formula: ,in These are the original values. This is the mean of the feature in the training set. This represents the standard deviation of the feature in the training set. The purpose of normalization is to eliminate differences in the dimensions of different indicators, such as "tumor size: cm" and "CEA: ng / mL", to avoid the impact of differences in numerical range on subsequent vector calculations.

[0086] Encoding Stage: A two-layer fully connected neural network (FCN) combined with layer normalization (LayerNorm) is used to construct the encoder. The network structure is as follows: Input layer (dimension = 1, corresponding to a single numerical feature) → First fully connected layer (output dimension = 2d, where d is the target embedding dimension, preferably d = 64~128) → ReLU activation function → Layer normalization → Second fully connected layer (output dimension = d) → Output semantic embedding vector For example, if the target embedding dimension d=64, for the numerical feature "maximum tumor diameter 3.2cm" in the CT report, after normalization, it is input into the encoder and finally outputs a 64-dimensional vector. .

[0087] 2. Text-based feature encoder, suitable for summarizing endoscopy report text.

[0088] Preprocessing stage: Clean the endoscopic text abstract, including removing special characters such as "、", ";", stop words such as "的", "见", and splitting the text into words or medical terms such as "胃窦部", "溃疡型肿物"; then use pre-trained word vectors in the medical field, such as the word vectors of BioBERT, to perform initial encoding on each term to obtain the word vector of each term, with dimension = 768.

[0089] Encoding stage: Use a lightweight Transformer encoder, preferably a 6-layer encoder, with a hidden layer dimension = 256, to encode the sequence of term word vectors and obtain the global semantic features of the text; then through a projection layer, which can be a fully connected network, project the 256-dimensional feature vector output by the Transformer to the target dimension d, and the calculation formula is: , where is the projection matrix, is the bias term, and finally output the semantic embedding vector ; for example, for the endoscopic text abstract "胃窦部见一不规则溃疡型肿物,质地硬,触之易出血", after encoding, a 64-dimensional vector is obtained.

[0090] 3. Hierarchical feature encoder, applicable to CT T staging and endoscopic invasion depth grading.

[0091] Preprocessing stage: Convert qualitative grading indicators into ordered numerical values. For example, convert CT T staging "T1, T2, T3, T4" to "1, 2, 3, 4", and convert endoscopic invasion depth "黏膜层, 肌层, 浆膜层" to "1, 2, 3"; this conversion is based on the progressive relationship of the clinical staging standard to ensure that the numerical size is consistent with the severity of the disease.

[0092] Encoding stage: Use an embedding layer combined with a fully connected network to construct an encoder. The embedding layer maps the ordered numerical values to low-dimensional vectors, with dimension = 2d, and then projects it to the target dimension d through a fully connected network, while adding a ReLU activation function and layer normalization, and finally output the semantic embedding vector ; taking CT T staging "T3", which is converted to the numerical value 3 as an example, after being processed by the embedding layer and the projection layer, a 64-dimensional vector is obtained.

[0093] It should be noted that the target embedding dimension d of all modal encoders should be consistent, preferably d=64~128, to ensure that similarity calculation can be performed on cross-modal vectors in the future. At the same time, the encoder parameters, such as the weights of the fully connected network and the attention weights of the Transformer, need to be optimized in the subsequent training process. The initial parameters can be randomly initialized or pre-trained parameters. For example, the Transformer parameters of the text encoder can be initialized using the parameters of the medical pre-trained model.

[0094] In some embodiments, step S4 involves constructing a cross-modal semantic alignment module based on contrastive learning to narrow the distance between feature vectors from different sources under the same semantic concept, thereby achieving semantic unification.

[0095] Please see Figure 2 , Figure 2 This is a schematic diagram of the cross-modal semantic alignment process provided in an embodiment of the present invention. According to the embodiment of the present invention, the construction and implementation process of this module is divided into "sample pair construction", "contrastive learning loss function design" and "loss function optimization".

[0096] In S201, positive and negative sample pairs are first constructed. Specifically, the core logic of sample pair construction is "the same semantics are positive, and different semantics are negative". By clearly defining positive and negative sample pairs, training samples are provided for comparative learning.

[0097] Positive sample pair construction: For semantic embedding vectors from different sources under the same patient and the same semantic concept, positive sample pairs are constructed. For example, under the "Tumor Spatial Scale Semantic Domain (TumorSize)," the CT tumor size vector of the same patient... Tumor size vector from endoscopy Since both describe the same clinical feature, they constitute a positive sample pair. If a third source vector exists under this semantic concept, such as an MRI tumor size vector. Then construct simultaneously and Two sets of positive sample pairs.

[0098] Negative sample pair construction: Negative sample pairs are divided into two categories: one is vector pairs of "same patient, different semantic concepts", and the other is vector pairs of "different patients, same semantic concept". Specifically:

[0099] For negative sample pairs with different semantics from the same patient, such as the "tumor spatial scale vector" of the same patient "and lymph node state vector" The two describe different clinical features, thus forming a negative sample pair. ;

[0100] For negative sample pairs with the same semantic meaning from different patients, such as the "tumor spatial scale vector" of patient A. "Tumor spatial scale vector" relative to patient B Although both belong to the same semantic concept, they correspond to different patients (whose conditions may differ), thus forming a negative sample pair. .

[0101] Understandably, when constructing sample pairs, it is necessary to ensure that each positive sample pair can be matched with a sufficient number of negative sample pairs, preferably 10 to 20 times the number of positive sample pairs, in order to ensure the effectiveness of contrastive learning. At the same time, the construction of sample pairs should be based on the data of the same training batch, and the batch size is preferably 64 to 256, in order to balance computational efficiency and learning effect.

[0102] In S202, a contrastive learning loss function is designed. Preferably, this invention adopts an InfoNCE (Information Noise Contrastive Estimation) style contrastive learning loss function, which can achieve vector alignment in the embedding space by maximizing the similarity of positive sample pairs and minimizing the similarity of negative sample pairs. Specifically, the mathematical expression of the loss function and the definition of each parameter are as follows:

[0103] For any positive sample pair ,in For anchor vectors, To and The contrastive loss of semantically consistent positive sample vectors The calculation formula is: ,

[0104] in: Representing vectors and The cosine similarity is calculated using the following formula: This is used to quantify the semantic similarity between two vectors, and its value ranges from 1 to 2. A larger value indicates greater semantic consistency.

[0105] This is a temperature parameter, ranging from 0.05 to 0.2. Its function is to adjust the sensitivity of the loss function to similarity differences: when... When the value is small, such as 0.05, the loss function is more sensitive to differences in similarity and is better able to distinguish pairs of samples with similarity. When the value is relatively large, such as 0.2, the sensitivity of the loss function decreases, which helps to avoid overfitting; in this invention, it is preferred that... To balance sensitivity and generalization ability;

[0106] Anchor vector The negative sample pool contains all samples from this batch that are related to... The vector that constitutes a negative sample pair, i.e. ;

[0107] Represents anchor vector The sum of the similarity indices of all negative sample vectors, Represents anchor vector With positive sample vector The similarity index, the entire score term reflects the "relative similarity advantage" of positive sample pairs among all sample pairs. The loss function forces the relative similarity advantage of positive sample pairs to be maximized by minimizing this log score.

[0108] Furthermore, to prevent all semantic vectors from collapsing to the same point in the embedding space, i.e., reducing the discriminative power of different semantic vectors, a semantic preservation term (Separation Term) needs to be added to the loss function. , used to push out vectors with different semantic concepts. The calculation formula is:

[0109] ,in, The number of semantic concepts, such as the aforementioned 6 semantic domains, then ); , Semantic concepts , The average vector under this semantic meaning is the mean of all sample vectors under this semantic meaning. For the preset similarity threshold, the preferred method is... , which represents the maximum similarity allowed for different semantic vectors;

[0110] When the similarity between different semantic vectors exceeds hour, A positive output contributes to the loss function, forcing the model to reduce the similarity between different semantic vectors; when the similarity is lower than... When this condition is met, the output of this item is 0, and it does not contribute.

[0111] Finally, the global loss function for cross-modal semantic alignment To compare the weighted sum of the loss and the semantic preservation term, the calculation formula is as follows:

[0112]

[0113] in, Represents all semantic concepts; Representing semantic concepts The set of all positive sample pairs below; The weight coefficients for semantic preservation terms are preferred. This is used to balance the contributions of contrastive loss and semantic preservation term, preventing any one loss term from dominating the training process.

[0114] In S203, the loss function is iteratively optimized using forward and backward propagation algorithms. Specifically, the optimization process involves updating the modal encoder parameters using the backward propagation algorithm to optimize the global loss. Minimize, thereby achieving cross-modal semantic alignment. For example... Figure 6 As shown, Figure 6 This is a comparison diagram of vector distribution before and after cross-modal semantic alignment provided in an embodiment of this application. The implementation steps are as follows:

[0115] 1. Initialize parameters: For modal encoders, all learnable parameters, such as fully connected network weights, Transformer attention weights, and embedding layer parameters, can be initialized, including numerical, textual, and hierarchical types. For text encoders, Transformer parameters can be initialized using medical pre-trained models, such as BioBERT-based parameters. Other parameters are initialized using He normality and are suitable for ReLU activation functions.

[0116] 2. Forward Propagation Loss Calculation: Batch data in the training set, containing multimodal features of the same patient, are processed sequentially through steps S1-S3 to obtain semantic embedding vectors under each semantic concept. Positive and negative sample pairs are constructed and substituted into... The formula calculates the global loss for the current batch;

[0117] 3. Backpropagation to update parameters: The Adaptive Momentum Optimizer (AdamW) is used to calculate the gradient of the loss function with respect to each parameter. The gradient calculation adopts an automatic differentiation framework, such as PyTorch's autograd. Then, the parameters are updated according to the gradient. The update formula is as follows:

[0118]

[0119] in Let be the parameters at time t. These are the updated parameters at time t+1. For the learning rate, the optimal value is selected. , For loss function pairs The gradient;

[0120] 4. Iterative Training: Repeat steps 2-3 above to perform multiple rounds of iterative training on the training set, preferably 50-100 rounds. After each round of training, calculate the results on the validation set. When the validation set loss no longer decreases for 5 consecutive rounds, training is stopped, and the modal encoder parameters at this point are saved as the fixed encoder for subsequent steps.

[0121] In some embodiments, step S5 resolves the inconsistency problem of multimodal data by quantifying semantic conflicts and dynamically allocating credibility weights, thus providing a reliable basis for subsequent feature fusion.

[0122] According to embodiments of the present invention, the purpose of semantic conflict detection is to identify whether there is "significant inconsistency" between different source vectors under the same semantic concept, that is, to determine whether the differences between vectors exceed a reasonable range. The present invention uses a combination of the "cosine distance method" and the "statistical bias method" to achieve conflict detection. Please refer to [link to relevant documentation]. Figure 3 , Figure 3 This is a schematic diagram of the semantic conflict detection process provided in an embodiment of the present invention. The specific process is as follows:

[0123] In S301, the cosine distance between different source vectors is calculated, specifically, for the same patient and the same semantic concept. The two source vectors below , such as the source of CT scans, and If the source is an endoscope, first calculate the cosine distance between the two. The relationship between cosine distance and cosine similarity is as follows: The range of values ​​is , The larger the value, the greater the semantic difference between the two vectors;

[0124] In S302, a predefined conflict threshold is defined. A preliminary assessment is made to determine whether there is a semantic conflict, specifically... The value range is 0.25~0.35, preferably... ;like If the two vectors are found to have a semantic conflict, it is preliminarily determined that there is a conflict. For example, the cosine distance between the CT tumor size vector and the endoscopic tumor size vector is 0.32, which is greater than the threshold of 0.3, so it is preliminarily determined that there is a conflict.

[0125] In S303, secondary verification is performed using the statistical bias method. Specifically, to avoid misjudgments caused by a single threshold judgment, such as accidental anomalies in a source vector, the statistical bias method is introduced for secondary verification. Specifically, the average bias of "source pair (a→b)" under the same semantic concept is pre-estimated in the training set. with standard deviation The average deviation The "vector" of all patients in the training set Corresponding clinical value with vector Corresponding clinical value The mean of the difference, i.e. Standard deviation This is the standard deviation of the difference; for the current patient, calculate its... actual difference ,like Exceeding ,in , where is the confidence coefficient, and is preferred. If the second determination corresponds to the 95% confidence interval, then there is a semantic conflict.

[0126] In S304, time correction is performed based on the examination time interval. Specifically, considering that the examination time interval may lead to data discrepancies, such as tumor growth within one month, a time correction mechanism is preferably introduced. Specifically, for each semantic concept... Predefined time-sensitive window For example, the semantic domain of "tumor spatial scale". Heaven, the semantic domain of "hematologic tumor markers" If the inspection time difference between the two data sources is... If so, the conflict threshold should be appropriately relaxed, such as by making Adjusted from 0.3 to 0.4, The value can be adjusted from 2 to 2.5, or the difference can be directly determined as a "reasonable time change" and not considered a semantic conflict.

[0127] In S305, the final conflict determination is made by combining the above steps. Specifically, if "cosine distance > threshold" and "actual difference exceeds confidence interval" and "time difference ≤ time sensitive window", then a semantic conflict is determined to exist; otherwise, it is determined to be without conflict.

[0128] According to embodiments of the present invention, the purpose of credibility estimation is to assign a dynamic credibility weight to the semantic embedding vector from each source. The value range is (0,1), and a larger weight indicates that the source data is more reliable. According to an embodiment of the present invention, the credibility weight is calculated by "multi-factor weighted synthesis".

[0129] For example, credibility weight It consists of three core factors, which assess the reliability of the source from three dimensions: "vector consistency", "historical reliability", and "metadata quality".

[0130] Consistency factor The consistency calculation is based on the current source vector and other source vectors under the same semantic concept, and the formula is as follows: ,in The number of sources under this semantic concept. For the current source vector, For other source vectors; The range of values ​​is Normalized to (0,1) by the sigmoid function, i.e. If a semantic conflict exists, It will decrease significantly, leading to This reduces the credibility of the source.

[0131] Historical reliability factor The correlation between this source data and the "true label of distant metastasis of gastric cancer" is calculated based on the historical performance of this source in the training set, specifically using the Pearson correlation coefficient. Measures such as the correlation coefficient between "CT tumor size" and metastasis labels, Its absolute value is taken to ensure non-negativity, and its range is [0,1]. For example, if the correlation coefficient between CT tumor size and metastasis label in the training set is 0.6, then the source... ;

[0132] Metadata quality factor The calculation is based on the metadata attributes of the source field. The metadata includes "detection error range," "time sensitivity," and "equipment accuracy level," etc. For example, the "detection error range" is divided into "low error, such as CT tumor size error ±0.2cm," "medium error, such as endoscopic tumor size error ±0.5cm," and "high error, such as blood NLR error ±0.3cm," respectively corresponding to... , , "Time sensitivity" is divided into "low sensitivity (e.g., CTT staging)" and "high sensitivity (e.g., blood CEA)," corresponding to... Add 0.1, subtract 0.1; final This is a weighted sum of the scores for each attribute, with a value range of [0,1].

[0133] The three normalized factors are weighted and fused using learnable weight parameters to obtain the final credibility weight. The formula is:

[0134]

[0135] in, , , These are the weighting coefficients of the three factors. These are bias terms, all of which are learnable parameters and need to be optimized during training. For the sigmoid function, ensure The output range is (0,1); the core logic of this formula is to learn the contribution of each factor to "credibility" through training data. For example, if the "consistency factor" has the greatest impact on reliability in the training set, then... It will be optimized to a larger value;

[0136] Parameters of the credibility estimation module ( , , , The parameters of the modal encoder are trained synchronously, with the training objective being to minimize the cross-modal semantic alignment loss. Specifically, during each backpropagation, not only are the encoder parameters updated, but the parameters of the credibility estimation module are also updated, enabling the model to learn the parameter combination that best reflects the "correlation between factors and credibility".

[0137] In some embodiments, the purpose of step S6 is to fuse vectors from multiple sources under the same semantic concept into a unified feature vector, thereby eliminating semantic conflicts and preserving the valid information from each source. According to embodiments of the present invention, this step employs a fusion strategy of "bias correction + weighted averaging".

[0138] Specifically, due to systematic errors inherent in different examination equipment or detection methods—for example, endoscopy generally estimates tumor size better than CT—it is necessary to first correct for biases in the vectors from each source to eliminate systematic biases. This includes:

[0139] Learn the source correction constant; specifically, learn a correction constant for a specific field of each source. , Used to characterize the systematic measurement bias of a source under a certain semantic concept; for example, for endoscopic sources in the "tumor spatial scale semantic domain", This indicates the average overestimation bias of the endoscopic tumor size relative to the "true tumor size," such as... ;

[0140] The learning method for the correction constant, exemplarily, The parameters of the modal encoder and the credibility estimation module are trained synchronously, with the training objective being to minimize the cross-modal semantic alignment loss. Specifically, in calculating the clinical value corresponding to the semantic embedding vector... When doing so, the correction constant must be subtracted first. ,Right now ,in The mapping function from vector to clinical value is preferably a single-layer fully connected network that projects a d-dimensional vector into a 1-dimensional scalar, as shown in the formula. , , Through training, It will be optimized to minimize the difference in clinical values ​​after correction from different sources;

[0141] Alternatively, if the downstream prediction model directly uses vector features instead of clinical values, then the semantic embedding vectors can be modified. Perform correction; the correction formula is as follows: ,in This is the projection matrix, used to convert scalar correction constants. Project onto a d-dimensional vector space to ensure that the corrected vectors still maintain semantic consistency.

[0142] After bias correction, a weighted average of the corrected vector or clinical values ​​is calculated using confidence weights to obtain the fused feature vector. Specifically, there are two fusion scenarios:

[0143] Scenario 1, clinical value fusion, is applicable to numerical semantic concepts.

[0144] For semantic concepts that represent clinical significance in numerical form, such as the "semantic domain of tumor spatial scale" and the "semantic domain of tumor markers," the fusion process is as follows:

[0145] Calculate the corrected clinical value for each source. ;

[0146] Based on credibility weight Calculate the weighted average fusion value ( (For semantic concepts), the formula is: ,

[0147] in This refers to the set of all sources under this semantic concept; for example, under the "tumor spatial scale semantic domain" there is CT, where... , , and endoscopy, among which , Two sources, then the fusion value ;

[0148] Vectorization of fused values: converting fused values Transformed into a d-dimensional vector through the embedding layer. As the fused feature vector of this semantic concept, the embedding layer formula is: ,in This is a pre-trained embedding layer that shares parameters with the embedding layer of the hierarchical feature encoder.

[0149] Scenario 2: Direct vector fusion, which is suitable for textual or complex semantic concepts.

[0150] For text summaries or semantic concepts containing multi-dimensional information, such as the "tumor invasion depth semantic domain," which includes both numerical staging and textual description, the corrected semantic embedding vectors are directly fused, as follows:

[0151] Calculate the corrected vector for each source. Based on credibility weight Calculate the weighted average fusion vector The formula is:

[0152] For example, the "tumor invasion depth semantic domain" contains CT vectors. ,That , and endoscope vector , Then the fusion vector ;

[0153] It should be noted that during the fusion process, if there is only one source for a certain semantic concept, such as the "semantic domain of distant metastasis signs" which only contains CT data, then the corrected vector from that source is directly used as the fused feature vector, with the confidence weight... .

[0154] In some embodiments, for step S7, the fused feature vector is processed by a downstream prediction model to output the predicted probability or classification result of distant metastasis of gastric cancer.

[0155] Specifically, firstly, the feature vectors after fusing all semantic concepts are concatenated to form a global feature vector. Specifically, if there exists... There are 1 semantic concept, and the feature vector after fusion of each semantic concept has a dimension of 1. Then the dimension of the global feature vector is ;For example, (The aforementioned 6 semantic domains) Then the dimension of the global feature vector is , recorded as ,in This indicates transpose.

[0156] Preferably, the downstream prediction model employs a hybrid multimodal Transformer (HMMT) model, which can effectively capture the relationships between different semantic concepts and improve prediction accuracy. Alternatively, convolutional neural networks (CNN), gradient boosting trees (XGBoost), or other models can also be used; the specific choice depends on the data scale and prediction requirements. The following uses the HMMT model as an example to illustrate the model construction process:

[0157] HMMT Model Structure: The HMMT model consists of a multi-layer Transformer encoder and a classification head. The specific structure is as follows:

[0158] Input layer: Receives global feature vectors ;

[0159] Location embedding layer: Adds location embedding to the vector of each semantic concept. The positional embedding is used to distinguish the order information of different semantic concepts. It employs sinusoidal positional encoding, with the formula:

[0160] ,in Index for semantic concepts ( ), Indexing the vector dimensions ( );

[0161] Transformer encoder layer: A 6-layer Transformer encoder is used, each layer containing a multi-head self-attention mechanism and a feed-forward network.

[0162] Multi-head self-attention mechanism: divides the input vector into... Head (preferred) Each head independently calculates attention weights to capture the relationships between different semantic concepts, using the following formula:

[0163]

[0164] in The query, key, and value are all global feature vectors with added position embeddings. For each head dimension, To output the projection matrix;

[0165] Feedforward network: Employs a two-layer fully connected network with a ReLU activation function added in between, as shown in the formula:

[0166]

[0167] in , , , ;

[0168] Classification Head: Employs a two-layer fully connected network to process the output (dimensions) of the Transformer encoder. Transformed through Global Average Pooling The dimensional vector is then passed through two fully connected layers to output the predicted probability, as shown in the formula: , ; ;

[0169] in The first output of the Transformer encoder A semantic concept vector, , , , , This represents the predicted probability of distant metastasis of gastric cancer, with a value ranging from (0,1).

[0170] Next, based on the model structure constructed above, model training and prediction are performed, specifically including:

[0171] Training data preparation: Divide the clinical data into training set, validation set and test set in a ratio of 7:2:1. The training set is used for model parameter update, the validation set is used for hyperparameter tuning and early termination judgment, and the test set is used to evaluate the final performance of the model. All data must include "fused feature vector" and "true label of distant metastasis of gastric cancer", where 1 indicates the presence of metastasis and 0 indicates no metastasis.

[0172] Training data example and annotation structure, for example, Table 1 below is an example of the original data structure (patient ID: GC2024001, after desensitization).

[0173] Table 1

[0174] Inspection type Feature fields Original value Inspection time Unit / Explanation CT imaging Maximum diameter of tumor 3.2 2024-01-10 cm T installment T3 2024-01-10 - Number of enlarged lymph nodes 2 2024-01-10 indivual low-density lesions in the liver yes 2024-01-10 1.2cm in diameter Endoscopy Tumor estimated diameter 4.0 2024-01-12 cm Immersion depth muscle layer 2024-01-12 - Surface morphology Ulcerative type 2024-01-12 - Blood test CEA 8.5 2024-01-09 ng / mL CA19-9 42.3 2024-01-09 U / mL PNI 43.2 2024-01-09 - NLR 3.1 2024-01-09 - Authentic Labels Remote transfer 1 - 1 = Existence, 0 = Non-existence

[0175] The annotation structure example is as follows (sensitized):

[0176] Patient basic information

[0177] project content Patient ID GC2024001 Check time window 3 days (meets the ≤30 days requirement)

[0178] Clinical characteristic data

[0179] (1) Tumor size

[0180] Check the source Feature terms numerical values unit CT Maximum diameter of tumor 3.2 cm Endoscopy Tumor estimated diameter 4.0 cm

[0181] (2). Infiltration Depth

[0182] Check the source Feature terms Original value Mapping value CT T installment T3 3 Endoscopy Immersion depth muscle layer 2

[0183] (3) Lymph Node Status

[0184] Check the source Feature terms numerical values unit CT Number of enlarged lymph nodes 2 indivual

[0185] (4) Tumor Marker Composite

[0186] Check the source Feature terms numerical values unit blood CEA 8.5 ng / mL blood CA19-9 42.3 U / mL

[0187] (5) Nutritional inflammation markers

[0188] Check the source Feature terms numerical values unit blood PNI 43.2 — blood NLR 3.1 —

[0189] (6) Distant Metastasis Sign

[0190] Check the source Feature terms numerical values describe CT low-density lesions in the liver yes 1.2cm in diameter

[0191] Gold Standard (Ground Truth)

[0192] project content Distant transfer Yes (1) Transfer site Liver Diagnosis method Pathological biopsy

[0193] Loss function selection: Due to class imbalance in distant metastasis data of gastric cancer, with metastatic samples typically accounting for less than 20%, the weighted cross-entropy loss function is preferred. The formula is:

[0194] ,in For the sample size, For the first The true label of each sample To predict probabilities, Here are the weighting coefficients, where , Number of samples without transfer The number of samples to be transferred is used to balance the loss contribution of positive and negative samples;

[0195] Model training: A two-stage training strategy is adopted, which is connected with the alignment training in step S4. Specifically,

[0196] The first stage, also known as alignment training, involves training only the modality encoder and the credibility estimation module, and then optimizing... The number of training rounds is 50-100.

[0197] The second stage, or prediction fine-tuning, involves freezing the modal encoder parameters or setting a small learning rate. Fine-tuning was performed, training the Transformer encoder and classification head of the HMMT model, and optimization was performed. At the same time, a regularization term is added to the fusion layer correction constant. (The formula is) , Total loss The training run consists of 30-80 rounds, employing an early stopping strategy, with the validation set... If the score does not drop for 5 consecutive rounds, stop training.

[0198] Prediction and Output: Input the fused feature vectors from the test set or new patients into the trained HMMT model, and output the predicted probabilities. ;like The threshold can be adjusted according to clinical needs. For example, if high sensitivity is desired, the threshold can be set to 0.3, in which case the prediction result will be "existence of distant gastric cancer metastasis"; otherwise, the prediction result will be "no distant gastric cancer metastasis". Simultaneously, the prediction probability is output. The confidence interval of the fusion results provides a reference for clinical decision-making; for example, such as Figure 7 As shown, Figure 7 The diagram provided in this application provides a schematic image of a slice confirming distant metastasis to the liver. The image shows a round nodule in the S6 segment of the liver, measuring 1.3 × 1.1 cm. The pathological morphology of the nodule is consistent with that of the primary gastric cancer, indicating that the cancer cells spread from the stomach to the liver, rather than being a primary cancer growing in the liver itself.

[0199] It is understood that, during the implementation of this invention, all patients' medical examination data have been anonymized, such as by removing personal identification information like names and ID numbers, in accordance with relevant laws and regulations and to protect patients' privacy.

[0200] Please see Figure 4 , Figure 4 This is a schematic diagram of the structure of a gastric cancer distant metastasis prediction system 5 based on multiple examinations provided in an embodiment of this application. Figure 4 As shown, system 400 includes:

[0201] The data acquisition module 401 is used to acquire multiple medical examination data of the same patient, wherein the medical examination data includes at least imaging examination data, endoscopic examination data and blood examination data.

[0202] The semantic mapping module 402 is used to map the various original features in the multiple medical examination data to a predefined unified semantic concept set, which contains multiple semantic domains for describing the clinical status of gastric cancer.

[0203] The semantic conversion module 403 is used to convert the original features under each semantic concept after mapping into semantic embedding vectors of a unified dimension using a modal encoder corresponding to the data type being inspected.

[0204] The semantic alignment module 404 is used to construct a cross-modal semantic alignment module. This module is based on a contrastive learning mechanism to calculate the similarity between semantic embedding vectors that belong to the same semantic concept but come from different medical examination data, and uses a loss function to shorten the distance between the semantic embedding vectors in the embedding space to achieve cross-modal semantic alignment.

[0205] The semantic conflict detection module 405 is used to perform semantic conflict detection and credibility estimation based on the output of the cross-modal semantic alignment module; the semantic conflict detection is used to identify whether there is a significant inconsistency between semantic embedding vectors that belong to the same semantic concept but come from different sources; the credibility estimation is used to assign a dynamic credibility weight to the semantic embedding vector from each source.

[0206] The feature vector fusion module 406 is used to perform weighted fusion of the corresponding semantic embedding vectors of the same semantic concept with multiple sources according to the credibility weight, so as to obtain the fused feature vector of the semantic concept.

[0207] The prediction module 407 is used to input the fused feature vector into the downstream prediction model and output the prediction result of distant metastasis of gastric cancer.

[0208] Those skilled in the art will clearly understand that the technical solutions of the embodiments of this application can be implemented by means of software and / or hardware. In this specification, "unit" and "module" refer to software and / or hardware that can independently complete or cooperate with other components to complete a specific function, wherein the hardware may be, for example, a field-programmable gate array (FPGA), an integrated circuit (IC), etc.

[0209] Each processing unit and / or module in the embodiments of this application can be implemented by an analog circuit that implements the functions described in the embodiments of this application, or by software that executes the functions described in the embodiments of this application.

[0210] Please see Figure 5 It shows a schematic diagram of the structure of an electronic device according to an embodiment of this application, which can be used to implement... Figure 1 The method in the illustrated embodiment. (As shown) Figure 5 As shown, the electronic device 500 may include:

[0211] The system includes at least one processor 501, at least one network interface 504, a user interface 503, a memory 505, and at least one communication bus 502. The communication bus 502 is used to enable connection and communication between the components. The user interface 503 may include buttons, and optionally include a standard wired or wireless interface. The network interface 504 may include, but is not limited to, a Bluetooth module, an NFC module, a Wi-Fi module, etc.

[0212] The processor 501 may include one or more processing cores and connect to various parts within the device 500 via various interfaces and lines. It implements the various functions and data processing of the device 500 by running or executing instructions, programs, code sets, or instruction sets stored in the memory 505, and by accessing data in the memory 505. Optionally, the processor 501 may be implemented using at least one hardware form of DSP, FPGA, or PLA. The processor 501 may also integrate one or more combinations of CPU, GPU, and modem. The CPU is mainly used to handle the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content required for display on the screen; and the modem is used for wireless communication. It is understood that the modem may not be integrated into the processor 501, but may be implemented through a separate chip.

[0213] Memory 505 may include random access memory (RAM) or read-only memory (ROM). Optionally, memory 505 includes a non-transitory computer-readable medium for storing instructions, programs, code, code sets, or instruction sets. Memory 505 may be divided into a program storage area and a data storage area, wherein the program storage area may be used to store instructions for implementing an operating system, instructions for implementing at least one function (such as touch functionality, audio playback functionality, image playback functionality, etc.), and instructions for implementing the aforementioned method embodiments; the data storage area may be used to store data involved in the relevant method embodiments. Memory 505 may also be at least one storage device located remotely from processor 501. Figure 5 As shown, the memory 505, which serves as a computer storage medium, may contain an operating system, a network communication module, a user interface module, and program instructions.

[0214] In particular, the methods and / or embodiments in this application can be implemented as computer software programs. For example, the embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowchart. When the computer program is executed by processor 501, it performs the functions defined in the methods of this application.

[0215] Another embodiment of this application provides a computer-readable storage medium having computer program instructions stored thereon, which can be executed by a processor to implement the methods and / or technical solutions of any one or more embodiments of this application described above.

[0216] The computer-readable storage medium may include, but is not limited to, any type of disk, including floppy disks, optical disks, DVDs, CD-ROMs, microdrives, as well as magneto-optical disks, ROMs, RAMs, EPROMs, EEPROMs, DRAMs, VRAMs, flash memory devices, magnetic cards or optical cards, nanosystems (including molecular memory ICs), or any type of medium or device suitable for storing instructions and / or data.

[0217] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

Claims

1. A method for predicting distant metastasis of gastric cancer based on multiple examinations, characterized in that, The methods and steps include the following: Acquire multiple medical examination data of the same patient, wherein the medical examination data includes at least imaging examination data, endoscopic examination data, and blood examination data; The original features in the various medical examination data are mapped to a predefined unified semantic concept set, which contains multiple semantic domains for describing the clinical status of gastric cancer. Using a modal encoder corresponding to the data type being inspected, the original features under each semantic concept after mapping are converted into semantic embedding vectors of a unified dimension. A cross-modal semantic alignment module is constructed. This module is based on a contrastive learning mechanism to calculate the similarity between semantic embedding vectors that belong to the same semantic concept but come from different medical examination data. It also uses a loss function to narrow the distance between the semantic embedding vectors in the embedding space to achieve cross-modal semantic alignment. Based on the output of the cross-modal semantic alignment module, semantic conflict detection and credibility estimation are performed; the semantic conflict detection is used to identify whether there are significant inconsistencies between semantic embedding vectors that belong to the same semantic concept but come from different sources. The credibility estimation is used to assign a dynamic credibility weight to the semantic embedding vector of each source; The semantic conflict detection includes: calculating the cosine distance between semantic embedding vectors from different sources under the same semantic concept; performing a preliminary conflict determination when the cosine distance exceeds a preset conflict threshold; performing a secondary verification based on the average deviation and standard deviation of source pairs under the same semantic concept in the training set to check whether the actual difference between the clinical values ​​corresponding to vectors from different sources for the current patient exceeds the confidence interval; and performing time correction on the inspection time interval of data from different sources based on a predetermined time-sensitive window for each semantic concept; determining that a semantic conflict exists when the cosine distance exceeds the preset conflict threshold, the actual difference exceeds the confidence interval, and the inspection time difference is not greater than the time-sensitive window. For the same semantic concept with multiple sources, a correction constant is learned for a specific field of each source. The correction constant is used to characterize the systematic measurement bias of that source on the semantic concept. Before weighted fusion, the clinical value mapped by the semantic embedding vector of the corresponding source or the semantic embedding vector itself is first corrected using the correction constant. The fusion process uses the aforementioned confidence weights to perform a weighted average on the corrected values ​​or vectors to obtain the final fused feature vector; The fused feature vector is input into the downstream prediction model, and the prediction result of distant metastasis of gastric cancer is output.

2. The method for predicting distant metastasis of gastric cancer based on multiple examinations according to claim 1, characterized in that, The construction of the cross-modal semantic alignment module specifically includes: Construct positive sample pairs by constructing semantic embedding vectors of the same patient and the same semantic concept from different medical examination data; Construct negative sample pairs from semantic embedding vectors of different semantic concepts, or semantic embedding vectors of the same semantic concept from different patients; By using a contrastive learning loss function based on cosine similarity, the vectors in positive sample pairs are made closer to each other in the embedding space, while the vectors in negative sample pairs are made farther apart in the embedding space. The contrastive learning loss function includes a temperature parameter to adjust the degree of attention given to difficult negative samples.

3. The method for predicting distant metastasis of gastric cancer based on multiple examinations according to claim 2, characterized in that, The credibility estimate is calculated based on one or more of the following factors through a learnable weighted composition function: the mean cosine similarity between the semantic embedding vector and other source vectors, the historical reliability statistic of the source field in the training set, and the quality factor indicated by the predefined metadata of the field. The credibility weight is constructed such that when a conflict detection determines that a semantic conflict exists, the credibility weight corresponding to the source of inconsistency will be automatically reduced.

4. The method for predicting distant metastasis of gastric cancer based on multiple examinations according to claim 3, characterized in that, The training process employs a two-stage approach, specifically as follows: In the first stage, the modality encoder and the credibility estimation module are trained using the contrastive learning loss function; in the second stage, the trained modality encoder is frozen, and the downstream prediction model is trained using the fused feature vector as input, and the weighted fusion layer and the correction constant are jointly optimized.

5. A method for predicting distant metastasis of gastric cancer based on multiple examinations according to any one of claims 1-4, characterized in that, The predefined unified semantic concept set is achieved by constructing a feature semantic map, which classifies the original feature fields from different medical examination data into a limited, predefined semantic domain according to their clinical significance.

6. A gastric cancer distant metastasis prediction system based on multiple examinations, characterized in that, include: The data acquisition module is used to acquire multiple medical examination data of the same patient, including at least imaging examination data, endoscopic examination data, and blood examination data. The semantic mapping module is used to map the various original features in the multiple medical examination data to a predefined unified semantic concept set, which contains multiple semantic domains for describing the clinical status of gastric cancer. The semantic transformation module is used to convert the original features under each semantic concept after mapping into semantic embedding vectors of a unified dimension using a modal encoder corresponding to the data type being inspected. The semantic alignment module is used to construct a cross-modal semantic alignment module. This module is based on a contrastive learning mechanism to calculate the similarity between semantic embedding vectors that belong to the same semantic concept but come from different medical examination data, and uses a loss function to shorten the distance between the semantic embedding vectors in the embedding space to achieve cross-modal semantic alignment. The semantic conflict detection module is used to perform semantic conflict detection and credibility estimation based on the output of the cross-modal semantic alignment module; the semantic conflict detection is used to identify whether there is a significant inconsistency between semantic embedding vectors that belong to the same semantic concept but come from different sources; The credibility estimation is used to assign a dynamic credibility weight to the semantic embedding vector of each source; The semantic conflict detection includes: calculating the cosine distance between semantic embedding vectors from different sources under the same semantic concept; performing a preliminary conflict determination when the cosine distance exceeds a preset conflict threshold; performing a secondary verification based on the average deviation and standard deviation of source pairs under the same semantic concept in the training set to check whether the actual difference between the clinical values ​​corresponding to vectors from different sources for the current patient exceeds the confidence interval; and performing time correction on the inspection time interval of data from different sources based on a predetermined time-sensitive window for each semantic concept; determining that a semantic conflict exists when the cosine distance exceeds the preset conflict threshold, the actual difference exceeds the confidence interval, and the inspection time difference is not greater than the time-sensitive window. The feature vector fusion module is used to learn a correction constant for a specific field of each source for the same semantic concept that exists from multiple sources. The correction constant is used to characterize the systematic measurement bias of the source on the semantic concept. Before weighted fusion, the clinical values ​​mapped by the semantic embedding vectors from the corresponding sources or the semantic embedding vectors themselves are first corrected using the correction constant. The fusion process uses the aforementioned confidence weights to perform a weighted average on the corrected values ​​or vectors to obtain the final fused feature vector; The prediction module is used to input the fused feature vector into the downstream prediction model and output the prediction result of distant metastasis of gastric cancer.

7. An electronic device, characterized in that, include: At least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-5.

8. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, The computer program instructions can be executed by a processor to implement the method as described in any one of claims 1-5.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-5.

Images