A multi-modal gastric cancer risk prediction method and system based on a gated attention mechanism
By employing a multimodal gastric cancer risk prediction method based on a gated attention mechanism, the problem of insufficient prediction stability under fluctuations in image and text quality is solved, achieving a more accurate and stable gastric cancer risk assessment.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIJING INFORMATION SCI & TECH UNIV
- Filing Date
- 2026-04-16
- Publication Date
- 2026-07-14
AI Technical Summary
Existing technologies lack stability in multimodal gastric cancer risk prediction under conditions of fluctuating image and text quality, making it difficult to coordinate the contribution relationship between images and text, thus affecting prediction reliability.
A multimodal gastric cancer risk prediction method based on gating attention mechanism is adopted. By performing self-attention and cross-attention calculations on image features and text features, gating fusion coefficients are generated, and the contributions of different modalities are dynamically adjusted and weighted fusion is performed to improve prediction stability.
It enhances the accuracy and stability of gastric cancer risk prediction and improves information focusing ability by dynamically adjusting the contribution of modal features.
Smart Images
Figure CN122392940A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of gastric cancer risk prediction technology, and in particular to a multimodal gastric cancer risk prediction method and system based on a gated attention mechanism. Background Technology
[0002] With the development of medical imaging equipment, digital health records, and multimodal feature fusion technology, data processing methods for individual gastric cancer risk assessment have gradually evolved from single image analysis or single text modeling to joint modeling of abdominal CT images, gastric organ identification results, and health text information. This type of technology provides a richer data foundation for gastric cancer risk prediction by simultaneously utilizing organ structure information, regional cue information, and semantic information.
[0003] The main problem with existing related technologies is that multimodal fusion lacks a mechanism to handle differences in the input quality of information from different modalities. In particular, it is difficult to coordinate the contribution relationship between images and text under conditions of fluctuating image quality and inaccurate text information, which leads to unstable risk representation and affects the reliability of gastric cancer risk prediction. Summary of the Invention
[0004] In view of the aforementioned existing problems, the present invention is proposed.
[0005] Therefore, this invention provides a multimodal gastric cancer risk prediction method based on a gated attention mechanism, which solves the problem of insufficient stability in multimodal gastric cancer risk prediction under conditions of fluctuation in image and text quality.
[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution: In a first aspect, the present invention provides a multimodal gastric cancer risk prediction method based on a gated attention mechanism, which includes acquiring abdominal CT images, gastric organ recognition results and health text information corresponding to a target sample, and performing preprocessing to obtain standardized abdominal CT images, standardized gastric organ recognition results and standardized health text information to form a target multimodal sample; Feature fusion is performed on standardized abdominal CT images and standardized gastric organ recognition results from target multimodal samples to construct organ prior enhanced images; Feature extraction was performed on the enhanced organ prior image and the standardized health text information to obtain image features and text features respectively; Self-attention output and cross-attention output are calculated from image features and text features. Based on the self-attention entropy value and cross-attention peak value, gated fusion coefficients for the image part and the text part are generated respectively. Based on the gating fusion coefficients of the image and text parts, the self-attention output and cross-attention output are weighted and fused respectively to obtain the gating fusion features of the image and text parts; The self-attention output and cross-attention output are calculated based on the gated fusion features of the image part and the text part respectively, and then added in equal proportions to obtain the fusion features of the image part and the text part. Classification based on fusion features determines the risk of gastric cancer in target samples.
[0007] As a preferred embodiment of the multimodal gastric cancer risk prediction method based on gated attention mechanism described in this invention, the specific steps for forming the target multimodal sample are as follows: Obtain the abdominal CT image and gastric organ recognition result corresponding to the target sample. Perform image standardization processing on the abdominal CT image to form a standardized abdominal CT image. Perform image standardization processing on the gastric organ recognition result to form a standardized gastric organ recognition result. Obtain the health text information corresponding to the target sample, clean and standardize the health text information to form standardized health text information; Standardized abdominal CT images, standardized gastric organ recognition results, and standardized health text information together constitute the target multimodal sample.
[0008] As a preferred embodiment of the multimodal gastric cancer risk prediction method based on gated attention mechanism described in this invention, the specific steps for constructing the organ prior enhanced image are as follows: Image features are extracted from standardized abdominal CT images in the target multimodal sample, and gastric organ features are extracted from standardized gastric organ recognition results in the target multimodal sample. The image features are multiplied with the stomach organ features, and the image features are enhanced through residual connection to obtain an enhanced prior image of the organ.
[0009] As a preferred embodiment of the multimodal gastric cancer risk prediction method based on gated attention mechanism described in this invention, the specific steps for obtaining image features and text features are as follows: Image features are obtained by performing convolution operations and positional encoding on organ-prior enhanced images; Text features are obtained by segmenting and encoding standardized health text information.
[0010] As a preferred embodiment of the multimodal gastric cancer risk prediction method based on gated attention mechanism described in this invention, the specific steps for generating gated fusion coefficients for the image portion and the text portion are as follows: Self-attention calculations are performed on image features and text features respectively, and the corresponding self-attention response weights are determined to obtain the self-attention outputs corresponding to image features and text features. Cross-attention calculation is performed based on image features and text features to determine the cross-attention response weights from image features to text features and from text features to image features, and to obtain the cross-attention outputs from image features to text features and from text features to image features. Based on the self-attention outputs corresponding to image features and text features, calculate the self-attention entropy values corresponding to image features and text features respectively. Based on the cross-attention outputs from image features to text features and from text features to image features, calculate the cross-attention peaks from image features to text features and from text features to image features respectively. Based on the self-attention entropy value corresponding to the image features and the cross-attention peak value from the image features to the text features, the gating fusion coefficients for the image part are generated; based on the self-attention entropy value corresponding to the text features and the cross-attention peak value from the text features to the image features, the gating fusion coefficients for the text part are generated.
[0011] As a preferred embodiment of the multimodal gastric cancer risk prediction method based on gated attention mechanism described in this invention, the specific steps for obtaining the gated fusion features are as follows: Based on the gating fusion coefficient of the image part, the self-attention output corresponding to the image features and the cross-attention output from the image features to the text features are weighted and fused to obtain the gating fusion features of the image part; Based on the gating fusion coefficient of the text part, the self-attention output corresponding to the text features and the cross-attention output from the text features to the image features are weighted and fused to obtain the gating fusion features of the text part.
[0012] As a preferred embodiment of the multimodal gastric cancer risk prediction method based on gated attention mechanism described in this invention, the specific steps for obtaining the fusion features of the image and text parts are as follows: Based on the gated fusion features of the image and text parts, the corresponding self-attention output is calculated; Based on the gated fusion features of the image and text parts, calculate the cross-attention output from image features to text features and from text features to image features; The self-attention output of the image part and the cross-attention output from image features to text features are added proportionally to form the image part fusion feature; the self-attention output of the text part and the cross-attention output from text features to image features are added proportionally to form the text part fusion feature.
[0013] As a preferred embodiment of the multimodal gastric cancer risk prediction method based on gating attention mechanism described in this invention, the determination of the risk of gastric cancer in the target sample refers to determining the risk of gastric cancer in the target sample by performing pooling and classification operations based on fusion features.
[0014] Secondly, the present invention provides a multimodal gastric cancer risk prediction system based on a gated attention mechanism, including a sample construction module for acquiring abdominal CT images, gastric organ recognition results and health text information corresponding to the target sample, and performing image and text preprocessing to form a target multimodal sample; The prior construction module is used to extract features and perform residual fusion based on standardized abdominal CT images and standardized gastric organ recognition results in the target multimodal samples to construct organ prior enhanced images; The feature extraction module is used to extract features from the prior enhanced image of the organ and the standardized health text information respectively, to obtain image features and text features; The gated fusion coefficient generation module is used to interact with image features and text features to obtain the self-attention outputs corresponding to image features and text features, as well as the cross-attention outputs from image features to text features and from text features to image features. It also calculates the self-attention entropy value and the cross-attention peak value to generate the gated fusion coefficients for the image part and the text part. The gated fusion module is used to perform weighted fusion of the self-attention output and the cross-attention output based on the gated fusion coefficients of the image part and the text part, respectively, to obtain the gated fusion features of the image part and the text part; The simple fusion module is used to perform self-attention and cross-attention calculations on the gated fusion features of the image and text parts, and then add the self-attention output and cross-attention output proportionally to form the fusion feature; The decision-making module is used to determine the risk of gastric cancer in the target sample based on the fusion features.
[0015] The beneficial effects of this invention are as follows: by performing self-attention and cross-attention calculations on image features and text features, and generating gating fusion coefficients for the image and text parts based on the self-attention entropy and cross-attention peak values of each part, the invention simultaneously reflects the concentration of feature distribution within a modality and the significance of the correspondence between modalities. Based on the importance of the features themselves and their association with another modality, the invention dynamically adjusts the contribution of different modalities to the task results, enhances the focusing ability of gastric cancer-related information, and improves the accuracy and stability of gastric cancer risk prediction. Attached Figure Description
[0016] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0017] Figure 1This is a flowchart of a multimodal gastric cancer risk prediction method based on a gating attention mechanism.
[0018] Figure 2 This is a schematic diagram of a multimodal gastric cancer risk prediction system based on a gated attention mechanism.
[0019] Figure 3 A flowchart for generating gating fusion coefficients. Detailed Implementation
[0020] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.
[0021] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.
[0022] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.
[0023] Reference Figures 1-3 This is one embodiment of the present invention, which provides a multimodal gastric cancer risk prediction method based on a gated attention mechanism, comprising the following steps: S1: Obtain the abdominal CT image, gastric organ recognition result and health text information corresponding to the target sample, and perform preprocessing to obtain standardized abdominal CT image, standardized gastric organ recognition result and standardized health text information to form target multimodal sample.
[0024] S1.1: Obtain the abdominal CT image and gastric organ recognition result corresponding to the target sample, perform image standardization processing on the abdominal CT image to form a standardized abdominal CT image, and perform image standardization processing on the gastric organ recognition result to form a standardized gastric organ recognition result.
[0025] Based on the abdominal CT images and gastric organ recognition results corresponding to the target sample, the parameters such as image orientation and spatial size are uniformly corrected and standardized.
[0026] Abdominal CT images were uniformly adjusted to standardized three-channel images with a size of 3×224×224.
[0027] The results of stomach organ identification were uniformly adjusted to standardized single-channel images with a size of 1×224×224.
[0028] When the image size is larger than the target size, the image is compressed; when the image size is smaller than the target size, the image is stretched to ensure consistency in spatial orientation and input scale.
[0029] S1.2: Obtain the health text information corresponding to the target sample, clean and standardize the health text information to form standardized health text information.
[0030] Remove duplicate spaces, duplicate punctuation marks, meaningless line breaks, garbled characters, and invalid symbols from the health text information; Standardize the expression of key information such as age, gender, medical history, symptoms, and examination records; and unify the conversion of synonyms, abbreviations, and colloquial expressions into standard terminology. All text information is integrated into a single piece of natural language to serve as standardized health text information.
[0031] S1.3: Standardized abdominal CT images, standardized gastric organ recognition results, and standardized health text information together constitute the target multimodal sample.
[0032] For each target sample, a corresponding match is performed between standardized abdominal CT images, standardized gastric organ recognition results, and standardized health text information.
[0033] Standardized abdominal CT images, standardized gastric organ identification results, and standardized health text information are combined and recorded in a fixed order to form a target multimodal sample.
[0034] S2: Based on the standardized abdominal CT images and standardized gastric organ recognition results in the target multimodal samples, feature fusion is performed to construct organ prior enhanced images.
[0035] S2.1: Extract image features based on standardized abdominal CT images in the target multimodal sample, and extract gastric organ features based on standardized gastric organ recognition results in the target multimodal sample.
[0036] Standardized abdominal CT images are input into a convolutional network, and then processed sequentially through a 3×3 convolution operation, normalization, and ReLU activation function to extract abdominal CT image features. The standardized abdominal CT images are 3×224×224 in size, and the image features after feature extraction are 16×224×224 in size.
[0037] The standardized gastric organ recognition result is input into another feature extraction branch, and then sequentially undergoes 3×3 convolution, normalization, ReLU activation function processing, 3×3 convolution, and Sigmoid activation function processing to extract gastric organ features. The standardized gastric organ recognition result has a size of 1×224×224, and after feature extraction, the gastric organ features have a size of 16×224×224.
[0038] S2.2: Multiply the image features with the stomach organ features, and enhance the image features through residual connection to obtain an enhanced prior image of the organ.
[0039] Based on the image features and stomach organ features extracted in the previous step, the two are first multiplied element-wise, so that the stomach organ features serve as spatial priors to constrain the image features, thus obtaining preliminary fused features.
[0040] The preliminary fusion features and image features are added element-wise using a residual connection method. While introducing gastric organ features as a spatial prior, the local texture information, edge contour information and global structural response of the image features are preserved to obtain the first round of fusion results.
[0041] The first-round fusion result was processed by 1×1 convolution. The result after convolution was added to the standardized abdominal CT image element by element through residual connection, and finally the organ prior enhancement image with size 3×224×224 was obtained.
[0042] S3: Extract features from the prior enhanced images of organs and the standardized health text information to obtain image features and text features respectively.
[0043] S3.1: Perform convolution operations and positional encoding on organ prior enhanced images to obtain image features.
[0044] Local receptive field modeling is performed on the organ prior enhanced image using convolution operations. The organ prior enhanced image is divided and mapped into a series of image patches. The image patches are flattened and transposed to convert them into feature sequences suitable for sequence modeling. A learnable classification token is concatenated before the sequence. This feature sequence is added to the corresponding positional encoding to introduce spatial location information between image patches, finally obtaining an image feature of size 768×197.
[0045] S3.2: Perform word segmentation and encoding based on standardized health text information to obtain text features.
[0046] The BERT model is used to segment and encode standardized health text information. The continuous text is split into several tokens and formed into a sequence. A special marker [CLS] is added to the beginning of the sequence, and a marker [SEP] is added between sentence boundaries or segments.
[0047] The token sequence is input into the BERT encoding layer, and text features of size 768×300 are constructed through word embedding, position embedding and segment embedding.
[0048] S4: Calculate the self-attention output and cross-attention output for image features and text features. Based on the self-attention entropy value and cross-attention peak value, generate the gating fusion coefficients for the image part and the text part, respectively.
[0049] S4.1: Perform self-attention calculations on image features and text features respectively, determine the corresponding self-attention response weights, and obtain the self-attention outputs corresponding to image features and text features.
[0050] Image features and text features are used as inputs to the first layer of a 12-layer Transformer encoder.
[0051] In each layer of the Transformer encoder from layer 1 to layer 5, query vector transformation, key vector transformation, and value vector transformation are performed on the current layer image features and the current layer text features respectively to obtain image query vector, image key vector, image value vector, text query vector, text key vector, and text value vector.
[0052] The correlation scores between different locations in the current layer image features are calculated based on the correlation between the image query vector and the image key vector. The correlation scores are then normalized to form the self-attention response weights of the current layer image features.
[0053] The correlation scores between different positions in the current layer text features are calculated based on the correlation between the text query vector and the text key vector. The correlation scores are then normalized to form the self-attention response weights of the current layer text features.
[0054] The image value vector is weighted and summarized according to the self-attention response weights of the image features to obtain the self-attention output corresponding to the image features of the current layer.
[0055] The text value vector is weighted and summarized according to the self-attention response weights of the text features to obtain the self-attention output corresponding to the text features of the current layer.
[0056] It should be noted that the 12-layer Transformer encoder consists of 12 Transformer coding layers arranged in series. Each Transformer coding layer includes layer normalization operation, attention calculation and fusion, layer normalization, and multilayer perceptron.
[0057] The Transformer coding layers from layer 1 to layer 5 are used to perform gated fusion interaction of image features and text features. In each layer, the image features and text features are first normalized, and then the interaction is completed through attention calculation and fusion operation. After layer normalization and multilayer perceptron processing, the image feature output and text feature output of the current layer are obtained, which are used as the input of image features and text features in the next Transformer coding layer, until the processing of the fifth Transformer coding layer is completed.
[0058] S4.2: Perform cross-attention calculation based on image features and text features, determine the cross-attention response weights from image features to text features and from text features to image features, and obtain the cross-attention outputs from image features to text features and from text features to image features.
[0059] In each of the Transformer coding layers 1 through 5 of the 12-layer Transformer encoder, the current layer's image features and text features are used as inputs for cross-attention computation.
[0060] The correlation score between the current layer image features and text features is calculated based on the correlation between the image query vector and the text key vector. The correlation score is then normalized to form the cross-attention response weight between the current layer image features and text features.
[0061] The correlation score between the text query vector and the image key vector is calculated based on the correlation between the text features and the image features of the current layer. The correlation score is then normalized to form the cross-attention response weight between the text features and the image features of the current layer.
[0062] The text value vector is weighted and summarized based on the cross-attention response weights from image features to text features to obtain the cross-attention output from image features to text features in the current layer.
[0063] The image value vector is weighted and summarized based on the cross-attention response weights from text features to image features to obtain the cross-attention output from text features to image features in the current layer.
[0064] S4.3: Based on the self-attention outputs corresponding to image features and text features, calculate the self-attention entropy values corresponding to image features and text features respectively. Based on the cross-attention outputs from image features to text features and from text features to image features, calculate the cross-attention peaks from image features to text features and from text features to image features respectively.
[0065] Calculate the information entropy for the value of each token in the self-attention output corresponding to the image feature, and average the information entropy of all tokens to form the self-attention entropy value corresponding to the image feature of the current layer.
[0066] Calculate the information entropy for the value of each token in the self-attention output corresponding to the text feature, and average the information entropy of all tokens to form the self-attention entropy value corresponding to the text feature of the current layer.
[0067] The maximum value of each token in the cross-attention output from the current layer's image features to text features is taken, and the average of the maximum values of all tokens is taken to form the peak value of the cross-attention output from the current layer's image features to text features.
[0068] The maximum value of each token in the cross-attention output from the current layer's text features to image features is taken, and the average of the maximum values of all tokens is taken to form the peak value of the cross-attention output from the current layer's text features to image features.
[0069] S4.4: Based on the self-attention entropy value corresponding to the image features and the cross-attention peak value from the image features to the text features, generate the gated fusion coefficients for the image part; based on the self-attention entropy value corresponding to the text features and the cross-attention peak value from the text features to the image features, generate the gated fusion coefficients for the text part.
[0070] The self-attention entropy value corresponding to the current layer image features and the cross-attention peak of the current layer image features to text features are combined in a fixed order, and then processed by a multilayer perceptron and a sigmoid function to obtain the gating fusion coefficient of the current layer image part.
[0071] The self-attention entropy value corresponding to the current layer text features and the cross-attention peak of the current layer text features to image features are combined in a fixed order, and then processed by a multilayer perceptron and a sigmoid function to obtain the gating fusion coefficient of the current layer text part.
[0072] S5: Based on the gating fusion coefficients of the image part and the text part, the self-attention output and the cross-attention output are weighted and fused respectively to obtain the gating fusion features of the image part and the text part.
[0073] S5.1: Based on the gating fusion coefficients of the image part, the self-attention output corresponding to the image features and the cross-attention output from the image features to the text features are weighted and fused to obtain the gating fusion features of the image part.
[0074] Based on the gating fusion coefficients of the current layer image portion, the self-attention output corresponding to the current layer image features and the cross-attention output from image features to text features are weighted and calculated to form the gating fusion features of the current layer image portion, expressed as: ; In the formula, For the first Gated fusion features of layer image components, For the first The gated fusion coefficients of the layer image portion, For the first Self-attention output of layer image features, For the first Cross-attention output from layer image features to text features. For network layer number, For image modality, For text modality.
[0075] S5.2: Based on the gating fusion coefficient of the text part, the self-attention output corresponding to the text feature and the cross-attention output from the text feature to the image feature are weighted and fused to obtain the gating fusion feature of the text part.
[0076] Based on the gating fusion coefficients of the current layer text portion, the self-attention output corresponding to the current layer text features and the cross-attention output from text features to image features are weighted and calculated to form the gating fusion features of the current layer text portion, expressed as: ; In the formula, For the first Gated fusion features of the text layer. For the first The gating fusion coefficient of the text layer. For the first Self-attention output of layered text features. For the first Cross-attention output from text features to image features. For network layer number, For image modality, This is a text modal.
[0077] S6: Calculate the self-attention output and cross-attention output based on the gated fusion features of the image part and the text part respectively, and add them in equal proportions to obtain the fusion features of the image part and the text part.
[0078] S6.1: Calculate the corresponding self-attention output based on the gating fusion features of the image and text parts.
[0079] The gated fusion features of the image portion and the gated fusion features of the text portion output from the 5th Transformer coding layer are used as the input to the 6th Transformer coding layer.
[0080] The query vector, key vector, and value vector are transformed into the current layer image features and the current layer text features respectively, to obtain the image query vector, image key vector, image value vector, text query vector, text key vector, and text value vector.
[0081] The correlation scores between different locations in the current layer image features are calculated based on the correlation between the image query vector and the image key vector. The correlation scores are then normalized to form the self-attention response weights of the current layer image features.
[0082] The image value vector is weighted and summarized according to the self-attention response weights corresponding to the image features to obtain the self-attention output corresponding to the current layer image features.
[0083] The correlation scores between different positions in the current layer text features are calculated based on the correlation between the text query vector and the text key vector. The correlation scores are then normalized to form the self-attention response weights of the current layer text features.
[0084] The text value vector is weighted and summarized according to the self-attention response weights corresponding to the text features to obtain the self-attention output corresponding to the text features of the current layer.
[0085] S6.2: Based on the gating fusion features of the image and text parts, calculate the cross-attention output from image features to text features and from text features to image features.
[0086] The correlation score between the current layer image features and text features is calculated based on the correlation between the image query vector and the text key vector. The correlation score is then normalized to form the cross-attention response weight between the current layer image features and text features.
[0087] The text value vector is weighted and summarized based on the cross-attention response weights from image features to text features to obtain the cross-attention output from image features to text features in the current layer.
[0088] The correlation score between the text query vector and the image key vector is calculated based on the correlation between the text features and the image features of the current layer. The correlation score is then normalized to form the cross-attention response weight between the text features and the image features of the current layer.
[0089] The image value vector is weighted and summarized based on the cross-attention response weights from text features to image features to obtain the cross-attention output from text features to image features in the current layer.
[0090] S6.3: The self-attention output of the image part and the cross-attention output from image features to text features are added proportionally to form the image part fusion feature; the self-attention output of the text part and the cross-attention output from text features to image features are added proportionally to form the text part fusion feature.
[0091] The self-attention output corresponding to the current layer image features and the cross-attention output from image features to text features are added in the same proportion to form the fused features of the current layer image part, represented as: ; In the formula, For the first Fusion features of layered image portions, For the first Self-attention output of layer image features, For the first Cross-attention output from layer image features to text features. For network layer number, For image modality, This is a text modal.
[0092] The self-attention output corresponding to the current layer text features and the cross-attention output from text features to image features are added in the same proportion to form the fused features of the current layer text part, represented as: ; In the formula, For the first The fusion features of the layered text portion For the first Self-attention output of layered text features. For the first Cross-attention output from text features to image features. For network layer number, For image modality, This is a text modal.
[0093] The fusion features of the current layer's image portion and the fusion features of the current layer's text portion are processed by layer normalization and multilayer perceptron and then used as the input to the next Transformer coding layer, until the processing of the 12th Transformer coding layer is completed.
[0094] S7: Classify based on fusion features to determine the risk of gastric cancer in the target sample.
[0095] The fused features of the image portion are subjected to average pooling and max pooling. The average pooling result and the max pooling result are added together to obtain the pooled features of the image portion.
[0096] The fused features of the text portion are subjected to average pooling and max pooling. The average pooling result and the max pooling result are added together to obtain the pooled features of the text portion.
[0097] The pooled features of the image and text are added proportionally and input into the classification head for risk classification, outputting the risk of the target sample having gastric cancer.
[0098] This embodiment also provides a multimodal gastric cancer risk prediction system based on a gated attention mechanism, including: The sample construction module is used to acquire the abdominal CT images, gastric organ recognition results and health text information corresponding to the target sample, and to perform image and text preprocessing to form target multimodal samples; The prior construction module is used to extract features and perform residual fusion based on standardized abdominal CT images and standardized gastric organ recognition results in the target multimodal samples to construct organ prior enhanced images; The feature extraction module is used to extract features from the prior enhanced image of the organ and the standardized health text information respectively, to obtain image features and text features; The gated fusion coefficient generation module is used to interact with image features and text features to obtain the self-attention outputs corresponding to image features and text features, as well as the cross-attention outputs from image features to text features and from text features to image features. It also calculates the self-attention entropy value and the cross-attention peak value to generate the gated fusion coefficients for the image part and the text part. The gated fusion module is used to perform weighted fusion of the self-attention output and the cross-attention output based on the gated fusion coefficients of the image part and the text part, respectively, to obtain the gated fusion features of the image part and the text part; The simple fusion module is used to perform self-attention and cross-attention calculations on the gated fusion features of the image and text parts, and then add the self-attention output and cross-attention output proportionally to form the fusion feature; The decision-making module is used to determine the risk of gastric cancer in the target sample based on the fusion features.
[0099] In summary, this invention first fuses features from standardized abdominal CT images and standardized gastric organ identification results; then, it extracts features from multimodal data to obtain image features and text features; within the Transformer encoder, it calculates intramodal self-attention and intermodal cross-attention for the image features and text features respectively; in the first five coding layers, it generates gating fusion coefficients based on the entropy value of the self-attention output and the peak value of the cross-attention output; and it performs weighted fusion on the self-attention output and cross-attention output of the two modalities respectively; and in the last seven coding layers, it adds the self-attention output and cross-attention output of the two modalities proportionally, balancing the efficiency of feature interaction and model complexity, enhancing the focusing ability of gastric cancer-related information, and improving the accuracy and stability of gastric cancer risk prediction.
[0100] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.
Claims
1. A multimodal gastric cancer risk prediction method based on a gated attention mechanism, characterized in that, include: The abdominal CT image, gastric organ recognition result, and health text information corresponding to the target sample are obtained and preprocessed to obtain standardized abdominal CT image, standardized gastric organ recognition result, and standardized health text information, forming a target multimodal sample. Feature fusion is performed on standardized abdominal CT images and standardized gastric organ recognition results from target multimodal samples to construct organ prior enhanced images; Feature extraction was performed on the enhanced organ prior image and the standardized health text information to obtain image features and text features respectively; Self-attention output and cross-attention output are calculated from image features and text features. Based on the self-attention entropy value and cross-attention peak value, gated fusion coefficients for the image part and the text part are generated respectively. Based on the gating fusion coefficients of the image and text parts, the self-attention output and cross-attention output are weighted and fused respectively to obtain the gating fusion features of the image and text parts; The self-attention output and cross-attention output are calculated based on the gated fusion features of the image part and the text part respectively, and then added in equal proportions to obtain the fusion features of the image part and the text part. Classification based on fusion features determines the risk of gastric cancer in target samples.
2. The multimodal gastric cancer risk prediction method based on gated attention mechanism as described in claim 1, characterized in that, The specific steps for forming the target multimodal sample are as follows: Obtain the abdominal CT image and gastric organ recognition result corresponding to the target sample. Perform image standardization processing on the abdominal CT image to form a standardized abdominal CT image. Perform image standardization processing on the gastric organ recognition result to form a standardized gastric organ recognition result. Obtain the health text information corresponding to the target sample, clean and standardize the health text information to form standardized health text information; Standardized abdominal CT images, standardized gastric organ recognition results, and standardized health text information together constitute the target multimodal sample.
3. The multimodal gastric cancer risk prediction method based on gated attention mechanism as described in claim 1, characterized in that, The specific steps for constructing the prior enhanced image of the organ are as follows: Image features are extracted from standardized abdominal CT images in the target multimodal sample, and gastric organ features are extracted from standardized gastric organ recognition results in the target multimodal sample. The image features are multiplied with the stomach organ features, and the image features are enhanced through residual connection to obtain an enhanced prior image of the organ.
4. The multimodal gastric cancer risk prediction method based on gated attention mechanism as described in claim 1, characterized in that, The specific steps for obtaining image features and text features are as follows: Image features are obtained by performing convolution operations and positional encoding on organ-prior enhanced images; Text features are obtained by segmenting and encoding standardized health text information.
5. The multimodal gastric cancer risk prediction method based on gated attention mechanism as described in claim 1, characterized in that, The specific steps for generating the gated fusion coefficients for the image portion and the text portion are as follows: Self-attention calculations are performed on image features and text features respectively, and the corresponding self-attention response weights are determined to obtain the self-attention outputs corresponding to image features and text features. Cross-attention calculation is performed based on image features and text features to determine the cross-attention response weights from image features to text features and from text features to image features, and to obtain the cross-attention outputs from image features to text features and from text features to image features. Based on the self-attention outputs corresponding to image features and text features, calculate the self-attention entropy values corresponding to image features and text features respectively. Based on the cross-attention outputs from image features to text features and from text features to image features, calculate the cross-attention peaks from image features to text features and from text features to image features respectively. Based on the self-attention entropy value corresponding to the image features and the cross-attention peak value from the image features to the text features, the gating fusion coefficients for the image part are generated; based on the self-attention entropy value corresponding to the text features and the cross-attention peak value from the text features to the image features, the gating fusion coefficients for the text part are generated.
6. The multimodal gastric cancer risk prediction method based on gated attention mechanism as described in claim 1, characterized in that, The specific steps for obtaining the gated fusion features are as follows: Based on the gating fusion coefficient of the image part, the self-attention output corresponding to the image features and the cross-attention output from the image features to the text features are weighted and fused to obtain the gating fusion features of the image part; Based on the gating fusion coefficient of the text part, the self-attention output corresponding to the text features and the cross-attention output from the text features to the image features are weighted and fused to obtain the gating fusion features of the text part.
7. The multimodal gastric cancer risk prediction method based on gated attention mechanism as described in claim 1, characterized in that, The specific steps for obtaining the fusion features of the image and text portions are as follows: Based on the gated fusion features of the image and text parts, the corresponding self-attention output is calculated; Based on the gated fusion features of the image and text parts, calculate the cross-attention output from image features to text features and from text features to image features; The self-attention output of the image part and the cross-attention output from image features to text features are added proportionally to form the image part fusion feature; the self-attention output of the text part and the cross-attention output from text features to image features are added proportionally to form the text part fusion feature.
8. The multimodal gastric cancer risk prediction method based on gated attention mechanism as described in claim 1, characterized in that, Determining the risk of gastric cancer in a target sample refers to determining the risk of gastric cancer in a target sample by performing pooling and classification operations based on fusion features.
9. A multimodal gastric cancer risk prediction system based on a gated attention mechanism, based on the multimodal gastric cancer risk prediction method based on a gated attention mechanism as described in any one of claims 1 to 8, characterized in that, include, The sample construction module is used to acquire the abdominal CT images, gastric organ recognition results and health text information corresponding to the target sample, and to perform image and text preprocessing to form target multimodal samples; The prior construction module is used to extract features and perform residual fusion based on standardized abdominal CT images and standardized gastric organ recognition results in the target multimodal samples to construct organ prior enhanced images; The feature extraction module is used to extract features from the prior enhanced image of the organ and the standardized health text information respectively, to obtain image features and text features; The gated fusion coefficient generation module is used to interact with image features and text features to obtain the self-attention outputs corresponding to image features and text features, as well as the cross-attention outputs from image features to text features and from text features to image features. It also calculates the self-attention entropy value and the cross-attention peak value to generate the gated fusion coefficients for the image part and the text part. The gated fusion module is used to perform weighted fusion of the self-attention output and the cross-attention output based on the gated fusion coefficients of the image part and the text part, respectively, to obtain the gated fusion features of the image part and the text part; The simple fusion module is used to perform self-attention and cross-attention calculations on the gated fusion features of the image and text parts, and then add the self-attention output and cross-attention output proportionally to form the fusion feature; The decision-making module is used to determine the risk of gastric cancer in the target sample based on the fusion features.