A risk prediction method for gastric cancer liver metastasis based on multi-head attention mechanism

Abstract

The present invention belongs to the medical field, and specifically relates to a method for predicting the risk of gastric cancer liver metastasis based on a multi-head attention mechanism, comprising: acquiring gastric cancer data, and preprocessing the gastric cancer data to obtain a preprocessing result; performing feature interaction on the preprocessing result using the multi-head attention mechanism to obtain feature interaction information; performing sample interaction on the preprocessing result using an improved multi-head attention mechanism to obtain sample interaction information; obtaining a gastric cancer liver metastasis risk result using a neural network according to the feature interaction information and the sample interaction information; calculating a loss function value according to the gastric cancer liver metastasis risk result, and updating a model parameter according to the loss function value; when a preset number of iterations is reached, obtaining a trained gastric cancer liver metastasis risk prediction model; the present invention simultaneously captures information between features and between samples through two multi-head attention mechanisms, and fuses information between features and between samples using a neural network, thereby enhancing feature representation and improving risk prediction accuracy.

Description

Technical Field

[0001] The present invention belongs to the medical field, and specifically relates to a method for predicting the risk of gastric cancer liver metastasis based on a multi-head attention mechanism. Background Art

[0002] In the past few years, there have been a lot of studies on metastasis risk prediction for cancer data in the form of tabular data, mainly using nomograms and machine learning methods. There are also studies that try to use neural networks to learn feature representations. However, these neural network-based methods do not fully utilize the power of neural networks because they only use simple multi-layer perceptrons, which have limited learning capabilities.

[0003] In recent years, with the popularity of transformer models, attention mechanisms have also received widespread attention and have been widely used in the fields of text and images. In the field of text, by introducing the attention mechanism, the model can automatically learn and selectively focus on important information in the input text. In the field of images, the attention mechanism can help the model focus on important areas or features during image processing, thereby improving the performance of tasks such as image recognition, classification, and target detection. In the field of table learning, the application of attention mechanisms is still immature, especially in the risk prediction of gastric cancer liver metastasis. Some existing risk prediction methods are often just simple linear models or integrated models, which ignore the information between gastric cancer data features and samples, resulting in deviations in prediction accuracy.

[0004] In summary, existing methods ignore the balance between features and samples of gastric cancer data and fail to fully extract information between features and samples, resulting in low risk prediction accuracy. Summary of the invention

[0005] In order to solve the shortcomings of the above existing gastric cancer liver metastasis risk prediction algorithms, the present invention adopts a gastric cancer liver metastasis risk prediction method based on a multi-head attention mechanism, including: obtaining gastric cancer data, inputting the gastric cancer data into a trained gastric cancer liver metastasis risk prediction model, and predicting the gastric cancer liver metastasis risk result;

[0006] The training process of the gastric cancer liver metastasis risk prediction model includes:

[0007] S1. Acquire gastric cancer data, and preprocess the gastric cancer data to obtain a preprocessing result; the preprocessing result includes multiple samples, each sample includes a gastric cancer liver metastasis label and feature;

[0008] S2. Use the multi-head attention mechanism to perform feature interaction on the preprocessing results to obtain feature interaction information;

[0009] S3. Use the improved multi-head attention mechanism to perform sample interaction on the preprocessing results to obtain sample interaction information;

[0010] S4. Obtain the risk result of gastric cancer liver metastasis using a neural network based on feature interaction information and sample interaction information;

[0011] S5. Calculate the loss function value according to the gastric cancer liver metastasis risk result, and update the model parameters according to the loss function value; when the preset number of iterations is reached, complete the model training and obtain a trained gastric cancer liver metastasis risk prediction model.

[0012] Preprocessing of gastric cancer data includes:

[0013] S11, acquiring gastric cancer data, and performing preliminary processing on the gastric cancer data to obtain gastric cancer data D1 after preliminary processing;

[0014] S12, using the SMOTE-NC oversampling method to process the gastric cancer data D1 to obtain balanced gastric cancer data D2;

[0015] S13. Perform feature embedding on the gastric cancer data D2.

[0016] The preliminary processing of gastric cancer data includes: converting the gastric cancer data to obtain structured tabular data; and cleaning the structured tabular data.

[0017] The gastric cancer data D1 includes multiple samples; processing the gastric cancer data D1 using the SMOTE-NC oversampling method includes: using the nearest neighbor algorithm to calculate K neighbor samples of the minority class sample of the gastric cancer data D1, and selecting the sample with the highest occurrence frequency from the K neighbor samples; constructing a new minority class sample according to the selected sample by random linear interpolation; synthesizing the new minority class sample with the gastric cancer data D1 to obtain balanced gastric cancer data D2.

[0018] The gastric cancer data D2 includes: discrete data and continuous data; feature embedding of the gastric cancer data D2 includes: calculating the one-hot encoding representation of the discrete data of the gastric cancer data D2, and using the embedding matrix to perform feature embedding on the one-hot encoding representation to obtain features embedded in the discrete data; and using the embedding matrix to perform feature embedding on the continuous data of the gastric cancer data D2 to obtain features embedded in the continuous data.

[0019] Using the multi-head attention mechanism to interact with the preprocessing results includes:

[0020] S21, stacking each feature of the sample in the preprocessing result vertically to obtain a feature matrix;

[0021] S22, linearly transform the feature matrix through multiple linear layers to obtain the matrix Q m , K m 、V m; Where m represents the mth head;

[0022] S23. According to Q m and K m Calculate the attention weight of the mth head; add the attention weight of the mth head to the matrix V m Multiply them together to get the attention output of the mth head;

[0023] S24. Concatenate the attention output results of each head and perform linear transformation on the concatenated results to obtain feature interaction information.

[0024] The sample interaction of the preprocessing results using the improved multi-head attention mechanism includes:

[0025] S31, stacking each feature of the sample in the preprocessing result horizontally, and then stacking the horizontally stacked samples vertically to obtain a sample matrix;

[0026] S32, linearly transform the sample matrix through multiple linear layers to obtain the matrix Q h , K h 、V h ; where h represents the hth head;

[0027] S33, according to Q h and K h Calculate the attention weight of the h-th head; add the attention weight of the h-th head to the matrix V h Multiply them together to get the attention output of the h-th head;

[0028] S34. Concatenate the attention output results of each head and perform linear transformation on the concatenated results to obtain sample interaction information.

[0029] The risk result of gastric cancer liver metastasis is obtained by using a neural network according to the feature interaction information and the sample interaction information, including: using layer normalization and a feedforward neural network to process the feature interaction information and the sample interaction information, and obtaining the processing result of the feature interaction information The processing results of the sample interaction information According to the processing results The weighted average result is calculated, and the risk result of gastric cancer liver metastasis is obtained using the fully connected layer according to the weighted average result; wherein i represents the index of the feature in the sample.

[0030] Using layer normalization and feedforward neural network to process feature interaction information and sample interaction information respectively includes:

[0031]

[0032]

[0033] Among them, LayerNorm is the layer normalization operation, GELU is the GELU activation function, and W FFN is the learnable weight matrix of the feedforward neural network, e i is the characteristic of the sample in the preprocessing result, is the sample interaction information, is the processing result of sample interaction information, is the feature interaction information, is the processing result of feature interaction information.

[0034] According to the processing results The weighted average calculation results include:

[0035]

[0036] Among them, W is the learnable weight matrix, W1 is the all-1 matrix, The result is weighted average.

[0037] Beneficial effects of the present invention:

[0038] 1. The present invention utilizes two multi-head attention mechanisms to simultaneously capture information between features and between samples, and utilizes a neural network to fuse information between features and between samples, thereby enhancing the representation of features and improving the accuracy of risk prediction; 2. The present invention calculates the interactive information between samples based on the improved multi-head row attention mechanism, which can supplement the diversity and quality of gastric cancer data, thereby eliminating the noise in each sample and making the information in the sample higher quality. When there are missing values ​​in a sample, the multi-head row attention mechanism can use the same feature in other samples to supplement the missing values ​​of the sample, thereby further enhancing the robustness of risk prediction; 3. The present invention solves the problem of imbalance in sample number through the SMOTE-NC oversampling method, thereby improving the robustness of risk prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 A flow chart of a method for predicting the risk of gastric cancer liver metastasis based on a multi-head attention mechanism provided in an embodiment of the present invention;

[0040] Figure 2 A schematic diagram of a gastric cancer dataset provided by an embodiment of the present invention;

[0041] Figure 3 A schematic diagram of ablation experiment results provided by an embodiment of the present invention;

[0042] Figure 4 A schematic diagram of comparative experimental results provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0043] The technical scheme in the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiment of the present invention. Obviously, the described embodiment is only a part of the embodiment of the present invention, not all of the embodiments. Based on the embodiment of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0044] like Figure 1 As shown, the embodiment of the present invention adopts a method for predicting the risk of gastric cancer liver metastasis based on a multi-head attention mechanism, including:

[0045] S1. Obtain gastric cancer data and pre-process the gastric cancer data to ensure the quality of the input data and to ensure that sufficient interaction of features or samples can be performed in the subsequent interaction process; the gastric cancer data includes whether liver metastasis occurs and related features; the related features include basic personal information, pathological information, and surgery and radiotherapy and chemotherapy information;

[0046] In the embodiment of the present invention, the basic personal information data includes the patient's age, marital status, race, gender, education level, occupation, family income, etc. Basic personal information is usually obtained through patient questionnaires, medical records, government census, medical information systems, hospital databases, etc.

[0047] In the embodiment of the present invention, the pathological information includes information related to the patient's gastric cancer tumor, such as the primary site of the tumor, the histological type of the tumor, the size of the tumor, the number of tumors, the depth of tumor invasion, etc. In order to ensure the authenticity of the symptom data, the pathological data can be obtained from medical websites, medical institution databases, etc.

[0048] In the embodiment of the present invention, the surgery and radiotherapy and chemotherapy information includes information related to the patient's gastric cancer treatment, such as surgical method, number of tumors removed, whether radiotherapy was performed, whether chemotherapy was performed, etc. In order to ensure the authenticity of symptom data, the surgery and radiotherapy and chemotherapy information can be obtained from medical websites, medical institution databases, etc.

[0049] In an embodiment of the present invention, the gastric cancer data can be obtained from some special disease databases. At present, the special disease database application system used for medical research in hospitals is usually established on the basis of the special disease database. The generation process of the special disease database can be specifically as follows: the original data of the hospital, such as patient medical records and medical records, is transformed into a structured disease patient database that can be used by the application system after a series of big data processing including data collection, cleaning, normalization, and structuring, as well as business processing such as patient sorting and field logic calculation. For a specialized disease research system for a single disease, the data used is the database of a single disease. For example, the database corresponding to the gastric cancer specialized disease research system is the gastric cancer patient database.

[0050] Preprocessing of gastric cancer data includes:

[0051] S11, acquiring gastric cancer data, and performing preliminary processing on the gastric cancer data to obtain gastric cancer data D1 after preliminary processing;

[0052] In an embodiment of the present invention, preliminary processing of gastric cancer data includes: performing data conversion on the gastric cancer data to obtain structured tabular data; performing data cleaning on the structured tabular data; and dividing the cleaned tabular data into gastric cancer liver metastasis results and related information.

[0053] S12, using the SMOTE-NC oversampling method to process the gastric cancer data D1 to obtain balanced gastric cancer data D2;

[0054] For imbalance processing, specifically, this embodiment first divides the gastric cancer data into a training set, a test set, and a validation set in a ratio of 7:2:1. The training set and the validation set are processed using SMOTE-NC. The processing of the training set includes: using the sampling nearest neighbor algorithm to calculate the K nearest neighbors of the minority class samples in the training set, and selecting the data with the highest frequency from the K nearest neighbors; constructing new minority class samples based on the selected data by random linear interpolation; combining the newly constructed minority class samples with the training set to generate balanced gastric cancer data D2; the same is true for the validation set.

[0055] S13. Perform feature embedding on gastric cancer data D2.

[0056] The gastric cancer data D2 includes: discrete data and continuous data;

[0057] By embedding features of discrete data and continuous data separately, they are mapped to the same dimension, making it convenient for subsequent interactions at the same time.

[0058] For discrete data, an embedding matrix Map it into the d-dimensional vector space and express it as:

[0059] e c =M c x c

[0060] Among them, x c is the unique hot encoding representation of discrete data, N c is the number of discrete data.

[0061] For continuous data, all continuous data share a common embedding matrix. Then multiply it by the eigenvalue of the continuous data to get the embedding result of each continuous data, expressed as:

[0062] e n =Mn x n

[0063] Among them, x n is a continuous data value, N n is the number of consecutive data.

[0064] In the embodiment of the present invention, the result after feature embedding will be processed into two formats as the input of the subsequent modules. The first format, which is the general input format of the multi-head attention mechanism, is to stack each feature of each sample vertically, and use a matrix to represent all the features of a sample. This format is used for feature interaction. The second format is to stack each feature of each sample horizontally, and then stack multiple samples that have been stacked horizontally vertically. This format is used for sample interaction.

[0065] S2. Use the multi-head attention mechanism to perform feature interaction on the preprocessing results and fully capture the interactive information between features;

[0066] In this embodiment of the present invention, step S2 includes:

[0067] S21, stacking each feature of the sample in the preprocessing result vertically to obtain a feature matrix;

[0068] S22, linearly transform the feature matrix through multiple linear layers to obtain the matrix Q m , K m 、V m ; Where m represents the mth head;

[0069] S23. According to Q m and K m Calculate the attention weight of the mth head; add the attention weight of the mth head to the matrix V m Multiply them together to get the attention output of the mth head

[0070] Calculate the two features e of the samples in the preprocessing results i and e j The attention weight is expressed as:

[0071]

[0072] Where N is the number of features of the sample, j' represents the feature number, and e j′ is the j'th feature, ψ(e i ,e j ) is the inner product of the two, expressed as:

[0073] ψ(e i ,e j )= <Wq e i ,W k e j >

[0074] Among them, W q and W k are two learnable weight matrices.

[0075] Calculate the i-th feature e i Interaction results with other features It is expressed as:

[0076]

[0077] Among them, W v is the learnable weight matrix, a i,j For feature e i With feature e j The attention weight.

[0078] By introducing H heads, the attention mechanism can be transformed into a multi-head attention mechanism, and the single self-attention result can be transformed into the result of H attentions. For the i-th feature, the result set generated by its H heads is expressed as Then, the results of multiple heads are concatenated to form the final output of feature interaction. Similarly, the final output of the sample interaction is obtained

[0079] S24. Output the attention results of each head The splicing is performed and a linear transformation is performed on the splicing result through a linear layer to obtain the interactive information between features.

[0080] S3. Use the improved multi-head attention mechanism, i.e., multi-head row attention mechanism, to perform sample interaction on the preprocessing results and fully capture the interaction information between samples;

[0081] Step S3 and step S2 are performed in parallel.

[0082] The multi-head row attention mechanism also takes the preprocessing result as input. The difference from the multi-head attention mechanism is that its input is a matrix obtained by stacking each feature in a sample horizontally and then stacking multiple samples vertically. In essence, it performs vector calculations on a certain sample in each batch of samples with other samples to achieve the purpose of sample interaction. If two gastric cancer patient samples are combined, the correlation between them can be found, reflecting the different or common disease risk factors between different patients, which can supplement the diversity and quality of gastric cancer data, thereby improving the generalization ability of the model, and can eliminate noise and supplement the missing values ​​of this sample from other samples, further enhancing the robustness of risk prediction. The operation process is consistent with the step S2, with only a difference in input, and the final output of sample interaction is obtained.

[0083] S4. The interacted data obtained in step S2 and step S3 are respectively input into the normalization and feedforward neural network, and then weighted average is performed to obtain the fused gastric cancer data, which is finally input into the linear layer to obtain the gastric cancer liver metastasis risk result; layer normalization and feedforward neural network can improve the training speed and stability of the neural network, and provide a deeper understanding and representation of the input data.

[0084] In the embodiment of the present invention, the above process solves the problem of ignoring feature interaction and sample interaction in the existing gastric cancer liver metastasis risk prediction algorithm, and obtains a more comprehensive feature representation. At the same time, the attention weight matrix obtained by the above steps can be used to understand the role of each feature in the final risk prediction result.

[0085] The results obtained by the two attention mechanisms are input into the layer normalization and feedforward neural network respectively, expressed as:

[0086]

[0087]

[0088] Among them, LayerNorm is the layer normalization operation, GELU is the GELU activation function, and W FFN is the learnable weight matrix of the feedforward neural network, e i is the characteristic of the sample in the preprocessing result, is the sample interaction information, is the processing result of sample interaction information, is the feature interaction information, is the processing result of feature interaction information, and i is the index of the feature in the sample.

[0089] The outputs of the two multi-head attentions after layer normalization and feedforward neural network are weighted averaged and expressed as:

[0090]

[0091] Among them, W is the learnable weight matrix, W1 is the all-1 matrix, The result is weighted average.

[0092] The execution subject of the present invention may be a disease factor data processing device, or a terminal or a server, and the specific form is not limited.

[0093] The gastric cancer dataset (called GastricCancer) used in the present invention comes from the National Cancer Institute's Surveillance, Epidemiology, and End Results database, referred to as the SEER database. The information of the data is as follows Figure 2 shown.

[0094] The present invention uses the following indicators to evaluate the performance of the algorithm: AUC (used to evaluate the performance of binary classifiers), F1 (the harmonic mean of precision and recall, which can comprehensively consider the performance of the classifier on positive and negative samples). The experiment is divided into ablation experiments and comparative experiments. The results are as follows: Figure 3 , Figure 4 As shown, the ablation experiment results show that the model of the present invention that performs both feature interaction and sample interaction at the same time achieves better results in AUC and F1 indicators compared to the model that only performs feature interaction or sample interaction, proving that the method of the present invention is more effective. Comparative experiments show that compared with other models, the model used in the present invention has significantly improved AUC and F1 scores; the method of the present invention is better than other methods in AUC and F1 scores in gastric cancer datasets, which is sufficient to prove its effectiveness in predicting the risk of gastric cancer liver metastasis.

[0095] All or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program. The program can be stored in a computer-readable storage medium, which may include: ROM, RAM, magnetic disk or optical disk, etc.

[0096] The above embodiments further illustrate the purpose, technical solutions and advantages of the present invention in detail. It should be understood that the above embodiments are only preferred implementation modes of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made to the present invention within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A risk prediction method for gastric cancer liver metastasis based on multi-head attention mechanism, It is characterized in that include: Obtain gastric cancer data, input the gastric cancer data into the trained gastric cancer liver metastasis risk prediction model, and predict the gastric cancer liver metastasis risk results; The training process of the gastric cancer liver metastasis risk prediction model includes: S1. Acquire gastric cancer data, and preprocess the gastric cancer data to obtain a preprocessing result; the preprocessing result includes multiple samples, each sample includes a gastric cancer liver metastasis label and feature; S2. Use the multi-head attention mechanism to perform feature interaction on the preprocessing results to obtain feature interaction information; S3. Use the improved multi-head attention mechanism to perform sample interaction on the preprocessing results to obtain sample interaction information; S4. Obtain the risk result of gastric cancer liver metastasis using a neural network based on feature interaction information and sample interaction information; S5. Calculate the loss function value according to the gastric cancer liver metastasis risk result, and update the model parameters according to the loss function value; when the preset number of iterations is reached, complete the model training and obtain a trained gastric cancer liver metastasis risk prediction model; Using the multi-head attention mechanism to perform feature interaction on the preprocessing result includes: stacking each feature of the sample in the preprocessing result vertically to obtain a feature matrix; using the multi-head attention mechanism to process the feature matrix to obtain feature interaction information; Using the multi-head attention mechanism to process the feature matrix includes: S21, linearly transform the feature matrix through multiple linear layers to obtain the matrix Q m , K m 、V m ; Where m represents the mth head; S22, according to Q m and K m Calculate the attention weight of the mth head; add the attention weight of the mth head to the matrix V m Multiply them together to get the attention output of the mth head; S23, concatenating the attention output results of each head, and performing linear transformation on the concatenated results to obtain feature interaction information; Using the multi-head row attention mechanism to perform sample interaction on the preprocessing results includes: stacking each feature of the sample in the preprocessing results horizontally, and then stacking the horizontally stacked samples vertically to obtain a sample matrix; using the multi-head attention mechanism to process the sample matrix to obtain sample interaction information; Using the multi-head attention mechanism to process the sample matrix includes: S31, linearly transform the sample matrix through multiple linear layers to obtain the matrix Q h , K h 、V h ; where h represents the hth head; S32, according to Q h and K h Calculate the attention weight of the h-th head; add the attention weight of the h-th head to the matrix V h Multiply them together to get the attention output of the h-th head; S33, concatenating the attention output results of each head, and performing linear transformation on the concatenated results to obtain sample interaction information; The risk result of gastric cancer liver metastasis is obtained by using a neural network according to the feature interaction information and the sample interaction information, including: using layer normalization and a feedforward neural network to process the feature interaction information and the sample interaction information, and obtaining the processing result of the feature interaction information The processing results of the sample interaction information According to the processing results Calculate the weighted average result, and use the fully connected layer to obtain the risk result of gastric cancer liver metastasis based on the weighted average result; where i represents the index of the feature in the sample; Using layer normalization and feedforward neural network to process feature interaction information and sample interaction information respectively includes: Among them, LayerNorm is the layer normalization operation, GELU is the GELU activation function, and W FFN is the learnable weight matrix of the feedforward neural network, e i is the characteristic of the sample in the preprocessing result, is the sample interaction information, is the feature interaction information.

2. According to claim 1, a method for predicting the risk of gastric cancer liver metastasis based on a multi-head attention mechanism, It is characterized in that Preprocessing of gastric cancer data includes: S11, obtaining gastric cancer data, and performing preliminary processing on the gastric cancer data to obtain gastric cancer data D after preliminary processing 1 ; S12. Using SMOTE-NC oversampling method to process gastric cancer data D 1 , and get the balanced gastric cancer data D 2 ; S13. Gastric cancer data D 2 Perform feature embedding.

3. According to claim 2, a method for predicting the risk of gastric cancer liver metastasis based on a multi-head attention mechanism, It is characterized in that The preliminary processing of gastric cancer data includes: converting the gastric cancer data to obtain structured tabular data; and cleaning the structured tabular data.

4. According to claim 2, a method for predicting the risk of gastric cancer liver metastasis based on a multi-head attention mechanism, It is characterized in that The gastric cancer data 1 Including multiple samples; using SMOTE-NC oversampling method to process gastric cancer data D 1 Including: Using the nearest neighbor algorithm to calculate gastric cancer data D 1 The K nearest neighbor samples of the minority class sample are selected, and the sample with the highest frequency is selected from the K nearest neighbor samples; a new minority class sample is constructed based on the selected sample by random linear interpolation; the new minority class sample is compared with the gastric cancer data D 1 Synthesize and obtain the balanced gastric cancer data D 2 .

5. According to claim 2, a method for predicting the risk of gastric cancer liver metastasis based on a multi-head attention mechanism, It is characterized in that The gastric cancer data 2 Includes: discrete data and continuous data; gastric cancer data D 2 Feature embedding includes: calculating gastric cancer data D 2 The one-hot encoding representation of discrete data is used to embed the one-hot encoding representation using the embedding matrix to obtain the features of discrete data embedding; the gastric cancer data D is embedded using the embedding matrix 2 The continuous data is feature embedded to obtain the features embedded in the continuous data.

6. The method for predicting the risk of gastric cancer liver metastasis based on a multi-head attention mechanism according to claim 1, It is characterized in that According to the processing results The weighted average calculation results include: Among them, W is the learnable weight matrix, W 1 is a matrix of all 1s, The result is weighted average.

Images