A prediction method and model for microvascular invasion of hepatocellular carcinoma

Abstract

The present invention discloses a prediction model for microvascular invasion of hepatocellular carcinoma, comprising: a feature extraction module, a modal attention module and a dual-channel example feature aggregation module. The present invention also discloses a method for predicting microvascular invasion of hepatocellular carcinoma. Compared with the previous two-dimensional-based deep learning model, the modal attention and dual-channel multi-instance deep learning model of the present invention inputs all two-dimensional slices of the three-dimensional tumor, which will not cause information loss, and therefore has higher accuracy. Compared with the three-dimensional model, it extracts the features of all slices through a two-dimensional feature extractor, and at the same time, a better feature representation can be obtained through a dual-channel multi-instance aggregator, so it not only has higher accuracy, but also requires fewer computing resources and is easier to train.

Description

Technical Field

[0001] The present invention relates to the technical field of imaging medicine and imaging omics, and more specifically to a method and model for predicting microvascular invasion of hepatocellular carcinoma. Background Art

[0002] Hepatocellular carcinoma (HCC) is one of the most common liver malignancies in the world. Treatment options for HCC include liver resection, transarterial chemoembolization, radiofrequency ablation, systemic therapy, and liver transplantation. Identifying risk factors for HCC recurrence can provide clinicians with more aggressive treatment strategies.

[0003] Studies have shown that microvascular invasion (MVI) is a key prognostic factor for postoperative recurrence in HCC patients. Currently, MVI can only be identified by postoperative histopathological examination. However, for HCC patients, if their MVI status can be accurately predicted before surgery, their postoperative recurrence rate will be reduced by formulating more reasonable treatment strategies (for example, expanding the tumor resection margin for high-risk MVI patients). Therefore, it is very necessary to predict MVI through preoperative medical imaging.

[0004] Many studies in the existing technology have predicted MVI through radiomics and achieved good prediction results. Since radiomics-based methods rely on expert knowledge, they require manual feature extraction, which is time-consuming and labor-intensive, and the prediction results are unstable. Therefore, end-to-end prediction methods based on deep learning have gradually become mainstream. There are two types of prediction methods for microvascular invasion of hepatocellular carcinoma based on deep learning: one is based on a two-dimensional deep learning method, which inputs a slice of a three-dimensional tumor into the model to obtain the prediction result. The other is based on a three-dimensional deep learning method, which inputs the entire three-dimensional tumor into the model to obtain the prediction result.

[0005] The current two-dimensional deep learning model only takes one slice of the tumor, ignoring the contextual information of the lesion. The three-dimensional deep learning model inputs the entire tumor, resulting in high computing resource consumption and difficulty in training. At the same time, it is also prone to overfitting due to the small amount of data. Summary of the invention

[0006] The purpose of the present invention is to provide a method and model for predicting microvascular invasion of hepatocellular carcinoma, which can solve the shortcoming of low accuracy in the prior art.

[0007] The present invention adopts the following technical solutions:

[0008] A prediction model for microvascular invasion of hepatocellular carcinoma includes a feature extraction module, a modality attention module, and a dual-channel multi-instance feature aggregation module.

[0009] The feature extraction module consists of five convolutional layers and one fully connected layer. Each convolutional layer is followed by a Relu activation function, a Batch normalization layer, and a pooling layer. The Batch normalization layer is used to accelerate model convergence, the Relu layer is used to increase the nonlinearity of the model, and the pooling layer is used to reduce the size of the feature map.

[0010] The modal attention module replaces the number of channels in senet with the number of MRI sequences. It includes three operations: compression, excitation and reweighting. The compression operation compresses all two-dimensional slices in all phases of three-dimensional tumor MRI into a modal vector. The excitation operation maps the vector into a set of weights. The reweighting operation multiplies the weights with the original five modal vectors for reweighting.

[0011] The dual-channel multi-instance feature aggregation module has two channels. The upper channel obtains a maximum score value through multi-instance maximum pooling and finds the key example feature. The lower channel calculates the similarity between other example features by calculating the cosine similarity between them and the key example feature. The more similar the example features are, the larger the weight is given. Then, the features of each example are added together according to the weight to obtain the features of the package. Finally, the features of the package are passed through a classifier to obtain the prediction score. Finally, the prediction score of the package is added to the prediction score of the key example feature to obtain the total prediction score.

[0012] A method for predicting microvascular invasion of hepatocellular carcinoma comprises the following steps:

[0013] S1: Collect hepatocellular carcinoma MRI data sets, each MRI data set includes five sequences: arterial phase, portal phase, delayed phase, diffusion sequence and T2-weighted sequence;

[0014] S2: A rectangular box was used to outline the region of interest of the tumor in each arterial phase MRI slice at a distance of 5–10 mm from the tumor edge;

[0015] S3: The arterial phase MRI is registered with the portal phase, delayed phase, diffusion sequence, and T2-weighted MRI, and then the three-dimensional tumor in each phase is cropped;

[0016] S4: The dataset was divided into a training set and a validation set in a ratio of 4:1, and a five-fold cross validation was performed;

[0017] S5: A single sequence 3D MRI lesion is considered as a multi-instance learning bag, and all slices in the bag are considered as examples;

[0018] S6: Before inputting the five sequence packets into the model, the slices are uniformly scaled to S*64*64, where S represents the number of slices and 64*64 represents the length and width of the slice;

[0019] S7: Each MRI slice of each modality first passes through a shared feature extraction module to obtain S*128-dimensional features;

[0020] S8: After concatenating the features of the five MRI modalities, the features of the example dimension become 5*128 dimensions, and then input into the modality attention module. Through the modality attention module, the modality that helps more in prediction is given a more important weight;

[0021] S9: Input the example features output from the modality attention module into the dual-channel multi-example feature aggregation module;

[0022] S10: In the first channel of the dual-channel multi-instance feature aggregation module, the key example features are found through multi-instance maximum pooling, and the prediction scores of the key example features are obtained. c m (B) , the formula is as follows:

[0023] c m (B) = max{W0h1,…,W0h n};

[0024] S11: In the second channel of the dual-channel multi-instance feature aggregation module, the similarity coefficient between the key example feature and other example features is obtained by calculating the cosine similarity between the key example feature and other example features. The formula is as follows:

[0025] ;

[0026] S12: According to the similarity coefficient, all the example features are weighted to obtain the features of a package:

[0027] ;

[0028] S13: Through a fully connected layer and activation function, the prediction score based on the package features is obtained, the formula is as follows:

[0029] c b (B) = W b b;

[0030] S14: Add the prediction score based on the key example features and the prediction score based on the package to obtain the final prediction score, which is as follows:

[0031] .

[0032] The advantages of the present invention are: compared with the previous two-dimensional deep learning model, the modal attention and dual-channel multi-instance deep learning model of the present invention inputs all two-dimensional slices of the three-dimensional tumor, which will not cause information loss, so the accuracy is higher. Compared with the three-dimensional model, the features of all slices are extracted by the two-dimensional feature extraction module, and a better feature representation can be obtained by the dual-channel multi-instance feature aggregation module, so not only the accuracy is higher, but also less computing resources are required and it is easier to train. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The present invention is described in detail below with reference to the embodiments and accompanying drawings, wherein:

[0034] Figure 1 : MRI images of hepatocytes in five sequences: arterial phase, portal phase, delayed phase, diffusion sequence and T2-weighted; among them, (A) arterial phase, (B) delayed phase, (C) portal phase, (D) T2-weighted, (E) diffusion sequence MRI images of hepatocytes;

[0035] Figure 2 :Processing workflow of hepatocellular carcinoma MRI images;

[0036] Figure 3 : A schematic diagram of the architecture of the feature extraction module of the present invention;

[0037] Figure 4 : The feature extraction module structure of two-dimensional slices in the implementation of the present invention;

[0038] Figure 5 : The dual-channel multi-instance feature aggregation module structure in the implementation of the present invention. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0040] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in sequences other than those illustrated or described herein.

[0041] Depending on the context, "if" as used herein may be interpreted as "when" or "when" or "in response to determining" or "in response to detecting."

[0042] When doctors diagnose diseases through medical images, they will check all two-dimensional slices of medical images (such as MRI). When at least one of them is found to be problematic, the patient will be judged to have a problem. When all slices are fine, it will be judged as no problem. The present invention constructs a method for predicting microvascular invasion of hepatocellular carcinoma based on modal attention and dual-channel multi-instance deep learning model.

[0043] The invention discloses a hepatocellular carcinoma microvascular invasion prediction model, comprising: a feature extraction module, a modal attention module and a dual-channel multi-example feature aggregation module.

[0044] 1. Feature extraction module

[0045] The feature extraction module structure consists of five convolutional layers and one fully connected layer. Each convolutional layer is followed by a Relu activation function, a Batch normalization layer, and a pooling layer. The Batch normalization layer is used to accelerate model convergence, the Relu layer is used to increase the nonlinearity of the model, and the pooling layer is used to reduce the size of the feature map. Figure 3 As shown in , the convolution kernel size of each convolution layer in the feature extraction module is 3×3, and the maximum pooling is 2×2. The number of channels in each convolution layer is 16, 32, 64, 128, and 256. The number of neurons in the fully connected layer is 128. Figure 4 As shown in the figure, (s, 1, 64, 64) represents the number of slices, channels, length and width of slices in MRI, respectively. In the feature extraction module, the input is a 1×64×64 MRI slice, and after five convolutional layers, the output feature dimension is 256×2×2. When flattened, the dimension of the feature map from the convolutional layer is 4096, which is reduced to 128 by the fully connected layer.

[0046] 2. Modality Attention Module

[0047] The modality attention module is modified from senet. By replacing the number of channels in senet with the number of MRI sequences, the model focuses on one or several modal MRIs. The modality attention module includes three operations: compression, excitation, and reweighting. The compression operation compresses all two-dimensional slices in all phases of three-dimensional tumor MRI into a modality vector. The excitation operation maps the vector into a set of weights. The reweighting operation multiplies the weights with the original five modality vectors for reweighting.

[0048] Here, let M = {m1, m2, m3, m4, m5} refer to the features of five MRI modalities. mk∈R S×128 (k=1, 2, 3, 4, 5) represents the modal feature tensor of the kth mode, where s represents the number of slices and 128 represents the dimension of the feature tensor of each slice. Global average pooling is used to compress the modal descriptor, as shown below:

[0049] ;

[0050] Where mk(i,j) represents the jth element in the i-th slice of mk. The excitation operator maps the descriptor to a set of modal weights. The SENET is simulated by two fully connected layers and a ReLU structure. It is described as follows:

[0051]

[0052] where d = {d1, d2, d3, d4, d5} is the modality descriptor vector, W1 and W2 are the weight vectors of the two fully connected layers, and a = {a1, a2, a3, a4, a5} is the attention vector.

[0053] The reweighting operator ensures that a specific MRI modality is emphasized by multiplying the attention vector a={a1, a2, a3, a4, a5} with the original multimodal features M={m1, m2, m3, m4, m5}. It is described as follows:

[0054]

[0055] in Represents the reweighted features of the five MRI modalities.

[0056] 3. Dual-channel multi-instance feature aggregation module

[0057] The dual-channel multi-instance feature aggregation module has two channels. The upper channel obtains a maximum score value through multi-instance max pooling and finds the key example feature. The lower channel calculates the similarity between other example features and the key example feature by calculating the cosine similarity between them. The more similar example features are given a larger weight, and then the features of each example are added together by weight to obtain the features of the package. Finally, the features of the package are passed through a classifier to obtain the prediction score. Finally, the prediction score of the package is added to the prediction score of the key example feature to obtain the total prediction score.

[0058] The dual-channel multi-instance feature aggregation module is used to operate on pathological image patches. Let B = {x1, ..., xn} represent a set of MRI slices of a specific modality, where x i Represents example i. xi After passing through the feature extraction module and the modality attention module, it can be projected into the embedding The first pass of the module uses an example feature classifier on each example feature embedding and performs multi-instance learning max pooling on the scores to obtain the key example features h m and the highest score c m (B) :

[0059] c m (B) = max{ W 0 h 1 ,…,W 0 h n};

[0060] in W 0 is a weight vector and represents a classifier. W 0 h i Indicates that the classifier is obtained h i The prediction score of . Then max{ W 0 h 1 ,…,W 0 h n} means in W 0 h i Get the maximum score.

[0061] The second pass aggregates all the example features into a package feature, which is then passed through the package classifier to get the prediction score of the package. Specifically, each example feature h i (include h m ) is converted into two vectors: and , the conversion process is as follows:

[0062] q i = W q h i , v i =W v h i , i = 1, . . . , n ;

[0063] in W q and W v Represents the weight matrix ，q i and v i yes h i Another linear representation of .

[0064] The feature extraction module is responsible for extracting example features, while the modality attention module simulates the doctor's decision-making process and helps the model focus on the most important MRI sequences. The two-stream example feature aggregation module aggregates the features of all examples into bag features, enabling the model to focus on key MRI slices. Each MRI modality is first processed by a feature sharing extraction module, which produces a feature tensor of dimension s×128. After concatenating the feature tensors of the five MRI modalities, the dimension of the example feature tensor becomes 5×128. The tensor is then input into the modality attention module, which assigns higher weights to the MRI modality with the highest prediction effect. The features of the example are then aggregated to form bag features, which are used to produce the final prediction results.

[0065] The present invention also discloses a method for predicting microvascular invasion of hepatocellular carcinoma, comprising the following steps:

[0066] S1: Collect hepatocellular carcinoma MRI data sets, each MRI data includes five sequences: arterial phase, portal venous phase, delayed phase, diffusion sequence and T2-weighted sequence.

[0067] S2: A radiologist with 5 years of experience used a rectangular frame to outline the region of interest of the tumor on each arterial phase MRI slice at a distance of 5–10 mm from the tumor edge, and another radiologist with 20 years of experience confirmed it;

[0068] S3: The arterial phase MRI is registered with the portal phase, delayed phase, diffusion sequence, and T2-weighted MRI, and then the three-dimensional tumor in each phase is cropped;

[0069] S4: The dataset was divided into a training set and a validation set in a ratio of 4:1, and a five-fold cross validation was performed;

[0070] S5: A single sequence 3D MRI lesion is considered as a multi-instance learning bag, and all slices in the bag are considered as examples;

[0071] S6: Before inputting the five sequence packets into the model, the slices are uniformly scaled to S*64*64, where S represents the number of slices and 64*64 represents the length and width of the slice;

[0072] S7: Each MRI slice of each modality first passes through a shared feature extraction module to obtain S*128-dimensional features;

[0073] S8: After concatenating the features of the five MRI modalities, the feature of the example dimension becomes 5*128 dimensions, and then input into the modality attention module. Through the modality attention module, the modules that are more helpful for prediction are given more important weights;

[0074] S9: Input the example features output from the modality attention module into the dual-channel multi-example feature aggregation module;

[0075] S10: In the first channel of the dual-channel multi-instance feature aggregation module, the key examples are found through multi-instance maximum pooling, and the prediction scores of the key examples are obtained. c m (B) , the formula is as follows:

[0076] c m (B) = max{W0h1,…,W0h n};

[0077] in, c m (B) represents the prediction score of the key example, h i represents the example features, and W0 represents the matrix coefficients of the example classifier.

[0078] S11: In the second channel of the dual-channel multi-instance feature aggregation module, the similarity coefficient between the key example and other examples is obtained by calculating the cosine similarity between the key example and other examples. The formula is as follows:

[0079] ;

[0080] U (h i ,h m ) represents the similarity coefficient, i ,q m > represents example q i and key examples q m ​The inner product of , which also represents the cosine similarity between them, is converted into U ( h i , h m ) normalize so that all U ( h i , h m ) and is 1.

[0081] S12: According to the similarity coefficient, all the example features are weighted to obtain the features of a package:

[0082] ;

[0083] b represents the characteristics of the package, v i Represents the transformation vector of the example, which converts all the example features into the same vector according to the similarity coefficient with the key example feature v i Weighted to get the characteristics of the package b .

[0084] S13: Through a fully connected layer and activation function, the prediction score based on the packet features is obtained, the formula is as follows:

[0085] c b ( B ) =W b b;

[0086] c b ( B ) represents the prediction score based on the package, W b Represents the matrix coefficients of the example classifier.

[0087] S14: Add the prediction score based on key example features and the prediction score based on the package to get the final prediction score c ( B ), the formula is as follows:

[0088]

[0089] in, c m ( B ) represents the prediction score based on the key example, c b ( B ) represents the bag-based prediction score, c ( B ) represents the final prediction score.

[0090] In this implementation, the above model is implemented using the Pytorch open source deep learning framework. All experiments were performed on a Dell T640 workstation with two independent NVIDIA GeForce RTX 2080Ti graphics cards and two Intel Xeon Silver 4110 CPUs. In the experiment, the cross entropy loss function and the adam optimizer were used. In the present invention, s represents the number of slices in the MRI volume, and 64 represents the width and height of the slice. Example

[0091] The experiment of this method verifies the model through indicators such as accuracy, sensitivity, specificity, negative predictive value NPV, positive predictive value PPV, and area under the curve AUC.

[0092] 1) First, in order to verify the effectiveness of each module of this method, ablation experiments were conducted on the models respectively. It is assumed that the features extracted by the feature extraction module are directly classified and predicted as becnmark, and then the modal attention module and the dual-channel multi-instance feature aggregation module are added to verify the prediction performance of the model. The experimental results are shown in Table 1:

[0093] Table 1

[0094] As shown in the figure, when only the benchmark is used, the model performance is the worst, with an accuracy of 67.14%, a sensitivity of 73.89%, a specificity of 60.3%, an NPV of 67.38%, a PPV of 67.77, and an AUC of 0.6710. When the modality attention module and the dual-channel multi-instance feature aggregation module are added, the prediction performance is improved. When all modules are added, the model performance is the best, with an accuracy of 76.43%, a sensitivity of 71.41%, a specificity of 80.36%, an NPV of 74.16%, a PPV of 81.74%, and an AUC of 0.7422.

[0095] 2) Since feature extraction has a great influence on the model effect, the model is prone to overfitting when the number of layers of the feature extraction module is deep, and the features may not be fully extracted when it is shallow. In order to determine the number of convolution layers of the feature extraction module, experiments were conducted on feature extraction modules with 4, 5, 6, 7, and 8 convolution layers. The results are shown in the following table:

[0096] Table 2

[0097] It can be seen from the above table that when the number of convolutional layers of the feature extraction module is 5, the prediction effect of the model is the best. When the number of convolutional layers is less than 5 or more than 5, the effect is poor.

[0098] 3) Doctors often focus on certain important MRI modalities when diagnosing diseases, so they have more accurate diagnostic results based on these modalities. In order to verify the effectiveness of the model, we tested the prediction performance of the model under a single MRI modality, and the results are as follows:

[0099]

[0100] Table 3

[0101] As can be seen from the table above, when only the arterial phase images are input, the overall prediction performance of the model is the best, with an accuracy of 72.50%, a sensitivity of 76.73%, a specificity of 68.81%, an NPV of 76.19%, a PPV of 72.30%, and an AUC of 0.69.74. When only the portal phase (Venous) and diffusion phase (ADC) are input, the prediction performance is second. When only the delayed phase (Delay) and T2-weighted phase images are input, the effect is the worst. These results are consistent with the clinical diagnostic experience of doctors, which also proves the effectiveness of the model.

[0102] 4) Since the data augmentation strategy also has a significant impact on the model prediction performance, we verified the impact of the model with and without data augmentation strategy. For the data augmentation strategy, we randomly flipped and rotated the MRI slices in each package by 90° vertically. The results are shown in the following table:

[0103]

[0104] Table 4

[0105] As can be seen from the table, when there is no random enhancement, the accuracy is 70.71%, the sensitivity is 70.26%, the specificity is 70.22%, the NPV is 69.75%, the PPV is 72.30%, and the AUC is 0.6804. When data enhancement is performed, the prediction performance of the model is greatly improved compared to without data enhancement. From the above structure, it can be seen that the enhancement combination of each slice in the package greatly increases the increase in the data of the package, thus greatly improving the prediction performance of the model compared to conventional data enhancement.

[0106] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for predicting microvascular invasion of hepatocellular carcinoma, characterized in that: The following steps are involved: S1: Collect hepatocellular carcinoma MRI data sets, each MRI data set includes five sequences: arterial phase, portal venous phase, delayed phase, diffusion sequence and T2-weighted sequence; S2: A rectangular box was used to outline the region of interest of the tumor in each arterial phase MRI slice at a distance of 5–10 mm from the tumor edge; S3: The arterial phase MRI is registered with the portal phase, delayed phase, diffusion sequence, and T2-weighted MRI, and then the three-dimensional tumor in each phase is cropped; S4: The dataset was divided into a training set and a validation set in a ratio of 4:1, and a five-fold cross validation was performed; S5: A single sequence 3D MRI lesion is considered as a multi-instance learning bag, and all slices in the bag are considered as examples; S6: Before inputting the five sequence packets into the model, the slices are uniformly scaled to S*64*64, where S represents the number of slices and 64*64 represents the length and width of the slice; S7: Each MRI slice of each modality first passes through a shared feature extraction module to obtain S*128-dimensional features; The extraction module consists of five convolutional layers and one fully connected layer. Each convolutional layer is followed by a Relu activation function, a batch normalization layer, and a pooling layer. The batch normalization layer is used to accelerate model convergence, the Relu layer is used to increase the nonlinearity of the model, and the pooling layer is used to reduce the size of the feature map. The convolution kernel size of each convolutional layer in the feature extraction module is 3×3, the maximum pool is 2×2, the number of channels in each convolutional layer is 16, 32, 64, 128, and 256, respectively, and the number of neurons in the fully connected layer is 128. S8: The features of the five MRI modalities are concatenated and input into the modality attention module. Through the modality attention module, the modality that helps the prediction most is given an important weight; The modality attention module replaces the number of channels in senet with the number of MRI sequences. The modality attention module includes three operations: compression, excitation, and reweighting. The specific method of the modal attention module is: M = {m1, m2, m3, m4, m5} refers to the features of the five MRI modalities, mk∈R S×128 (k=1, 2, 3, 4, 5) represents the modal feature tensor of the kth mode, where s represents the number of slices and 128 represents the dimension of the feature tensor of each slice; global average pooling is used to compress and obtain the modal descriptor, as shown below: ； Where mk(i, j) represents the jth element in the i-th slice of mk; The excitation operator maps the modality descriptor to a set of modality weights; the SENET is simulated by two fully connected layers and the ReLU structure, as described below: a=W2relu(W1d) where d = {d1, d2, d3, d4, d5} is the modality descriptor vector, W1 and W2 are the weight vectors of the two fully connected layers, and a = {a1, a2, a3, a4, a5} is the attention vector; The reweighting operator emphasizes the MRI modality by multiplying the attention vector a = {a1, a2, a3, a4, a5} with the original multimodal features M = {m1, m2, m3, m4, m5}, as described below: ,in Represents the reweighted features of the five MRI modalities; B = {x1, ..., xn} represents a set of MRI slices of a modality, where x i Represents example i, x i After passing through the feature extraction module and the modality attention module, it is projected into the embedding h i ; S9: Input the example features output from the modality attention module into the dual-channel multi-example feature aggregation module; S10: In the first channel of the dual-channel multi-instance feature aggregation module, find the key example feature h m , and get the key example feature h m The prediction score c m (B), where the key example feature h m The prediction score c m (B) is the feature h obtained for each example i The maximum score among the predicted scores of ; Including h m Each example feature h i is transformed into two vectors: q i and v i , the transformation process is as follows: q i =W q h i ,v i =W v h i ,i=1,...,n; where W q and W v represents the weight matrix, q i and v i Yes i Another linear representation of ; S11: In the second channel of the dual-channel multi-instance feature aggregation module, by calculating the key example feature h m The cosine similarity with other example features is used to obtain the similarity coefficient between other example features and key example features. The formula is as follows: ; Among them, U(h i ,h m ) represents the similarity coefficient, i ,q m > indicates q i and q m The inner product of​ S12: According to the similarity coefficient, all the example features v i Weighted, we get the characteristics of a package: ； Among them, b represents the characteristics of the package, v i Denotes the example feature h i The transformation vector of S13: Through a fully connected layer and activation function, the prediction score c based on the package features is obtained b (B) ; S14: Add the prediction score based on the key example features and the prediction score based on the package to obtain the final prediction score, which is as follows:

Images