Intelligent diagnosis method for biliary atresia based on multimodal medical data

Abstract

The present invention proposes an intelligent biliary atresia recognition model based on multimodal medical data, which belongs to the field of data processing. In the data preprocessing stage, the key position information of the multimodal data is acquired; in the feature extraction stage, the visual image feature encoding and numerical data feature encoding are used to provide a basis for subsequent modal fusion; in the modal fusion stage, through multi-level fine-grained fusion, the image features are first innovatively fused using the masked self-attention mechanism for fine-grained fusion, and then multimodal fusion is performed at the overall level, effectively improving the fusion effect under modality loss, thereby effectively capturing the key features between and within the modalities and interactively fusing them, providing a basis for high-precision prediction. In order to improve the model training efficiency and prediction accuracy, a multi-loss training method is used, which not only incorporates the multimodal fusion feature loss, but also adds a certain proportion of the loss of single-modal calculation. Finally, the intelligent recognition and prediction of biliary atresia are performed by training the model weights.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence and intelligent identification of biliary atresia (G06F 19 / 00), and specifically relates to a biliary atresia identification method based on multimodal medical data. Background Art

[0002] Biliary atresia (BA) is a rare idiopathic fibro-obliterative bile duct disease that occurs in the perinatal period. The incidence of BA ranges from 1:20,000 in North America and Europe to 1:5,000–1.1:10,000 in Asia. [1-5] It is the leading cause of liver transplantation in children. Without treatment, BA inevitably progresses to end-stage liver disease and leads to death within the first two years of life. Since 1968, Kasai portoenterostomy and liver transplantation have significantly prolonged survival and improved quality of life. [6] Early age at the time of Kasai portoenterostomy is one of the most important factors in predicting surgical outcome, with the best results when performed before 30 to 45 days of age. [7-9] ; After that, the chances of success decrease significantly with age. Unfortunately, the median age at the time of Kasai surgery is about 60 days, and there is no improvement over time. [7，8] The main reason for the delay is the lack of effective and practical screening methods, so early diagnosis is a prominent and ongoing clinical challenge.

[0003] With the rapid development of artificial intelligence (AI) in recent years, computer-aided diagnosis (CAD) has become a research hotspot. Deep learning, a powerful AI technique, has been widely applied to medical data analysis. Deep neural networks are capable of learning complex representations of data, achieving state-of-the-art performance in various medical imaging tasks.

[0004] Some studies have attempted to use clinical information and laboratory test data to build an artificial neural network to assist in the diagnosis of biliary atresia. [1] There are also studies that add manually extracted ultrasound image features to these types of data.

[10] However, manual feature extraction relies on the doctor's experience, and simple artificial neural networks and multivariate logistic regression analysis methods cannot fully fit the complex relationship between features. Using deep learning technology to analyze gallbladder images for diagnosis can achieve expert-level accuracy.

[11] However, using only a single modality will ignore other information that is meaningful for diagnosis and is not enough to achieve high-precision auxiliary diagnosis.

[0005] Doctors' diagnosis of biliary atresia relies on a multi-faceted examination of the patient. However, the multimodal medical data generated by laboratory tests and ultrasound examinations at multiple locations performed by the patient have not been systematically collected and effectively utilized. The diagnostic performance of all current computer-assisted diagnosis methods for biliary atresia still has much room for improvement. More research is needed to further improve computer-assisted diagnosis systems and apply them to clinical practice.

[0006] A multimodal learning framework should be able to learn from data of different modalities without requiring different model architectures. A unified multimodal model will combine different types of data, align the representations of data from different modalities, and use them for downstream tasks. In the past few years, there has been a transition from architectures applied to specific modalities, such as convolutional neural networks for images or recurrent neural networks for text, to a relatively new architecture, the Transformer, which has shown good performance on a variety of input and output modalities and tasks.

[12] However, there is no good solution to the problem of modality loss and multimodal information redundancy in medical data, and different types of tasks require separate designs. Summary of the Invention

[0007] To align closely with physicians' analytical thinking, we drew inspiration from clinicians' analytical and judgment processes and proposed a multimodal deep learning model. This model effectively addresses modality loss and fusion issues within multimodal analysis, comprehensively analyzing the multimodal data generated by patients undergoing multi-faceted examinations. We had professional physicians collect and annotate large amounts of multimodal data related to biliary atresia and applied this multimodal model to develop an intelligent identification model for biliary atresia.

[0008] Because clinicians often use ultrasound images and clinical data to identify biliary atresia, we enabled computers to analyze ultrasound images, clinical information, and laboratory data based on physician experience to arrive at a final classification result. Therefore, we proposed a multimodal deep learning method that can be adapted to assist in the identification task using multimodal medical data. We used collected medical data related to biliary atresia for training and testing, verifying the model's effectiveness in assisting medical decision-making for biliary atresia.

[0009] Biliary atresia multimodal data includes ultrasound images, clinical data, and laboratory test data. Ultrasound images include the gallbladder, fibrous plaques, liver capsule, and liver parenchyma. Clinical data include gender, age, premature delivery, gestational age, weight at onset, jaundice, clay-like stools, and dark urine. Laboratory test data include total bilirubin (TB), direct bilirubin (DB), total bile acid (TBA), total protein (TP), albumin (ALB), alanine aminotransferase (ALT), aspartate aminotransferase (AST), gamma-glutamyl transpeptidase (GGT), alkaline phosphatase (ALP), platelets (PLT), white blood cells (WBC), prothrombin time (PT), and activated partial thromboplastin time (APTT).

[0010] To comprehensively analyze diverse medical data, multimodal models must be able to adapt to different modalities and effectively integrate them for analysis. Our proposed model consists of four main components: data mapping and enhancement, feature extraction, modality fusion, and multi-loss joint training. For multimodal input, data processing methods corresponding to different modalities are used. Different strategies are employed for handling missing values. For missing ultrasound images of tissue or organ locations, gallbladder images of the same patient are used to fill in the gaps. If the gallbladder is also missing, images of fibrous plaques are used. If both gallbladder and fibrous plaque images of the same patient are missing, null values ​​are used to explicitly indicate their absence. Missing numerical data is interpolated using the mean value from the training set. For multimodal information interaction and fusion, features are extracted using separate data encoders from different modalities and then fused in a high-dimensional space. To improve training stability and model generalization, a converged optimization approach using multiple loss functions is employed. As a general approach, these four components can be viewed as plug-and-play modules, allowing for flexible switching to meet task requirements.

[0011] The present invention comprises the following steps:

[0012] Step S1: data preprocessing;

[0013] Step S11: The doctor labels the ultrasound image for organs, including the locations of four tissues: gallbladder, fibrous plaque, liver capsule, and liver parenchyma, as well as a label indicating whether it is biliary atresia.

[0014] Step S12: Clean the data using outlier analysis;

[0015] Step S2: feature extraction and missing processing;

[0016] Step S21: extract four tissue positions from the ultrasound image obtained by processing in step S11, use the visual model to extract features from the ultrasound image of each tissue, and obtain image features of the four tissue positions.

[0017] Step S22: Encode the clinical data and laboratory test data obtained in step S12 by one-hot encoding and normalization to obtain numerical features;

[0018] Step S23: Modally fill in missing data: If the ultrasound image of a tissue or organ location is missing, fill it in with the gallbladder image of the same patient. If the gallbladder is also missing, fill it in with the fibrous plaque image. If both the gallbladder image and the fibrous plaque image of the same patient are missing, fill it in with a null value to explicitly indicate its absence. If numerical data is missing, interpolate and fill it in with the mean value in the training set.

[0019] Step S3: training a biliary atresia intelligent recognition model;

[0020] Step S31: Based on the ultrasound image features of multiple tissues obtained in step S2, the proposed masked self-attention mechanism is used to perform fine-grained fusion of the ultrasound image features of each tissue.

[0021] Step S32: For the fused multi-position ultrasound image features, multi-information fusion is performed with the numerical features through a multi-layer perceptron.

[0022] Step S33: The fusion features obtained in step S32 are passed through the fully connected layer to predict the result, and the loss is calculated using the cross entropy function.

[0023] Step S34: Calculate the cross entropy loss between the predicted result obtained by the fully connected layer of the features extracted in step S2 and the true value.

[0024] Step S35: The losses calculated in steps S33 and S34 are used together to calculate the gradient and backpropagate the training model to obtain the model weights.

[0025] Based on a thorough analysis of the existing deep model applications and the lack of multimodal medical data, the present invention sets up a model with pluggable module design and masked self-attention mechanism to effectively deal with the training difficulties caused by modality loss, and improves the effective information fusion effect through a multi-level and multi-granular approach in the modal fusion stage. Experimental results prove that the present invention has reached a very advanced level in intelligent identification of biliary atresia based on multimodal medical data.

[0026] The beneficial effects of the present invention are:

[0027] 1. Based on multimodal medical data and expert diagnostic experience, the model achieves an AUC accuracy of 0.986 for intelligent diagnosis of biliary atresia, exceeding that of an expert doctor with 10 years of experience.

[0028] 2. This study is based on the application of multimodal medical data. In multimodal fusion and multimodal loss processing, it uses an innovative multi-level fine-grained fusion scheme and a masked self-attention mechanism, which has important research significance and huge application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 This is a schematic diagram of the overall process of the intelligent recognition model for biliary atresia based on multimodal medical data proposed in the present invention.

[0030] Figure 2 This is a multi-loss training structure diagram of the biliary atresia intelligent recognition model based on multimodal medical data in the present invention.

[0031] Figure 3 This is a visual explanation diagram of the results analysis in this invention

[0032] Figure 4 The results of the present invention in the retrospective experimental test are shown in FIG

[0033] Figure 5 This is the result diagram of the present invention in the prospective experimental test

[0034] Figure 6 This is a graph showing the improvement in accuracy brought about by the present invention in assisting doctors in analysis and judgment; af is the improvement result of retrospective experimental test, and gl is the improvement result of prospective experimental test

[0035] Figure 7 This is a simplified model experimental structure diagram of the present invention

[0036] Figure 8 This is a flow chart of the application of the present invention. DETAILED DESCRIPTION

[0037] In order to make the objectives, technical solutions and advantages of the present invention more clear, the present invention is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings. It should be understood that the embodiments provided below are only intended to fully and comprehensively disclose the present invention and fully convey the technical concept of the present invention to those skilled in the art, and are not intended to limit the scope of the present invention.

[0038] See also Figure 1 When the present invention is used for intelligent identification of biliary atresia, the specific processing steps are as follows:

[0039] Step S1: data preprocessing;

[0040] Step S11: Extract the ultrasound image annotation information, such as Figure 8 As shown on the left,

[0041] Step S12: Perform one-hot encoding on the discrete values ​​in the clinical data and laboratory test data, including gender, whether premature, whether jaundice, whether clay stool, whether dark urine, and perform normalization on the continuous values, such as Figure 1 Data preprocessing includes data cleaning, data discretization, and data normalization;

[0042] Step S2: feature extraction;

[0043] Step S21: Extract four tissue positions from the ultrasound image obtained in step S11, and use the visual model to extract features from the ultrasound image of each tissue. The extracted features are represented as f i , where i∈0,1,2,3 represent four positions respectively, and the visual encoder used is Swin Transformer V2, as shown Figure 1 As shown in the Visual Encoder;

[0044] Step S22: The formatted clinical data and laboratory test data obtained in step S12 are concatenated and passed through the MLP Encoder to obtain numerical features, which are expressed as f c ,like Figure 1 As shown in the lower right corner;

[0045] Step S23: Modal filling of missing data: If the ultrasound image of a tissue or organ is missing, fill it with the gallbladder image of the same patient. If the gallbladder is also missing, fill it with the fibrous plaque image. If both the gallbladder image and the fibrous plaque image of the same patient are missing, fill it with a null value to explicitly indicate its missingness. If numerical data is missing, interpolation is performed using the mean value in the training set. This step corresponds to Figure 1 Missing completion module;

[0046] Step S3: training a biliary atresia intelligent recognition model;

[0047] Step S31: Based on the ultrasound image features of multiple tissues obtained in step S2, the proposed mask self-attention mechanism is used to perform fine-grained fusion of the ultrasound image features of each tissue, such as Figure 1 As shown in the Multi-Head Self-MaskedAttention module, specifically:

[0048] MultiHead(Q,K,V)=Concat(head1,…,head h )W O (1)

[0049] head i=Attention(Q′,K′,V′) (2)

[0050]

[0051] The projection matrix and Where i represents the projection matrix number required for the calculation in the i-th head. K T is the transpose of K. M Q 、M K and M V It is the mask matrix corresponding to Q, K and V, and its dimensions are the same as Q, K and V. q d k d v Corresponding to the dimensions of Q, K, and V respectively, d modal represents the modal dimension, which is equal to 4 here. Q, K, and V are the image features f extracted in step S21. i Combined stitching. Projection matrix and W O All are randomly initialized before model training; the head in Formula 1 is calculated by Formula 2, the attention in Formula 2 is calculated by Formula 3, and Q′, K′, and V′ in Formula 2 are calculated by Formulas 4, 5, and 6.

[0052] Step S32: For the multi-position ultrasound image features after fusion, multi-information fusion is performed with the numerical features through a multi-layer perceptron. This step is reflected in Figure 1 The Multimodal Fusion module obtains the fusion features;

[0053] Step S33: The fusion feature obtained in step S32 is passed through the fully connected layer to obtain the biliary atresia recognition result of the fusion feature, and the loss is calculated using the cross entropy function, such as Figure 2 As shown, the fusion features obtained by Multimodal Fusion are passed through the MLP Head to obtain the prediction value and calculate the loss with the label;

[0054] Step S34: Calculate the ultrasound image features of the four tissue positions extracted in step S2 and the biliary atresia recognition results of the ultrasound images of the four tissue positions obtained through the fully connected layer, and calculate the cross entropy loss, such as Figure 2 As shown, the four tissue location image features and numerical features are passed through their corresponding MLP Head to obtain prediction values ​​and calculate the loss with the label;

[0055] Step S35: Calculate the biliary atresia recognition result of the numerical features extracted in step S2 and the numerical features obtained through the fully connected layer, and calculate the cross entropy loss, such as Figure 2As shown on the right, the four tissue location image features and numerical features are passed through their corresponding MLP heads to obtain prediction values ​​and calculate losses with the labels, for a total of 5 loss terms;

[0056] Step S36: The losses calculated in step S33 and step S34 are used together to calculate the gradient and inverse

[0057] Propagate the training model, that is:

[0058]

[0059] in represents the multimodal fusion feature obtained in step 32 and the unimodal visual feature f obtained in step S21 i and the numerical feature f obtained in step S22 c , where K = 4, representing images of four tissue locations, i belongs to K. α and β represent the weights used for the loss, both set to 0.5 here. represents the total loss of the calculation, represents the cross entropy loss function;

[0060] Step S4:

[0061] Step S41: Based on the biliary atresia recognition model and model weights obtained in step S3, the ultrasound images of four tissue locations, clinical data, and laboratory examination data are input, and the model outputs the final recognition prediction results.

[0062] The proposed intelligent biliary atresia recognition model based on multimodal medical data was tested on a collected dataset divided into training, validation, and test sets in a ratio of 6:1:3. The training set was first used to train each model. Subsequently, the validation set was used to obtain the corresponding weights for each model and determine the optimal parameters of the present invention. Finally, the test set was used to obtain predictions and corresponding evaluation metrics.

[0063] Figure 3 After the training model is complete, Grad-CAM is used to obtain feature maps of the predicted image, providing a visual explanation. Figures A and D show example images of a gallbladder, fibrous plaques, liver capsule, and liver parenchyma, respectively. The top row represents the source image, and the bottom row represents the feature maps obtained using Grad-CAM. Red in the figures indicates the areas of greatest interest to the model, providing supplementary information for doctors to prioritize.

[0064] Figure 4This is the prediction result of the present invention on the retrospectively collected data set, and is compared with the analysis results of radiologists. The ROC curve is used to analyze the performance of the proposed model, expert and unimodal model on the test data set. Figure a shows the ROC curve of the multimodal model we proposed, and the corresponding diagnostic results provided by the expert. Figure b is the ROC curve of various simplified multimodal models under the conditions of their respective data sets. CI and LTD represent clinical information and laboratory test data, respectively. Figure c shows the ROC curve generated by the unimodal model for gallbladder data. Figure d shows the ROC curve generated by the unimodal model for triangular cord sign (TCS) data. Figure e shows the ROC curve generated by the unimodal model for liver capsule data. Figure f shows the ROC curve generated by the unimodal model for liver parenchyma data. Figure g shows the ROC curve generated by the unimodal model for clinical information and laboratory data. Each unimodal model is composed of the feature extractor of the corresponding modality of the multimodal model plus the MLP Head of the corresponding modality. The corresponding area under the ROC curve (AUC) value is represented in the legend of each figure. As shown in Figure 4 As shown, the model recognition performance exceeded that of radiologists, one of whom had 10 years of experience in diagnosing biliary atresia. Figure 5 This is the ROC curve result of partial prospective data collection for verification. The model recognition performance is close to that of doctors with more than 10 years of experience in diagnosing biliary atresia.

[0065] Figure 6 is the change in the diagnostic accuracy of radiologists with the assistance of the present invention, where af is the improvement result of retrospective experimental tests, and gl is the improvement result of prospective experimental tests. AUC represents the corresponding area under the ROC curve, Sensitivity represents sensitivity, Specificity represents specificity, Accuracy represents accuracy, PPV and NPV represent positive predictive value and negative predictive value respectively, and the higher these indicators are, the better. The gray asterisks in the figure represent the performance of the model, and the colors are used to distinguish different doctors. The circles represent the diagnostic results without the assistance of the model, and the squares represent the diagnostic results with the assistance of the model. All radiologists have different degrees of improvement in diagnostic performance with the assistance of the present invention.

[0066] Since the design of the present invention adopts a plug-and-play modular design, the input mode can be simplified, such as Figure 7 As shown, the input can be only gallbladder images, fibrous plaque images, clinical data (or other combinations), and the model can still be trained and predicted without changing the model.

[0067] Figure 8 A schematic diagram showing the process of the present invention in practical application deployment, where the annotated data is extracted and input into the model, and the model outputs the final recognition result and a visual explanation diagram.

[0068] In summary, this method proposes an intelligent identification method for biliary atresia based on multimodal medical data through improvements in data preprocessing, multimodal fusion, modality loss processing and prediction model. Experiments show that this model can assist doctors in screening and diagnosing the rare disease biliary atresia, while promoting the application of intelligent identification of other rare diseases.

[0069] The above description is only a specific embodiment of the present invention. Any feature disclosed in this specification, unless otherwise stated, can be replaced by other equivalent or alternative features with similar purposes; all disclosed features, or all steps in the methods or processes, except for mutually exclusive features and / or steps, can be combined in any way.

[0070] [1]Chung PHY, Zheng S, Tam PKH. Biliary Atresia: East versus West. Seminars in pediatric surgery. 2020Aug; 29(4):150950.

[0071] [2]McKiernan PJ, Baker AJ, Kelly DA. The Frequency and Outcome ofBiliary Atresia in the UK and Ireland. Lancet (London, England). 2000Jan; 355(9197):25–29.

[0072] [3]Lupo PJ,Isenburg JL,Salemi JL,Mai CT,Liberman RF,Canfield MA,etal.Population-Based Birth Defects Data in the United States,2010-2014:A Focuson Gastrointestinal Defects.Birth defects research.2017Nov;109(18):1504–1514.

[0073] [4]Tiao MM,Tsai SS,Kuo HW,Chen CL,Yang CY.Epidemiological Features ofBiliary Atresia,a National Study 1996-2003.Journal of gastroenterology andhepatology.2008Jan;23(1):62–66.

[0074] [5]Wada H,Muraji T,Yokoi A,Okamoto T,Sato S,Takamizawa S,etal.Insignificant Seasonal and Geographical Variation in Incidence of BiliaryAtresia in Japan:A Regional Survey of over 20Years.Journal of pediatricsurgery.2007Dec;42(12):2090–2092.

[0075] [6]Kasai M,Kimura S,Asakura Y,Suzuki H,Taira Y,Ohashi E.SurgicalTreatment of Biliary Atresia.Journal of pediatric surgery.1968;3(6):665–675.

[0076] [7]Hopkins PC,Yazigi N,Nylund CM.Incidence of Biliary Atresia andTiming of Hepatoportoenterostomy in the United States.The Journal ofpediatrics.2017Aug;187:253–257.

[0077] [8]Parolini F,Boroni G,Milianti S,Tonegatti L,Armellini A,GarciaMagne M,et al.Biliary Atresia:20-40-Year Follow-up with Native Liver in anItalian Centre.Journal of pediatric surgery.2019 Jul;54(7):1440–1444.

[0078] [9]Serinet MO,Wildhaber BE,Brou′e P,Lachaux A,Sarles J,Jacquemin E,etal.Impact of Age at Kasai Operation on Its Results in Late Childhood andAdolescence:A Rational Basis for Biliary Atresia Screening.Pediatrics.2009May;123(5):1280–1286.

[0079]

[10] Liu,J.et al.Diagnostic Value and Effectiveness of an ArtificialNeural Network in Biliary Atresia.Front.Pediatr.8,409(2020)

[0080]

[11] Dai,S.Y.et al.Development and Assessment of Screening Nomogramfor Biliary Atresia Based on Hepatobiliary UltrasonographicFeatures.Front.Pediatr.9,625451(2021).

[0081]

[12] Zhou,W.et al.Ensembled deep learning model outperforms humanexperts in diagnosing biliary atresia from sonographic gallbladder images.NatCommun 12,1259(2021).

[0082]

[13] Stahlschmidt,S.R.,Ulfenborg,B.&Synnergren,J.Multimodal deeplearning for biomedical data fusion:a review.Briefings in Bioinformatics 23,bbab569(2022).

Claims

1. A method for identifying biliary atresia based on multimodal medical data, characterized in that: The following steps are involved: Step S1: data preprocessing; Step S11: extracting ultrasound image annotation information; Step S12: Perform one-hot encoding on discrete values ​​in clinical data and laboratory test data, including gender, premature birth, jaundice, clay stool, and dark urine, and perform normalization on continuous values; Step S2: feature extraction; Step S21: Extract four tissue locations, namely gallbladder, fibrous plaque, liver capsule and liver parenchyma, from the ultrasound image obtained in step S11. Use the visual model to extract features from the ultrasound image of each tissue. The extracted features are expressed as f i , where i∈0,1,2,3 represent four positions respectively; Step S22: The formatted clinical data and laboratory test data obtained in step S12 are concatenated and represented as f c , represents the numerical feature; Step S23: Modally fill in the missing data: if the ultrasound image of a tissue or organ location is missing, fill it in with the gallbladder image of the same patient; if the gallbladder is also missing, fill it in with the fibrous plaque image; if both the gallbladder image and the fibrous plaque image of the same patient are missing, fill it in with a null value to explicitly indicate their missingness; If numerical data is missing, interpolation is performed using the mean value in the training set; Step S3: training a biliary atresia intelligent recognition model; Step S31: for the ultrasound image features of multiple tissues obtained in step S2, the proposed multi-head masked self-attention mechanism is used to perform fine-grained fusion of the ultrasound image features of each tissue; Step S32: For the fused multi-position ultrasound image features, multi-information fusion is performed with the numerical features through a multi-layer perceptron to obtain fused features; Step S33: Passing the fused features obtained in step S32 through a fully connected layer to obtain a biliary atresia recognition result of the fused features, and calculating the loss using a cross entropy function; Step S34: Calculate the ultrasound image features of the four tissue locations extracted in step S2 and the biliary atresia recognition results of the ultrasound images of the four tissue locations obtained through the fully connected layer, and calculate the cross entropy loss; Step S35: Calculate the biliary atresia recognition result of the numerical features extracted in step S2 and the numerical features obtained through the fully connected layer, and calculate the cross entropy loss; Step S36: The losses calculated in step S33 and step S34 are used together to calculate the gradient and back-propagate the training model to obtain the model weight; Step S4: Biliary atresia intelligent recognition model reasoning; Step S41: Based on the biliary atresia recognition model and model weights obtained in step S3, the ultrasound images of four tissue locations, clinical data, and laboratory examination data are input, and the model outputs the final recognition prediction results.

2. The method according to claim 1, characterized in that In step S31, a masked self-attention mechanism is used for the image features at the four positions. In addition to calculating the corresponding weight matrices for Q, K, and V in the attention mechanism, two mask matrices are added so that Q only represents its own features, and K and V represent other features except themselves. Then, the attention score is calculated so that the image features at each position can pay attention to and integrate the information of other partial features, and then all partial features are spliced ​​to form a complete image fusion feature. The proposed masked self-attention mechanism performs fine-grained fusion of ultrasound image features of each tissue, specifically: MultiHead(Q,K,V)=Concat(head1,...,head h )W O (1) head i =Attention(Q′,K′,V′) (2) Q′=(Q·M Q )W i Q (4) The projection matrix and Where i represents the projection matrix number required for the calculation in the i-th head, K T is the transpose of K, M Q 、M K and M V is the mask matrix corresponding to Q, K and V, with the same dimensions as Q, K and V, d q d k d v Corresponding to the dimensions of Q, K, and V respectively, d modal represents the modal dimension, which is equal to 4 here, Q, K, and V are the image features f extracted by step S21 i Combined stitching, projection matrix W i Q , W i K , W i V and W O All are randomly initialized before model training; the head in formula (1) is calculated by formula (2), the attention in formula (2) is calculated by formula (3), and Q′, K′, and V′ in formula (2) are calculated by formulas (4), (5), and (6).

3. The method according to claim 2, characterized in that In step S35, the loss of the entire model is calculated, the unimodal information is added, and the two-stream structure is used to train the model: All loss calculations use cross entropy loss.

4. An information processing device comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 3 is implemented.

Images