Intelligent diagnosis method for biliary atresia based on multi-modal medical data

Abstract

The invention provides an intelligent biliary atresia recognition model based on multi-modal medical data, and belongs to the field of data processing. The method comprises the following steps: acquiring key position information of multi-modal data in a data preprocessing stage; in the feature extraction stage, a basis is provided for subsequent modal fusion through visual image feature coding and numerical data feature coding; in the modal fusion stage, through multi-level fine-grained fusion, firstly, fine-grained fusion is innovatively carried out on image features by using a mask self-attention mechanism, and then multi-modal fusion is carried out on the overall level, so that the fusion effect under the condition of modal deficiency is effectively improved, key features between modals and in the modals are effectively captured and interactively fused, and the fusion efficiency is improved. And a basis is provided for high-precision prediction. In order to improve model training efficiency and prediction precision, a multi-loss training mode is used, multi-modal fusion feature loss is incorporated, and loss of single-modal calculation is added in a certain proportion. And finally carrying out intelligent recognition prediction on biliary atresia through the weight of the training model.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence and intelligent identification of biliary atresia (G06F 19 / 00), and specifically relates to a biliary atresia identification method based on multimodal medical data. Background Art

[0002] Biliary atresia (BA) is a rare idiopathic fibrous obliterative cholangiopathy that occurs in the perinatal period. The incidence of BA varies from 1:20,000 in North America and Europe to 1:5000–1.1:10,000 in Asia. [1-5] It is the leading cause of liver transplantation in children. If untreated, BA inevitably progresses to end-stage liver disease and leads to death within the first two years of life. Since 1968, Kasai portoenterostomy and liver transplantation have significantly prolonged survival and improved quality of life. [6] Early age at the time of Kasai portoenterostomy is one of the most important factors in predicting surgical outcome, with best results when performed before 30 to 45 days of age. [7-9] ; after that, the chances of success decrease significantly with age. Unfortunately, the median age at the time of the Kasai procedure is about 60 days, with no improvement over time. [7 , 8] The main reason for the delay is the lack of effective and practical screening methods, so early diagnosis is a prominent and ongoing clinical challenge.

[0003] In recent years, with the rapid development of artificial intelligence technology, computer-aided diagnosis has become a current research hotspot. Deep learning is a powerful artificial intelligence technology that has been widely used in the analysis of medical data. Deep neural networks are able to learn complex representations of data, thereby achieving state-of-the-art performance in various medical imaging tasks.

[0004] Some studies have attempted to use clinical information and laboratory test data to build an artificial neural network to assist in the diagnosis of biliary atresia. [1] There are also studies that add manually extracted ultrasound image features to these types of data.

[10] However, manual feature extraction relies on the doctor's experience, and simple artificial neural networks and multivariate logistic regression analysis methods cannot fully fit the complex relationship between features. Using deep learning technology to analyze gallbladder images for diagnosis can achieve expert-level accuracy

[11] However, using only a single modality will ignore other information that is meaningful to the diagnosis and is not sufficient to achieve high-precision auxiliary diagnosis.

[0005] Doctors' diagnosis of biliary atresia relies on a multi-faceted examination of the patient. However, the multimodal medical data generated by laboratory tests and ultrasound examinations at multiple locations performed by the patient have not been systematically collected and effectively utilized. The diagnostic performance of all current computer-aided diagnosis methods for biliary atresia still has much room for improvement. More research is needed to further improve the computer-aided diagnosis system and apply it to clinical practice.

[0006] A multimodal learning framework should be able to learn from data of different modalities without requiring different model architectures. A unified multimodal model will combine different types of data, align representations of different modal data, and use them for downstream tasks. Over the past few years, there has been a transition from architectures applied to specific modalities, such as convolutional neural networks for images or recurrent neural networks for text, to a relatively new architecture, the Transformer, which has shown good performance on a variety of input and output modalities and tasks.

[12] However, there is no good solution to the problem of missing modalities in medical data and the problem of redundant multimodal information, and different types of tasks need to be designed separately. Summary of the invention

[0007] In order to be as close to the doctors' analysis ideas as possible, we borrowed the analysis and judgment process of clinicians and proposed a multimodal deep learning model that can effectively deal with the problem of modal loss and fusion in multimodality, and comprehensively analyze the multimodal data generated by the multi-faceted examination of patients. We collected and annotated a large amount of multimodal data related to biliary atresia by professional doctors, and applied this multimodal model to develop an intelligent recognition model for biliary atresia.

[0008] Since clinicians use ultrasound images and clinical data to identify biliary atresia, we enable the computer to analyze ultrasound images, clinical information, and laboratory information based on the doctor's diagnosis and treatment experience, and then obtain the final classification result. Therefore, we proposed a multimodal deep learning method that can adapt to the auxiliary recognition task of multimodal medical data. We used the collected medical data related to biliary atresia for training and testing to verify the effectiveness of the model in assisting medical decision-making for biliary atresia.

[0009] The multimodal data of biliary atresia include ultrasound images, clinical data and laboratory test data. The ultrasound images include four tissue locations: gallbladder, fibrous plaques, liver capsule and liver parenchyma; clinical data include gender, age, premature birth, gestational age, weight at onset, jaundice, clay stools, and dark urine; laboratory test data include total bilirubin (TB), direct bilirubin (DB), total bile acid (TBA), total protein (TP), albumin (ALB), alanine aminotransferase (ALT), aspartate aminotransferase (AST), γ-glutamyl transpeptidase (GGT), alkaline phosphatase (ALP), platelets (PLT), white blood cells (WBC), prothrombin time (PT) and activated partial thromboplastin time (APTT).

[0010] In order to comprehensively analyze medical data of different forms, multimodal models need to be able to adapt to different modal data and effectively fuse and analyze them. The model we proposed mainly consists of four parts: data mapping and enhancement, feature extraction, modal fusion, and multi-loss joint training. For multimodal input, the data processing methods corresponding to different modalities are adopted. For the processing of missing values, different modalities adopt different strategies. If the ultrasound image of the tissue or organ position is missing, the gallbladder image of the same patient is used to fill it. If the gallbladder is also missing, the fibrous plaque image is used to fill it. If both the gallbladder image and the fibrous plaque image of this patient are missing, they are filled with null values ​​to explicitly indicate their missingness; if the numerical data is missing, the mean value in the training set is used for interpolation filling. For multimodal information interaction and fusion, different modalities are used to extract features using data encoders separately and then fuse features in high-dimensional space. In order to improve the stability of training and the generalization of the model, multiple loss functions are used for aggregation optimization. As a general method, these four parts can be regarded as plug-and-play modules, and each part can be switched freely to meet the task requirements.

[0011] The present invention comprises the following steps:

[0012] Step S1: data preprocessing;

[0013] Step S11: The doctor annotates the ultrasound image with organs, including the locations of four tissues: gallbladder, fibrous plaque, liver capsule, and liver parenchyma, as well as a label indicating whether it is biliary atresia;

[0014] Step S12: Clean the data using outlier analysis;

[0015] Step S2: feature extraction and missing processing;

[0016] Step S21: extract four tissue positions from the ultrasound image obtained by step S11, use the visual model to extract features from the ultrasound image of each tissue, and obtain image features of the four tissue positions.

[0017] Step S22: Encode the clinical data and laboratory test data obtained in step S12 by one-hot encoding and normalization to obtain numerical features;

[0018] Step S23: Modally fill the missing data: if the ultrasound image of the tissue or organ location is missing, fill it with the gallbladder image of the same patient; if the gallbladder is also missing, fill it with the fibrous plaque image; if both the gallbladder image and the fibrous plaque image of the same patient are missing, fill it with a null value to explicitly indicate its missingness. If the numerical data is missing, use the mean value in the training set for interpolation filling;

[0019] Step S3: training a biliary atresia intelligent recognition model;

[0020] Step S31: For the ultrasound image features of multiple tissues obtained by processing in step S2, the proposed masked self-attention mechanism is used to perform fine-grained fusion of the ultrasound image features of each tissue.

[0021] Step S32: For the fused multi-position ultrasound image features, multi-information fusion is performed with the numerical features through a multi-layer perceptron.

[0022] Step S33: The fusion features obtained in step S32 are used to predict the results through a fully connected layer, and the loss is calculated using a cross entropy function.

[0023] Step S34: Calculate the cross entropy loss between the predicted result obtained by the fully connected layer of the features extracted in step S2 and the true value.

[0024] Step S35: The losses calculated in step S33 and step S34 are used together to calculate the gradient and back-propagate the training model to obtain the model weight.

[0025] Based on a thorough analysis of the existing deep model applications and the insufficiency of multimodal medical data, the present invention sets up a model of pluggable module design and masked self-attention mechanism to effectively deal with the training difficulty caused by the lack of modalities, and improves the effective information fusion effect through a multi-level and multi-granularity approach in the modal fusion stage. Experimental results prove that the present invention has reached a very advanced level in the intelligent identification of biliary atresia based on multimodal medical data.

[0026] The beneficial effects of the present invention are:

[0027] 1. The present invention is based on multimodal medical data and follows expert diagnostic experience, so that the model's intelligent diagnosis AUC accuracy for biliary atresia reaches 0.986, which exceeds that of an expert doctor with 10 years of experience.

[0028] 2. This study is based on the application of multimodal medical data. In multimodal fusion and multimodal missing processing, it uses innovative multi-level fine-grained fusion schemes and masked self-attention mechanisms, which have important research significance and huge application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 This is a schematic diagram of the overall process of the biliary atresia intelligent recognition model based on multimodal medical data proposed in the present invention.

[0030] Figure 2 This is a multi-loss training structure diagram of the biliary atresia intelligent recognition model based on multimodal medical data in the present invention.

[0031] Figure 3 A visual explanation diagram of the result analysis in this invention

[0032] Figure 4 The results of the present invention in the retrospective experimental test are shown in FIG.

[0033] Figure 5 The results of the prospective experimental test of the present invention are shown in FIG.

[0034] Figure 6 This is a graph showing the improvement in accuracy brought about by the present invention assisting doctors in analysis and judgment; af is the improvement result of retrospective experimental test, and gl is the improvement result of prospective experimental test

[0035] Figure 7 This is a simplified model experimental structure diagram of the present invention

[0036] Figure 8 The present invention is applied to the flow chart. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention is further described in detail below in combination with specific implementation methods and with reference to the accompanying drawings. It should be understood that the embodiments provided below are only for the purpose of fully and comprehensively disclosing the present invention and fully conveying the technical concept of the present invention to those skilled in the art, but are not intended to limit the scope of the present invention.

[0038] See also Figure 1 When the present invention is used for intelligent identification of biliary atresia, the specific processing steps are as follows:

[0039] Step S1: data preprocessing;

[0040] Step S11: Extracting ultrasound image annotation information, such as Figure 8 As shown on the left,;

[0041] Step S12: Perform one-hot encoding on the discrete values ​​in the clinical data and laboratory test data, including gender, premature birth, jaundice, clay stool, and dark urine, and perform normalization on the continuous values, such as Figure 1 Data preprocessing includes data cleaning, data discretization, and data normalization;

[0042] Step S2: feature extraction;

[0043] Step S21: extract four tissue positions from the ultrasound image obtained in step S11, and use the visual model to extract features from the ultrasound image of each tissue. The extracted features are represented as f i , where i∈0,1,2,3 represent four positions respectively, and the visual encoder used is Swin Transformer V2, such as Figure 1 As shown in the Visual Encoder;

[0044] Step S22: The formatted clinical data and laboratory test data obtained in step S12 are concatenated and then passed through the MLP Encoder to obtain numerical features, which are expressed as f c ,like Figure 1 As shown in the lower right corner;

[0045] Step S23: Modally fill in the missing data: If the ultrasound image of the tissue or organ location is missing, fill it with the gallbladder image of the same patient; if the gallbladder is also missing, fill it with the fibrous plaque image; if both the gallbladder image and the fibrous plaque image of the same patient are missing, fill it with a null value to explicitly indicate its missingness. If the numerical data is missing, use the mean value in the training set for interpolation filling. This step corresponds to Figure 1 Missing completion module;

[0046] Step S3: training a biliary atresia intelligent recognition model;

[0047] Step S31: For the ultrasound image features of multiple tissues obtained in step S2, the proposed masked self-attention mechanism is used to perform fine-grained fusion of the ultrasound image features of each tissue, such as Figure 1 As shown in the Multi-Head Self-MaskedAttention module, specifically:

[0048] MultiHead(Q,K,V)=Concat(head1,…,head h )W O (1)

[0049] head i=Attention(Q′,K′,V′) (2)

[0050]

[0051] The projection matrix and Where i represents the projection matrix number required for calculation in the i-th head. T is the transpose of K. M Q 、M K and M V It is the mask matrix corresponding to Q, K and V, and its dimension is the same as Q, K and V. q d k d v Corresponding to the dimensions of Q, K, and V respectively, d modal represents the modal dimension, which is equal to 4 here. Q, K, and V are the image features f extracted in step S21. i Combined stitching. Projection matrix and W O All are randomly initialized before model training; the head in Formula 1 is calculated by Formula 2, the Attention in Formula 2 is calculated by Formula 3, and Q′, K′, V′ in Formula 2 are calculated by Formulas 4, 5, and 6;

[0052] Step S32: For the fused multi-position ultrasound image features, the multi-information fusion is performed with the numerical features through a multi-layer perceptron. This step is reflected in Figure 1 The Multimodal Fusion module obtains the fusion features;

[0053] Step S33: The fusion feature obtained in step S32 is passed through the fully connected layer to obtain the biliary atresia recognition result of the fusion feature, and the loss is calculated using the cross entropy function, such as Figure 2 As shown, the fusion features obtained by Multimodal Fusion are passed through the MLP Head to obtain the prediction value and calculate the loss with the label;

[0054] Step S34: Calculate the ultrasound image features of the four tissue locations extracted in step S2 and the biliary atresia recognition results of the ultrasound images of the four tissue locations obtained through the fully connected layer, and calculate the cross entropy loss, such as Figure 2 As shown, the four tissue location image features and numerical features are passed through their corresponding MLP Heads to obtain prediction values ​​and calculate losses with the labels;

[0055] Step S35: Calculate the biliary atresia recognition result of the numerical features extracted in step S2 and the numerical features obtained through the fully connected layer, and calculate the cross entropy loss, such as Figure 2As shown on the right, the four tissue location image features and numerical features are passed through their corresponding MLP Heads to obtain prediction values ​​and calculate losses with the labels, with a total of 5 loss items;

[0056] Step S36: The losses calculated in step S33 and step S34 are used together to calculate the gradient and inverse

[0057] Propagate the training model, that is:

[0058]

[0059] in represents the multimodal fusion feature obtained in step 32 and the unimodal visual feature f obtained in step S21 i and the numerical feature f obtained in step S22 c , where K = 4, represents the images of four tissue locations, i belongs to K. α and β represent the weights used for the loss, both of which are set to 0.5 here. represents the total loss of the calculation, represents the cross entropy loss function;

[0060] Step S4:

[0061] Step S41: Based on the biliary atresia recognition model and model weights obtained in step S3, ultrasound images of four tissue locations, clinical data and laboratory examination data are input, and the model outputs the final recognition prediction results.

[0062] The present invention tests the proposed biliary atresia intelligent recognition model based on multimodal medical data in a collected data set, which is divided into a training set, a validation set, and a test set in a ratio of 6:1:3. First, the training set is used to train each model, and then the validation set is used to obtain the corresponding weights of each model and determine the optimal parameters of the present invention. Finally, the test set is used to obtain the predicted value and the corresponding evaluation index.

[0063] Figure 3 After the training model of the present invention is completed, the feature map of the predicted image is obtained by Grad-CAM to provide a visual explanation. Figures ad respectively show example images of gallbladder images, fibrous plaques, liver capsules and liver parenchyma. The upper row shows the source image, and the lower row shows the feature map obtained by using Grad-CAM. The red color in the figure shows the area that the model pays the most attention to, which can provide auxiliary information for doctors as the key areas of attention.

[0064] Figure 4It is the prediction result of the present invention on the retrospectively collected data set, and is compared with the analysis result of the radiologist. The ROC curve is used to analyze the performance of the proposed model, expert and unimodal model on the test data set. Figure a shows the ROC curve of the multimodal model we proposed, and the corresponding diagnosis results provided by the expert. Figure b is the ROC curve of various simplified multimodal models under the conditions of their respective data sets. CI and LTD represent clinical information and laboratory test data, respectively. Figure c shows the ROC curve generated by the unimodal model of gallbladder data. Figure d shows the ROC curve generated by the unimodal model for triangular cord sign (TCS) data. Figure e shows the ROC curve generated by the unimodal model for liver capsule data. Figure f shows the ROC curve generated by the unimodal model for liver parenchyma data. Figure g shows the ROC curve generated by the unimodal model for clinical information and laboratory data. Each unimodal model is composed of the feature extractor of the corresponding modality of the multimodal model plus the MLP Head of the corresponding modality. The corresponding area under the ROC curve (AUC) value is represented in the legend of each figure. As shown in Figure 4 As shown, the model recognition performance exceeded that of radiologists, one of whom had 10 years of experience in diagnosing biliary atresia. Figure 5 This is the ROC curve result of collecting partial prospective data for verification. The model recognition performance is close to that of doctors with more than 10 years of experience in diagnosing biliary atresia.

[0065] Figure 6 It is the change in the diagnostic accuracy of radiologists with the assistance of the present invention, where af is the improvement result of retrospective experimental tests, and gl is the improvement result of prospective experimental tests. AUC represents the corresponding area under the ROC curve, Sensitivity represents sensitivity, Specificity represents specificity, Accuracy represents accuracy, PPV and NPV represent positive predictive value and negative predictive value respectively, and the higher these indicators are, the better. The gray asterisks in the figure represent the performance of the model, and the colors are used to distinguish different doctors. The circles represent the diagnostic results without the assistance of the model, and the squares represent the diagnostic results with the assistance of the model. All radiologists have different degrees of improvement in diagnostic performance with the assistance of the present invention.

[0066] Since the design of the present invention adopts a plug-and-play modular design, the input mode can be simplified, such as Figure 7 As shown, the input can be only gallbladder images, fibrous plaque images, clinical data (or other combinations), and the model can still be trained and predicted without changing the model.

[0067] Figure 8 A schematic diagram showing the process of the present invention in actual application deployment, wherein the annotated data is extracted and input into the model, and the model outputs the final recognition result and a visual explanation diagram.

[0068] In summary, this method proposes an intelligent identification method for biliary atresia based on multimodal medical data through improvements in data preprocessing, multimodal fusion, modal missing processing and prediction models. Experiments show that this model can assist doctors in screening and diagnosing the rare disease biliary atresia, while promoting the application of intelligent identification of other rare diseases.

[0069] The above description is only a specific implementation mode of the present invention. Any feature disclosed in this specification, unless otherwise stated, can be replaced by other alternative features that are equivalent or have similar purposes; all the disclosed features, or all the steps in the methods or processes, except for mutually exclusive features and / or steps, can be combined in any way.

[0070] [1]Chung PHY, Zheng S, Tam PKH. Biliary Atresia: East versus West. Seminars in pediatric surgery. 2020Aug; 29(4):150950.

[0071] [2]McKiernan PJ, Baker AJ, Kelly DA. The Frequency and Outcome ofBiliary Atresia in the UK and Ireland. Lancet (London, England). 2000Jan; 355(9197):25–29.

[0072] [3]Lupo PJ,Isenburg JL,Salemi JL,Mai CT,Liberman RF,Canfield MA,etal.Population-Based Birth Defects Data in the United States,2010-2014:A Focuson Gastrointestinal Defects.Birth defects research.2017Nov;109(18):1504–1514.

[0073] [4]Tiao MM,Tsai SS,Kuo HW,Chen CL,Yang CY.Epidemiological Features ofBiliary Atresia,a National Study 1996-2003.Journal of gastroenterology andhepatology.2008Jan;23(1):62–66.

[0074] [5]Wada H,Muraji T,Yokoi A,Okamoto T,Sato S,Takamizawa S,etal.Insignificant Seasonal and Geographical Variation in Incidence of BiliaryAtresia in Japan:A Regional Survey of over 20Years.Journal of pediatricsurgery.2007Dec;42(12):2090–2092.

[0075] [6]Kasai M,Kimura S,Asakura Y,Suzuki H,Taira Y,Ohashi E.SurgicalTreatment of Biliary Atresia.Journal of pediatric surgery.1968;3(6):665–675.

[0076] [7]Hopkins PC,Yazigi N,Nylund CM.Incidence of Biliary Atresia andTiming of Hepatoportoenterostomy in the United States.The Journal ofpediatrics.2017Aug;187:253–257.

[0077] [8]Parolini F,Boroni G,Milianti S,Tonegatti L,Armellini A,GarciaMagne M,et al.Biliary Atresia:20-40-Year Follow-up with Native Liver in anItalian Centre.Journal of pediatric surgery.2019 Jul;54(7):1440–1444.

[0078] [9]Serinet MO,Wildhaber BE,Brou′e P,Lachaux A,Sarles J,Jacquemin E,etal.Impact of Age at Kasai Operation on Its Results in Late Childhood andAdolescence:A Rational Basis for Biliary Atresia Screening.Pediatrics.2009May;123(5):1280–1286.

[0079]

[10] Liu,J.et al.Diagnostic Value and Effectiveness of an ArtificialNeural Network in Biliary Atresia.Front.Pediatr.8,409(2020)

[0080]

[11] Dai,S.Y.et al.Development and Assessment of Screening Nomogramfor Biliary Atresia Based on Hepatobiliary UltrasonographicFeatures.Front.Pediatr.9,625451(2021).

[0081]

[12] Zhou,W.et al.Ensembled deep learning model outperforms humanexperts in diagnosing biliary atresia from sonographic gallbladder images.NatCommun 12,1259(2021).

[0082]

[13] Stahlschmidt,S.R.,Ulfenborg,B.&Synnergren,J.Multimodal deeplearning for biomedical data fusion:a review.Briefings in Bioinformatics 23,bbab569(2022).

Claims

1. A method for identifying biliary atresia based on multimodal medical data, characterized in that: The following steps are involved: Step S1: data preprocessing; Step S11: extracting ultrasound image annotation information; Step S12: Perform one-hot encoding on discrete values ​​in clinical data and laboratory test data, including gender, premature birth, jaundice, clay stool, and dark urine, and perform normalization on continuous values; Step S2: feature extraction; Step S21: extract four tissue positions, namely gallbladder, fibrous plaque, liver capsule and liver parenchyma, from the ultrasound image obtained in step S11, and use the visual model to extract features from the ultrasound image of each tissue. The extracted features are represented as f i , where i∈0,1,2,3 represent four positions respectively; Step S22: The formatted clinical data and laboratory test data obtained in step S12 are concatenated and represented as f c , represents the numerical feature; Step S23: modally fill the missing data: if the ultrasound image of a tissue or organ position is missing, fill it with the gallbladder image of the same patient; if the gallbladder is also missing, fill it with the fibrous plaque image; if both the gallbladder image and the fibrous plaque image of the patient are missing, fill it with a null value to explicitly indicate its missingness; If numerical data is missing, interpolation is performed using the mean value in the training set; Step S3: training a biliary atresia intelligent recognition model; Step S31: for the ultrasound image features of multiple tissues obtained by processing in step S2, the proposed multi-head masked self-attention mechanism is used to perform fine-grained fusion of the ultrasound image features of each tissue; Step S32: for the fused multi-position ultrasound image features, multi-information fusion is performed with the numerical features through a multi-layer perceptron to obtain fused features; Step S33: Passing the fused features obtained in step S32 through a fully connected layer to obtain a biliary atresia recognition result of the fused features, and calculating the loss using a cross entropy function; Step S34: calculating the ultrasonic image features of the four tissue positions extracted in step S2 and the biliary atresia recognition results of the ultrasonic images of the four tissue positions obtained through the fully connected layer, and calculating the cross entropy loss; Step S35: Calculate the biliary atresia recognition result of the numerical features extracted in step S2 and the numerical features obtained through the fully connected layer, and calculate the cross entropy loss; Step S36: The losses calculated in step S33 and step S34 are used together to calculate the gradient and back-propagate the training model to obtain the model weight; Step S4: Biliary atresia intelligent recognition model reasoning; Step S41: Based on the biliary atresia recognition model and model weights obtained in step S3, ultrasound images of four tissue locations, clinical data and laboratory examination data are input, and the model outputs the final recognition prediction results.

2. The method according to claim 1, characterized in that In step S31, for the image features at the four positions, a masked self-attention mechanism is used. For Q, K, and V in the attention mechanism, in addition to calculating the corresponding weight matrices, two Mask matrices are added so that Q only represents its own features, and K and V represent other features except themselves; then the attention score is calculated so that the image features at each position can pay attention to and fuse the information of other partial features, and then all partial features are spliced ​​to form a complete image fusion feature; the proposed masked self-attention mechanism performs fine-grained fusion of ultrasound image features of each tissue, specifically: MultiHead(Q,K,V)=Concat(head1,...,head h )W O (1) head i =Attention(Q′,K′,V′) (2) Q′=(Q·M Q )W i Q (4) The projection matrix and Where i represents the projection matrix number required for calculation in the i-th head, K T is the transpose of K, M Q 、M K and M V is the mask matrix corresponding to Q, K and V, with the same dimensions as Q, K and V, d q ,d k ,d v Corresponding to the dimensions of Q, K, and V respectively, d modal represents the modal dimension, which is equal to 4 here, Q, K, and V are the image features f extracted by step S21 i Combined stitching, projection matrix W i Q , W i K , W i V and W O All are randomly initialized before model training; the head in formula (1) is calculated by formula (2), the attention in formula (2) is calculated by formula (3), and Q′, K′, and V′ in formula (2) are calculated by formulas (4), (5), and (6).

3. The method according to claim 2, characterized in that In step S35, the loss of the entire model is calculated, the unimodal information is added, and the two-stream structure is used to train the model: All loss calculations use cross entropy loss.

4. An information processing device, comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 3 is implemented.

Images