An intelligent report interpretation method and system based on deep learning

Abstract

The present invention relates to the field of deep learning technology, and particularly to a deep learning-based intelligent report interpretation method and system, the method comprising: obtaining a physical examination report containing multiple physical examination items, and identifying text information and image information; inputting the text information and image information into a report recognition model, extracting image features of the physical examination items through an image encoder, and extracting text features of the physical examination items through a text encoder; dynamically allocating weights of image features and text features through an attention fusion layer, aligning the weighted image features to the dimensions of text features, and then splicing them with the weighted text features to obtain fused features; classifying the fused features through a classifier, identifying the focus items in the physical examination report, and outputting health categories corresponding to the focus items; inputting the focus items and the health categories corresponding to the focus items into a knowledge graph to obtain an interpretation result of the physical examination report; the present invention improves the accuracy and efficiency of physical examination report interpretation.

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology, and in particular to a deep learning-based intelligent report interpretation method and system. Background Art

[0002] Traditional medical report interpretation typically involves doctors or nurses with sufficient medical knowledge interpreting each report item by item to analyze the patient's health status. Subjective judgment by doctors can lead to discrepancies in interpretation, reducing the reliability of medical reports. Furthermore, manual interpretation is inefficient and unable to cope with the large volume of reports required, impacting the quality of medical services.

[0003] Therefore, it is necessary to provide a solution that can improve the accuracy and efficiency of interpreting physical examination reports. Summary of the Invention

[0004] To solve the above problems, the present invention provides an intelligent report interpretation method and system based on deep learning, which can improve the accuracy and efficiency of physical examination report interpretation.

[0005] In order to achieve the above object, the present invention provides the following technical solutions:

[0006] In one aspect, an embodiment of the present invention provides a method for intelligent report interpretation based on deep learning, the method comprising the following steps:

[0007] Obtain a medical examination report containing multiple medical examination items and identify text and image information in the medical examination report;

[0008] The text and image information are input into a report recognition model consisting of an image encoder, a text encoder, an attention fusion layer, and a classifier. The image encoder extracts the image features of the physical examination items, and the text encoder extracts the text features of the physical examination items. The attention fusion layer dynamically assigns weights to the image features and text features, aligns the weighted image features to the dimensions of the text features, and then concatenates them with the weighted text features to obtain fused features. The classifier classifies the fused features, identifies the items of concern in the physical examination report, and outputs the health category corresponding to the items of concern.

[0009] Input the items of interest and the health categories corresponding to the items of interest into the knowledge graph to obtain the interpretation results of the physical examination report.

[0010] Optionally, the report recognition model is trained in the following manner:

[0011] Construct a multimodal model consistent with the reported recognition model architecture;

[0012] Obtain a sample set, input the sample set into the multimodal model, freeze all layers of the image encoder and the text encoder, iterate the multimodal model, and gradually adjust the learning rate according to the number of iterations and the first loss value of the multimodal model;

[0013] After the training parameters of the multimodal model reach the number of warm-up steps, the image encoder and text encoder are unfrozen layer by layer according to the second loss value of the multimodal model, and the learning rate is gradually adjusted according to the number of iterations and the second loss value of the multimodal model until the second loss value of the multimodal model is lower than the second loss threshold, thereby obtaining a report recognition model; wherein, the input data of the sample set is a sample report containing graphic and text information, and the output data is the focus items and corresponding health categories in the sample report.

[0014] Optionally, dynamically allocating weights of image features and text features through an attention fusion layer includes:

[0015] Calculate the attention weight α of text features to image features through cross attention ij , and the attention weight β of the image feature V to the text feature T kl :

[0016] Use Softmax function to adjust the attention weight α ij and attention weight β kl Normalize and get image weight and text weight.

[0017] Optionally, the attention weight of the text feature to the image feature is calculated by the following formula:

[0018]

[0019] Among them, T is the text feature, V is the image feature, α ij represents the attention weight of the i-th text token to the j-th image patch, V is the image feature matrix, T is the text feature matrix, Q v is the mapping matrix from image features to text feature queries, K t is the mapping matrix from text features to image feature keys, V ik is the mapping matrix from image features to text feature queries, Q v , K t is a learnable matrix, k is the index variable of the image patch, d v is the image feature dimension, d t is the text feature dimension.

[0020] Optionally, the attention weight of the image feature V to the text feature T is calculated by the following formula:

[0021]

[0022] Among them, β kl represents the attention weight of the k-th image patch to the l-th text token, Q t is the mapping matrix from text features to image feature queries, K v is the mapping matrix from image features to text feature keys, Q t , K v is a learnable matrix, and m is the index variable of the text token.

[0023] Optionally, the learning rate is calculated as follows:

[0024] cut

[0025] lr(t) new =lr(t)·δ;

[0026]

[0027] Among them, t is the current training step, lr(t) new is the learning rate of the tth iteration, lr(t) is the baseline learning rate of the tth iteration, δ and γ are adjustment factors, 0<δ<1, 0<γ<1, cut is the number of iterations in which the loss value drops below the amplitude threshold continuously, lr base is the initial learning rate, w s is the number of warm-up steps, t s is the total number of training steps, L1 represents the first loss value, and L2 represents the second loss value.

[0028] Optionally, the first loss value of the multimodal model is calculated using the following formula:

[0029] L1=λ·L align +(1-λ)·L class ;

[0030]

[0031] Among them, L1 align represents the cross attention alignment loss, A i,i Represents the cross attention matrix of text features to image features, A i,i =Softmax(α ij );N ′ is the batch size, N is the number of patches of image features, M is the number of tokens of text features, L1 class Represents the first classification loss value, Y ih represents a single-bit efficient encoding of the true label, The probability that the i-th sample belongs to class h for multimodal model training.

[0032] Optionally, the second loss value of the multimodal model is calculated using the following formula:

[0033] L2=-L align +L2 class +L2 KL ;

[0034]

[0035] Among them, L2 represents the second loss value, L2 var Indicates the variance penalty value, L2 class Represents the second classification loss value, L2 KL represents the KL divergence value; q represents the variational distribution of the parameter θ, θ represents the hyperparameter of the model, and q(θ) represents the posterior distribution of the parameter θ. represents the variance parameter, E q(θ) represents the expected value of parameter θ, Z (y＝h) represents the indicator function, p(y=h|x,θ) represents the probability that the input fusion feature x belongs to the healthy category h, y represents the true label, H represents the total number of healthy categories, σ 2 represents the variance of the posterior distribution q(θ), p(θ) represents the prior distribution of the parameter θ, represents the variance of the prior distribution p(θ).

[0036] On the other hand, an embodiment of the present invention provides an intelligent report interpretation system based on deep learning, comprising:

[0037] at least one processor;

[0038] at least one memory for storing at least one program;

[0039] When the at least one program is executed by the at least one processor, the at least one processor implements the above method.

[0040] The beneficial effects of the present invention are: the present invention discloses an intelligent report interpretation method and system based on deep learning, the present invention extracts image features of physical examination items through an image encoder, and extracts text features of physical examination items through a text encoder; the weights of image features and text features are dynamically assigned through an attention fusion layer, and by dynamically assigning weights, more attention can be paid to key information, thereby improving the accuracy and efficiency of report interpretation. Through multimodal fusion, not only details can be captured, but also deep semantics can be understood, accurate interpretation can be achieved, and the quality of medical decision-making can be improved. After aligning the weighted image features to the dimensions of the text features, and then splicing them with the weighted text features, fusion features are obtained; the fusion features are classified and processed by a classifier, and the items of concern in the physical examination report are identified, and the health categories corresponding to the items of concern are output. The present invention significantly improves the accuracy and efficiency of report interpretation through multimodal feature fusion and loss optimization. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0042] Figure 1 This is a flowchart of an intelligent report interpretation method based on deep learning according to an embodiment of the present invention;

[0043] Figure 2 This is a structural diagram of an intelligent report interpretation system based on deep learning in an embodiment of the present invention. DETAILED DESCRIPTION

[0044] The following will be combined with the embodiments and drawings to clearly and completely describe the concept, specific structure and technical effects disclosed in the present invention, so as to fully understand the purpose, scheme and effect disclosed in the present invention. It should be noted that the embodiments and features in the embodiments of this application can be combined with each other unless there is a conflict.

[0045] refer to Figure 1 ,like Figure 1 An intelligent report interpretation method based on deep learning provided by an embodiment of the present invention is shown, and the method includes the following steps:

[0046] S100, obtaining a physical examination report containing multiple physical examination items, and identifying text information and image information in the physical examination report;

[0047] Specifically, the physical examination report can be uploaded and submitted in the form of scanned images, PDF, etc. After receiving the physical examination report uploaded by the user, OCR is used to identify the text information in the physical examination report, and keywords are used to match the inspection name in the text information to obtain the result data corresponding to the project name. At the same time, image recognition technology is used to extract key indicators in the image information to ensure the comprehensiveness and accuracy of the data. Then, the acquired data is preprocessed, including data cleaning, format unification and feature extraction, to prepare for the input of the subsequent deep learning model. After the standardized data (text information and image information) in the physical examination report, it is convenient to use the deep learning model to perform feature learning and pattern recognition on the data.

[0048] S200: Input text information and image information into a report recognition model comprising an image encoder, a text encoder, an attention fusion layer, and a classifier. The image encoder extracts image features of the physical examination items, and the text encoder extracts text features of the physical examination items. The attention fusion layer dynamically assigns weights to image features and text features, aligns the weighted image features to the dimensions of the text features, and then concatenates them with the weighted text features to obtain fused features. The classifier classifies the fused features, identifies the items of concern in the physical examination report, and outputs the health category corresponding to the items of concern.

[0049] Specifically, the multimodal model uses a dual-branch parallel encoding and cross-modal attention fusion architecture. The image encoder is based on the ViT model to extract global semantic features of image information; the text encoder is based on a pre-trained language model of the BERT model to capture the contextual semantic information of text information; the cross-modal attention fusion layer realizes inter-modal feature interaction through dynamic weight allocation; the fused features are input into the classifier to output the final health category. Health categories are divided into five levels: normal, mild abnormality, moderate abnormality, severe abnormality, and critical. Based on the classification results, the system automatically marks the physical examination items that require special attention. The items of concern include physical examination items that fall below the preset health category. After multiple physical examination items of special concern are formed into a focus item, the classifier is used to match the corresponding health category. For example, if the blood pressure index and cholesterol index show moderate abnormality, the blood pressure index and cholesterol index will be marked as physical examination items that require special attention.

[0050] S300: Input the focus items and the health categories corresponding to the focus items into the knowledge graph to obtain the interpretation results of the physical examination report.

[0051] Specifically, the knowledge graph is built based on the medical knowledge base, which can associate project information with disease knowledge and provide accurate interpretation. The interpretation results include the possibility of the disease, influencing factors and preventive measures, helping users to fully understand their own health status and take effective measures to manage their health. For example, blood pressure indicators and cholesterol indicators are taken as focus items and associated with cardiovascular diseases. The interpretation results show a moderate abnormal risk, and regular monitoring, adjustment of diet and increase of exercise are recommended to prevent the occurrence of cardiovascular diseases. Through the intelligent report interpretation method, users can not only quickly grasp the key information of the physical examination, but also obtain personalized health guidance and improve the efficiency of health management. The present invention effectively integrates deep learning technology and expert knowledge to provide users with accurate and convenient health services.

[0052] As an improvement to the above embodiment, the report recognition model is trained in the following manner:

[0053] Construct a multimodal model consistent with the reported recognition model architecture;

[0054] The report recognition model is based on the training of a multimodal model, which includes an image encoder, a text encoder, an attention fusion layer, and a classifier. The attention fusion layer uses dynamic weight allocation to achieve feature interaction between text and image information; the classifier is used to output multiple health categories.

[0055] Obtain a sample set, input the sample set into the multimodal model, freeze all layers of the image encoder and the text encoder, iterate the multimodal model, and gradually adjust the learning rate according to the number of iterations and the first loss value of the multimodal model;

[0056] After the training parameters of the multimodal model reach the number of warm-up steps, the image encoder and text encoder are unfrozen layer by layer according to the second loss value of the multimodal model, and the learning rate is gradually adjusted according to the number of iterations and the second loss value of the multimodal model until the second loss value of the multimodal model is lower than the second loss threshold, thereby obtaining a report recognition model; wherein, the input data of the sample set is a sample report containing graphic and text information, and the output data is the focus items and corresponding health categories in the sample report.

[0057] It should be noted that in the warm-up stage, all layers of the image encoder and text encoder are frozen, and the cross-attention layer and classifier are iteratively trained until the loss value of the multimodal model is lower than the first loss threshold; in the thawing stage, the image encoder, text encoder, attention fusion layer and classifier are iteratively trained, and the image encoder and text encoder are gradually thawed. The various layers of the image encoder and text encoder are gradually thawed in order from high to low levels. When the loss value of the multimodal model is lower than the second loss threshold, the training of the multimodal model is completed and the report recognition model is obtained.

[0058] Specifically, the image encoder uses the ViT model, and the text encoder uses the BERT model. The ViT model extracts image features, while the BERT model extracts text features. If the text features contain keywords related to the health category (such as test results), the text weight is automatically increased, giving more attention to the text semantics. If the image contains feature regions strongly associated with the health category (such as lesion areas), the image weight is increased. This adaptively assigns dynamic weights to images and text, improving the robustness and generalization of cross-modal tasks. The hyperparameters for initializing the multimodal model include: setting the pre-trained weights for the ViT and BERT models, and setting the number of attention heads in the cross-attention layer. The input dimension of the classifier is the total dimension of the concatenated output features of the ViT and BERT models, and the output dimension is the total number of health categories. The pre-trained weights for the ViT model are set based on ViTB / 16, and the pre-trained weights for the BERT model are set based on BERTbase, preserving their underlying feature extraction capabilities. Only the parameters of the cross-attention layer and the classifier are fine-tuned, reducing training time and computational cost. By loading the pre-trained ViT model and BERT model, and freezing the parameters of all layers to prevent them from being updated during training.

[0059] The ViT model normalizes image information to 256×256 pixels and normalizes it to the range [0,1]. The ViT model's Transformer architecture introduces local window attention, capturing global features while preserving local details. The BERT model performs word segmentation and word embedding on text information to generate sequential input.

[0060] The cross-attention layer captures intermodal correlations and enhances feature fusion. The classifier accurately identifies health levels, deriving focus items and corresponding health categories, improving prediction accuracy. A multi-stage training strategy optimizes model performance, ensuring efficient and stable health assessment.

[0061] Through a learnable attention weight matrix, the contribution of features between modalities is adaptively allocated (images have higher weights in early layers, and text has increased weights in later layers), significantly improving the accuracy of health category recognition.

[0062] An adaptive learning strategy is adopted to dynamically adjust model parameters to adapt to different data distributions, further enhancing the generalization and robustness of the model, ensuring that stable and reliable interpretation results can be provided in different scenarios, achieving efficient fusion of the model on multimodal data, and significantly improving the accuracy of health assessment.

[0063] As an improvement to the above embodiment, the method of dynamically allocating weights of image features and text features through the attention fusion layer includes:

[0064] The attention weight of the text feature T on the image feature V is calculated by cross attention. The calculation formula is:

[0065]

[0066] Among them, α ij represents the attention weight of the i-th text token to the j-th image patch, V is the image feature matrix, T is the text feature matrix, Q v is the mapping matrix from image features to text feature queries, K t is the mapping matrix from text features to image feature keys, V ik is the mapping matrix from image features to text feature queries, Q v , K t is a learnable matrix, k is the index variable of the image patch, d v is the image feature dimension, d t is the text feature dimension.

[0067] It should be noted that Q v , K t It is a learnable weight matrix used to project image features and text features into a dynamic weight space, learn the interaction pattern of image and text features, and dynamically assign weights. If the image feature dimension d v =512, text feature dimension d t =512, then Q v , K t is a 512×512 matrix. V is the image feature matrix, with a shape of (N,d v ), T is the text feature matrix, the shape is (M,d t ), N is the number of image patches, and M is the number of tokens of text features.

[0068] The attention weight of the image feature V on the text feature T is calculated by cross attention. The calculation formula is:

[0069]

[0070] Among them, β kl represents the attention weight of the k-th image patch to the l-th text token, Q t is the mapping matrix from text features to image feature queries, K v is the mapping matrix from image features to text feature keys, Q t , K v is a learnable matrix, and m is the index variable of the text token.

[0071] It should be noted that d v is the image feature dimension, d tis the text feature dimension.

[0072] Use Softmax function to adjust the attention weight α ij and attention weight β kl Normalize and get image weight and text weight.

[0073] Through the cross-attention mechanism, the model can more comprehensively understand the complex relationship between images and text, improving performance on multimodal tasks. Experiments have demonstrated significant results in tasks such as image captioning and question answering, validating its effectiveness. Further optimization of model parameters can improve the accuracy of cross-modal information fusion and enhance the model's generalization capabilities.

[0074] As an improvement to the above embodiment, the learning rate is calculated as follows:

[0075] lr(t) new =lr(t)·δ cut ;

[0076]

[0077] Among them, t is the current training step, lr(t) new is the learning rate of the tth iteration, lr(t) is the baseline learning rate of the tth iteration, δ and γ are adjustment factors, 0<δ<1, 0<γ<1, cut is the number of iterations in which the loss value drops below the amplitude threshold continuously, lr base is the initial learning rate, w s is the number of warm-up steps, t s is the total number of training steps, L1 represents the first loss value, and L2 represents the second loss value.

[0078] Specifically, the number of warm-up steps is set to 1000 and the total number of training steps is set to 50000. The base learning rate of the ViT model and the BERT model is 1×10 6 (fine-tuning after unfreezing), the base learning rate of the cross attention layer and the classifier is 1×10 4 The default learning rate for other parameters is 1×10 5 The larger the loss value, the larger the learning rate. If the loss value decreases by less than the threshold, the learning rate is reduced. By dynamically adjusting the learning rate, the model converges quickly in the early stages of training and is finely optimized in the later stages, effectively improving overall performance and generalization ability.

[0079] In some embodiments, the first loss value of the multimodal model is calculated using the following formula:

[0080] L1=λ1·L align +(1-λ1)·L class ;

[0081]

[0082] Among them, L1 align represents the cross attention alignment loss, A i,i Represents the cross attention matrix of text features to image features, A i,i =Softmax(α ij );N ′ is the batch size, N is the number of patches of image features, M is the number of tokens of text features, L1 class Represents the first classification loss value, Y ih Indicates a valid encoding of the true label (One-Hot encoding, when the i-th sample belongs to category h, Y ih is 1), is the probability that the i-th sample trained by the multimodal model belongs to category h; λ1 is the weight coefficient, 0<λ1<1.

[0083] It's important to note that freezing the pre-trained encoder, iterating the multimodal model, and dynamically adjusting the learning rate effectively balances the optimization requirements of feature extraction and multimodal alignment. This dynamic adjustment strategy allows the multimodal model to adaptively balance convergence speed and accuracy during training, demonstrating greater robustness and adaptability in complex tasks.

[0084] In some embodiments, the second loss value of the multimodal model is calculated using the following formula:

[0085] L2=L2 var +L2 class +L2 KL ;

[0086]

[0087] Among them, L2 represents the second loss value, L2 var Indicates the variance penalty value, L2 class Represents the second classification loss value, L2 KL represents the KL divergence value; q represents the variational distribution of the parameter θ, θ represents the hyperparameter of the model, and q(θ) represents the posterior distribution of the parameter θ. represents the variance parameter, E q(θ) represents the expected value of parameter θ, Z (y＝h) represents the indicator function, p(y=h|x,θ) represents the probability that the input fusion feature x belongs to the healthy category h, y represents the true label, H represents the total number of healthy categories, σ 2 represents the variance of the posterior distribution q(θ), p(θ) represents the prior distribution of the parameter θ, represents the variance of the prior distribution p(θ); λ2, λ3, and λ4 are all weight coefficients ranging from 0 to 1, and λ2+λ3+λ4=1.

[0088] It should be noted that x represents the input fusion feature, and y represents the true health level label of the input sample; when the true label y is equal to the health category h, Z (y＝h) =1; otherwise Z (y＝h) = 0. p(y = h|x,θ) represents the probability that the input fusion feature x belongs to category h (h = 1, 2, ..., 5 correspond to normal, mild abnormality, moderate abnormality, severe abnormality, and critical respectively). Represents the model's estimate of the uncertainty of the output, with all classes sharing the same variance.

[0089] The second loss value of the multimodal model includes the variance penalty value, the second classification loss value and the KL divergence value; the variance penalty value The variance used to encourage model predictions Don't be too big and avoid being overconfident; when When the variance of the true data distribution is close, the weight of the classification error is reduced, reflecting the impact of uncertainty on the prediction. The parameter θ is constrained to follow the prior distribution p(θ) to prevent overfitting. By combining the variance penalty, the second classification loss, and the KL divergence, the classification accuracy and the uncertainty of the multimodal model are balanced.

[0090] During training, the weight coefficients λ2, λ3, and λ4 are continuously adjusted to optimize the overall loss function and improve prediction accuracy. Simultaneously, changes in the variance parameters are monitored to ensure the model's uncertainty estimates are reasonable, avoiding overconfidence or overconservatism. Ultimately, this achieves efficient fusion of multimodal data and precise health level classification. This not only accurately identifies health levels but also effectively quantifies prediction uncertainty.

[0091] refer to Figure 2 , an embodiment of the present invention further provides an intelligent report interpretation system based on deep learning, comprising:

[0092] at least one processor;

[0093] at least one memory for storing at least one program;

[0094] When the at least one program is executed by the at least one processor, the at least one processor implements the above method.

[0095] The contents of the above method embodiments are all applicable to this embodiment. The functions specifically implemented by this embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments, which will not be repeated here.

[0096] Although the description of the present disclosure has been quite detailed and particularly describes several embodiments, it is not intended to be limited to any of these details or embodiments or any particular embodiment, but should be considered to provide a broad possible interpretation of these claims by reference to the appended claims in view of the prior art, thereby effectively covering the intended scope of the present disclosure. In addition, the above description of the present disclosure is based on the embodiments foreseen by the inventors, which is intended to provide a useful description, and those non-substantial changes to the present disclosure that have not yet been foreseen may still represent equivalent changes to the present disclosure.

Claims

1. An intelligent report interpretation method based on deep learning, characterized in that: The method comprises the following steps: Obtain a medical examination report containing multiple medical examination items and identify text and image information in the medical examination report; The text and image information are input into a report recognition model consisting of an image encoder, a text encoder, an attention fusion layer, and a classifier. The image encoder extracts the image features of the physical examination items, and the text encoder extracts the text features of the physical examination items. The attention fusion layer dynamically assigns weights to the image features and text features, aligns the weighted image features to the dimensions of the text features, and then concatenates them with the weighted text features to obtain fused features. The classifier classifies the fused features, identifies the items of concern in the physical examination report, and outputs the health category corresponding to the items of concern. Input the items of interest and the corresponding health categories into the knowledge graph to obtain the interpretation results of the physical examination report; The report recognition model is trained in the following way: Construct a multimodal model consistent with the reported recognition model architecture; Obtain a sample set, input the sample set into the multimodal model, freeze all layers of the image encoder and the text encoder, iterate the multimodal model, and gradually adjust the learning rate according to the number of iterations and the first loss value of the multimodal model; After the training parameters of the multimodal model reach the preheating step number, the image encoder and text encoder are unfrozen layer by layer according to the second loss value of the multimodal model, and the learning rate is gradually adjusted according to the number of iterations and the second loss value of the multimodal model until the second loss value of the multimodal model is lower than the second loss threshold, thereby obtaining a report recognition model; wherein the input data of the sample set is a sample report containing image and text information, and the output data is the focus items and corresponding health categories in the sample report; The second loss value of the multimodal model is calculated by the following formula: <h2 style=";text-align:left;direction:ltr">L2=L2<h2 style=";text-align:left;direction:ltr"> var <h2 style=";text-align:left;direction:ltr"> +L2<h2 style=";text-align:left;direction:ltr"> class <h2 style=";text-align:left;direction:ltr"> +L2<h2 style=";text-align:left;direction:ltr"> KL <h2 style=";text-align:left;direction:ltr"> ; Among them, L2 represents the second loss value, L2 var Indicates the variance penalty value, L2 class Represents the second classification loss value, L2 KL represents the KL divergence value; q represents the variational distribution of the parameter θ, θ represents the hyperparameter of the model, and q(θ) represents the posterior distribution of the parameter θ. represents the variance parameter, E q(θ) represents the expected value of parameter θ, Z (y＝h) represents the indicator function, p(y=h|x,θ) represents the probability that the input fusion feature x belongs to the healthy category h, y represents the true label, H represents the total number of healthy categories, σ 2 represents the variance of the posterior distribution q(θ), p(θ) represents the prior distribution of the parameter θ, represents the variance of the prior distribution p(θ).

2. The method according to claim 1, characterized in that The dynamic allocation of weights of image features and text features through the attention fusion layer includes: Calculate the attention weight α of text features to image features through cross attention ij , and the attention weight β of the image feature V to the text feature T kl : Use Softmax function to adjust the attention weight α ij and attention weight β kl Normalize and get image weight and text weight.

3. The method according to claim 2, characterized in that The attention weight of the text feature to the image feature is calculated by the following formula: Among them, T is the text feature, V is the image feature, α ij represents the attention weight of the i-th text token to the j-th image patch, Q v is the mapping matrix from image features to text feature queries, K t is the mapping matrix from text features to image feature keys, V ik is the mapping matrix from image features to text feature queries, Q v , K t is a learnable matrix and k is the index variable of the image patch.

4. The method according to claim 3, characterized in that The attention weight of the image feature V to the text feature T is calculated by the following formula: Among them, β kl represents the attention weight of the k-th image patch to the l-th text token, Q t is the mapping matrix from text features to image feature queries, K v is the mapping matrix from image features to text feature keys, Q t , K v is a learnable matrix, and m is the index variable of the text token.

5. The method according to claim 1, wherein The learning rate is calculated as follows: lr(t) new =lr(t)·δ cut ; Among them, t is the current training step, lr(t) new is the learning rate of the tth iteration, lr(t) is the baseline learning rate of the tth iteration, δ and γ are adjustment factors, 0<δ<1, 0<γ<1, cut is the number of iterations in which the loss value drops below the amplitude threshold continuously, lr base is the initial learning rate, w s is the number of warm-up steps, t s is the total number of training steps, L1 represents the first loss value, and L2 represents the second loss value.

6. The method according to claim 5, characterized in that The first loss value of the multimodal model is calculated by the following formula: L1⼝λ·L align +(1-λ)·L class ; Among them, L1 align represents the cross attention alignment loss, A i,i Represents the cross attention matrix of text features to image features; N ′ is the batch size, N is the number of patches of image features, M is the number of tokens of text features, L1 class Represents the first classification loss value, Y ih represents a single-bit efficient encoding of the true label, The probability that the i-th sample belongs to class h for multimodal model training.

7. An intelligent report interpretation system based on deep learning, characterized in that: include: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method according to any one of claims 1 to 6.

Images