An intelligent auxiliary method and system based on multi-modal deep learning

Abstract

The application discloses an intelligent auxiliary method and system based on multi-modal deep learning, relates to medical image processing and data processing technology, and comprises the following steps: text features are extracted based on illness description text data, and image features of medical image data are extracted; the extracted text features and image features are respectively input into a pre-trained LSTM, so that the LSTM outputs context illness text features and local lesion features; the output context illness text features and local lesion features are weighted and fused by using an attention fusion mechanism; the weighted and fused features are encoded by using an encoder, so that text-related lesion features are output by the encoder; and the text-related lesion features and the local lesion features are decoded by using a decoder, so that a predicted lesion condition is output based on the decoder. Through fusion of text and image features in cross modalities, the application improves the prediction accuracy of the model.

Description

Technical Field

[0001] This application relates to the fields of medical image processing and data processing technology, and in particular to an intelligent assistance method and system based on multimodal deep learning. Background Technology

[0002] With the development of artificial intelligence, its application in medicine is becoming increasingly widespread. For example, neural networks are used to learn from a large number of medical diagnostic images to achieve disease classification. To improve the accuracy of disease classification, multimodal fusion techniques, such as the fusion of image and text modalities, have been proposed. Image-text fusion can supplement the missing information in a single image or text, making the neural network model more powerful and the classification results more accurate.

[0003] Existing technologies involve extracting features from image and text modalities, training these features separately, and finally performing classification through decision-level fusion. For example, Chinese patent number 2023116944924 discloses "A Multimodal Medical Semantic Alignment Method for Electronic Medical Records Based on Contrastive Learning," which extracts and trains features for image and text modalities separately, and then performs decision fusion. However, this method ignores the correlation between images and text, and only processes image and text features separately, reducing the prediction accuracy of the neural network model. Summary of the Invention

[0004] This application provides an intelligent assistance method and system based on multimodal deep learning. By fusing cross-modal text and image features, it improves the prediction accuracy of the model and enhances the assistance effect of artificial intelligence on medical data.

[0005] This application proposes an intelligent assistance method based on multimodal deep learning, including:

[0006] Obtain textual data describing the patient's condition, as well as medical imaging data at the corresponding diagnosis time;

[0007] Text features are extracted from the disease description text data, and image features are extracted from the medical image data;

[0008] The extracted text features and image features are respectively input into a pre-trained LSTM to utilize the LSTM to output contextual disease text features and local lesion features.

[0009] The output contextual features of the patient's condition and the local lesion features are weighted and fused using an attention fusion mechanism.

[0010] The weighted fusion features are encoded using an encoder to output text-related lesion features through the encoder output;

[0011] The text-associated lesion features and local lesion features are decoded using a decoder to output a predicted lesion condition based on the decoder output.

[0012] Optionally, extracting text features based on the disease description text data includes:

[0013] The text data describing the illness is segmented into words.

[0014] Based on multiple pre-configured disease-representative terms, the word segments obtained by word segmentation are processed, and the similarity between each word segment and the disease-representative terms is calculated. The frequency of occurrence of each word segment with a similarity greater than a preset threshold is also calculated. The disease-representative terms include fixed word segments and dynamic word segments.

[0015] Assign a first weight to fixed word segments, and assign weights to corresponding word segments based on their frequency of occurrence, with higher frequency words having higher weights;

[0016] Text features are extracted based on each word segment with a similarity greater than a preset threshold and its weight.

[0017] Optionally, the frequency of occurrence of each word segment with a similarity greater than a preset threshold is calculated by including each word segment with a similarity greater than a preset threshold in the statistical frequency of the same dynamic word segment, so as to calculate the frequency of occurrence of the dynamic word segment.

[0018] Assigning a primary weight to fixed-value segments and assigning weights to dynamic-value segments based on their frequency of occurrence includes:

[0019] Determine the relationship between fixed and dynamic word segmentation, assign a first weight to the fixed word segmentation based on the determined relationship, and assign a corresponding second weight based on the frequency of occurrence of the dynamic word segmentation, so that the sum of the first weight and each second weight is 1.

[0020] Optionally, extracting image features from the medical image data includes:

[0021] Based on the statistical analysis of dynamic word segmentation, the appropriate organ or tissue is determined;

[0022] The medical image data is segmented to obtain multiple sub-regions;

[0023] Calculate the similarity between the segmented sub-regions and typical illustrations of the corresponding organs or tissues, and determine the local image based on multiple sub-regions with similarity greater than a preset similarity threshold;

[0024] Based on the identified local image, extract image features.

[0025] Optionally, the extracted text features and image features are input into a pre-trained LSTM to utilize the LSTM output contextual text features and local lesion features, including:

[0026] ;

[0027] in, , These represent the hidden states of the corresponding feature at step n and step (n-1), respectively. For the characteristics of the nth step length, , These represent the cell states of the corresponding features at step n and step (n-1), respectively.

[0028] Optionally, the output contextual features of the patient's condition and local lesion features can be weighted and fused using an attention fusion mechanism, including:

[0029] ;

[0030] in, As a feature of weighted fusion, , For trainable weights, , Weighted parameters are assigned to the context-based textual features of the patient's condition and the local lesion features, respectively. , , , These are the first and second weight matrices representing the local lesion features, respectively. The weight matrix represents the textual features of the patient's condition. The lesion features are output by LSTM. The contextual features of the disease text output by LSTM. To predict the current fusion features in the sequence, Z is the bilinear weight matrix, representing the j-th fusion feature in the predicted sequence.

[0031] Optionally, the training process may also include calculating the cross-entropy loss between the predicted lesion status and the actual lesion distribution based on the patient's historical medical data, in order to perform model training.

[0032] Optionally, the text-associated lesion features and local lesion features are decoded using a decoder to predict the lesion situation based on the decoder output, including:

[0033] The corresponding region of the medical image data at the current diagnosis time is superimposed with a portion of the predicted lesion condition output, so as to fuse the predicted lesion condition output with the corresponding region of the medical image data at the current diagnosis time.

[0034] The system calculates the sharpness of the fused local areas, highlights image areas with sharpness below a preset sharpness threshold, and...

[0035] Based on the determined dynamic word segmentation, annotations are performed on the highlighted regions.

[0036] This application also proposes an intelligent assistance system based on multimodal deep learning, including a processor and a memory. The memory stores a computer program, which, when executed by the processor, implements the steps of the aforementioned intelligent assistance method based on multimodal deep learning.

[0037] The method in this application improves the prediction accuracy of the model by fusing cross-modal text and image features, effectively enhancing the auxiliary effect of artificial intelligence on medical data.

[0038] The above description is merely an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0039] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings.

[0040] Figure 1 This is a schematic diagram of the basic process of the intelligent assistance method based on multimodal deep learning in this embodiment. Detailed Implementation

[0041] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0042] This application proposes an intelligent assistance method based on multimodal deep learning, such as... Figure 1 As shown, it includes the following steps:

[0043] In step S101, the patient's medical condition description text data and the corresponding medical image data at the time of diagnosis are obtained. In some embodiments, the patient's medical condition description text data may include the patient's historical medical records, the patient's medical condition description, etc., combined to form the medical condition description text data. In some examples, the corresponding medical image data may be obtained by combining the time when the patient's medical images were collected. For example, medical images at the corresponding time can be extracted based on the patient's medical history.

[0044] In step S102, text features are extracted based on the disease description text data, and image features are extracted from the medical image data. In some embodiments, this can be achieved based on TF-IDF or Word2Vec, and the image features of the medical image data can be extracted based on CNN, etc.

[0045] In step S103, the extracted text features and image features are respectively input into a pre-trained LSTM to output contextual disease text features and local lesion features. In a specific example, based on LSTM, the input sequence, such as a text sequence composed of text features extracted from historical disease descriptions, and the corresponding extracted image feature sequence, can be processed separately to output contextual disease text features and local lesion features using LSTM.

[0046] In step S104, the output contextual disease text features and local lesion features are weighted and fused using an attention fusion mechanism. In this embodiment, the contextual disease text features and local lesion features are fused through cross-modal attention fusion, thereby improving the correlation between the image and the text, and thus improving the prediction accuracy of the model.

[0047] In step S105, the weighted fusion features are encoded using an encoder to output text-related lesion features through the encoder.

[0048] In step S106, the text-associated lesion features and local lesion features are decoded using a decoder to output a predicted lesion condition based on the decoder output. This embodiment of the application, by encoding the weighted fused features using an encoder and further decoding the encoded features with local lesion features using a decoder, enables the association of the patient's condition text features. Through the fusion of cross-modal text and image features, the prediction accuracy of the model is improved, effectively enhancing the auxiliary effect of artificial intelligence on medical data.

[0049] In some embodiments, extracting text features based on the disease description text data includes:

[0050] The disease description text data is processed by word segmentation, for example, word segmentation is performed on the combined disease description text data.

[0051] Based on multiple pre-configured disease-representative terms, the similarity between each segmented word and the disease-representative terms is calculated, and the frequency of occurrence of each segmented word with a similarity greater than a preset threshold is statistically calculated. Disease-representative terms include fixed and dynamic segments. In some examples, corresponding disease-representative terms can be configured for each typical case. In some examples, the disease is also associated with factors such as age and gender. In this example, disease-representative terms are divided into fixed and dynamic segments. For example, fixed segments can be age and gender, while dynamic segments can be feature descriptions of a specific case, such as abdominal pain or swelling caused by a certain disease. A first weight is assigned to the fixed segments, and weights are assigned to the corresponding segments based on their frequency of occurrence, with higher frequencies resulting in higher weights. In specific examples, since patients' conditions are individual, this application uses a first weight for the fixed segments and dynamic weights for the dynamic segments of the patient's condition to effectively balance the influence of both on the condition, improving the recognition accuracy of subsequent models.

[0052] Text features are extracted based on each word segment with a similarity greater than a preset threshold and its weight.

[0053] In some embodiments, calculating the frequency of occurrence of each word segment with a similarity greater than a preset threshold includes: including each word segment with a similarity greater than the preset threshold in the statistical frequency of the same dynamic word segment to calculate the frequency of occurrence of the dynamic word segment. In this embodiment, by including words with a similarity greater than a specified threshold in the same dynamic word segment, the deviation caused by potential non-standardization in the word segmentation description of the disease description text data can be reduced.

[0054] Assigning a first weight to fixed word segments and assigning weights to corresponding word segments based on the frequency of occurrence of dynamic word segments includes: determining the association between fixed word segments and dynamic word segments, assigning a first weight to fixed word segments based on the determined association, and assigning corresponding second weights based on the frequency of occurrence of dynamic word segments, so that the sum of the first weight and each second weight is 1.

[0055] In some embodiments, extracting image features from the medical image data includes:

[0056] Based on the statistical dynamic word segmentation, the appropriate organ or tissue can be determined. For example, in some examples, the description of the illness includes heart pain, abdominal pain, etc., so the appropriate organ or tissue can be roughly determined.

[0057] The medical image data is segmented to obtain multiple sub-regions. In a specific example, based on the aforementioned identified organs or tissues, these are designated as key observation areas in the medical images.

[0058] The similarity between segmented sub-regions and typical images of the corresponding organs or tissues is calculated. Local images are then identified based on multiple sub-regions with similarity scores exceeding a preset similarity threshold. For example, for the heart, typical images can be used to identify local regions in medical images, eliminating the need for manual selection of regions of interest and improving subsequent assistance to doctors.

[0059] Based on the identified local image, extract image features.

[0060] In some embodiments, the extracted text features and image features are respectively input into a pre-trained LSTM to utilize the LSTM to output contextual disease text features and local lesion features, including:

[0061]

[0062] in, , These represent the hidden states of the corresponding feature at step n and step (n-1), respectively. For the characteristics of the nth step length, , These represent the cell states of the corresponding features at step n and step (n-1), respectively. In a specific example, the extracted text features and image features are input into an LSTM, thereby using two LSTMs to output the contextual disease text features and local lesion features, respectively.

[0063] In some embodiments, weighted fusion of the output contextual disease text features and local lesion features using an attention fusion mechanism includes:

[0064]

[0065] in, As a feature of weighted fusion, , For trainable weights, , Weighted parameters are assigned to the context-based textual features of the patient's condition and the local lesion features, respectively. , , , These are the first and second weight matrices representing the local lesion features, respectively. The weight matrix represents the textual features of the patient's condition. For cross-modal attention, The lesion features are output by LSTM. The contextual features of the disease text output by LSTM. To predict the current fusion features in the sequence, Z is the bilinear weight matrix, representing the j-th fusion feature in the predicted sequence.

[0066] In some embodiments, the training process further includes calculating the cross-entropy loss between the predicted lesion status and the actual lesion distribution based on the patient's historical medical data, in order to perform model training.

[0067] In some embodiments, decoding the text-associated lesion features and local lesion features using a decoder to predict the lesion condition based on the decoder output includes:

[0068] This method involves overlaying a portion of the corresponding region from the medical imaging data at the current diagnosis time with a portion of the predicted lesion area to fuse the predicted lesion with the corresponding region from the medical imaging data at the current diagnosis time. In other words, it allows for the fusion of relevant image regions, preserving local details before and after prediction.

[0069] The system calculates the sharpness of local areas in the fusion process and highlights image areas with a sharpness below a preset threshold. Highlighting blurry areas allows doctors to determine the possible trend of disease progression in a timely manner, thereby improving the effectiveness of auxiliary diagnosis and treatment.

[0070] Based on the determined dynamic word segmentation, the highlighted areas are labeled. The labeling of the areas based on the aforementioned dynamic word segmentation can further supplement the auxiliary effect of the final output image containing lesions.

[0071] This application also proposes an intelligent assistance system based on multimodal deep learning, including a processor and a memory. The memory stores a computer program, which, when executed by the processor, implements the steps of the aforementioned intelligent assistance method based on multimodal deep learning.

[0072] Furthermore, although exemplary embodiments have been described herein, their scope includes any and all embodiments based on this disclosure that have equivalent elements, modifications, omissions, combinations (e.g., schemes involving intersections of various embodiments), adaptations, or changes. They are not limited to the examples described in this specification or during the implementation of this application, and such examples are to be construed as non-exclusive.

[0073] The above description is intended to be illustrative and not restrictive. For example, the above examples (or one or more of them) can be used in combination with each other. Other embodiments can be used by those skilled in the art when reading the above description.

[0074] The above embodiments are merely exemplary embodiments of this disclosure. Those skilled in the art can make various modifications or equivalent substitutions to this invention within the scope of the disclosure, and such modifications or equivalent substitutions should also be considered to fall within the protection scope of this invention.

Claims

1. An intelligent assistance method based on multi-modal deep learning, characterized in that, The method comprises the following steps: obtaining patient's disease description text data and medical image data at corresponding diagnosis time; extracting text features based on the disease description text data and image features of the medical image data; inputting the extracted text features and image features into a pre-trained LSTM respectively to output contextual disease text features and local lesion features by the LSTM; performing weighted fusion on the output contextual disease text features and local lesion features by using an attention fusion mechanism; encoding the weighted and fused features by using an encoder to output text-related lesion features by the encoder; decoding the text-related lesion features and local lesion features by using a decoder to output predicted lesion conditions based on the decoder; extracting text features based on the disease description text data comprises the following steps: performing word segmentation on the disease description text data; calculating the similarity between each segmented word and a disease representative word based on a plurality of pre-configured disease representative words, and counting the frequency of each segmented word with a similarity greater than a preset threshold, wherein the disease representative words include fixed segmented words and dynamic segmented words; configuring a first weight for the fixed segmented words, and configuring a weight for the corresponding dynamic segmented words according to the frequency of the dynamic segmented words, wherein the higher the frequency, the greater the weight; extracting text features according to the segmented words with a similarity greater than the preset threshold and their weights; extracting image features of the medical image data comprises the following steps: determining the adapted organ or tissue according to the counted dynamic segmented words; segmenting the medical image data to obtain a plurality of sub-regions; calculating the similarity between the segmented sub-regions and the typical patterns of the adapted organ or tissue to determine local images according to the plurality of sub-regions with a similarity greater than a preset similarity threshold; extracting image features according to the determined local images; inputting the extracted text features and image features into a pre-trained LSTM respectively to output contextual disease text features and local lesion features by the LSTM comprises the following steps: ； wherein, , are the hidden states of the respective features at the th step, and the th step, respectively, is the feature at the th step, , are the cell states of the respective features at the th step, and the th step, respectively. performing weighted fusion on the output contextual disease text features and local lesion features by using an attention fusion mechanism comprises the following steps: ； in, As a feature of weighted fusion, , For trainable weights, , Weighted parameters are assigned to the context-based textual features of the patient's condition and the local lesion features, respectively. , , , These are the first and second weight matrices representing the local lesion features, respectively. The weight matrix represents the textual features of the patient's condition. For cross-modal attention, The lesion features are output by LSTM. The contextual features of the disease text output by LSTM. To predict the current fusion features in the sequence, For the predicted sequence of the first A fusion feature, It is a bilinear weight matrix. 2.The intelligent assistance method based on multi-modal deep learning of claim 1, wherein, counting the frequency of each segmented word with a similarity greater than a preset threshold comprises counting each segmented word with a similarity greater than a preset threshold into the statistical frequency of the same dynamic segmented word to count the frequency of the dynamic segmented word; configuring a first weight for the fixed segmented words and a weight for the corresponding dynamic segmented words according to the frequency of the dynamic segmented words comprises the following steps: determining the association between the fixed segmented words and the dynamic segmented words, configuring a first weight for the fixed segmented words according to the determined association, and configuring a second weight for the corresponding dynamic segmented words according to the frequency of the dynamic segmented words, so that the sum of the first weight and each second weight is 1. 3.The intelligent assistance method based on multi-modal deep learning of claim 1, wherein, In the training process, the cross-entropy loss between the predicted lesion conditions and the actual lesion distribution is calculated according to the patient's historical disease data to perform model training. 4.The intelligent assistance method based on multi-modal deep learning of claim 3, wherein, decoding the text-related lesion features and local lesion features by using a decoder to output predicted lesion conditions based on the decoder comprises the following steps: superimpose the corresponding region of the medical image data at the current diagnosis time and the partial region of the output predicted lesion condition to fuse the output predicted lesion condition and the corresponding region of the medical image data at the current diagnosis time; calculate the local region definition of the fusion, and highlight the image region whose definition is lower than a preset definition threshold, and perform labeling based on the highlighted region according to the determined dynamic word segmentation.

5. An intelligent assistant system based on multi-modal deep learning, characterized by, The system comprises a processor and a memory, and the memory stores a computer program which, when executed by the processor, implements the steps of the intelligent auxiliary method based on multi-modal deep learning according to any one of claims 1 to 4.

Images