Intelligent auxiliary method and system based on multi-modal deep learning

Abstract

The invention discloses an intelligent auxiliary method and system based on multi-modal deep learning, and relates to a medical image processing and data processing technology, and the method comprises the steps: extracting text features based on illness state description text data, and extracting image features of medical image data; according to the extracted text features and the image features, inputting a pre-trained LSTM respectively, and outputting contextual illness state text features and local focus features by using the LSTM; carrying out weighted fusion on the output contextual illness state text features and local focus features by using an attention fusion mechanism; encoding the weighted fusion features by using an encoder so as to output text-associated focus features through the encoder; and decoding the lesion features associated with the text and the local lesion features by using a decoder so as to output a predicted lesion condition based on the decoder. Through fusion of cross-modal text and image features, the prediction precision of the model is improved.

Description

Technical Field

[0001] The present application relates to the field of medical image processing and data processing technology, and in particular to an intelligent assistance method and system based on multimodal deep learning. Background Art

[0002] With the development of artificial intelligence, its application in medicine is becoming more and more common. For example, neural networks are used to learn a large number of medical diagnostic images to achieve the purpose of disease classification. In order to improve the accuracy of disease classification, multimodal fusion technology has been further proposed, such as the fusion of image modality and text modality. Image and text fusion can supplement the missing information of a single image or text, making the classification ability of the neural network model stronger and the classification results more accurate.

[0003] The existing technology includes extracting features from image modalities and text modalities, then training these features separately, and finally classifying through decision-level fusion. For example, the Chinese patent with patent number 2023116944924 discloses "a multimodal medical semantic alignment method for electronic medical records based on contrastive learning", which is to extract and train features from image and text modalities separately, and finally perform decision fusion. However, this method ignores the correlation between images and texts, and only processes image and text features separately, which reduces the prediction accuracy of the neural network model. Summary of the invention

[0004] The embodiments of the present application provide an intelligent assistance method and system based on multimodal deep learning, which improves the prediction accuracy of the model and the assistance effect of artificial intelligence based on medical data by fusing cross-modal text and image features.

[0005] The present application embodiment proposes an intelligent assistance method based on multimodal deep learning, including: Obtain the patient's condition description text data and the medical imaging data at the corresponding diagnosis time; Extracting text features based on the condition description text data, and extracting image features of the medical image data; The extracted text features and image features are respectively input into the pre-trained LSTM to use the LSTM to output the contextual disease text features and local lesion features; The output contextual disease text features and local lesion features are weightedly fused using the attention fusion mechanism; The weighted fused features are encoded using an encoder, so as to output text-related lesion features through the encoder; The text-associated lesion features and the local lesion features are decoded by a decoder to output a predicted lesion condition based on the decoder.

[0006] Optionally, extracting text features based on the condition description text data includes: Performing word segmentation processing on the condition description text data; Based on the pre-configured multiple disease representative words, the similarity between each segmentation word and the disease representative word is calculated, and the occurrence frequency of each segmentation word whose similarity is greater than a preset threshold is statistically calculated, wherein the disease representative word includes fixed segmentation words and dynamic segmentation words; A first weight is configured for a fixed participle, and a weight is configured for a corresponding participle according to the frequency of occurrence of the dynamic participle, wherein the higher the frequency of occurrence, the greater the weight; Extract text features based on each word and its weight whose similarity is greater than a preset threshold.

[0007] Optionally, the statistical calculation of the occurrence frequency of each segmentation word with a similarity greater than a preset threshold includes: counting each segmentation word with a calculated similarity greater than the preset threshold into the statistical frequency of the same dynamic segmentation word, so as to calculate the occurrence frequency of the dynamic segmentation word; Configuring a first weight for a fixed participle, and configuring a weight for a corresponding participle according to the frequency of occurrence of a dynamic participle includes: Determine the association relationship between the fixed participle and the dynamic participle, configure a first weight for the fixed participle according to the determined association relationship, and configure a corresponding second weight according to the occurrence frequency of the dynamic participle, so that the sum of the first weight and each second weight is 1.

[0008] Optionally, extracting image features of the medical image data includes: According to the statistical dynamic segmentation, the suitable organs or tissues are determined; Segmenting the medical image data to obtain a plurality of sub-regions; Calculating the similarity between the segmented sub-regions and the typical illustration of the adapted organ or tissue, so as to determine the local image according to a plurality of sub-regions having similarities greater than a preset similarity threshold; Image features are extracted based on the determined local image.

[0009] Optionally, the extracted text features and image features are respectively input into a pre-trained LSTM, so as to utilize the LSTM to output contextual disease text features and local lesion features, including: ; in, , The corresponding features are Step length, and The hidden state of step length, For the The characteristics of step length, , The corresponding features are Step length, and The cell state of the step.

[0010] Optionally, the output contextual disease text features and local lesion features are weightedly fused using an attention fusion mechanism, including: ; in, is the weighted fusion feature, , are trainable weights, , are weighted parameters based on the context-based disease text features and local lesion features, , , , are the first and second weight matrices of the local lesion features, is the weight matrix of the disease text features, is the local lesion feature output by LSTM, The condition text feature of the context output by LSTM, is the current fusion feature in the prediction sequence, To predict the first Fusion features, is the bilinear weight matrix.

[0011] Optionally, the training process also includes calculating the cross entropy loss between the predicted lesion condition and the actual lesion distribution based on the patient's historical medical data to perform model training.

[0012] Optionally, decoding the text-associated lesion features and the local lesion features using a decoder, so as to output a predicted lesion condition based on the decoder includes: Superimposing a region corresponding to the medical image data at the current diagnosis time with a portion of the output predicted lesion condition, so as to fuse the output predicted lesion condition with the corresponding region of the medical image data at the current diagnosis time; Calculate the clarity of the fused local area, highlight the image area whose clarity is below a preset clarity threshold, and, According to the determined dynamic segmentation, annotation is performed based on the highlighted area.

[0013] An embodiment of the present application also proposes an intelligent assistance system based on multimodal deep learning, including a processor and a memory, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the steps of the intelligent assistance method based on multimodal deep learning as described above are implemented.

[0014] The method of the embodiment of the present application improves the prediction accuracy of the model by fusing cross-modal text and image features, and effectively improves the auxiliary effect of artificial intelligence on medical data.

[0015] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings: Figure 1 The following is a schematic diagram of the basic process of the intelligent assistance method based on multimodal deep learning in this embodiment. DETAILED DESCRIPTION

[0017] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0018] The present application embodiment proposes an intelligent assistance method based on multimodal deep learning, such as Figure 1 As shown, the following steps are included: In step S101, the patient's condition description text data and the medical image data at the corresponding diagnosis time are obtained. In some embodiments, the patient's condition description text data may include the patient's historical medical history, the patient's condition description, etc., which are combined to form the condition description text data. In some examples, the corresponding medical image data can be obtained in combination with the time when the patient collects the medical image. For example, the medical image at the corresponding time can be extracted according to the patient's condition history.

[0019] In step S102, text features are extracted based on the text data describing the condition, and image features of the medical image data are extracted. In some embodiments, this can be achieved based on TF-IDF or Word2Vec, and image features of the medical image data can be extracted based on CNN.

[0020] In step S103, the pre-trained LSTM is inputted according to the extracted text features and image features, so as to output the contextual text features of the disease and the local lesion features by using the LSTM. In a specific example, based on the LSTM, the input sequence, such as the text sequence composed of the text features extracted from the historical disease description text and the corresponding extracted image feature sequence, can be respectively outputted by using the LSTM to output the contextual text features of the disease and the local lesion features.

[0021] In step S104, the output contextual disease text features and local lesion features are weighted fused using an attention fusion mechanism. In the embodiment of the present application, the contextual disease text features and local lesion features are fused by cross-modal attention fusion, thereby improving the correlation between the image and the text, thereby improving the prediction accuracy of the model.

[0022] In step S105, the weighted fused features are encoded using an encoder to output text-associated lesion features through the encoder.

[0023] In step S106, the text-associated lesion features are decoded with the local lesion features by a decoder, so as to output the predicted lesion condition based on the decoder. The embodiment of the present application encodes the weighted fused features by an encoder, and further decodes the encoded features with the local lesion features by a decoder, so as to associate the text features of the patient's condition, and improves the prediction accuracy of the model by fusing cross-modal text and image features, effectively improving the auxiliary effect of artificial intelligence on medical data.

[0024] In some embodiments, extracting text features based on the condition description text data includes: The condition description text data is segmented, for example, the condition description text data is segmented.

[0025] Based on the pre-configured multiple disease representative words, the segmentation processing obtains each segmentation word, calculates the similarity between each segmentation word and the disease representative word, and statistically calculates the frequency of occurrence of each segmentation word whose similarity is greater than the preset threshold, wherein the disease representative word includes fixed segmentation words and dynamic segmentation words. In some examples, corresponding disease representative words can be configured for each typical case, and in some examples, the disease is also associated with, for example, age and gender. In this example, the disease representative words are divided into fixed segmentation words and dynamic segmentation words. For example, fixed segmentation words can be age and gender, etc., and dynamic segmentation words can be characteristic descriptions of a certain case, such as abdominal pain, swelling, etc. caused by a certain disease. Configure a first weight for the fixed segmentation words, and configure a weight for the corresponding segmentation words according to the frequency of occurrence of the dynamic segmentation words, wherein the higher the frequency of occurrence, the greater the weight. In the specific example, since the patient's condition is individual, in this application, by configuring a first weight for the fixed segmentation words, and configuring a dynamic weight for the dynamic segmentation words of the patient's condition, the degree of influence of the two on the condition can be well balanced, and the recognition accuracy of the subsequent model can be improved.

[0026] Extract text features based on each word and its weight whose similarity is greater than a preset threshold.

[0027] In some embodiments, the statistical calculation of the occurrence frequency of each segmentation word with a similarity greater than a preset threshold includes: counting each segmentation word with a similarity greater than the preset threshold into the statistical frequency of the same dynamic segmentation word, so as to count the occurrence frequency of the dynamic segmentation word. In the embodiment of the present application, by counting words with a similarity greater than a specified threshold into the same dynamic segmentation word, the deviation caused by the possible non-standardization of the segmentation description in the condition description text data can be reduced.

[0028] Configuring a first weight for a fixed participle and configuring a weight for a corresponding participle according to the frequency of occurrence of a dynamic participle includes: determining an association relationship between the fixed participle and the dynamic participle, configuring a first weight for the fixed participle according to the determined association relationship, and configuring a corresponding second weight according to the frequency of occurrence of the dynamic participle, so that the sum of the first weight and each second weight is 1.

[0029] In some embodiments, extracting the image features of the medical image data includes: According to the dynamic segmentation statistics, the suitable organs or tissues are determined. For example, in some examples, the disease descriptions include heart pain, abdominal pain, etc., so that the suitable organs or tissues can be roughly determined.

[0030] The medical image data is segmented to obtain a plurality of sub-regions. In a specific example, based on the aforementioned determined organs or tissues, they are used as key observation sites in the medical image.

[0031] The similarity between the segmented sub-regions and the typical illustrations of the adapted organs or tissues is calculated to determine the local image based on multiple sub-regions whose similarity is greater than a preset similarity threshold. For example, for the heart, the local area in the medical image can be determined by the typical illustration, without the need to manually select the area of ​​interest in the image, thereby improving the subsequent assistance effect for doctors.

[0032] Image features are extracted based on the determined local image.

[0033] In some embodiments, the extracted text features and image features are respectively input into a pre-trained LSTM to output contextual disease text features and local lesion features using the LSTM, including: ; in, , The corresponding features are Step length, and The hidden state of step length, For the The characteristics of step length, , The corresponding features are Step length, and In the specific example, the extracted text features and image features are input into LSTM respectively, so that two LSTMs are used to output the contextual disease text features and the local lesion features respectively.

[0034] In some embodiments, the output contextual disease text features and local lesion features are weightedly fused using an attention fusion mechanism, including: ; in, is the weighted fusion feature, , are trainable weights, , are weighted parameters based on the context-based disease text features and local lesion features, , , , are the first and second weight matrices of the local lesion features, is the weight matrix of the disease text features, is the cross-modal attention, is the local lesion feature output by LSTM, The condition text feature of the context output by LSTM, is the current fusion feature in the prediction sequence, To predict the first Fusion features, is the bilinear weight matrix.

[0035] In some embodiments, the training process also includes calculating the cross entropy loss between the predicted lesion condition and the actual lesion distribution based on the patient's historical medical data to perform model training.

[0036] In some embodiments, decoding the text-associated lesion features and the local lesion features using a decoder to output a predicted lesion condition based on the decoder includes: The corresponding area of ​​the medical image data at the current diagnosis time is superimposed with the partial area of ​​the output predicted lesion condition, so as to fuse the output predicted lesion condition with the corresponding area of ​​the medical image data at the current diagnosis time. That is, the corresponding image areas can be fused, and the local details before and after the prediction can be performed.

[0037] Calculate the clarity of the fused local area, highlight the image area whose clarity is lower than the preset clarity threshold, and highlight the blurred area so that doctors can immediately identify the possible development trend of the disease, thereby improving the effect of auxiliary diagnosis and treatment.

[0038] According to the determined dynamic segmentation, the highlighted area is labeled. According to the aforementioned dynamic segmentation, the area is marked to further supplement the auxiliary effect of the finally output image containing the lesion condition.

[0039] An embodiment of the present application also proposes an intelligent assistance system based on multimodal deep learning, including a processor and a memory, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the steps of the intelligent assistance method based on multimodal deep learning as described above are implemented.

[0040] In addition, although exemplary embodiments have been described herein, the scope includes any and all embodiments based on the present disclosure with equivalent elements, modifications, omissions, combinations (e.g., various embodiments intersecting schemes), adaptations or changes. It is not limited to the examples described in this specification or during the implementation of this application, and its examples will be interpreted as non-exclusive.

[0041] The above description is intended to be illustrative rather than restrictive. For example, the above examples (or one or more of them) can be used in combination with each other. For example, those of ordinary skill in the art may use other embodiments when reading the above description.

[0042] The above embodiments are merely exemplary embodiments of the present disclosure. Those skilled in the art may make various modifications or equivalent substitutions to the present invention within the essence and protection scope of the present disclosure, and such modifications or equivalent substitutions shall also be deemed to fall within the protection scope of the present invention.

Claims

1. An intelligent assistance method based on multimodal deep learning, characterized in that: include: Obtain the patient's condition description text data and the medical imaging data at the corresponding diagnosis time; Extracting text features based on the condition description text data, and extracting image features of the medical image data; The extracted text features and image features are respectively input into the pre-trained LSTM to use the LSTM to output the contextual disease text features and local lesion features; The output contextual disease text features and local lesion features are weightedly fused using the attention fusion mechanism; The weighted fused features are encoded using an encoder, so as to output text-related lesion features through the encoder; The text-associated lesion features and the local lesion features are decoded by a decoder to output a predicted lesion condition based on the decoder.

2. The intelligent assistance method based on multimodal deep learning according to claim 1, characterized in that: Extracting text features based on the disease description text data includes: Performing word segmentation processing on the condition description text data; Based on the pre-configured multiple disease representative words, the similarity between each segmentation word and the disease representative word is calculated, and the occurrence frequency of each segmentation word whose similarity is greater than a preset threshold is statistically calculated, wherein the disease representative word includes fixed segmentation words and dynamic segmentation words; A first weight is configured for a fixed participle, and a weight is configured for a corresponding participle according to the frequency of occurrence of the dynamic participle, wherein the higher the frequency of occurrence, the greater the weight; Extract text features based on each word and its weight whose similarity is greater than a preset threshold.

3. The intelligent assistance method based on multimodal deep learning as claimed in claim 2, characterized in that: Statistically calculating the occurrence frequency of each word whose similarity is greater than a preset threshold includes: counting each word whose similarity is greater than the preset threshold into the statistical frequency of the same dynamic word, so as to calculate the occurrence frequency of the dynamic word; Configuring a first weight for a fixed word segmentation, and configuring a weight for a corresponding word segmentation according to the frequency of occurrence of a dynamic word segmentation includes: Determine the association relationship between the fixed participle and the dynamic participle, configure a first weight for the fixed participle according to the determined association relationship, and configure a corresponding second weight according to the occurrence frequency of the dynamic participle, so that the sum of the first weight and each second weight is 1.

4. The intelligent assistance method based on multimodal deep learning according to claim 2, characterized in that: Extracting the image features of the medical image data includes: According to the statistical dynamic segmentation, the suitable organs or tissues are determined; Segmenting the medical image data to obtain a plurality of sub-regions; Calculating the similarity between the segmented sub-regions and the typical illustration of the adapted organ or tissue, so as to determine the local image according to a plurality of sub-regions having similarities greater than a preset similarity threshold; Image features are extracted based on the determined local image.

5. The intelligent assistance method based on multimodal deep learning according to claim 4, characterized in that: The extracted text features and image features are respectively input into the pre-trained LSTM to use LSTM to output the contextual disease text features and local lesion features, including: ； in, , The corresponding features are Step length, and The hidden state of step length, For the The characteristics of step length, , The corresponding features are Step length, and The cell state of the step.

6. The intelligent assistance method based on multimodal deep learning according to claim 4, characterized in that: The output contextual disease text features and local lesion features are weightedly fused using the attention fusion mechanism, including: ； in, is the weighted fusion feature, , are trainable weights, , are weighted parameters based on the context-based disease text features and local lesion features, , , , are the first and second weight matrices of the local lesion features, is the weight matrix of the disease text features, is the cross-modal attention, is the local lesion feature output by LSTM, The condition text feature of the context output by LSTM, is the current fusion feature in the prediction sequence, To predict the first Fusion features, is the bilinear weight matrix.

7. The intelligent assistance method based on multimodal deep learning according to claim 6, characterized in that: The training process also includes calculating the cross entropy loss between the predicted lesion condition and the actual lesion distribution based on the patient's historical condition data to perform model training.

8. The intelligent assistance method based on multimodal deep learning according to claim 7, characterized in that: Decoding the lesion features associated with the text and the local lesion features by using a decoder, so as to output the predicted lesion conditions based on the decoder, including: Superimposing a region corresponding to the medical image data at the current diagnosis time with a partial region of the output predicted lesion condition, so as to fuse the output predicted lesion condition with the corresponding region of the medical image data at the current diagnosis time; Calculate the clarity of the fused local area, highlight the image area whose clarity is below a preset clarity threshold, and, According to the determined dynamic segmentation, annotation is performed based on the highlighted area.

9. An intelligent assistance system based on multimodal deep learning, characterized in that: It includes a processor and a memory, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the steps of the intelligent assistance method based on multimodal deep learning as described in any one of claims 1 to 8 are implemented.

Images