Multi-modal medical image and report association retrieval and diagnosis enhancement method

Abstract

The multi-modal medical image and report association retrieval and diagnosis enhancement method comprises the following steps: S1, preprocessing a medical image and a report text to respectively obtain the preprocessed medical image and report text; s2, utilizing a deep learning model and a natural language processing model to perform feature extraction on the preprocessed medical image and report text to obtain visual features of the medical image and semantic features of the report text; and S3, constructing a multi-modal association retrieval model, fusing the visual features of the medical image and the semantic features of the report text, and realizing association retrieval of the medical image and the report text by calculating the similarity between the features. The invention aims to provide a multi-modal medical image and report association retrieval and diagnosis enhancement method so as to solve the problems that in the prior art, medical images and report texts are lack of effective association, retrieval is inaccurate and diagnosis analysis is not deep.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of medical information processing, in particular to a multi-modal medical image and report association retrieval and diagnosis enhancement method. BACKGROUND

[0002] In the process of medical diagnosis, medical images (such as X-ray, CT, MRI, etc.) and corresponding report texts are important basis for doctors to diagnose diseases, but there are many problems at present: medical image data and report text data are independent of each other, lacking effective association logic, doctors need to consult separately when diagnosing, consuming a lot of time and effort, for example, when dealing with a large number of patient cases, it is difficult to quickly and accurately find the relevant historical report text of the current image, affecting the diagnosis efficiency; the existing retrieval method cannot make full use of the complementary information of multi-modal data, only based on image features or text keywords, cannot comprehensively consider the image visual information and text semantic information, resulting in inaccurate retrieval results, which is difficult to meet the clinical needs; the fusion analysis of multi-modal data in the diagnosis process is not deep enough, and the semantic information in the report text cannot be effectively used to enhance the understanding and diagnosis of medical images, for example, in the diagnosis of complex diseases, relying solely on image analysis may miss key information, while the description in the report text may provide important clues, but the existing technology cannot combine the two organically, affecting the diagnosis accuracy. SUMMARY

[0003] The purpose of the present application is to provide a multi-modal medical image and report association retrieval and diagnosis enhancement method to solve the problems of lack of effective association between medical images and report texts, inaccurate retrieval and not deep diagnosis analysis in the prior art.

[0004] In order to solve the above problems, the technical scheme of the present application is as follows: The multi-modal medical image and report association retrieval and diagnosis enhancement method comprises the following steps: S1: pre-processing the medical image and the report text to obtain the pre-processed medical image and the report text respectively; S2: using a deep learning model and a natural language processing model to extract features from the pre-processed medical image and the report text respectively to obtain visual features of the medical image and semantic features of the report text; S3: constructing a multi-modal association retrieval model, fusing the visual features of the medical image and the semantic features of the report text, and realizing the association retrieval of the medical image and the report text by calculating the similarity between the features; S4: fusing the medical image and the report text retrieved by association, using the semantic information in the report text to enhance the understanding and diagnosis of the medical image, and using a machine learning model to classify and predict the fused multi-modal data to provide diagnosis suggestions for doctors.

[0005] Further, the preprocessing of the medical image includes image denoising, enhancement, and normalization operations.

[0006] Further, the preprocessing of the report text includes text cleaning, word segmentation, and part-of-speech tagging operations.

[0007] Further, the visual features of the medical image are extracted using a deep learning model, which is a convolutional neural network (CNN).

[0008] Further, the semantic features of the report text are extracted using a natural language processing model, which is BERT.

[0009] Further, the similarity between the features is calculated using a cosine similarity algorithm.

[0010] Further, the association retrieval module also introduces semantic understanding and reasoning logic to optimize the retrieval results.

[0011] Further, the semantic understanding and reasoning logic uses knowledge graph technology.

[0012] Further, the fused multi-modal data is classified and predicted using a machine learning model, which is a support vector machine (SVM).

[0013] Further, the fusion analysis includes superimposing the lesion location, size, and morphological description information related to the medical image in the report text on the corresponding area of the medical image in a visual annotation manner.

[0014] The beneficial effects of the present application are: 1. Through the association retrieval of medical images and report texts, doctors can quickly obtain historical report texts related to the current image, reducing the time for reviewing materials and improving the diagnosis efficiency. For example, in actual clinical applications, doctors can find report texts of similar cases in a short time, refer to the diagnosis experience therein, and speed up the diagnosis process of the current patient.

[0015] 2. The multi-modal association retrieval model considers the visual features of medical images and the semantic features of report texts, further optimizes the retrieval results through semantic understanding and reasoning logic, makes the retrieval results more accurate and relevant, and better meets the clinical needs. For example, during the retrieval process, not only can the appearance similarity of the image be matched, but also the semantic information in the text can be combined to find the truly relevant materials of the current case.

[0016] 3. Integrating semantic information from the report text with medical images can assist doctors in identifying potential lesions and provide diagnostic suggestions, thereby improving diagnostic accuracy. For example, in the diagnosis of some complex diseases, the descriptions in the report text can guide doctors to focus on areas in the images that are easily overlooked, avoiding misdiagnosis and missed diagnosis. Attached Figure Description

[0017] The invention will be further described below with reference to the accompanying drawings: Figure 1 This is a flowchart of the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] A method for multimodal medical image and report association retrieval and diagnostic enhancement includes the following steps: S1: Preprocess the medical images and report text to obtain the preprocessed medical images and report text respectively; S2: Use deep learning models and natural language processing models to extract features from the preprocessed medical images and report texts to obtain the visual features of the medical images and the semantic features of the report texts. S3: Construct a multimodal association retrieval model that integrates the visual features of medical images and the semantic features of report texts. By calculating the similarity between features, the association retrieval of medical images and report texts can be achieved. S4: The retrieved medical images and report texts are fused and analyzed. The semantic information in the report text is used to enhance the understanding and diagnosis of medical images. A machine learning model is used to classify and predict the fused multimodal data to provide diagnostic suggestions for doctors.

[0020] Therefore, the above steps, through preprocessing, feature extraction, correlation retrieval, and fusion analysis of medical images and report texts, effectively correlate multimodal data, solving the problems of lack of effective correlation between medical images and report texts, inaccurate retrieval, and insufficient diagnostic analysis. It enables rapid acquisition of relevant historical reports, optimizes retrieval results by integrating multimodal information, and provides diagnostic suggestions to doctors, thereby improving overall diagnostic efficiency and accuracy.

[0021] Further, the preprocessing of the medical images includes image denoising, enhancement, and normalization. This can remove image noise interference, improve image contrast and consistency, make image features clearer and standardized, provide high-quality input for subsequent feature extraction, improve the accuracy of visual feature extraction, and thus ensure the reliability of correlation retrieval and diagnostic analysis.

[0022] Image denoising: For CT images, a Gaussian filter algorithm is used. Assume that the CT image is a two-dimensional matrix I(x, y), where x and y represent the coordinates of the pixel points. The formula of Gaussian filtering is: ; where is the mean, is the standard deviation. By convolving the Gaussian filter with the CT image, that is, ; the noise in the image can be removed, and the denoised image is obtained.

[0023] Image enhancement: The histogram equalization method is used to enhance the X-ray image. First, calculate the gray level histogram H(i) of the X-ray image, where i represents the gray level. Then calculate the cumulative distribution function: .

[0024] By mapping the original image gray value I(x, y) to the new gray value , where N is the total number of pixel points in the image, and L is the total number of gray levels, the enhanced image is obtained, which improves the contrast of the image.

[0025] Image normalization: For MRI images, the pixel values are normalized to the interval [0, 1]. Assume that the pixel value range of the MRI image is [a, b], and the normalization formula is: , the normalized image is obtained, which facilitates the processing of the subsequent feature extraction model Further, the preprocessing of the report text includes text cleaning, word segmentation, and part-of-speech tagging. This can remove irrelevant characters, split the text into basic units for analysis and label the part-of-speech, standardize the text format, lay the foundation for semantic feature extraction, improve the accuracy and effectiveness of text semantic features, and help precise correlation retrieval.

[0026] Text cleaning: Use regular expressions to remove special characters and punctuation from the report text. For example, for text, by using the regular expression re.sub('[^a-zA-Z0-9\s]','', text), remove non-alphabetic, numeric, and space characters from the text, and get the cleaned text text_{clean}.

[0027] Tokenization: Use the tokenization function of the NLTK toolkit to tokenize the cleaned text. For example, for text tex tclea , by , split the text into individual words words.

[0028] Part-of-speech tagging: Use the part-of-speech tagging function of the NLTK toolkit to tag the part-of-speech of the tokenized words. For example, for the word list words, by , get the part-of-speech tagging result tagged words for each word, which provides the basis for subsequent semantic analysis.

[0029] Further, the visual features of the medical image are extracted using a deep learning model, which is a convolutional neural network (CNN).

[0030] A ResNet-based convolutional neural network (CNN) model is constructed to extract visual features of medical images. Assume that the input medical image is I, with dimensions , where H is the height, W is the width, and C is the number of channels. The ResNet model includes multiple convolutional layers, pooling layers, and fully connected layers.

[0031] Convolutional layer: Convolve the image with a convolution kernel to extract local features of the image. For example, a convolutional layer has a kernel size of , where is the input channel number, is the output channel number. The convolution operation formula is: ; where O(x,y) is the convolution output result, and K is the convolution kernel.

[0032] Pooling layer: Use the max-pooling operation to downsample the feature map output by the convolutional layer, reduce the size of the feature map, and reduce the computational complexity. For example, the max-pooling kernel size is , and the step is 2, the max-pooling operation is performed on the feature map F, and the downsampled feature map F pooled is obtained.

[0033] Fully connected layer: the feature map output by the pooling layer is flattened and input into the fully connected layer to obtain the visual feature vector V of the medical image.

[0034] The visual features of the medical image are extracted using a convolutional neural network (CNN). CNN can effectively capture local features and spatial information of the image, accurately extract key visual features such as lesion morphology and texture in the medical image, and provide rich image feature support for multi-modal correlation retrieval, enhancing the relevance of the retrieval.

[0035] Further, the semantic features of the report text are extracted using a natural language processing model, which is BERT.

[0036] The pre-trained BERT model is used to extract the semantic features of the report text. Assuming that the input report text is T, after tokenization, special mark (such as [CLS], [SEP]) addition and other processing, it is input into the BERT model. The BERT model contains multiple Transformer blocks, each of which contains a multi-head attention mechanism and a feedforward neural network.

[0037] Multi-head attention mechanism: parallel attention calculation is performed on different representation subspaces of the input text to obtain the semantic relationship between different positions in the text. For example, for the embedded representation E of the input text, the attention output A is calculated through the multi-head attention mechanism.

[0038] Feedforward neural network: further feature transformation and nonlinear activation are performed on the attention output to obtain the semantic feature representation of the text. After processing by multiple Transformer blocks, the output corresponding to the [CLS] mark is taken as the semantic feature vector S of the report text.

[0039] The semantic features of the report text are extracted using the BERT model. BERT can deeply understand the context semantics of the text and accurately capture semantic information such as diagnostic descriptions and key symptoms in the report, providing accurate text features for multi-modal correlation retrieval and improving the accuracy of cross-modal correlation.

[0040] Further, the similarity between the features is calculated using the cosine similarity algorithm.

[0041] The visual feature vector V of the medical image and the semantic feature vector S of the report text are fused, and the cosine similarity algorithm is used to calculate the similarity between them. The cosine similarity formula is: ; where is the dot product of vectors V and S, and respectively. By calculating the similarity, the medical images and report texts with higher similarity are associated and matched to obtain the associated retrieval results.

[0042] The cosine similarity algorithm is used to calculate the feature similarity, which can quantify the correlation degree between the visual features of medical images and the semantic features of report texts, effectively measure the matching degree of the two, provide scientific similarity judgment basis for associated retrieval, and improve the accuracy of retrieval results.

[0043] Further, the associated retrieval module also introduces semantic understanding and reasoning logic to optimize the retrieval results.

[0044] The knowledge graph technology is used to optimize the retrieval results. A medical knowledge graph is constructed, in which the nodes represent medical entities (such as diseases, symptoms, examinations, etc.), and the edges represent the relationships between entities (such as causes, symptom manifestations, diagnosis methods, etc.). For the retrieved medical images and report texts, the medical entities involved are matched with the nodes in the knowledge graph, and through the reasoning mechanism of the knowledge graph, the relationships between entities are analyzed to further filter and sort the retrieval results, improving the accuracy and relevance of the retrieval results.

[0045] The associated retrieval module introduces semantic understanding and reasoning logic to optimize the retrieval results, which can deeply mine the potential semantic association of multi-modal data, exclude irrelevant or low-relevance retrieval results, and make the retrieval results more meet the clinical needs, improving the accuracy and effectiveness of retrieval.

[0046] Further, the semantic understanding and reasoning logic uses knowledge graph technology.

[0047] The associated retrieved medical images and report texts are fused and analyzed. For example, when a doctor views a medical image, the description information related to the image in the report text (such as lesion location, size, shape, etc.) is marked on the image in a visual manner to assist the doctor in more intuitively understanding the image content and discovering potential lesions.

[0048] The knowledge graph technology is used to realize semantic understanding and reasoning logic, which can integrate medical knowledge, establish the association between diseases, symptoms, and other entities, and further filter and sort the retrieval results through reasoning, enhancing the clinical relevance and reliability of the retrieval results, and assisting doctors in more accurately understanding the association between images and reports.

[0049] Further, the machine learning model used for classification and prediction of the fused multi-modal data is a support vector machine (SVM).

[0050] The fused multi-modal data is classified and predicted by using a support vector machine (SVM) model. The visual feature vector V of the medical image and the semantic feature vector S of the report text are spliced into a new feature vector F = [V, S] as the input of the SVM model. First, the SVM model is trained using the labeled multi-modal data samples (including the feature vector F and the corresponding diagnosis label y), and the parameters of the SVM model are determined by optimizing the objective function (such as minimizing the structural risk). In the prediction stage, the feature vector of the multi-modal data to be diagnosed is input into the trained SVM model to obtain the prediction result of the type and severity of the disease, and to provide a diagnosis suggestion for the doctor.

[0051] Further, the fusion analysis includes superimposing the lesion location, size, and morphological description information related to the medical image in the report text in a visual annotation manner on the corresponding region of the medical image. The independent state of the medical image data and the report text data is broken, and the image and the text are preprocessed, feature extraction is performed, and a multi-modal correlation retrieval model is constructed to fuse the features to calculate the similarity, thereby realizing effective correlation retrieval of the two. Doctors can quickly obtain the historical report text related to the current image, for example, when processing a large number of patient cases, the time for reviewing the materials can be saved, and the diagnosis efficiency is improved. On the other hand, the multi-modal correlation retrieval model integrates the visual features of the medical image and the semantic features of the report text, and changes the one-sidedness of the previous retrieval based only on the image features or the text keywords. At the same time, the semantic understanding and reasoning logic (such as using knowledge graph technology) are introduced to optimize the retrieval result, so that the retrieval result is more accurate and more suitable for clinical needs, and a large amount of irrelevant information is avoided.

[0052] The content described in the embodiments of the present specification is only a list of implementation forms of the inventive concept, and the protection scope of the present application should not be regarded as being limited to the specific forms stated in the embodiments, and the protection scope of the present application also extends to equivalent technical means that can be thought of by those skilled in the art according to the inventive concept.

Claims

1. A method for multimodal medical image and report association retrieval and diagnostic enhancement, characterized in that, Includes the following steps: S1: Preprocess the medical images and report text to obtain the preprocessed medical images and report text respectively; S2: Use deep learning models and natural language processing models to extract features from the preprocessed medical images and report texts to obtain the visual features of the medical images and the semantic features of the report texts. S3: Construct a multimodal association retrieval model that integrates the visual features of medical images and the semantic features of report texts. By calculating the similarity between features, the association retrieval of medical images and report texts can be achieved. S4: The retrieved medical images and report texts are fused and analyzed. The semantic information in the report text is used to enhance the understanding and diagnosis of medical images. A machine learning model is used to classify and predict the fused multimodal data to provide diagnostic suggestions for doctors.

2. The multimodal medical image and report association retrieval and diagnostic enhancement method according to claim 1, characterized in that, The preprocessing of medical images includes image denoising, enhancement, and normalization operations.

3. The multimodal medical image and report association retrieval and diagnostic enhancement method according to claim 1, characterized in that, The preprocessing of the report text includes text cleaning, word segmentation, and part-of-speech tagging.

4. The multimodal medical image and report association retrieval and diagnostic enhancement method according to claim 1, characterized in that, The method utilizes a deep learning model to extract visual features from medical images, wherein the deep learning model is a convolutional neural network (CNN).

5. The multimodal medical image and report association retrieval and diagnostic enhancement method according to claim 1, characterized in that, The semantic features of the report text are extracted using a natural language processing model, wherein the natural language processing model is BERT.

6. The multimodal medical image and report association retrieval and diagnostic enhancement method according to claim 1, characterized in that, The similarity between the calculated features is achieved using the cosine similarity algorithm.

7. The multimodal medical image and report association retrieval and diagnostic enhancement method according to claim 1, characterized in that, The associated retrieval module also incorporates semantic understanding and reasoning logic to optimize the retrieval results.

8. The multimodal medical image and report association retrieval and diagnostic enhancement method according to claim 7, characterized in that, The semantic understanding and reasoning logic employs knowledge graph technology.

9. The multimodal medical image and report association retrieval and diagnostic enhancement method according to claim 1, characterized in that, The fused multimodal data is classified and predicted using a machine learning model, which is a support vector machine (SVM).

10. The multimodal medical image and report association retrieval and diagnostic enhancement method according to claim 1, characterized in that, The fusion analysis includes overlaying the descriptions of the location, size, and shape of lesions related to medical images in the report text onto the corresponding areas of the medical images in a visual annotation manner.

Images