Medical image report generation method and device based on multi-modal data

Abstract

The invention relates to a medical image report generation method and device based on multi-modal data. The method comprises the following steps: acquiring original multi-modal data and preprocessing the original multi-modal data, wherein the multi-modal data comprises image data, complaint text data and clinical examination text data; predicting and generating a corresponding medical image report by using the trained multi-modal deep learning model; the prediction process specifically comprises the following steps: based on preprocessed image data, extracting image features by using a trained convolutional neural network model; based on the preprocessed chief complaint text data and clinical examination text data, using the trained artificial neural network model to extract semantic features; fusing the image features and the corresponding semantic features to obtain fused features; and inputting the fusion features into a full connection layer, and mapping to obtain a medical image report. Compared with the prior art, the method has the advantages of improving the reliability of medical image report generation, reducing the manual load and the like.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence deep learning technology, and in particular to a method and device for generating a medical imaging report based on multimodal data. Background Art

[0002] Orthodontic diagnosis refers to malocclusion, which is a clinical manifestation of abnormal development of teeth, jaws, and craniofacial area under the interaction of internal and external environment and genetics in children's growth and development period. The purpose of oral orthodontic treatment is to obtain the most perfect aesthetic, stable, coordinated, and balanced jaw-jaw relationship for the patient's individual dental and facial features within the normal oral function and physiological range. However, the causes of malocclusion are complicated, and the dental and jaw relationship varies greatly. It is difficult for doctors to immediately propose a treatment plan that is more in line with the patient's actual situation after a preliminary examination of the patient. It requires close cooperation between patients and doctors, and after careful analysis, the main problems of the patient's dental and jaw deformity are listed, the causes of malocclusion are clarified, the mechanism of malocclusion is analyzed, and then the next treatment plan is proposed. Orthodontic diagnosis often forms an individualized diagnostic system based on various clinical information of the patient.

[0003] At present, medical imaging data plays an important role in assisting orthodontic diagnosis and treatment plan formulation. Clinically, the interpretation of medical images is usually done by doctors who review, analyze and describe medical images and give imaging reports, which then serve as a reference for the final orthodontic diagnosis and treatment plan formulation. However, with the development of imaging technology, the amount of imaging data of different patients has increased sharply, and the format and professional terms of the reports are mostly unified and repeated text descriptions. Therefore, relying entirely on manpower to write reports will waste a lot of manpower and time costs. Therefore, the automatic generation system of medical imaging reports has emerged. For example, Chinese patent CN 115205880A discloses a method and device for generating medical imaging reports, wherein the method obtains a target medical image, processes the acquired image based on a semantic association model of the target medical image, obtains corresponding multimodal topic distribution information, and then generates multi-level text description information to obtain a corresponding medical imaging report. Although this method can interpret the potential semantics of medical images. However, this method only considers image data, performs single topic modeling, and only relies on topic distribution to generate text descriptions, which limits the reliability of the generated report.

[0004] Since orthodontic diagnosis involves many types of examination and analysis, including medical history, clinical examination, functional analysis, model analysis, X-ray examination, X-ray cephalometric analysis, etc., considering more content when generating medical imaging reports will effectively improve the reliability of the reports. At present, the medical imaging report generation technology based on multimodal deep learning algorithms has gradually become a trend. Orthodontic multimodal data includes patients' imaging data, such as lateral head positioning films, curved surface tomography films, frontal phases, lateral phases, and intraoral photos of patients. In addition, it also includes patients' cone beam computed tomography images (CBCT images) and temporomandibular joint magnetic resonance imaging (temporomandibular joint MRI images). The patient's clinical examination and chief complaint are also within the scope of orthodontic diagnosis. Therefore, the accurate generation of medical imaging reports depends on the comprehensive analysis of patients' clinical multimodal data, which in turn assists in forming an effective diagnosis of patients' oral orthodontics. Based on this, how to further improve the reliability of medical imaging report generation and reduce manual load has become a problem that needs to be solved in this field. Summary of the invention

[0005] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide a method and device for generating medical imaging reports based on multimodal data, which can integrate multimodal data of oral orthodontics, improve the reliability of medical imaging report generation and reduce manual workload.

[0006] The purpose of the present invention can be achieved by the following technical solutions:

[0007] According to a first aspect of the present invention, a method for generating a medical imaging report based on multimodal data is provided, comprising the following steps: obtaining original multimodal data and performing preprocessing, the multimodal data comprising imaging data, chief complaint text data and clinical examination text data; based on the preprocessed multimodal data, using a trained multimodal deep learning model to perform prediction and generate a corresponding medical imaging report; wherein the multimodal deep learning model comprises a trained convolutional neural network model, an artificial neural network model and a fully connected layer, and the prediction process specifically comprises: based on the preprocessed imaging data, using the trained convolutional neural network model to extract image features; based on the preprocessed chief complaint text data and clinical examination text data, using the trained artificial neural network model to extract semantic features; fusing the image features with corresponding semantic features to obtain fused features; and inputting the fused features into the fully connected layer to map and obtain the medical imaging report.

[0008] As a preferred technical solution, the preprocessing process specifically includes image reconstruction, image segmentation and normalization processing.

[0009] As an optimal technical solution, the process of training the convolutional neural network model and the artificial neural network model specifically includes: dividing the preprocessed multimodal data into a training set, a validation set and a test set; based on the image data in the training set, training the pre-trained convolutional neural network model using a gradient descent algorithm; based on the chief complaint text data and clinical examination text data in the training set, training the pre-trained artificial neural network model using a gradient descent algorithm; using the validation set to respectively determine the optimal parameters of the convolutional neural network model and the artificial neural network model; and using the test set to evaluate the optimized convolutional neural network model and the artificial neural network model.

[0010] As a preferred technical solution, the training process of the multimodal deep learning model specifically includes: dividing the preprocessed multimodal data into a training set and a validation set; based on preset initial model parameters, iteratively training the multimodal deep learning model using the training set and a gradient descent algorithm; and determining the optimal parameters of the multimodal deep learning model using the validation set.

[0011] As a preferred technical solution, a method for fusing the image features with corresponding semantic features includes vector splicing.

[0012] As a preferred technical solution, the image data include cone beam computed tomography images and temporomandibular joint magnetic resonance images as well as lateral head positioning films, curved surface tomography films, frontal photos, lateral photos and photos of the inside of the oral cavity.

[0013] According to a second aspect of the present invention, there is provided a device for generating a medical image report based on multimodal data, comprising a data acquisition and preprocessing module and a medical image report generating module, wherein the data acquisition and preprocessing module is used to acquire and preprocess the original multimodal data, wherein the multimodal data comprises image data, chief complaint text data and clinical examination text data; the medical image report generating module is used to predict based on the preprocessed multimodal data using a trained multimodal deep learning model to generate a corresponding medical image report; wherein the multimodal deep learning model comprises a trained convolutional neural network model, an artificial neural network model and a fully connected layer, and the prediction process specifically comprises: based on the preprocessed image data, extracting image features using the trained convolutional neural network model; based on the preprocessed chief complaint text data and clinical examination text data, extracting semantic features using the trained artificial neural network model; fusing the image features with the corresponding semantic features to obtain fused features; and inputting the fused features into the fully connected layer to map and obtain the medical image report.

[0014] As a preferred technical solution, the data acquisition and preprocessing module includes an image reconstruction unit, an image segmentation unit and a normalization processing unit, which are respectively used for image reconstruction, image segmentation and normalization processing.

[0015] As a preferred technical solution, the medical imaging report generation module includes a single-modal model training unit, which is used to train the convolutional neural network model and the artificial neural network model. The specific training process includes: dividing the preprocessed multimodal data into a training set, a validation set and a test set; based on the image data in the training set, training the pre-trained convolutional neural network model using a gradient descent algorithm; based on the chief complaint text data and clinical examination text data in the training set, training the pre-trained artificial neural network model using a gradient descent algorithm; using the validation set to respectively determine the optimal parameters of the convolutional neural network model and the artificial neural network model; and using the test set to evaluate the optimized convolutional neural network model and the artificial neural network model.

[0016] As a preferred technical solution, the medical imaging report generation module also includes a multimodal model training unit, which is used to train the multimodal deep learning model. The specific training process includes: dividing the preprocessed multimodal data into a training set and a validation set; based on preset initial model parameters, iteratively training the multimodal deep learning model using the training set and a gradient descent algorithm; and determining the optimal parameters of the multimodal deep learning model using the validation set.

[0017] Compared with the prior art, the present invention has the following beneficial effects:

[0018] 1. The medical imaging report generation method provided by the present invention utilizes oral orthodontic multimodal data including imaging data, chief complaint text data and clinical examination text data to construct and train a multimodal deep learning model for automatically generating a medical impact report. In the prediction process, features are extracted using pre-trained single-modal models, and then fused to obtain fused features, which are then mapped into medical imaging report texts through a fully connected layer. This method can make full use of a variety of imaging data and text data. Compared with a single imaging data source or clinical examination text data, the input of chief complaint text data is also added, and the data volume is richer. The predicted report is closer to the actual situation of the patient and more reliable. At the same time, no manual writing is required, which reduces the manual load.

[0019] 2. The medical imaging report generation method provided by the present invention, when training two single-modal models, the convolutional neural network model and the artificial neural network model, extracts features according to the characteristics of different data types respectively, and then fuses them, so as to facilitate more flexible adjustment, improve the generalization ability of the model, and adapt to various types of data features;

[0020] 3. The medical imaging report generating device provided by the present invention adopts the medical imaging report generating method proposed by the present invention, has a simple structure and high computational efficiency, can improve the speed of automatic generation of medical imaging reports, can be integrated into the clinical medical process for use, improve the automation level of orthodontic diagnosis, and liberate manpower. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 A schematic flow chart of the method provided by the present invention;

[0022] Figure 2 A schematic diagram of the process flow of multimodal deep learning model prediction in the method provided by the present invention. DETAILED DESCRIPTION

[0023] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0024] Example

[0025] like Figure 1 As shown, this embodiment provides a method for generating a medical imaging report based on multimodal data. The first step of the method is to obtain the original multimodal data and perform preprocessing. The multimodal data includes imaging data, chief complaint text data, and clinical examination text data. The second step is to use a trained multimodal deep learning model to predict based on the preprocessed multimodal data and generate a corresponding medical imaging report. Among them, the multimodal deep learning model includes a trained convolutional neural network model (CNN), an artificial neural network model (ANN), and a fully connected layer. The prediction process specifically includes: based on the preprocessed imaging data, using the trained convolutional neural network model to extract image features; based on the preprocessed chief complaint text data and clinical examination text data, using the trained artificial neural network model to extract semantic features; fusing image features with corresponding semantic features to obtain fused features; inputting the fused features into the fully connected layer to map and obtain a medical imaging report. Specifically:

[0026] (1) Data acquisition and preprocessing, database establishment

[0027] The collected multimodal orthodontic data include: patient imaging data, such as lateral head positioning radiographs, curved plane tomography, frontal and lateral views, intraoral photographs of patients, CBCT and MRI images of the temporomandibular joint. In addition, the patient's clinical examination text data and chief complaint text data are also considered. In the process of data collection, relevant ethical standards and laws and regulations will be strictly observed to protect patient privacy and data security; informed consent of patients will be obtained, and ethical review and approval will be conducted when necessary.

[0028] Among them, the information from different sources of each sample is associated together through the patient's unique index (sample number). The marked patients are first divided into training set, validation set and test set according to the ratio of 70%, 15%, and 15%. After completing the classification of the patients, the clinical information of the samples is used to complete the data set classification according to the number index. The final data set is a patient's data including information from multiple sources. This information corresponds to the sample through a unique index to form an available orthodontic multimodal database. The annotation processing of image data and the standardization of text data formats, professional terms, etc. can be performed by orthodontic experts first. The processed data is stored in the data set and can be called at any time for model training and prediction.

[0029] Data preprocessing is performed on orthodontic multimodal data. For multi-layer three-dimensional data, key images are extracted and the temporomandibular joint area of ​​interest is segmented so that the model can better focus on the lesion area.

[0030] The image data is preprocessed and annotated, such as image reconstruction, image segmentation, marking key structures, etc. In addition, the acquired image data is subjected to noise reduction and data normalization.

[0031] Both clinical examination text data and chief complaint text data are converted into standardized text, and finally into standardized vectors that can be recognized by neural networks.

[0032] Adjust the size of the orthodontic image, standardize the grayscale of the image data, etc. to ensure the consistency of the input model.

[0033] (2) Orthodontic single modality data feature extraction

[0034] Using the transfer learning strategy, the CNN model and ANN model are trained separately using the training set data, and the performance evaluation and parameter tuning are performed using the validation set data. Both the CNN model and the ANN model are pre-trained on public datasets, such as the ImageNet dataset.

[0035] During training, first load the pre-trained model, then fine-tune the model and train the adjusted model using the training set data. Monitor the performance of the model during training and make adjustments and optimizations as needed to improve the accuracy of the model. After training is completed, the valid feature information is stored in each unimodal model.

[0036] Specifically:

[0037] When training the CNN model, the image data in the training set and the gradient descent algorithm are used for training, and the data in the validation set are used for parameter optimization. The CNN model with the optimal parameters is used to extract image features during prediction.

[0038] Train the ANN model using the text data of the chief complaint and clinical examination in the training set and the gradient descent algorithm, and optimize the parameters using the data in the validation set. The ANN model with the optimal parameters is used to extract semantic features during prediction, including the extraction of text information features such as physique, secondary sexual characteristics, and metabolite levels.

[0039] (3) Multimodal data fusion training

[0040] The best performing models of the above single-modal models on the validation set are taken out respectively, and the pre-trained models are loaded into the multimodal deep learning model by back-end fusion. Finally, the model is trained using the gradient descent algorithm, the hyperparameters of the multimodal deep learning model are adjusted, the parameters of the entire multimodal deep learning model are fine-tuned, and the model is trained multiple times until the multimodal deep learning model converges well.

[0041] The multimodal deep learning model framework constructed in this embodiment includes a CNN model, an ANN model and a fully connected layer. In some other embodiments, multiple CNN models or ANN models may be included.

[0042] In the prediction process, after using the CNN model to extract image features and the ANN model to extract semantic features, the two features are concatenated together through vector concatenation to form a new feature vector. The fused feature is a feature vector that integrates multimodal information.

[0043] The fully connected layer consists of two linear layers, which are connected after the feature layer of the CNN model and the feature layer of the ANN model to map the obtained fusion features into orthodontic medical imaging reports.

[0044] (4) Model evaluation and validation

[0045] Use the test set data to evaluate and verify the trained model to assess its generalization ability and performance on unseen data, analyze the model's prediction results, and use appropriate evaluation indicators to evaluate the model's accuracy, recall, precision, F1 score, etc.

[0046] (5) Model deployment and application

[0047] The trained multimodal deep learning model is deployed in a clinical setting to assist doctors in orthodontic diagnosis.

[0048] Monitor the effectiveness of multimodal deep learning models in actual clinical applications, and continuously optimize and improve them based on feedback.

[0049] Furthermore, the present embodiment also provides a medical image report generation device based on multimodal data, including a data acquisition and preprocessing module and a medical image report generation module. The data acquisition and preprocessing module is used to acquire and preprocess the original multimodal data, and the multimodal data includes image data, chief complaint text data and clinical examination text data; the medical image report generation module is used to predict based on the preprocessed multimodal data using a trained multimodal deep learning model to generate a corresponding medical image report; wherein the multimodal deep learning model includes a trained convolutional neural network model, an artificial neural network model and a fully connected layer, and the prediction process specifically includes: based on the preprocessed image data, extracting image features using the trained convolutional neural network model; based on the preprocessed chief complaint text data and clinical examination text data, extracting semantic features using the trained artificial neural network model; fusing image features with corresponding semantic features to obtain fused features; inputting the fused features into the fully connected layer to map and obtain a medical image report. The data acquisition and preprocessing module also includes an image reconstruction unit, an image segmentation unit and a normalization processing unit, which are respectively used for image reconstruction, image segmentation and normalization processing. The medical imaging report generation module includes a single-modality model training unit and a multi-modality model training unit, which are used to train CNN models, ANN models, and multi-modality deep learning models, respectively. The execution process of each module and unit of the device is the same as the execution process of each step of the aforementioned method, which will not be repeated here.

[0050] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A method for generating a medical imaging report based on multimodal data, characterized in that: The following steps are involved: Acquiring and preprocessing original multimodal data, wherein the multimodal data includes image data, chief complaint text data, and clinical examination text data; Based on the preprocessed multimodal data, the trained multimodal deep learning model is used to make predictions and generate corresponding medical imaging reports; The multimodal deep learning model includes a trained convolutional neural network model, an artificial neural network model and a fully connected layer, and the prediction process specifically includes: Based on the preprocessed image data, the trained convolutional neural network model is used to extract image features; Based on the preprocessed chief complaint text data and clinical examination text data, the trained artificial neural network model is used to extract semantic features; Fusing the image features with corresponding semantic features to obtain fused features; The fused features are input into the fully connected layer and mapped to obtain the medical imaging report.

2. The method for generating a medical imaging report based on multimodal data according to claim 1, characterized in that: The preprocessing process specifically includes image reconstruction, image segmentation and normalization processing.

3. The method for generating a medical imaging report based on multimodal data according to claim 1, characterized in that: The process of training the convolutional neural network model and the artificial neural network model specifically includes: Divide the preprocessed multimodal data into training set, validation set and test set; Based on the image data in the training set, training the pre-trained convolutional neural network model using a gradient descent algorithm; Based on the chief complaint text data and clinical examination text data in the training set, training the pre-trained artificial neural network model using a gradient descent algorithm; Determining the optimal parameters of the convolutional neural network model and the artificial neural network model respectively using the validation set; The test set is used to evaluate the optimized convolutional neural network model and artificial neural network model.

4. The method for generating a medical imaging report based on multimodal data according to claim 1, characterized in that: The training process of the multimodal deep learning model specifically includes: Divide the preprocessed multimodal data into training set and validation set; Based on the preset initial model parameters, the multimodal deep learning model is iteratively trained using the training set and the gradient descent algorithm; The validation set is used to determine optimal parameters of the multimodal deep learning model.

5. The method for generating a medical imaging report based on multimodal data according to claim 1, characterized in that: The method of fusing the image features with corresponding semantic features includes vector concatenation.

6. The method for generating a medical imaging report based on multimodal data according to claim 1, characterized in that: The imaging data include cone beam computed tomography images, magnetic resonance imaging of the temporomandibular joint, and lateral head positioning films, curved plane films, frontal photos, lateral photos, and photos of the inside of the oral cavity.

7. A medical imaging report generation device based on multimodal data, characterized in that: Including data acquisition and preprocessing module and medical imaging report generation module, The data acquisition and preprocessing module is used to acquire and preprocess original multimodal data, wherein the multimodal data includes image data, chief complaint text data, and clinical examination text data; The medical imaging report generation module is used to generate a corresponding medical imaging report by predicting based on the preprocessed multimodal data using a trained multimodal deep learning model; wherein the multimodal deep learning model includes a trained convolutional neural network model, an artificial neural network model and a fully connected layer, and the prediction process specifically includes: extracting image features based on the preprocessed imaging data using a trained convolutional neural network model; extracting semantic features based on the preprocessed chief complaint text data and clinical examination text data using a trained artificial neural network model; fusing the image features with the corresponding semantic features to obtain fused features; and inputting the fused features into the fully connected layer to map and obtain the medical imaging report.

8. The device for generating a medical imaging report based on multimodal data according to claim 7, characterized in that: The data acquisition and preprocessing module includes an image reconstruction unit, an image segmentation unit and a normalization processing unit, which are used for image reconstruction, image segmentation and normalization processing respectively.

9. The device for generating a medical imaging report based on multimodal data according to claim 7, characterized in that: The medical image report generation module includes a single-modal model training unit, which is used to train the convolutional neural network model and the artificial neural network model. The specific training process includes: dividing the preprocessed multimodal data into a training set, a validation set and a test set; based on the image data in the training set, training the pre-trained convolutional neural network model using a gradient descent algorithm; based on the chief complaint text data and clinical examination text data in the training set, training the pre-trained artificial neural network model using a gradient descent algorithm; using the validation set to respectively determine the optimal parameters of the convolutional neural network model and the artificial neural network model; using the test set to evaluate the optimized convolutional neural network model and the artificial neural network model.

10. The device for generating medical imaging reports based on multimodal data according to claim 9, characterized in that: The medical imaging report generation module also includes a multimodal model training unit, which is used to train the multimodal deep learning model. The specific training process includes: dividing the preprocessed multimodal data into a training set and a validation set; based on preset initial model parameters, iteratively training the multimodal deep learning model using the training set and a gradient descent algorithm; and determining the optimal parameters of the multimodal deep learning model using the validation set.

Images