Medical image report text generation model training method and device

Abstract

The invention provides a medical image report text generation model training method and device, and relates to the technical field of artificial intelligence. The method comprises the following steps: acquiring a medical image marked with an organ type according to a medical image sequence, and acquiring a report text marked with the organ type according to a medical image report; and training a combined model constructed by an image encoder and a text generation model according to the medical image marked with the organ type and the report text marked with the organ type to obtain a medical image report text generation model. The device executes the method. According to the medical image report text generation model training method and device provided by the embodiment of the invention, text representation is performed based on all medical images of different organs of a human body, and the medical image report is automatically generated according to the medical image report text generation model, so that subsequent medical image report data analysis can be comprehensively and accurately assisted.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method and device for training a medical imaging report text generation model. Background Art

[0002] With the rapid increase in medical imaging data and the urgent need for precision medicine, the complexity of medical imaging is also increasing. Traditional image analysis methods are no longer able to meet clinical needs. Breakthroughs in AI technology, particularly deep learning algorithms, enable computers to extract features and perform pattern recognition from massive amounts of imaging data, greatly improving the speed and accuracy of image analysis.

[0003] Current medical imaging reports are often limited to analyzing a single site, such as generating chest-related medical reports based solely on chest CT images. However, single-site diagnosis ignores systemic pathological connections. For example, the lungs may have significant correlations with cardiovascular and cerebrovascular diseases. If this potential connection is not considered, it can lead to a one-sided diagnosis. Medical imaging report text generation models trained using the same method are also limited to single-site analysis. Summary of the Invention

[0004] In response to the problems in the prior art, embodiments of the present invention provide a method and device for training a medical imaging report text generation model, which can at least partially solve the problems in the prior art.

[0005] In one aspect, the present invention provides a method for training a medical imaging report text generation model, comprising:

[0006] Acquire medical images marked with organ types according to medical image sequences, and acquire report texts marked with organ types according to medical image reports;

[0007] The medical image sequence includes a collection of all medical images corresponding to different organs of each human body, the medical image report includes a collection of all report texts corresponding to different organs of each human body, and the report text marked with organ type includes the final text representation of the first training stage;

[0008] A combined model constructed by an image encoder and a text generation model is trained based on medical images labeled with organ types and report texts labeled with organ types to obtain a medical image report text generation model.

[0009] The step of acquiring medical images marked with organ types based on medical image sequences includes:

[0010] performing organ classification on the medical image sequence based on a preset sequence classification model to obtain medical images marked with organ types;

[0011] The preset sequence classification model includes a ResNet18 model and a first fully connected layer connected in sequence.

[0012] The performing organ classification on the medical image sequence based on a preset sequence classification model to obtain medical images marked with organ types includes:

[0013] Performing feature extraction on the medical image sequence based on the ResNet18 model to obtain image features of each slice image;

[0014] Predicting the image features of each slice image based on the first fully connected layer to obtain the predicted probability of each slice image belonging to a different organ category;

[0015] The average values of the category prediction probabilities corresponding to different organs are calculated, and the organ type corresponding to the maximum average value is selected to label the medical image.

[0016] The step of obtaining a report text marked with an organ type based on a medical imaging report includes:

[0017] Performing text segmentation on the medical imaging report based on a preset report segmentation model to obtain a report text marked with organ types;

[0018] The preset report segmentation model is a Transformer model having a preset number of encoder layers and a second fully connected layer connected in sequence.

[0019] The step of performing text segmentation on the medical imaging report based on a preset report segmentation model to obtain a report text marked with organ types includes:

[0020] An encoder based on a preset number of layers sequentially performs word segmentation processing on the medical imaging report to obtain a final text representation of the first training stage;

[0021] The final text representation of the first training stage is predicted based on the second fully connected layer, and the report text is obtained and labeled with the organ type corresponding to the report text.

[0022] The method of training a combined model constructed by an image encoder and a text generation model based on medical images marked with organ types and report text marked with organ types to obtain a medical image report text generation model includes:

[0023] Perform data alignment on medical images labeled with organ types and report texts labeled with organ types;

[0024] Extracting features from the medical image marked with organ types based on the image encoder to obtain an image feature vector;

[0025] Randomly selecting the final text representation of the first training stage to obtain a partial final text representation, and fusing the partial final text representation with the image feature vector based on an attention mechanism to obtain fused data;

[0026] The fused data is predicted based on the text generation model to obtain the final text representation of the second training stage, and the final text representation of the second training stage is constrained and trained using the cross entropy loss function to obtain the medical imaging report text generation model.

[0027] In one aspect, the present invention provides a medical imaging report text generation model training device, comprising:

[0028] an acquisition unit, configured to acquire medical images marked with organ types based on medical image sequences, and acquire report text marked with organ types based on medical image reports;

[0029] The medical image sequence includes a collection of all medical images corresponding to different organs of each human body, the medical image report includes a collection of all report texts corresponding to different organs of each human body, and the report text marked with organ type includes the final text representation of the first training stage;

[0030] The training unit is used to train a combined model constructed by an image encoder and a text generation model based on medical images marked with organ types and report texts marked with organ types to obtain a medical image report text generation model.

[0031] In another aspect, an embodiment of the present invention provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the following method is implemented:

[0032] Acquire medical images marked with organ types according to medical image sequences, and acquire report texts marked with organ types according to medical image reports;

[0033] The medical image sequence includes a collection of all medical images corresponding to different organs of each human body, the medical image report includes a collection of all report texts corresponding to different organs of each human body, and the report text marked with organ type includes the final text representation of the first training stage;

[0034] A combined model constructed by an image encoder and a text generation model is trained based on medical images labeled with organ types and report texts labeled with organ types to obtain a medical image report text generation model.

[0035] An embodiment of the present invention provides a computer-readable storage medium, including:

[0036] The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following method is implemented:

[0037] Acquire medical images marked with organ types according to medical image sequences, and acquire report texts marked with organ types according to medical image reports;

[0038] The medical image sequence includes a collection of all medical images corresponding to different organs of each human body, the medical image report includes a collection of all report texts corresponding to different organs of each human body, and the report text marked with organ type includes the final text representation of the first training stage;

[0039] A combined model constructed by an image encoder and a text generation model is trained based on medical images labeled with organ types and report texts labeled with organ types to obtain a medical image report text generation model.

[0040] An embodiment of the present invention further provides a computer program product, comprising a computer program. When the computer program is executed by a processor, the computer program implements the following method:

[0041] Acquire medical images marked with organ types according to medical image sequences, and acquire report texts marked with organ types according to medical image reports;

[0042] The medical image sequence includes a collection of all medical images corresponding to different organs of each human body, the medical image report includes a collection of all report texts corresponding to different organs of each human body, and the report text marked with organ type includes the final text representation of the first training stage;

[0043] A combined model constructed by an image encoder and a text generation model is trained based on medical images labeled with organ types and report texts labeled with organ types to obtain a medical image report text generation model.

[0044] The embodiments of the present invention provide a method and device for training a medical imaging report text generation model, which obtains medical images marked with organ types based on medical imaging sequences, and obtains report texts marked with organ types based on medical imaging reports; wherein, the medical imaging sequence includes an image collection of all medical images corresponding to different organs of each human body, the medical imaging report includes a collection of all report texts corresponding to different organs of each human body, and the report text marked with organ types includes the final text representation of the first training stage; a combined model constructed by an image encoder and a text generation model is trained based on the medical images marked with organ types and the report text marked with organ types to obtain a medical imaging report text generation model, which performs text representation based on all medical images of different organs of the human body and automatically generates a medical imaging report based on the medical imaging report text generation model, thereby comprehensively and accurately assisting subsequent medical imaging report data analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0046] Figure 1 It is a flowchart of a medical imaging report text generation model training method provided by one embodiment of the present invention.

[0047] Figure 2 It is a structural diagram of a medical imaging report text generation model training device provided by one embodiment of the present invention.

[0048] Figure 3 A schematic diagram of the physical structure of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0049] To make the purpose, technical solutions and advantages of the embodiments of the present invention more clear, the embodiments of the present invention are further described in detail below with reference to the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention. It should be noted that, unless there is a conflict, the embodiments and features in the embodiments of this application can be combined with each other in any manner.

[0050] Figure 1 FIG. 1 is a flow chart of a method for training a medical imaging report text generation model according to an embodiment of the present invention. Figure 1 As shown, the medical imaging report text generation model training method provided by the embodiment of the present invention includes:

[0051] Step S1: obtaining a medical image marked with an organ type according to a medical image sequence, and obtaining a report text marked with an organ type according to a medical image report;

[0052] The medical image sequence includes an image collection of all medical images corresponding to different organs of each human body, the medical image report includes a collection of all report texts corresponding to different organs of each human body, and the report text marked with organ type includes the final text representation of the first training stage.

[0053] Step S2: training a combined model constructed by an image encoder and a text generation model based on medical images labeled with organ types and report texts labeled with organ types to obtain a medical image report text generation model.

[0054] In the above step S1, the device obtains medical images marked with organ types based on the medical image sequence, and obtains report text marked with organ types based on the medical image report;

[0055] The medical image sequence includes a collection of all medical images corresponding to different organs of each human body, the medical image report includes a collection of all report texts corresponding to different organs of each human body, and the report text labeled with organ type includes the final text representation of the first training phase. The apparatus may be a computer device, such as a server, that executes the method.

[0056] Medical Image Sequence Recording A medical image sequence representing the i-th organ of the v-th human body, where each organ contains M slice images. Medical images are usually presented in a sequence format (e.g., CT or MRI multi-slice sequence), where each slice reflects the anatomical characteristics of a different human organ, making feature extraction and classification more complex. The method of obtaining medical images labeled with organ types based on the medical image sequence includes:

[0057] performing organ classification on the medical image sequence based on a preset sequence classification model to obtain medical images marked with organ types;

[0058] The preset sequence classification model includes a ResNet18 model and a first fully connected layer connected in sequence. A medical image sequence can be input into the preset sequence classification model, and the preset sequence classification model outputs the predicted probability of each slice image belonging to a different organ category.

[0059] The performing organ classification on the medical image sequence based on a preset sequence classification model to obtain medical images marked with organ types includes:

[0060] Performing feature extraction on the medical image sequence based on the ResNet18 model to obtain image features of each slice image;

[0061] Based on the first fully connected layer, the image features of each slice image are predicted to obtain the category prediction probability of each slice image belonging to different organs; the calculation formula is as follows:

[0062]

[0063] in, is the predicted probability that the j-th slice image belongs to the i-th organ of the v-th human body, fc1 is the first fully connected layer, ResNet is the ResNet18 model, is the jth slice image of the i-th organ of the v-th person. The sequence classification model automatically classifies the human body part to which a sequence of medical images belongs based on input. The sequence classification model includes a ResNet18 model as an image feature extractor and a fully connected layer (fc) as a classifier.

[0064] Calculate the average of the class prediction probabilities corresponding to different organs, and select the organ type corresponding to the maximum average value to label the medical image. The calculation formula is as follows:

[0065]

[0066] in, is the average value of the predicted probability of the category corresponding to the i-th organ. An example is as follows:

[0067] Suppose there are 5 slices, and the category prediction probabilities of different organs corresponding to each slice are [0.7, 0.2, 0.1], [0.6, 0.3, 0.1], [0.8, 0.1, 0.1], [0.5, 0.4, 0.1], [0.9, 0.05, 0.05], and the calculated average value is [0.7, 0.21, 0.09]. In the category prediction results, category 1 has the highest probability.

[0068] Since an image contains multiple slices, individual slices may contain noise or insufficient feature information of specific parts. By averaging the category prediction probabilities of some slices in the sequence, the information of all slices can be effectively integrated, reducing the interference of single abnormal slices or artifacts on the prediction results, thereby achieving more accurate organ classification.

[0069] Medical imaging reports are usually full-text descriptions, and each sentence or phrase may correspond to a different body part. Manual annotation is time-consuming and labor-intensive, and is prone to ambiguity and inconsistency. An automated report segmentation method is needed to accurately map each paragraph of text in the report to the corresponding body part.

[0070] The step of obtaining a report text marked with an organ type according to a medical imaging report includes:

[0071] Performing text segmentation on the medical imaging report based on a preset report segmentation model to obtain a report text marked with organ types;

[0072] The preset report segmentation model is a Transformer model having a preset number of encoder layers and a second fully connected layer connected in sequence. A medical imaging report can be input into the preset report segmentation model, which then outputs the organ type corresponding to the report text.

[0073] The performing text segmentation on the medical imaging report based on a preset report segmentation model to obtain a report text marked with organ types includes:

[0074] The medical imaging report is segmented in turn by an encoder based on a preset number of layers to obtain the final text representation of the first training phase; the specific value of the preset number of layers can be set independently according to actual conditions, and 6 layers can be selected. The medical imaging report is input into the first encoder in the preset report segmentation model for word segmentation to obtain a text representation:

[0075]

[0076] in, There are s text representations, where s corresponds to the organ i, N is the number of words in the image report, and d is the dimension of the text representation. The formula for calculating the text representation of the next layer is as follows:

[0077] f t s(l+1) =E t (f t s(l) )

[0078] Among them, E t is a layer of Transformer encoder, f t s(l) is the text representation of the first layer. Referring to the above example, after 6 layers of encoding operations, the final text representation f of the first training stage is obtained t s(6) .

[0079] The final text representation of the first training phase is predicted based on the second fully connected layer, and the report text is obtained and labeled with the organ type corresponding to the report text. t s(6)To the second fully connected layer fc2, the output of the second fully connected layer fc2 is used as the organ type of the report text. The calculation formula is as follows:

[0080]

[0081] in, It reports the organ type to which the sth character representation belongs in the text, and softmax is the classifier.

[0082] The report segmentation model can automatically segment the description part corresponding to each sentence based on an input image description. Assume that the input image description is f t s1 ,f t s2 ,...,f t sN , the report segmentation mark is a code of the same length, denoted as Each element is labeled with the body part described by the corresponding position text. Assuming the labels include "brain," "chest and lungs," and "other," let the brain be labeled 1, the chest and lungs be labeled 2, and the other be labeled 3. The Transformer's multi-head attention mechanism captures long-range dependencies between sentences in a report, enabling a better understanding of context and precise location of the organ type described by the text. Through positional encoding and hierarchical feature extraction, the model can comprehensively extract a representation of the report text and adapt to complex syntax and polysemous descriptions.

[0083] Training the preset report segmentation model includes:

[0084] The text processing model is trained using unlabeled medical imaging report text to obtain the weight of the text processing model;

[0085] The weights of the text processing model are transferred to the preset report segmentation model to obtain a preset basic report segmentation model. To fully utilize the semantic information of large-scale unlabeled report text, a transfer learning strategy is used to pre-train and fine-tune the segmentation model. First, a text processing model is trained on a large amount of unlabeled report text to learn common text features. Subsequently, the weights obtained from this training are transferred to the report segmentation model and fine-tuned on labeled data to meet the requirements of the report segmentation task. By migrating its weights, the performance of the report segmentation model is further improved.

[0086] The preset basic report segmentation model is trained by introducing pre-training text with data augmentation processing that randomly adjusts characters to obtain the preset report segmentation model. The pre-training text training may include medical imaging report training text with labeled organ types and medical terminology training text with labeled organ types. The medical imaging report training text with labeled organ types is the text to be used when training the preset report segmentation model. The medical terminology training text with labeled organ types is used to train the preset report segmentation model. This medical data training text usually comes from medical literature, medical reference books, etc., and the text corresponding to the organ type has been labeled.

[0087] In order to increase the diversity of data the model is exposed to during the training phase and improve the generalization ability of the model, a random editing-based enhancement method (i.e., a data enhancement processing method that randomly adjusts characters) is introduced for the input text sequence and its corresponding label sequence. During the training process, a character is randomly inserted or deleted at each position with a probability of 2%, and the label sequence is adjusted accordingly. For example, if a character is inserted at a certain position, the label of the newly added character is set to the label of the character to its left, and some words in the text are randomly replaced with synonyms to increase the diversity of the text. While maintaining semantic integrity, the order of sentences is randomly adjusted, and some small grammatical errors or spelling errors are randomly injected into the text. This data enhancement method effectively expands the training data distribution of the model and reduces the risk of overfitting.

[0088] In addition, jointly training the report segmentation task with other related tasks (such as medical term recognition and text classification) can enhance the model's ability to understand medical text. Through multi-task learning, the model can share underlying text features and optimize on multiple tasks simultaneously, thereby improving overall performance.

[0089] In step S2 above, the apparatus trains a combined model constructed by an image encoder and a text generation model based on the medical images labeled with organ types and the report text labeled with organ types to obtain a medical image report text generation model. The training of a combined model constructed by an image encoder and a text generation model based on the medical images labeled with organ types and the report text labeled with organ types to obtain a medical image report text generation model includes:

[0090] Data alignment is performed on medical images marked with organ types and report texts marked with organ types; data alignment of medical images marked with organ types and report texts is a basic step to ensure semantic consistency between images and corresponding text descriptions.

[0091] Based on the image encoder, feature extraction is performed on the medical image marked with the organ type to obtain an image feature vector; the global or local features of the image are extracted through the image encoder, and the feature vector is input into the text generation model to complete the conversion from image to text. The ultimate goal is to output the report text of the medical image.

[0092] The final text representation of the first training stage is randomly selected to obtain a partial final text representation, and the partial final text representation and the image feature vector are fused based on the attention mechanism to obtain fused data; medical images and texts belong to different modalities, and there are huge differences in their feature distribution and expression forms (images are continuous pixel data, and texts are discrete symbol data). In diagnostic tasks, the attention mechanism can automatically and dynamically assign feature weights based on the characteristics of the input data (such as image content and text context). For example, based on certain data features, the main focus can be on the detailed information of the image, such as local features such as resolution and texture. For other data features, the overall contour information or text descriptive features in the image are highlighted.

[0093] The fused data is predicted based on the text generation model to obtain the final text representation of the second training phase, and the final text representation of the second training phase is constrained and trained using a cross-entropy loss function to obtain the medical imaging report text generation model. The cross-entropy loss function is expressed according to the following formula:

[0094]

[0095] Where T is the text length of the final text representation of the second training stage, V is the number of all segmented words in the final text representation of the second training stage, is the encoding of the target word at time step t in the final text representation of the second training phase, is the predicted probability distribution of the model at time step t.

[0096] By training the model parameters with cross entropy loss, the robustness of the category prediction probability can be ensured, so that the model will not fit the wrong distribution due to data bias during training, thereby enhancing the generalization ability of different image sequences.

[0097] Since the medical report texts actually used include: clinical description text, image description text, and image diagnosis text, there may be missing text. Since some report texts are incomplete, the accuracy of the auxiliary diagnosis model trained directly with incomplete report texts is reduced. Therefore, during the model training process, the complete final text representation is used as the output, and randomly selected parts of the final text representation and image feature vectors are used as input. This can train the model's text completion capabilities and ensure that it can be completed while outputting the text content in actual use. For example, when describing a lung nodule, if the specific shape or size of the lesion is missing in the actual image description text, this model can complete the information through partial context and image features, making the output report text more accurate.

[0098] The medical imaging report text generation model training method provided by the embodiment of the present invention has the following beneficial technical effects:

[0099] First, by automatically classifying images by organ, it is possible to separate medical imaging tasks for multiple diseases, avoiding feature confusion caused by differences in disease types. Medical imaging features of different organs vary significantly, and organ type classification can provide independent input data for subsequent models in advance, allowing the models to focus more on modeling specific organ characteristics. Automatic classification models can reduce the workload of manual labeling, especially when there are large numbers of unlabeled medical images. After independent training, the organ classification model can quickly assign organ type attributes to images.

[0100] Second, medical reports often contain diagnostic information for multiple organs. By properly breaking down this information into segments, the segmentation model can independently model the description of each organ. Segmenting medical imaging reports by organ allows for the creation of higher-quality image-text paired data. After the medical report is segmented, the text segmentation model can filter out content unrelated to the target organ, making the text's diagnostic information for specific diseases more precise. Subsequent models can then learn accurate feature information from this high-quality text, avoiding confusion in diagnosing multiple diseases.

[0101] Third, through joint training, the unique characteristics of different diseases can be learned without model confusion caused by data feature conflicts. After joint training on multiple diseases, the model can improve its generalization ability for diseases with similar characteristics by sharing some features. High-quality image-text correlation data provides a strong interactive signal for the model, enabling cross-modal learning. The model not only accurately understands abnormalities in images, but also provides professional and diverse disease reports when generating text content.

[0102] The embodiment of the present invention provides a method for training a medical imaging report text generation model, which obtains medical images marked with organ types based on medical imaging sequences, and obtains report texts marked with organ types based on medical imaging reports; wherein, the medical imaging sequence includes an image collection of all medical images corresponding to different organs of each human body, the medical imaging report includes a collection of all report texts corresponding to different organs of each human body, and the report text marked with organ types includes the final text representation of the first training stage; a combined model constructed by an image encoder and a text generation model is trained based on the medical images marked with organ types and the report text marked with organ types to obtain a medical imaging report text generation model, and by performing text representation based on all medical images of different organs of the human body and automatically generating a medical imaging report based on the medical imaging report text generation model, it can comprehensively and accurately assist subsequent medical imaging report data analysis.

[0103] Furthermore, the step of acquiring medical images marked with organ types based on the medical image sequence includes:

[0104] Organ classification is performed on the medical image sequence based on a preset sequence classification model to obtain medical images marked with organ types; the description can be made with reference to the above embodiment and will not be repeated here.

[0105] The preset sequence classification model includes a ResNet18 model and a first fully connected layer connected in sequence. The above description can be referred to in detail and will not be repeated here.

[0106] Furthermore, the performing organ classification on the medical image sequence based on a preset sequence classification model to obtain medical images marked with organ types includes:

[0107] Based on the ResNet18 model, feature extraction is performed on the medical image sequence to obtain image features of each slice image; the above embodiment can be referred to for description and will not be repeated here.

[0108] Based on the first fully connected layer, the image features of each slice image are predicted to obtain the category prediction probability of each slice image belonging to a different organ; please refer to the above embodiment for description and no further details will be given.

[0109] The average values of the class prediction probabilities corresponding to different organs are calculated, and the organ type corresponding to the maximum average value is selected to label the medical image.

[0110] Furthermore, obtaining a report text marked with an organ type according to the medical imaging report includes:

[0111] The medical imaging report is subjected to text segmentation based on a preset report segmentation model to obtain a report text marked with organ types; the description can be made with reference to the above embodiment and will not be repeated here.

[0112] The preset report segmentation model is a Transformer model having a preset number of encoder layers and a second fully connected layer connected in sequence.

[0113] Furthermore, the text segmentation of the medical imaging report is performed based on a preset report segmentation model to obtain a report text marked with organ types, including:

[0114] The encoder based on the preset number of layers performs word segmentation processing on the medical imaging report in sequence to obtain the final text representation of the first training stage; the description can be made with reference to the above embodiment and will not be repeated here.

[0115] The final text representation of the first training phase is predicted based on the second fully connected layer to obtain and label the report text with the organ type corresponding to the report text.

[0116] Furthermore, the combined model constructed by the image encoder and the text generation model is trained based on the medical images marked with organ types and the report text marked with organ types to obtain the medical image report text generation model, including:

[0117] Data alignment is performed on the medical image marked with the organ type and the report text marked with the organ type; the above-mentioned embodiment can be referred to for description and will not be repeated here.

[0118] Based on the image encoder, feature extraction is performed on the medical image marked with the organ type to obtain an image feature vector; the description can be made with reference to the above embodiment and will not be repeated here.

[0119] The final text representation of the first training stage is randomly selected to obtain a partial final text representation, and the partial final text representation and the image feature vector are fused based on the attention mechanism to obtain fused data; the above embodiment can be referred to for description and will not be repeated here.

[0120] The fused data is predicted based on the text generation model to obtain a final text representation of the second training phase. The final text representation of the second training phase is constrained and trained using a cross-entropy loss function to obtain the medical imaging report text generation model. This can be explained with reference to the above embodiment and will not be repeated here.

[0121] Figure 2 FIG. 1 is a schematic diagram of a medical imaging report text generation model training device according to an embodiment of the present invention. Figure 2As shown, the medical imaging report text generation model training device provided by the embodiment of the present invention includes an acquisition unit 201 and a training unit 202, wherein:

[0122] The acquisition unit 201 is used to acquire medical images marked with organ types based on a medical image sequence, and to acquire report text marked with organ types based on a medical image report; wherein the medical image sequence includes an image collection of all medical images corresponding to different organs of each human body, the medical image report includes a collection of all report texts corresponding to different organs of each human body, and the report text marked with organ types includes the final text representation of the first training stage; the training unit 202 is used to train a combined model constructed by an image encoder and a text generation model based on the medical images marked with organ types and the report text marked with organ types to obtain a medical image report text generation model.

[0123] Specifically, the acquisition unit 201 in the device is used to acquire medical images marked with organ types based on medical image sequences, and to acquire report texts marked with organ types based on medical image reports; wherein the medical image sequence includes an image collection of all medical images corresponding to different organs of each human body, the medical image report includes a collection of all report texts corresponding to different organs of each human body, and the report text marked with organ types includes the final text representation of the first training stage; the training unit 202 is used to train a combined model constructed by an image encoder and a text generation model based on the medical images marked with organ types and the report texts marked with organ types to obtain a medical image report text generation model.

[0124] The medical imaging report text generation model training device provided by an embodiment of the present invention obtains medical images marked with organ types based on medical image sequences, and obtains report texts marked with organ types based on medical imaging reports; wherein, the medical image sequence includes an image collection of all medical images corresponding to different organs of each human body, the medical imaging report includes a collection of all report texts corresponding to different organs of each human body, and the report text marked with organ types includes the final text representation of the first training stage; based on the medical images marked with organ types and the report texts marked with organ types, a combined model constructed by an image encoder and a text generation model is trained to obtain a medical imaging report text generation model, and by performing text representation based on all medical images of different organs of the human body and automatically generating a medical imaging report according to the medical imaging report text generation model, it can comprehensively and accurately assist subsequent medical imaging report data analysis.

[0125] The embodiment of the present invention provides an embodiment of a medical imaging report text generation model training device, which can be specifically used to execute the processing flow of the above-mentioned method embodiments. Its functions are not repeated here, and reference can be made to the detailed description of the above-mentioned method embodiments.

[0126] Figure 3 A schematic diagram of the physical structure of a computer device provided in an embodiment of the present invention is shown in FIG. Figure 3 As shown, the computer device includes: a memory 301, a processor 302, and a computer program stored in the memory 301 and executable on the processor 302. When the processor 302 executes the computer program, the following method is implemented:

[0127] Acquire medical images marked with organ types according to medical image sequences, and acquire report texts marked with organ types according to medical image reports;

[0128] The medical image sequence includes a collection of all medical images corresponding to different organs of each human body, the medical image report includes a collection of all report texts corresponding to different organs of each human body, and the report text marked with organ type includes the final text representation of the first training stage;

[0129] A combined model constructed by an image encoder and a text generation model is trained based on medical images labeled with organ types and report texts labeled with organ types to obtain a medical image report text generation model.

[0130] This embodiment discloses a computer program product, which includes a computer program. When the computer program is executed by a processor, the following method is implemented:

[0131] Acquire medical images marked with organ types according to medical image sequences, and acquire report texts marked with organ types according to medical image reports;

[0132] The medical image sequence includes a collection of all medical images corresponding to different organs of each human body, the medical image report includes a collection of all report texts corresponding to different organs of each human body, and the report text marked with organ type includes the final text representation of the first training stage;

[0133] A combined model constructed by an image encoder and a text generation model is trained based on medical images labeled with organ types and report texts labeled with organ types to obtain a medical image report text generation model.

[0134] This embodiment provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the following method is implemented:

[0135] Acquire medical images marked with organ types according to medical image sequences, and acquire report texts marked with organ types according to medical image reports;

[0136] The medical image sequence includes a collection of all medical images corresponding to different organs of each human body, the medical image report includes a collection of all report texts corresponding to different organs of each human body, and the report text marked with organ type includes the final text representation of the first training stage;

[0137] A combined model constructed by an image encoder and a text generation model is trained based on medical images labeled with organ types and report texts labeled with organ types to obtain a medical image report text generation model.

[0138] Compared with the technical solutions in the prior art, the embodiments of the present invention provide a method for training a medical imaging report text generation model, which obtains medical images marked with organ types based on medical image sequences, and obtains report texts marked with organ types based on medical imaging reports; wherein, the medical image sequences include an image collection of all medical images corresponding to different organs of each human body, the medical imaging reports include a collection of all report texts corresponding to different organs of each human body, and the report texts marked with organ types include the final text representation of the first training stage; a combined model constructed by an image encoder and a text generation model is trained based on the medical images marked with organ types and the report texts marked with organ types to obtain a medical imaging report text generation model, which performs text representation based on all medical images of different organs of the human body and automatically generates medical imaging reports based on the medical imaging report text generation model, thereby comprehensively and accurately assisting subsequent medical imaging report data analysis.

[0139] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0140] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0141] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0142] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0143] Throughout this specification, reference to terms such as "one embodiment," "a specific embodiment," "some embodiments," "for example," "example," "specific example," or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0144] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for training a model for generating text for medical imaging reports, characterized in that: include: Acquire medical images marked with organ types according to medical image sequences, and acquire report texts marked with organ types according to medical image reports; The medical image sequence includes a collection of all medical images corresponding to different organs of each human body, the medical image report includes a collection of all report texts corresponding to different organs of each human body, and the report text marked with organ type includes the final text representation of the first training stage; A combined model constructed by an image encoder and a text generation model is trained based on medical images labeled with organ types and report texts labeled with organ types to obtain a medical image report text generation model.

2. The medical imaging report text generation model training method according to claim 1, characterized in that: The step of acquiring medical images marked with organ types based on medical image sequences includes: performing organ classification on the medical image sequence based on a preset sequence classification model to obtain medical images marked with organ types; The preset sequence classification model includes a ResNet18 model and a first fully connected layer connected in sequence.

3. The medical imaging report text generation model training method according to claim 2, characterized in that: The performing organ classification on the medical image sequence based on a preset sequence classification model to obtain medical images marked with organ types includes: Performing feature extraction on the medical image sequence based on the ResNet18 model to obtain image features of each slice image; Predicting the image features of each slice image based on the first fully connected layer to obtain the predicted probability of each slice image belonging to a different organ category; The average values of the category prediction probabilities corresponding to different organs are calculated, and the organ type corresponding to the maximum average value is selected to label the medical image.

4. The medical imaging report text generation model training method according to claim 1, characterized in that: The step of obtaining a report text marked with an organ type according to a medical imaging report includes: Performing text segmentation on the medical imaging report based on a preset report segmentation model to obtain a report text marked with organ types; The preset report segmentation model is a Transformer model having a preset number of encoder layers and a second fully connected layer connected in sequence.

5. The medical imaging report text generation model training method according to claim 4, characterized in that: The performing text segmentation on the medical imaging report based on a preset report segmentation model to obtain a report text marked with organ types includes: An encoder based on a preset number of layers sequentially performs word segmentation processing on the medical imaging report to obtain a final text representation of the first training stage; The final text representation of the first training stage is predicted based on the second fully connected layer, and the report text is obtained and labeled with the organ type corresponding to the report text.

6. The method for training a model for generating medical imaging report text according to any one of claims 1 to 5, characterized in that: The method of training a combined model constructed by an image encoder and a text generation model based on medical images marked with organ types and report text marked with organ types to obtain a medical image report text generation model includes: Perform data alignment on medical images labeled with organ types and report texts labeled with organ types; Extracting features from the medical image marked with organ types based on the image encoder to obtain an image feature vector; Randomly selecting the final text representation of the first training stage to obtain a partial final text representation, and fusing the partial final text representation with the image feature vector based on an attention mechanism to obtain fused data; The fused data is predicted based on the text generation model to obtain the final text representation of the second training stage, and the final text representation of the second training stage is constrained and trained using the cross entropy loss function to obtain the medical imaging report text generation model.

7. A medical imaging report text generation model training device, characterized in that: include: an acquisition unit, configured to acquire medical images marked with organ types based on medical image sequences, and acquire report text marked with organ types based on medical image reports; The medical image sequence includes a collection of all medical images corresponding to different organs of each human body, the medical image report includes a collection of all report texts corresponding to different organs of each human body, and the report text marked with organ type includes the final text representation of the first training stage; The training unit is used to train a combined model constructed by an image encoder and a text generation model based on medical images marked with organ types and report texts marked with organ types to obtain a medical image report text generation model.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images