RadFM-based medical multi-modal model pre-training method and apparatus, and related product

Abstract

The invention discloses a pre-training method and device for a medical multi-modal model based on RadFM and a related product. For each group of training data, performing staged prediction on image visual features, image local features and image global features corresponding to the medical sample images through the initial pre-training model to obtain a prediction description text set corresponding to the medical sample images, wherein one prediction description text in the prediction description text set corresponds to a prediction result of one stage; and for the multiple groups of training data, according to the difference between the multiple medical description text sets and the multiple prediction description text sets, adjusting parameters of the initial pre-training model until the adjusted model meets a pre-training cut-off condition, and ending pre-training to obtain a medical multi-modal pre-training model. Therefore, the medical multi-modal pre-training model is obtained through staged training, and the model can learn different granularity information of the image, so that the interpretability of model prediction is improved.

Description

Technical Field

[0001] The present application relates to the field of medical model training technology, and in particular to a pre-training method, device and related products of a medical multimodal model based on RadFM. Background Art

[0002] Existing multimodal models usually use black box technologies such as deep learning to predict medical data to obtain medical prediction results. However, although black box technology can improve the prediction accuracy of the model to a certain extent, its internal prediction logic is opaque, making the prediction interpretability of the model poor, making it difficult for medical personnel to trust the medical prediction results obtained through deep learning prediction. Therefore, how to improve the prediction interpretability of the model is a key issue that technicians in this field are concerned about. Summary of the invention

[0003] Based on the above problems, the present application provides a pre-training method, device and related products of a medical multimodal model based on RadFM to improve the predictive interpretability of the model. The embodiments of the present application disclose the following technical solutions:

[0004] In a first aspect, the present application discloses a pre-training method for a medical multimodal model based on RadFM, comprising:

[0005] Acquiring medical multimodal data, wherein the medical multimodal data includes multiple sets of training data, each set of training data includes a medical sample image and a medical description text set corresponding to the medical sample image;

[0006] For each set of training data, feature extraction is performed on the medical sample image to obtain image visual features, image local features and image global features corresponding to the medical sample image;

[0007] For each set of training data, the image visual features, image local features and image global features corresponding to the medical sample image are predicted in stages through an initial pre-training model to obtain a set of predicted description texts corresponding to the medical sample image, wherein one predicted description text in the predicted description text set corresponds to a prediction result of one stage, and the backbone network of the initial pre-training model includes RadFM;

[0008] For the multiple groups of training data, according to the differences between multiple medical description text sets and multiple prediction description text sets, the parameters of the initial pre-training model are adjusted until the adjusted model meets the pre-training cutoff condition, and the pre-training is completed to obtain a medical multimodal pre-training model.

[0009] In an achievable implementation, the step of predicting the image visual features, image local features, and image global features corresponding to the medical sample image in stages through the initial pre-training model to obtain a predicted description text set corresponding to the medical sample image includes:

[0010] In the first prediction stage, the image visual features corresponding to the medical sample image are predicted by an initial pre-trained model to obtain a label-level description text corresponding to the medical sample image;

[0011] In the second prediction stage, the local image features corresponding to the medical sample image are predicted by the initial pre-training model to obtain a phrase-level description text corresponding to the medical sample image;

[0012] In the third prediction stage, the global image features corresponding to the medical sample image are predicted by the initial pre-trained model to obtain a sentence-level description text corresponding to the medical sample image;

[0013] The tag-level description text, the phrase-level description text and the sentence-level description text are used as a predicted description text set corresponding to the medical sample image.

[0014] In an achievable implementation, before acquiring the medical multimodal data, the method further includes:

[0015] Acquire target training data and a plurality of initial medical images, wherein the target training data includes a target medical image and a target description text set corresponding to the target medical image, and the plurality of initial medical images are obtained by generating an adversarial network;

[0016] According to the target medical image and the target description text set, the multiple initial medical images are processed to obtain medical sample images corresponding to the multiple initial medical images respectively;

[0017] The acquiring of medical multimodal data comprises:

[0018] The target description text set is assigned to multiple medical sample images to obtain multiple groups of training data, and the multiple groups of training data are used as medical multimodal data, wherein the target description text set and one medical sample image constitute a group of training data.

[0019] In an achievable implementation, the processing of the multiple initial medical images according to the target medical image and the target description text set to obtain medical sample images corresponding to the multiple initial medical images respectively includes:

[0020] Determining abnormal image features of abnormalities in the target medical image according to the target description text set;

[0021] The abnormal image feature and the multiple initial medical images are fused to obtain medical sample images corresponding to the multiple initial medical images.

[0022] In an achievable implementation, the label-level description text includes abnormality category text corresponding to the abnormality in the medical sample image, the phrase-level description text includes abnormality position text corresponding to the abnormality in the medical sample image, and the sentence-level description text includes abnormality semantic text corresponding to the abnormality in the medical sample image.

[0023] In an achievable implementation, each set of training data further includes a medical sample problem for the medical sample image. For each set of training data, the image visual features, image local features, and image global features corresponding to the medical sample image are predicted in stages through an initial pre-training model to obtain a predicted description text set corresponding to the medical sample image, including:

[0024] For each set of training data, the image visual features, image local features and image global features corresponding to the medical sample images are predicted in stages based on the medical sample problem through an initial pre-training model to obtain a predicted description text set corresponding to the medical sample images.

[0025] In an achievable implementation, the extracting features of the medical sample image to obtain image visual features, image local features, and image global features corresponding to the medical sample image includes:

[0026] Extracting visual features of the medical sample image to obtain image visual features corresponding to the medical sample image;

[0027] Performing local feature extraction on the medical sample image to obtain local image features corresponding to the medical sample image;

[0028] Perform global feature extraction on the medical sample image to obtain image global features corresponding to the medical sample image.

[0029] In a second aspect, the present application discloses a pre-training device for a medical multimodal model based on RadFM, comprising:

[0030] A multimodal data acquisition unit, used to acquire medical multimodal data, wherein the medical multimodal data includes multiple sets of training data, each set of training data includes a medical sample image and a set of medical description texts corresponding to the medical sample image;

[0031] An image feature extraction unit, used for performing feature extraction on the medical sample image for each set of training data, to obtain image visual features, image local features and image global features corresponding to the medical sample image;

[0032] A feature phased prediction unit, for predicting the image visual features, image local features and image global features corresponding to the medical sample image in phases through an initial pre-training model for each set of training data, to obtain a set of predicted description texts corresponding to the medical sample image, wherein one predicted description text in the predicted description text set corresponds to a prediction result of one phase, and the backbone network of the initial pre-training model includes RadFM;

[0033] The pre-training model obtaining unit is used to adjust the parameters of the initial pre-training model according to the differences between the multiple medical description text sets and the multiple prediction description text sets for the multiple sets of training data, until the adjusted model meets the pre-training cutoff condition, and the pre-training is completed to obtain a medical multimodal pre-training model.

[0034] In an achievable implementation, the feature stage-by-stage prediction unit is specifically used to:

[0035] In the first prediction stage, the image visual features corresponding to the medical sample image are predicted by an initial pre-trained model to obtain a label-level description text corresponding to the medical sample image;

[0036] In the second prediction stage, the local image features corresponding to the medical sample image are predicted by the initial pre-training model to obtain a phrase-level description text corresponding to the medical sample image;

[0037] In the third prediction stage, the global image features corresponding to the medical sample image are predicted by the initial pre-trained model to obtain a sentence-level description text corresponding to the medical sample image;

[0038] The tag-level description text, the phrase-level description text and the sentence-level description text are used as a predicted description text set corresponding to the medical sample image.

[0039] In one feasible embodiment, the device further includes:

[0040] A target training data acquisition unit, configured to acquire target training data and a plurality of initial medical images, wherein the target training data includes a target medical image and a target description text set corresponding to the target medical image, and the plurality of initial medical images are obtained by generating an adversarial network;

[0041] An initial medical image processing unit, configured to process the multiple initial medical images according to the target medical image and the target description text set, to obtain medical sample images corresponding to the multiple initial medical images respectively;

[0042] The multimodal data acquisition unit is specifically used for:

[0043] The target description text set is assigned to multiple medical sample images to obtain multiple groups of training data, and the multiple groups of training data are used as medical multimodal data, wherein the target description text set and one medical sample image constitute a group of training data.

[0044] In an achievable implementation, the initial medical image processing unit is specifically used to:

[0045] Determining abnormal image features of abnormalities in the target medical image according to the target description text set;

[0046] The abnormal image feature and the multiple initial medical images are fused to obtain medical sample images corresponding to the multiple initial medical images.

[0047] In an achievable implementation, the label-level description text includes abnormality category text corresponding to the abnormality in the medical sample image, the phrase-level description text includes abnormality position text corresponding to the abnormality in the medical sample image, and the sentence-level description text includes abnormality semantic text corresponding to the abnormality in the medical sample image.

[0048] In an achievable implementation, the feature stage-by-stage prediction unit is specifically used to:

[0049] For each set of training data, the image visual features, image local features and image global features corresponding to the medical sample images are predicted in stages based on the medical sample problem through an initial pre-training model to obtain a predicted description text set corresponding to the medical sample images.

[0050] In an achievable implementation, the image feature extraction unit is specifically used to:

[0051] Extracting visual features of the medical sample image to obtain image visual features corresponding to the medical sample image;

[0052] Performing local feature extraction on the medical sample image to obtain local image features corresponding to the medical sample image;

[0053] Perform global feature extraction on the medical sample image to obtain image global features corresponding to the medical sample image.

[0054] In a third aspect, an embodiment of the present application provides an electronic device, including:

[0055] Memory for storing computer programs;

[0056] A processor is used to implement the steps of the above-mentioned pre-training method of the RadFM-based medical multimodal model when executing the computer program.

[0057] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned pre-training method of the medical multimodal model based on RadFM are implemented.

[0058] Compared with the prior art, this application has the following beneficial effects:

[0059] In the present application, medical multimodal data is first acquired, and for each set of training data, feature extraction is performed on the medical sample images to obtain image visual features, image local features and image global features corresponding to the medical sample images; thereafter, for each set of training data, the image visual features, image local features and image global features corresponding to the medical sample images are predicted in stages through an initial pre-training model to obtain a predicted description text set corresponding to the medical sample images; finally, for multiple sets of training data, the parameters of the initial pre-training model are adjusted according to the differences between multiple medical description text sets and multiple predicted description text sets until the adjusted model meets the pre-training cutoff condition, and the pre-training is completed to obtain a medical multimodal pre-training model.

[0060] It should be noted that medical multimodal data includes multiple sets of training data, each set of training data includes medical sample images and medical description text sets corresponding to the medical sample images, and a prediction description text in the prediction description text set corresponds to the prediction result of a stage. It can be seen that in this application, the initial pre-training model can predict different image features of medical sample images at different stages, so that the initial pre-training model can learn the image granularity information of the image corresponding to different image features, and can output the corresponding prediction description text at each prediction stage, which clarifies the internal prediction logic of the model to a certain extent, thereby improving the interpretability of the model prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0062] Figure 1 A flowchart of a pre-training method for a medical multimodal model based on RadFM provided in an embodiment of the present application;

[0063] Figure 2 An image schematic diagram of phased prediction in a pre-training method of a medical multimodal model based on RadFM provided in an embodiment of the present application;

[0064] Figure 3 A flowchart of model training in a pre-training method of a medical multimodal model based on RadFM provided in an embodiment of the present application;

[0065] Figure 4 A schematic diagram of the structure of a pre-training device for a medical multimodal model based on RadFM provided in an embodiment of the present application;

[0066] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0067] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0068] It should be noted that the pre-training method, device and related products of a medical multimodal model based on RadFM provided in this application are used in the field of medical model training technology. The above is only an example and does not limit the application field of the method and device names provided in this application.

[0069] As described above, existing multimodal models usually use black box technologies such as deep learning to predict medical data to obtain medical prediction results. However, although black box technology can improve the prediction accuracy of the model to a certain extent, its internal prediction logic is opaque, which makes the prediction interpretability of the model poor, making it difficult for medical personnel to trust the medical prediction results obtained through deep learning prediction. For example: the multimodal model can predict "medical data A" to obtain "prediction result A corresponding to medical data A". At this time, the prediction result A is only a conclusion, resulting in low trust in the conclusion by medical personnel. Therefore, how to improve the predictive interpretability of the model is a key issue that technicians in this field are concerned about.

[0070] Therefore, the inventors proposed the technical solution of this application, aiming to improve the predictive interpretability of the model. Specifically, in this application, for each set of training data in the medical multimodal data, feature extraction is first performed on the medical sample image to obtain the image visual features, image local features and image global features corresponding to the medical sample image; thereafter, for each set of training data in the medical multimodal data, the image visual features, image local features and image global features corresponding to the medical sample image are predicted in stages through the initial pre-training model to obtain the predicted description text set corresponding to the medical sample image; finally, for multiple sets of training data in the medical multimodal data, according to the differences between multiple medical description text sets and multiple prediction description text sets, the parameters of the initial pre-training model are adjusted until the adjusted model meets the pre-training cutoff condition, and the pre-training is completed to obtain the medical multimodal pre-training model.

[0071] It can be seen that in the present application, the initial pre-trained model can perform phased predictions on the image visual features, image local features and image global features corresponding to the medical sample images, so that the initial pre-trained model can learn the image granularity information of the image corresponding to different image features, and can output the corresponding prediction description text in each prediction stage, which clarifies the internal prediction logic of the model to a certain extent, thereby improving the interpretability of the model prediction and also improving the reliability of the model prediction.

[0072] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0073] The following is an example of a pre-training method for a medical multimodal model based on RadFM provided by the present application. Figure 1 , which is a flowchart of a pre-training method of a medical multimodal model based on RadFM provided in an embodiment of the present application, such as Figure 1 As shown, the method includes:

[0074] S101: Acquire medical multimodal data.

[0075] It should be noted that the medical multimodal data includes multiple groups of training data, wherein each group of training data includes a medical sample image and a medical description text set corresponding to the medical sample image, and each medical description text in the medical description text set is a text describing different image features in the medical sample image. The image modality type of the medical sample image in each group of training data may be different, the image modality type may be a CT modality type, and the image modality type may be an MRI modality type, which is not specifically limited here.

[0076] Furthermore, since the current training model has few medical image samples, in order to improve the model's fitting ability, the present application proposes that medical image simulation samples can be generated through a generative adversarial network, thereby increasing the number of medical image samples. Specifically, before obtaining medical multimodal data, the present application can first obtain target training data, wherein the target training data includes a target medical image and a target description text set corresponding to the target medical image, and can use a generative adversarial network to generate multiple initial medical images, wherein the multiple initial medical images include medical image simulation samples generated by a generator in the generative adversarial network.

[0077] After that, the target description text set can continue to be extracted and processed using the generative adversarial network to determine the abnormal image features that are abnormal in the target medical image (i.e., the image features corresponding to the diseased tissue). Finally, the abnormal image features and multiple initial medical images are fused one by one based on the generative adversarial network to obtain medical sample images corresponding to the multiple initial medical images, so that the medical image retains both the normal tissue in the medical image and the diseased tissue in the medical image. The medical sample images corresponding to each initial medical image have the same abnormal image features, which increases the number of model samples to a certain extent. It should also be noted that in the present application, multiple different abnormal image features can also be fused to an initial medical image, which is not specifically limited here, and the fusion process can also be selected in actual applications.

[0078] Furthermore, since the medical sample images corresponding to each initial medical image have the same abnormal image features, the target description text set can be assigned to multiple medical sample images to obtain multiple sets of training data, that is, the target description text set and one medical sample image constitute a set of training data. In addition, multiple sets of training data can be used as medical multimodal data, and the target training data and multiple sets of training data can be used together as medical multimodal data to facilitate the subsequent training of the medical multimodal pre-training model.

[0079] S102: For each set of training data, extract features of the medical sample images to obtain image visual features, image local features and image global features corresponding to the medical sample images.

[0080] In this step, first, visual feature extraction can be performed on the medical sample image to obtain the image visual features corresponding to the medical sample image, then local feature extraction can be performed on the medical sample image to obtain the image local features corresponding to the medical sample image, and finally global feature extraction can be performed on the medical sample image to obtain the image global features corresponding to the medical sample image. In this way, before the initial pre-training model is pre-trained, the present application can pre-extract features of the medical sample image, so that the subsequent initial pre-training model can input the image features corresponding to the stage for model training during staged prediction.

[0081] It should be noted that the feature extraction process of the medical sample image is not limited in the present application. For example, the local features of the image corresponding to the medical sample image can be extracted first, or the global features of the image corresponding to the medical sample image can be extracted first. Among them, the image visual features are used for the model training of the initial pre-trained model in the first prediction stage, the image local features are used for the model training of the initial pre-trained model in the second prediction stage, and the image global features are used for the model training of the initial pre-trained model in the third prediction stage.

[0082] It should also be noted that in the present application, the medical sample images can be subjected to color feature extraction, texture feature extraction, shape feature extraction and other extraction methods to obtain the image visual features corresponding to the medical sample images, and the image visual features can enable the initial pre-trained model to predict the abnormal category text corresponding to the abnormality in the medical sample images. In the present application, the local image features corresponding to the medical sample images can be extracted by means of the FAST algorithm and other methods, and the local image features can enable the initial pre-trained model to predict the abnormal position text corresponding to the abnormality in the medical sample images. In the present application, the global image features corresponding to the medical sample images can be extracted by means of the image segmentation algorithm and other methods, and the global image features can enable the initial pre-trained model to predict the abnormal semantic text corresponding to the abnormality in the medical sample images. It should be noted that the feature extraction method is not limited in the present application, and the feature extraction method can be selected according to the actual situation in practical applications.

[0083] S103: For each set of training data, the image visual features, image local features and image global features corresponding to the medical sample images are predicted in stages through an initial pre-training model to obtain a predicted description text set corresponding to the medical sample images.

[0084] In this step, one prediction description text in the prediction description text set corresponds to a prediction result of one stage, and the stages include the first stage, the second stage and the third stage. Next, the staged prediction process of the image features of the medical sample image in this application is introduced.

[0085] First, in the first prediction stage, the image visual features corresponding to the medical sample image can be predicted by the initial pre-trained model to obtain the label-level description text corresponding to the medical sample image. After that, in the second prediction stage, the image local features corresponding to the medical sample image can be predicted by the initial pre-trained model to obtain the phrase-level description text corresponding to the medical sample image. Finally, in the third prediction stage, the image global features corresponding to the medical sample image can be predicted by the initial pre-trained model to obtain the sentence-level description text corresponding to the medical sample image. At this time, the label-level description text, the phrase-level description text and the sentence-level description text can be used as the predicted description text set corresponding to the medical sample image, that is, in each prediction stage, the description text corresponding to the prediction stage can be predicted, and then the description texts corresponding to all prediction stages are used as the predicted description text set corresponding to the medical sample image. In this way, in the present application, the model can learn different image granularity information at different stages, which supports the interpretability of the model prediction to a certain extent.

[0086] It should be noted that the label-level description text includes the abnormal category text corresponding to the abnormality in the medical sample image, the phrase-level description text includes the abnormal position text corresponding to the abnormality in the medical sample image, and the sentence-level description text includes the abnormal semantic text corresponding to the abnormality in the medical sample image. It can be understood that in this application, three training tasks are designed for the initial pre-training model, and the training objectives of each training task are different (that is, the training model can predict the output of abnormal category text, abnormal position text and abnormal semantic text), and the interpretability of the model can be improved by gradually training the model.

[0087] Among them, the abnormal category text indicates the category of abnormal tissue in the medical sample image. For example, if the lung lobe is shown in the medical sample image, the abnormal category text is "lung lobe", which is convenient for subsequent rapid classification and indexing; the abnormal position text indicates the position corresponding to the category of abnormal tissue in the medical sample image. For example, if the lung lobe is shown in the medical sample image, the abnormal position text is "right upper lobe", which can show the specific position of the abnormal tissue in the medical sample image. Compared with the abnormal category text, the description of the abnormal position text is more specific, which helps to understand the context of the abnormal tissue in the medical sample image; the abnormal semantic text indicates the category and location of the abnormal tissue in the medical sample image, as well as the morphology and size of the abnormal tissue. For example, if the lung lobe is shown in the medical sample image, the abnormal semantic text is "strip-like shadows and scars can be seen in the right upper lobe". Compared with the abnormal position text, the description of the abnormal semantic text is more specific, which provides comprehensive information about the abnormal tissue and can provide medical personnel with explainable prediction content.

[0088] like Figure 2 As shown, Figure 2 An image schematic diagram of phased prediction in a pre-training method of a medical multimodal model based on RadFM provided in an embodiment of the present application. Figure 2 (a) shows that the label-level description text corresponding to the medical sample image can be predicted by the image visual features corresponding to the medical sample image, namely, "lobe", "lung", "scar", "opaque" and "apex of lung"; Figure 2 (b) shows that the phrase-level description text corresponding to the medical sample image can be predicted by the local image features corresponding to the medical sample image, namely, "right upper lobe", "left lung apex", "interstitial fibrosis" and "cavitary lesion"; Figure 2 (c) shows that the sentence-level description text corresponding to the medical sample image can be predicted by the global image features corresponding to the medical sample image, namely, "strip-like shadows and scars are visible in the right upper lobe" and "irregular shadows are visible in the left lung apex". It should be noted that in this application, the label-level description text and the phrase-level description text can not only indicate the category and location of abnormal tissue, but also the abnormal conditions of the abnormal tissue (i.e., "opaque", "cavitary lesions", etc.). And what is proposed in this application is only a partial example, which is not specifically limited here.

[0089] It should also be noted that in the present application, the backbone network of the initial pre-trained model includes RadFM, wherein the RadFM can be understood as a multimodal medical image generation model, that is, in the present application, the model that already has the ability to generate medical images can be pre-trained again, and a model with better image language generation capabilities can be obtained, and the training efficiency of the model can be improved to a certain extent. And the model structure of the initial pre-trained model includes a visual encoder, a perceptron module, and a large language model, so that the initial pre-trained model can be pre-trained to obtain image text generated based on the input medical image. And the training tasks of the first prediction stage in the present application can also include mask target classification, image feature random shuffling, and cross-modal image-text comparison learning, the training tasks of the second prediction stage can also include mask target classification, image feature random shuffling, and image-text matching, and the training tasks of the third prediction stage can also include mask target classification, image feature random shuffling, cross-modal image sentence topic tasks, and image-based language models.

[0090] Among them, masked target classification can be used to predict the feature category and attribute label of the masked target; random shuffling of image features can be used to randomly change the feature order in the local image triplet according to a preset probability to enhance the robustness of the model to align the image target features and their semantic labels; cross-modal image-text contrast learning can be used to calculate the image-text contrast loss function and align the visual and text feature spaces to obtain better image and text representations; image-text matching can be used to learn multimodal representations of images and texts to capture fine-grained alignment of vision and language; cross-modal image-sentence topic tasks can be used to predict the information content jointly described by the image and sentence text; image-based language models can be used to generate corresponding description text for a given image, which has the generalization ability to convert visual information into coherent text.

[0091] In another achievable implementation, each set of training data also includes medical sample questions for medical sample images. Specifically, for each set of training data, the image visual features, image local features, and image global features corresponding to the medical sample images can be predicted in stages based on the medical sample questions through the initial pre-training model to obtain a set of predicted description texts corresponding to the medical sample images. That is, the initial pre-training model can predict the medical sample images in stages under the influence of the medical sample questions, so that in the present application, it is possible to train and obtain a medical multimodal pre-training model that can generate medical description texts based only on medical images, and it is also possible to train and obtain a medical multimodal pre-training model that can generate medical description texts based on medical images and medical questions, which improves the reliability of the model-generated text to a certain extent.

[0092] S104: For the multiple sets of training data, according to the differences between the multiple medical description text sets and the multiple prediction description text sets, the parameters of the initial pre-training model are adjusted until the adjusted model meets the pre-training cutoff condition, and the pre-training is completed to obtain a medical multimodal pre-training model.

[0093] In this step, a set of training data corresponds to a medical description text set and a prediction description text set, that is, the parameters of the initial pre-training model can be adjusted according to the medical description text set and the prediction description text set corresponding to the multiple sets of training data. The pre-training cutoff condition includes the training cutoff condition of the first prediction stage, the training cutoff condition of the second prediction stage, and the training cutoff condition of the third prediction stage, so that the parameters of the initial pre-training model can be adjusted under the joint action of these three training cutoff conditions until a medical multimodal pre-training model is obtained.

[0094] On the basis of obtaining the medical multimodal pre-trained model, in this application, the medical multimodal pre-trained model can also be fine-tuned based on the medical knowledge graph to further improve the prediction accuracy and interpretability of the model, thereby achieving accurate identification and interpretation of abnormal tissues in medical images. And in this application, the medical multimodal pre-trained model can also be fine-tuned based on the retrieval enhancement method, so that the external explicit knowledge and model parameters can be combined for joint prediction, so that the model can further generate reliable medical description text, thereby alleviating the model hallucination problem existing in the related technology.

[0095] like Figure 3 As shown, Figure 3 A flowchart of model training in a pre-training method of a medical multimodal model based on RadFM provided in an embodiment of the present application. Figure 3 First, the initial pre-trained model can be pre-trained to obtain a medical multimodal pre-trained model after pre-training. On this basis, the medical multimodal pre-trained model can also be fine-tuned to obtain a fine-tuned model for subsequent downstream tasks. It should be noted that in the present application, downstream tasks can be selected in actual applications to achieve model fine-tuning, and the downstream tasks are not limited here.

[0096] In summary, in this embodiment, the initial pre-trained model can perform stage-by-stage predictions on the image visual features, image local features, and image global features corresponding to the medical sample images, so that the initial pre-trained model can learn the image granularity information of the image corresponding to different image features, and can output the corresponding prediction description text in each prediction stage, which clarifies the internal prediction logic of the model to a certain extent, thereby improving the interpretability of the model prediction, as well as the reliability of the model prediction, thereby also enabling the prediction effect of the medical multimodal pre-training model to be improved.

[0097] The following is an introduction to a pre-training device for a medical multimodal model based on RadFM provided in an embodiment of the present application. The pre-training device for a medical multimodal model based on RadFM described below and the pre-training method for a medical multimodal model based on RadFM described above can be referred to each other. Figure 4 , which is a schematic diagram of the structure of a pre-training device for a medical multimodal model based on RadFM provided in an embodiment of the present application, such as Figure 4 As shown, the pre-training device of the RadFM-based medical multimodal model includes:

[0098] A multimodal data acquisition unit 401 is used to acquire medical multimodal data, wherein the medical multimodal data includes multiple sets of training data, each set of training data includes a medical sample image and a medical description text set corresponding to the medical sample image;

[0099] An image feature extraction unit 402 is used to extract features of the medical sample image for each set of training data, and obtain image visual features, image local features, and image global features corresponding to the medical sample image;

[0100] The feature phased prediction unit 403 is used to predict the image visual features, image local features and image global features corresponding to the medical sample image in phases through the initial pre-training model for each set of training data, and obtain a set of predicted description texts corresponding to the medical sample image, wherein one predicted description text in the predicted description text set corresponds to a prediction result of one phase, and the backbone network of the initial pre-training model includes RadFM;

[0101] The pre-training model obtaining unit 404 is used to adjust the parameters of the initial pre-training model according to the differences between the multiple medical description text sets and the multiple prediction description text sets for the multiple sets of training data, until the adjusted model meets the pre-training cutoff condition, and the pre-training is completed to obtain a medical multimodal pre-training model.

[0102] In an achievable implementation, the feature stage-by-stage prediction unit 403 is specifically used to:

[0103] In the first prediction stage, the image visual features corresponding to the medical sample image are predicted by an initial pre-trained model to obtain a label-level description text corresponding to the medical sample image;

[0104] In the second prediction stage, the local image features corresponding to the medical sample image are predicted by the initial pre-training model to obtain a phrase-level description text corresponding to the medical sample image;

[0105] In the third prediction stage, the global image features corresponding to the medical sample image are predicted by the initial pre-trained model to obtain a sentence-level description text corresponding to the medical sample image;

[0106] The tag-level description text, the phrase-level description text and the sentence-level description text are used as a predicted description text set corresponding to the medical sample image.

[0107] In one feasible embodiment, the device further includes:

[0108] A target training data acquisition unit, configured to acquire target training data and a plurality of initial medical images, wherein the target training data includes a target medical image and a target description text set corresponding to the target medical image, and the plurality of initial medical images are obtained by generating an adversarial network;

[0109] An initial medical image processing unit, configured to process the multiple initial medical images according to the target medical image and the target description text set, to obtain medical sample images corresponding to the multiple initial medical images respectively;

[0110] The multimodal data acquisition unit 401 is specifically used for:

[0111] The target description text set is assigned to multiple medical sample images to obtain multiple groups of training data, and the multiple groups of training data are used as medical multimodal data, wherein the target description text set and one medical sample image constitute a group of training data.

[0112] In an achievable implementation, the initial medical image processing unit is specifically used to:

[0113] Determining abnormal image features of abnormalities in the target medical image according to the target description text set;

[0114] The abnormal image feature and the multiple initial medical images are fused to obtain medical sample images corresponding to the multiple initial medical images.

[0115] In an achievable implementation, the label-level description text includes abnormality category text corresponding to the abnormality in the medical sample image, the phrase-level description text includes abnormality position text corresponding to the abnormality in the medical sample image, and the sentence-level description text includes abnormality semantic text corresponding to the abnormality in the medical sample image.

[0116] In an achievable implementation, the feature stage-by-stage prediction unit 403 is specifically used to:

[0117] For each set of training data, the image visual features, image local features and image global features corresponding to the medical sample images are predicted in stages based on the medical sample problem through an initial pre-training model to obtain a predicted description text set corresponding to the medical sample images.

[0118] In an achievable implementation, the image feature extraction unit 402 is specifically used to:

[0119] Extracting visual features of the medical sample image to obtain image visual features corresponding to the medical sample image;

[0120] Performing local feature extraction on the medical sample image to obtain local image features corresponding to the medical sample image;

[0121] Perform global feature extraction on the medical sample image to obtain image global features corresponding to the medical sample image.

[0122] The pre-training device of the medical multimodal model based on RadFM provided in the embodiment of the present application has the same beneficial effects as the pre-training method of the medical multimodal model based on RadFM provided in the above-mentioned embodiment, so they are not described in detail.

[0123] Electronic device embodiment

[0124] See also Figure 5 , which is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application, such as Figure 5 As shown, including:

[0125] A memory 11, used for storing computer programs;

[0126] The processor 12 is used to implement the steps of the pre-training method of the medical multimodal model based on RadFM described in any of the above method embodiments when executing the computer program.

[0127] In this embodiment, the device may be a vehicle-mounted computer, a PC (Personal Computer), or a terminal device such as a smart phone, a tablet computer, a PDA, or a portable computer.

[0128] The device may include a memory 11, a processor 12 and a bus 13. Among them, the memory 11 includes at least one type of readable storage medium, and the readable storage medium includes a flash memory, a hard disk, a multimedia card, a card-type memory (for example, SD or DX memory, etc.), a magnetic memory, a disk, an optical disk, etc. In some embodiments, the memory 11 may be an internal storage unit of the device, such as a hard disk of the device. In other embodiments, the memory 11 may also be an external storage device of the device, such as a plug-in hard disk equipped on the device, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. Further, the memory 11 may also include both an internal storage unit of the device and an external storage device. The memory 11 can not only be used to store application software and various types of data installed on the device, such as program code for executing a pre-training method of a medical multimodal model based on RadFM, but can also be used to temporarily store data that has been output or is to be output.

[0129] In some embodiments, the processor 12 can be a central processing unit (CPU), a controller, a microcontroller, a microprocessor or other data processing chip, used to run the program code stored in the memory 11 or process data, such as executing the program code of the pre-training method of the medical multimodal model based on RadFM.

[0130] The bus 13 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0131] Furthermore, the device may also include a network interface 14, which may optionally include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is generally used to establish a communication connection between the device and other electronic devices.

[0132] Optionally, the device may further include a user interface 15, which may include a display (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 15 may also include a standard wired interface and a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an OLED (Organic Light-Emitting Diode) touch device, etc. The display may also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the device and to display a visual user interface.

[0133] Figure 5 Only the device with components 11-15 is shown, and it can be understood by those skilled in the art that Figure 5 The structure shown does not constitute a limitation of the device, and may include fewer or more components than shown, or combine certain components, or arrange the components differently.

[0134] Readable storage medium embodiment

[0135] The embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the pre-training method of the medical multimodal model based on RadFM described in any of the above method embodiments are implemented. Among them, the storage medium may include: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0136] It should also be noted that the "first" and "second" (if any) in the names such as "first" and "second" mentioned in the embodiments of the present application are only used as name identifiers and do not represent the first or second in order.

[0137] The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0138] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0139] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0140] The above is a detailed introduction to the pre-training method, device and related products of a medical multimodal model based on RadFM provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of the present application.

Claims

1. A pre-training method for a medical multimodal model based on RadFM, characterized in that: include: Acquiring medical multimodal data, wherein the medical multimodal data includes multiple sets of training data, each set of training data includes a medical sample image and a medical description text set corresponding to the medical sample image; For each set of training data, feature extraction is performed on the medical sample image to obtain image visual features, image local features and image global features corresponding to the medical sample image; For each set of training data, the image visual features, image local features and image global features corresponding to the medical sample image are predicted in stages through an initial pre-training model to obtain a set of predicted description texts corresponding to the medical sample image, wherein one predicted description text in the predicted description text set corresponds to a prediction result of one stage, and the backbone network of the initial pre-training model includes RadFM; For the multiple groups of training data, according to the differences between multiple medical description text sets and multiple prediction description text sets, the parameters of the initial pre-training model are adjusted until the adjusted model meets the pre-training cutoff condition, and the pre-training is completed to obtain a medical multimodal pre-training model.

2. The method according to claim 1, characterized in that The method of predicting the image visual features, image local features and image global features corresponding to the medical sample image in stages through the initial pre-training model to obtain a predicted description text set corresponding to the medical sample image includes: In the first prediction stage, the image visual features corresponding to the medical sample image are predicted by an initial pre-trained model to obtain a label-level description text corresponding to the medical sample image; In the second prediction stage, the local image features corresponding to the medical sample image are predicted by the initial pre-training model to obtain a phrase-level description text corresponding to the medical sample image; In the third prediction stage, the global image features corresponding to the medical sample image are predicted by the initial pre-trained model to obtain a sentence-level description text corresponding to the medical sample image; The tag-level description text, the phrase-level description text and the sentence-level description text are used as a predicted description text set corresponding to the medical sample image.

3. The method according to claim 1, characterized in that Before acquiring the medical multimodal data, the method further includes: Acquire target training data and a plurality of initial medical images, wherein the target training data includes a target medical image and a target description text set corresponding to the target medical image, and the plurality of initial medical images are obtained by generating an adversarial network; According to the target medical image and the target description text set, the multiple initial medical images are processed to obtain medical sample images corresponding to the multiple initial medical images respectively; The acquiring of medical multimodal data comprises: The target description text set is assigned to multiple medical sample images to obtain multiple groups of training data, and the multiple groups of training data are used as medical multimodal data, wherein the target description text set and one medical sample image constitute a group of training data.

4. The method according to claim 3, characterized in that The step of processing the multiple initial medical images according to the target medical image and the target description text set to obtain medical sample images respectively corresponding to the multiple initial medical images includes: Determining abnormal image features of abnormalities in the target medical image according to the target description text set; The abnormal image feature and the multiple initial medical images are fused to obtain medical sample images corresponding to the multiple initial medical images.

5. The method according to claim 2, characterized in that: The label-level description text includes abnormality category text corresponding to the abnormality in the medical sample image, the phrase-level description text includes abnormality position text corresponding to the abnormality in the medical sample image, and the sentence-level description text includes abnormality semantic text corresponding to the abnormality in the medical sample image.

6. The method according to claim 1, characterized in that Each set of training data also includes a medical sample problem for the medical sample image. For each set of training data, the image visual features, image local features and image global features corresponding to the medical sample image are predicted in stages through an initial pre-training model to obtain a predicted description text set corresponding to the medical sample image, including: For each set of training data, the image visual features, image local features and image global features corresponding to the medical sample images are predicted in stages based on the medical sample problem through an initial pre-training model to obtain a predicted description text set corresponding to the medical sample images.

7. The method according to claim 1, characterized in that The extracting features of the medical sample image to obtain image visual features, image local features and image global features corresponding to the medical sample image includes: Extracting visual features of the medical sample image to obtain image visual features corresponding to the medical sample image; Performing local feature extraction on the medical sample image to obtain local image features corresponding to the medical sample image; Perform global feature extraction on the medical sample image to obtain image global features corresponding to the medical sample image.

8. A pre-training device for a medical multimodal model based on RadFM, characterized in that: include: A multimodal data acquisition unit, used to acquire medical multimodal data, wherein the medical multimodal data includes multiple sets of training data, each set of training data includes a medical sample image and a set of medical description texts corresponding to the medical sample image; An image feature extraction unit, used for performing feature extraction on the medical sample image for each set of training data, to obtain image visual features, image local features and image global features corresponding to the medical sample image; A feature phased prediction unit, for predicting the image visual features, image local features and image global features corresponding to the medical sample image in phases through an initial pre-training model for each set of training data, to obtain a set of predicted description texts corresponding to the medical sample image, wherein one predicted description text in the predicted description text set corresponds to a prediction result of one phase, and the backbone network of the initial pre-training model includes RadFM; The pre-training model obtaining unit is used to adjust the parameters of the initial pre-training model according to the differences between the multiple medical description text sets and the multiple prediction description text sets for the multiple sets of training data, until the adjusted model meets the pre-training cutoff condition, and the pre-training is completed to obtain a medical multimodal pre-training model.

9. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the pre-training method of the medical multimodal model based on RadFM as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the pre-training method of the medical multimodal model based on RadFM as described in any one of claims 1 to 7 are implemented.

Images