A pre-training method, device, and related products for a RadFM-based medical multimodal model.

Abstract

This application discloses a pre-training method, apparatus, and related products for a RadFM-based medical multimodal model. For each set of training data, an initial pre-trained model performs staged predictions on the visual features, local features, and global features of the medical sample image, obtaining a set of predicted descriptive texts corresponding to the medical sample image. Each predicted descriptive text in the set corresponds to a prediction result for one stage. For multiple sets of training data, the parameters of the initial pre-trained model are adjusted based on the differences between multiple medical descriptive text sets and multiple predicted descriptive text sets until the adjusted model meets the pre-training cutoff condition. Pre-training then concludes, resulting in a medical multimodal pre-trained model. Thus, this application obtains a medical multimodal pre-trained model through staged training, enabling the model to learn information at different granularities of the image, thereby improving the interpretability of the model's predictions.

Description

Technical Field

[0001] This application relates to the field of medical model training technology, and in particular to a pre-training method, device and related products for a medical multimodal model based on RadFM. Background Technology

[0002] Existing multimodal models typically employ black-box techniques such as deep learning to predict medical data and obtain predictive results. However, while black-box techniques can improve prediction accuracy to some extent, their opaque internal prediction logic results in poor interpretability, making it difficult for medical professionals to trust medical predictions obtained through deep learning. Therefore, improving the interpretability of model predictions is a key issue of concern for those skilled in the art. Summary of the Invention

[0003] To address the aforementioned problems, this application provides a pre-training method, apparatus, and related products for a RadFM-based medical multimodal model, thereby improving the predictive interpretability of the model. The embodiments of this application disclose the following technical solutions:

[0004] Firstly, this application discloses a pre-training method for a RadFM-based medical multimodal model, comprising:

[0005] Acquire medical multimodal data, wherein the medical multimodal data includes multiple sets of training data, each set of training data including medical sample images and a set of medical descriptive texts corresponding to the medical sample images;

[0006] For each set of training data, feature extraction is performed on the medical sample images to obtain the image visual features, image local features, and image global features corresponding to the medical sample images;

[0007] For each set of training data, the image visual features, image local features and image global features corresponding to the medical sample image are predicted in stages through the initial pre-trained model to obtain the predicted description text set corresponding to the medical sample image. One predicted description text in the predicted description text set corresponds to the prediction result of one stage. The backbone network of the initial pre-trained model includes RadFM.

[0008] For the multiple sets of training data, the parameters of the initial pre-trained model are adjusted according to the differences between multiple medical description text sets and multiple predicted description text sets until the adjusted model meets the pre-training cutoff condition, and the pre-training ends to obtain a medical multimodal pre-trained model.

[0009] In one feasible implementation, the step of performing staged predictions on the visual features, local features, and global features corresponding to the medical sample image using an initial pre-trained model to obtain a predicted descriptive text set corresponding to the medical sample image includes:

[0010] In the first prediction stage, the visual features of the medical sample image are predicted by the initial pre-trained model to obtain the label-level descriptive text corresponding to the medical sample image;

[0011] In the second prediction stage, the local features of the image corresponding to the medical sample image are predicted by the initial pre-trained model to obtain the phrase-level descriptive text corresponding to the medical sample image;

[0012] In the third prediction stage, the global features of the image corresponding to the medical sample image are predicted by the initial pre-trained model to obtain the sentence-level description text corresponding to the medical sample image;

[0013] The tag-level description text, the phrase-level description text, and the sentence-level description text are used as the predicted description text set corresponding to the medical sample image.

[0014] In one feasible implementation, prior to acquiring the medical multimodal data, the method further includes:

[0015] Acquire target training data and multiple initial medical images, wherein the target training data includes target medical images and target description text sets corresponding to the target medical images, and the multiple initial medical images are obtained through a generative adversarial network;

[0016] Based on the target medical image and the target description text set, the plurality of initial medical images are processed to obtain medical sample images corresponding to the plurality of initial medical images respectively;

[0017] The acquisition of medical multimodal data includes:

[0018] The target description text set is assigned to multiple medical sample images to obtain multiple sets of training data, and the multiple sets of training data are used as medical multimodal data, wherein the target description text set and a medical sample image constitute a set of training data.

[0019] In one feasible implementation, the step of processing the plurality of initial medical images according to the target medical image and the target descriptive text set to obtain medical sample images corresponding to the plurality of initial medical images includes:

[0020] Based on the target description text set, determine the abnormal image features present in the target medical image;

[0021] The abnormal image features and the multiple initial medical images are fused to obtain medical sample images corresponding to the multiple initial medical images respectively.

[0022] In one feasible implementation, the tag-level descriptive text includes anomaly category text corresponding to the anomaly in the medical sample image, the phrase-level descriptive text includes anomaly location text corresponding to the anomaly in the medical sample image, and the sentence-level descriptive text includes anomaly semantic text corresponding to the anomaly in the medical sample image.

[0023] In one feasible implementation, each set of training data further includes a medical sample question for the medical sample image. For each set of training data, a pre-trained model is used to predict the image visual features, local image features, and global image features corresponding to the medical sample image in stages to obtain a predicted descriptive text set corresponding to the medical sample image, including:

[0024] For each set of training data, the initial pre-trained model performs staged predictions on the image visual features, image local features, and image global features corresponding to the medical sample image based on the medical sample problem, thereby obtaining the predicted description text set corresponding to the medical sample image.

[0025] In one feasible implementation, the step of extracting features from the medical sample image to obtain the image visual features, image local features, and image global features corresponding to the medical sample image includes:

[0026] Visual features are extracted from the medical sample image to obtain the image visual features corresponding to the medical sample image;

[0027] Local feature extraction is performed on the medical sample image to obtain the local image features corresponding to the medical sample image;

[0028] Global feature extraction is performed on the medical sample image to obtain the global image features corresponding to the medical sample image.

[0029] Secondly, this application discloses a pre-training device for a RadFM-based medical multimodal model, comprising:

[0030] A multimodal data acquisition unit is used to acquire medical multimodal data, wherein the medical multimodal data includes multiple sets of training data, and each set of training data includes a medical sample image and a set of medical description texts corresponding to the medical sample image;

[0031] The image feature extraction unit is used to extract features from the medical sample image for each set of training data to obtain the image visual features, image local features and image global features corresponding to the medical sample image;

[0032] The feature-stage prediction unit is used to perform staged prediction of the image visual features, image local features and image global features corresponding to the medical sample image for each set of training data through an initial pre-trained model, so as to obtain a set of predicted description texts corresponding to the medical sample image, wherein one predicted description text in the set of predicted description texts corresponds to the prediction result of one stage, and the backbone network of the initial pre-trained model includes RadFM.

[0033] The pre-trained model acquisition unit is used to adjust the parameters of the initial pre-trained model based on the differences between multiple sets of medical description texts and multiple sets of predicted description texts for the multiple sets of training data, until the adjusted model meets the pre-training cutoff condition, and the pre-training ends to obtain a medical multimodal pre-trained model.

[0034] In one feasible implementation, the feature-stage prediction unit is specifically used for:

[0035] In the first prediction stage, the visual features of the medical sample image are predicted by the initial pre-trained model to obtain the label-level descriptive text corresponding to the medical sample image;

[0036] In the second prediction stage, the local features of the image corresponding to the medical sample image are predicted by the initial pre-trained model to obtain the phrase-level descriptive text corresponding to the medical sample image;

[0037] In the third prediction stage, the global features of the image corresponding to the medical sample image are predicted by the initial pre-trained model to obtain the sentence-level description text corresponding to the medical sample image;

[0038] The tag-level description text, the phrase-level description text, and the sentence-level description text are used as the predicted description text set corresponding to the medical sample image.

[0039] In one feasible implementation, the device further includes:

[0040] A target training data acquisition unit is used to acquire target training data and multiple initial medical images, wherein the target training data includes target medical images and target description text sets corresponding to the target medical images, and the multiple initial medical images are obtained through a generative adversarial network;

[0041] An initial medical image processing unit is used to process the plurality of initial medical images according to the target medical image and the target description text set to obtain medical sample images corresponding to the plurality of initial medical images respectively;

[0042] The multimodal data acquisition unit is specifically used for:

[0043] The target description text set is assigned to multiple medical sample images to obtain multiple sets of training data, and the multiple sets of training data are used as medical multimodal data, wherein the target description text set and a medical sample image constitute a set of training data.

[0044] In one feasible implementation, the initial medical image processing unit is specifically used for:

[0045] Based on the target description text set, determine the abnormal image features present in the target medical image;

[0046] The abnormal image features and the multiple initial medical images are fused to obtain medical sample images corresponding to the multiple initial medical images respectively.

[0047] In one feasible implementation, the tag-level descriptive text includes anomaly category text corresponding to the anomaly in the medical sample image, the phrase-level descriptive text includes anomaly location text corresponding to the anomaly in the medical sample image, and the sentence-level descriptive text includes anomaly semantic text corresponding to the anomaly in the medical sample image.

[0048] In one feasible implementation, the feature-stage prediction unit is specifically used for:

[0049] For each set of training data, the initial pre-trained model performs staged predictions on the image visual features, image local features, and image global features corresponding to the medical sample image based on the medical sample problem, thereby obtaining the predicted description text set corresponding to the medical sample image.

[0050] In one feasible implementation, the image feature extraction unit is specifically used for:

[0051] Visual features are extracted from the medical sample image to obtain the image visual features corresponding to the medical sample image;

[0052] Local feature extraction is performed on the medical sample image to obtain the local image features corresponding to the medical sample image;

[0053] Global feature extraction is performed on the medical sample image to obtain the global image features corresponding to the medical sample image.

[0054] Thirdly, embodiments of this application provide an electronic device, including:

[0055] Memory, used to store computer programs;

[0056] A processor is used to implement the steps of the pre-training method for the RadFM-based medical multimodal model when executing the computer program.

[0057] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-described pre-training method for a RadFM-based medical multimodal model.

[0058] Compared with the prior art, this application has the following beneficial effects:

[0059] This application first acquires medical multimodal data, and for each set of training data, extracts features from medical sample images to obtain the corresponding visual features, local features, and global features of the medical sample images. Then, for each set of training data, an initial pre-trained model is used to predict the corresponding visual features, local features, and global features of the medical sample images in stages, obtaining a set of predicted descriptive texts for the medical sample images. Finally, for multiple sets of training data, the parameters of the initial pre-trained model are adjusted based on the differences between multiple sets of medical descriptive texts and multiple sets of predicted descriptive texts until the adjusted model meets the pre-training cutoff condition, at which point the pre-training is completed, resulting in a medical multimodal pre-trained model.

[0060] It should be noted that the medical multimodal data includes multiple sets of training data. Each set of training data includes medical sample images and corresponding sets of medical descriptive texts. One predicted descriptive text in the predicted descriptive text set corresponds to the prediction result at one stage. Therefore, this application enables the initial pre-trained model to predict different image features of medical sample images at different stages. This allows the initial pre-trained model to learn the image granularity information corresponding to different image features, and it can output the corresponding predicted descriptive text at each prediction stage. This clarifies the internal prediction logic of the model to a certain extent, thereby improving the interpretability of the model's predictions. Attached Figure Description

[0061] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0062] Figure 1 A flowchart illustrating a pre-training method for a RadFM-based medical multimodal model provided in this application embodiment;

[0063] Figure 2 A schematic diagram of staged prediction in a pre-training method for a RadFM-based medical multimodal model provided in this application embodiment;

[0064] Figure 3 A flowchart illustrating model training in a RadFM-based pre-training method for a medical multimodal model, provided in this application embodiment;

[0065] Figure 4 A schematic diagram of the structure of a pre-training device for a RadFM-based medical multimodal model provided in this application embodiment;

[0066] Figure 5 This is a schematic diagram of an electronic device structure provided in an embodiment of this application. Detailed Implementation

[0067] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0068] It should be noted that the pre-training method, device and related products of the RadFM-based medical multimodal model provided in this application are for the field of medical model training technology. The above is only an example and does not limit the application field of the method and device provided in this application.

[0069] As described earlier, existing multimodal models typically employ black-box techniques such as deep learning to predict medical data and obtain medical prediction results. However, while black-box techniques can improve the model's prediction accuracy to some extent, their opaque internal prediction logic results in poor interpretability, making it difficult for medical personnel to trust the medical predictions obtained through deep learning. For example, the multimodal model might predict "medical data A" to obtain "the prediction result A corresponding to medical data A," but in this case, prediction result A is merely a conclusion, leading to low trust from medical personnel. Therefore, improving the predictive interpretability of the model is a key issue of concern for those skilled in the art.

[0070] Therefore, the inventors propose the technical solution of this application to improve the predictive interpretability of the model. Specifically, in this application, firstly, for each set of training data in the medical multimodal data, feature extraction is performed on the medical sample images to obtain the image visual features, image local features, and image global features corresponding to the medical sample images; then, for each set of training data in the medical multimodal data, the image visual features, image local features, and image global features corresponding to the medical sample images are predicted in stages using an initial pre-trained model to obtain the predicted descriptive text set corresponding to the medical sample images; finally, for multiple sets of training data in the medical multimodal data, the parameters of the initial pre-trained model are adjusted according to the differences between multiple medical descriptive text sets and multiple predicted descriptive text sets until the adjusted model meets the pre-training cutoff condition, and the pre-training ends to obtain the medical multimodal pre-trained model.

[0071] As can be seen, in this application, the initial pre-trained model can make staged predictions on the visual features, local features, and global features of the medical sample image. This allows the initial pre-trained model to learn the image granularity information corresponding to different image features, and can output corresponding prediction description text at each prediction stage. This clarifies the internal prediction logic of the model to a certain extent, thereby improving the interpretability of the model prediction and enhancing its reliability.

[0072] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0073] The following examples illustrate a pre-training method for a RadFM-based medical multimodal model provided in this application. See also... Figure 1 The figure is a flowchart of a pre-training method for a RadFM-based medical multimodal model provided in an embodiment of this application. Figure 1 As shown, the method includes:

[0074] S101: Acquire medical multimodal data.

[0075] It should be noted that the medical multimodal data includes multiple sets of training data. Each set of training data includes medical sample images and a set of medical descriptive texts corresponding to the medical sample images. Each medical descriptive text in the set of medical descriptive texts is a text that describes different image features in the medical sample images. The image modality type of the medical sample images in each set of training data may be different. The image modality type may be CT modality type or MRI modality type. No specific limitation is made here.

[0076] Furthermore, since the current training model has relatively few medical image samples, this application proposes to increase the number of medical image samples by generating simulated medical image samples through generative adversarial networks (GANs) to improve the model's fitting ability. Specifically, before acquiring medical multimodal data, this application first acquires target training data, which includes target medical images and target descriptive text sets corresponding to the target medical images. It also proposes to generate multiple initial medical images using a GAN, including simulated medical image samples generated by the generator in the GAN.

[0077] Following this, generative adversarial networks (GANs) can be used to extract and process the target description text set to identify anomalous image features (i.e., image features corresponding to diseased tissue) in the target medical image. Finally, the anomalous image features and multiple initial medical images are fused one by one using the GAN to obtain medical sample images corresponding to each initial medical image. This ensures that the medical image retains both normal and diseased tissues. Each initial medical image corresponds to a medical sample image that contains the same anomalous image features, which increases the number of model samples to some extent. It should also be noted that this application can fuse multiple different anomalous image features from a single initial medical image; no specific limitation is made here, and the fusion process can be selected in practical applications.

[0078] Furthermore, since each initial medical image corresponds to a medical sample image that shares the same anomalous image features, the target description text set can be assigned to multiple medical sample images to obtain multiple sets of training data. That is, the target description text set and one medical sample image constitute one set of training data. These multiple sets of training data can be used as medical multimodal data, or the target training data and multiple sets of training data can be combined as medical multimodal data, facilitating the subsequent training of medical multimodal pre-trained models.

[0079] S102: For each set of training data, perform feature extraction on the medical sample image to obtain the image visual features, image local features, and image global features corresponding to the medical sample image.

[0080] In this step, visual features are first extracted from the medical sample images to obtain the corresponding visual features. Next, local features are extracted to obtain the corresponding local features. Finally, global features are extracted to obtain the corresponding global features. Thus, before pre-training the initial pre-trained model, features can be extracted from the medical sample images, facilitating the input of the corresponding image features for each stage of the initial pre-trained model during staged prediction.

[0081] It should be noted that this application does not limit the feature extraction process of medical sample images. For example, local features of the medical sample image can be extracted first, or global features of the medical sample image can be extracted first. Specifically, visual features are used for model training in the first prediction stage of the initial pre-trained model, local features are used for model training in the second prediction stage, and global features are used for model training in the third prediction stage.

[0082] It should also be noted that this application can extract features from medical sample images using methods such as color feature extraction, texture feature extraction, and shape feature extraction to obtain the corresponding visual features of the medical sample images. These visual features enable the initial pre-trained model to predict the abnormal category text corresponding to anomalies in the medical sample images. This application can also extract local image features from medical sample images using algorithms such as FAST. These local image features enable the initial pre-trained model to predict the abnormal location text corresponding to anomalies in the medical sample images. Furthermore, this application can extract global image features from medical sample images using image segmentation algorithms. These global image features enable the initial pre-trained model to predict the abnormal semantic text corresponding to anomalies in the medical sample images. It should be noted that this application does not limit the feature extraction methods; in practical applications, the feature extraction method can be selected according to the actual situation.

[0083] S103: For each set of training data, the visual features, local features and global features of the image corresponding to the medical sample image are predicted in stages using the initial pre-trained model to obtain the predicted description text set corresponding to the medical sample image.

[0084] In this step, a prediction description text in the prediction description text set corresponds to the prediction result of one stage, and the stages include the first stage, the second stage, and the third stage. The following describes the staged prediction process for image features of medical sample images in this application.

[0085] First, in the initial prediction stage, the visual features of the medical sample image are predicted using an initial pre-trained model to obtain the corresponding label-level descriptive text. Then, in the second prediction stage, the local features of the medical sample image are predicted using the initial pre-trained model to obtain the corresponding phrase-level descriptive text. Finally, in the third prediction stage, the global features of the medical sample image are predicted using the initial pre-trained model to obtain the corresponding sentence-level descriptive text. The label-level, phrase-level, and sentence-level descriptive texts can then be used as the predicted descriptive text set for the medical sample image. That is, in each prediction stage, the descriptive text corresponding to that stage can be predicted, and the descriptive texts corresponding to all prediction stages are then used as the predicted descriptive text set for the medical sample image. Thus, in this application, the model can learn different image granularity information at different stages, which to some extent supports the interpretability of the model's predictions.

[0086] It should be noted that the label-level descriptive text includes the anomaly category text corresponding to the anomaly in the medical sample image; the phrase-level descriptive text includes the anomaly location text corresponding to the anomaly in the medical sample image; and the sentence-level descriptive text includes the anomaly semantic text corresponding to the anomaly in the medical sample image. Understandably, this application designs three training tasks for the initial pre-trained model, each with a different training objective (i.e., training the model to predict and output anomaly category text, anomaly location text, and anomaly semantic text). The interpretability of the model can be improved through the process of progressively training the model.

[0087] Among them, the anomaly category text indicates the category of abnormal tissue present in the medical sample image. For example, if a lung lobe is shown in the medical sample image, the anomaly category text would be "lung lobe," which facilitates subsequent rapid classification and indexing. The anomaly location text indicates the location of the abnormal tissue category in the medical sample image. For example, if a lung lobe is shown in the medical sample image, the anomaly location text would be "right upper lobe," which shows the specific location of the abnormal tissue in the medical sample image. Compared to the anomaly category text, the anomaly location text is more specific and helps to understand the context of the abnormal tissue in the medical sample image. The anomaly semantic text indicates the category and location of the abnormal tissue in the medical sample image, as well as the morphology and size of the abnormal tissue. For example, if a lung lobe is shown in the medical sample image, the anomaly semantic text would be "a linear shadow is visible in the right upper lobe, and a scar is visible." Compared to the anomaly location text, the anomaly semantic text is more specific and provides comprehensive information about the abnormal tissue, which can provide interpretable predictive content for medical personnel.

[0088] like Figure 2 As shown, Figure 2 This is a schematic diagram of a staged prediction in a pre-training method for a RadFM-based medical multimodal model provided in an embodiment of this application. Figure 2 (a) shows that the label-level descriptive text corresponding to a medical sample image can be predicted using the image visual features corresponding to the medical sample image, namely “lung lobe”, “lung”, “scar”, “opaque” and “lung apex”. Figure 2 (b) shows that phrase-level descriptive text corresponding to medical sample images can be predicted using local image features, namely “right upper lobe”, “left apex”, “interstitial fibrosis” and “cavitary lesion”. Figure 2 (c) illustrates how sentence-level descriptive text corresponding to a medical sample image can be predicted using the global image features, namely, "linear shadows and scars visible in the upper right lobe" and "irregular shadows visible in the apex of the left lung." It should be noted that in this application, the label-level and phrase-level descriptive texts can not only indicate the category and location of abnormal tissue, but also the abnormal conditions present in the abnormal tissue (i.e., "opaque," "cavitary lesion," etc.). Furthermore, the examples presented in this application are only partial examples and are not specifically limited herein.

[0089] It should also be noted that the backbone network of the initial pre-trained model in this application includes RadFM, which can be understood as a multimodal medical image generation model. This means that the model already capable of generating medical images can be re-pre-trained to obtain a model with better image-to-text generation capabilities, thus improving training efficiency to some extent. The model structure of the initial pre-trained model includes a visual encoder, a perceptron module, and a large language model, facilitating pre-training to obtain image-to-text generated from input medical images. Furthermore, the training tasks in the first prediction stage of this application may also include masked object classification, random image feature shuffling, and cross-modal image-to-text comparison learning; the training tasks in the second prediction stage may also include masked object classification, random image feature shuffling, and image-to-text matching; and the training tasks in the third prediction stage may also include masked object classification, random image feature shuffling, cross-modal image-to-sentence topic task, and image-based language modeling.

[0090] Among these, masked target classification can be used to predict the feature category and attribute label of the masked target; random shuffling of image features can be used to randomly change the feature order in local image triples according to a preset probability to enhance the robustness of the model in aligning image target features and their semantic labels; cross-modal image-text contrast learning can be used to calculate the image-text contrast loss function, align the visual and text feature spaces to obtain higher quality image and text representations; image-text matching can be used to learn multimodal representations of images and text to capture fine-grained alignment between vision and language; cross-modal image-sentence topic task can be used to predict the information content jointly described by the image and sentence text; and image-based language model can be used to generate corresponding descriptive text for a given image, which has the generalization ability to convert visual information into coherent text.

[0091] In another feasible implementation, each set of training data also includes a medical sample question for the medical sample image. Specifically, for each set of training data, the initial pre-trained model can perform staged predictions on the visual features, local features, and global features of the corresponding medical sample image based on the medical sample question, thereby obtaining a set of predicted descriptive texts for the medical sample image. This means that the initial pre-trained model can perform staged predictions on the medical sample image under the influence of the medical sample question. This allows for the training of both a multimodal medical pre-trained model capable of generating medical descriptive text solely based on medical images and a multimodal medical pre-trained model capable of generating medical descriptive text based on both medical images and medical questions, thus improving the reliability of the generated text to a certain extent.

[0092] S104: For the multiple sets of training data, adjust the parameters of the initial pre-trained model according to the differences between multiple medical description text sets and multiple predicted description text sets until the adjusted model meets the pre-training cutoff condition, and the pre-training ends to obtain the medical multimodal pre-trained model.

[0093] In this step, a set of training data corresponds to a medical descriptive text set and a predicted descriptive text set. This means that the parameters of the initial pre-trained model can be adjusted based on the medical descriptive text sets and predicted descriptive text sets corresponding to multiple sets of training data. The pre-training cutoff conditions include training cutoff conditions for the first prediction stage, the second prediction stage, and the third prediction stage. These three training cutoff conditions work together to adjust the parameters of the initial pre-trained model until a medical multimodal pre-trained model is obtained.

[0094] Building upon the existing medical multimodal pre-trained model, this application further refines the model using a medical knowledge graph to enhance its predictive accuracy and interpretability, thereby enabling accurate identification and interpretation of abnormal tissues in medical images. Additionally, this application utilizes retrieval enhancement techniques to fine-tune the medical multimodal pre-trained model, allowing for joint prediction by combining external explicit knowledge and model parameters. This further enables the model to generate reliable medical descriptive text, mitigating the model illusion problem present in related technologies.

[0095] like Figure 3 As shown, Figure 3 This document presents a flowchart illustrating model training in a RadFM-based pre-training method for a medical multimodal model, as provided in an embodiment of this application. Figure 3 The first step is to pre-train the initial pre-trained model to obtain a pre-trained medical multimodal model. Based on this, the pre-trained model can be fine-tuned to obtain a fine-tuned model, which is convenient for subsequent downstream tasks. It should be noted that in this application, downstream tasks can be selected in practical applications to achieve model fine-tuning; no restrictions are placed on the downstream tasks.

[0096] In summary, this embodiment enables the initial pre-trained model to predict the visual features, local features, and global features of the medical sample image in stages. This allows the initial pre-trained model to learn the image granularity information corresponding to different image features, and it can output corresponding prediction description text at each prediction stage. This clarifies the internal prediction logic of the model to a certain extent, thereby improving the interpretability and reliability of the model prediction, and ultimately enhancing the prediction performance of the medical multimodal pre-trained model.

[0097] The following describes a pre-training device for a RadFM-based medical multimodal model provided in an embodiment of this application. The RadFM-based pre-training device described below corresponds to and can be referred to in conjunction with the RadFM-based pre-training method described above. See also... Figure 4 The figure is a schematic diagram of the structure of a pre-training device for a RadFM-based medical multimodal model provided in an embodiment of this application. Figure 4 As shown, the pre-training device for the RadFM-based medical multimodal model includes:

[0098] The multimodal data acquisition unit 401 is used to acquire medical multimodal data, wherein the medical multimodal data includes multiple sets of training data, and each set of training data includes a medical sample image and a set of medical description texts corresponding to the medical sample image;

[0099] The image feature extraction unit 402 is used to extract features from the medical sample image for each set of training data to obtain the image visual features, image local features and image global features corresponding to the medical sample image;

[0100] The feature-stage prediction unit 403 is used to perform staged prediction of the image visual features, image local features and image global features corresponding to the medical sample image for each set of training data through an initial pre-trained model, so as to obtain a set of predicted description texts corresponding to the medical sample image, wherein one predicted description text in the set of predicted description texts corresponds to the prediction result of one stage, and the backbone network of the initial pre-trained model includes RadFM.

[0101] The pre-trained model acquisition unit 404 is used to adjust the parameters of the initial pre-trained model based on the differences between multiple sets of medical description texts and multiple sets of predicted description texts for the multiple sets of training data, until the adjusted model meets the pre-training cutoff condition, and the pre-training ends to obtain a medical multimodal pre-trained model.

[0102] In one feasible implementation, the feature-stage prediction unit 403 is specifically used for:

[0103] In the first prediction stage, the visual features of the medical sample image are predicted by the initial pre-trained model to obtain the label-level descriptive text corresponding to the medical sample image;

[0104] In the second prediction stage, the local features of the image corresponding to the medical sample image are predicted by the initial pre-trained model to obtain the phrase-level descriptive text corresponding to the medical sample image;

[0105] In the third prediction stage, the global features of the image corresponding to the medical sample image are predicted by the initial pre-trained model to obtain the sentence-level description text corresponding to the medical sample image;

[0106] The tag-level description text, the phrase-level description text, and the sentence-level description text are used as the predicted description text set corresponding to the medical sample image.

[0107] In one feasible implementation, the device further includes:

[0108] A target training data acquisition unit is used to acquire target training data and multiple initial medical images, wherein the target training data includes target medical images and target description text sets corresponding to the target medical images, and the multiple initial medical images are obtained through a generative adversarial network;

[0109] An initial medical image processing unit is used to process the plurality of initial medical images according to the target medical image and the target description text set to obtain medical sample images corresponding to the plurality of initial medical images respectively;

[0110] The multimodal data acquisition unit 401 is specifically used for:

[0111] The target description text set is assigned to multiple medical sample images to obtain multiple sets of training data, and the multiple sets of training data are used as medical multimodal data, wherein the target description text set and a medical sample image constitute a set of training data.

[0112] In one feasible implementation, the initial medical image processing unit is specifically used for:

[0113] Based on the target description text set, determine the abnormal image features present in the target medical image;

[0114] The abnormal image features and the multiple initial medical images are fused to obtain medical sample images corresponding to the multiple initial medical images respectively.

[0115] In one feasible implementation, the tag-level descriptive text includes anomaly category text corresponding to the anomaly in the medical sample image, the phrase-level descriptive text includes anomaly location text corresponding to the anomaly in the medical sample image, and the sentence-level descriptive text includes anomaly semantic text corresponding to the anomaly in the medical sample image.

[0116] In one feasible implementation, the feature-stage prediction unit 403 is specifically used for:

[0117] For each set of training data, the initial pre-trained model performs staged predictions on the image visual features, image local features, and image global features corresponding to the medical sample image based on the medical sample problem, thereby obtaining the predicted description text set corresponding to the medical sample image.

[0118] In one feasible implementation, the image feature extraction unit 402 is specifically used for:

[0119] Visual features are extracted from the medical sample image to obtain the image visual features corresponding to the medical sample image;

[0120] Local feature extraction is performed on the medical sample image to obtain the local image features corresponding to the medical sample image;

[0121] Global feature extraction is performed on the medical sample image to obtain the global image features corresponding to the medical sample image.

[0122] The RadFM-based medical multimodal model pre-training device provided in this application embodiment has the same beneficial effects as the RadFM-based medical multimodal model pre-training method provided in the above embodiments, and therefore will not be described again.

[0123] Electronic device examples

[0124] See Figure 5 This figure is a schematic diagram of an electronic device structure provided in an embodiment of this application, such as... Figure 5 As shown, it includes:

[0125] Memory 11 is used to store computer programs;

[0126] The processor 12 is configured to implement the steps of the pre-training method for the RadFM-based medical multimodal model described in any of the above method embodiments when executing the computer program.

[0127] In this embodiment, the device can be an in-vehicle computer, a PC (Personal Computer), or a terminal device such as a smartphone, tablet computer, handheld computer, or portable computer.

[0128] The device may include a memory 11, a processor 12, and a bus 13. The memory 11 includes at least one type of readable storage medium, such as flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), magnetic storage, magnetic disk, optical disk, etc. In some embodiments, the memory 11 may be an internal storage unit of the device, such as the device's hard disk. In other embodiments, the memory 11 may be an external storage device of the device, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, Flash Card, etc. Furthermore, the memory 11 may include both internal and external storage units. The memory 11 can be used not only to store application software and various types of data installed on the device, such as program code executing pre-training methods for RadFM-based medical multimodal models, but also to temporarily store data that has been output or will be output.

[0129] In some embodiments, processor 12 may be a central processing unit (CPU), controller, microcontroller, microprocessor or other data processing chip, used to run program code stored in memory 11 or process data, such as executing program code for a pre-training method of a RadFM-based medical multimodal model.

[0130] This bus 13 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 5 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0131] Furthermore, the device may also include a network interface 14, which may optionally include a wired interface and / or a wireless interface (such as a Wi-Fi interface, a Bluetooth interface, etc.), typically used to establish communication connections between the device and other electronic devices.

[0132] Optionally, the device may further include a user interface 15, which may include a display, an input unit such as a keyboard, and optionally, a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen, etc. The display may also be appropriately referred to as a screen or display unit, used to display information processed in the device and to display a visual user interface.

[0133] Figure 5 Only devices with components 11-15 are shown; those skilled in the art will understand that... Figure 5 The structure shown does not constitute a limitation on the device and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0134] Readable storage medium embodiments

[0135] This application also provides a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the steps of the pre-training method for the RadFM-based medical multimodal model described in any of the above method embodiments. The storage medium may include various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0136] It should also be noted that the "first" and "second" in the names such as "first" and "second" (if they exist) mentioned in the embodiments of this application are only used as name identifiers and do not represent the first and second in order.

[0137] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0138] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0139] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0140] The foregoing has provided a detailed description of a pre-training method, apparatus, and related products for a RadFM-based medical multimodal model. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A pre-training method for a medical multimodal model based on RadFM, characterized in that, include: Acquire medical multimodal data, wherein the medical multimodal data includes multiple sets of training data, each set of training data including medical sample images and a set of medical descriptive texts corresponding to the medical sample images; For each set of training data, color feature extraction, texture feature extraction, and shape feature extraction are performed on the medical sample image to obtain the image visual features corresponding to the medical sample image; local feature extraction is performed on the medical sample image based on the FAST algorithm to obtain the image local features corresponding to the medical sample image; Global feature extraction is performed on the medical sample image to obtain the global image features corresponding to the medical sample image; For each set of training data, the visual features, local features, and global features of the medical sample image are predicted in stages using an initial pre-trained model to obtain a predicted descriptive text set corresponding to the medical sample image, including: In the first prediction stage, the visual features of the medical sample image are predicted by the initial pre-trained model to obtain the label-level descriptive text corresponding to the medical sample image; In the second prediction stage, the local features of the image corresponding to the medical sample image are predicted by the initial pre-trained model to obtain the phrase-level descriptive text corresponding to the medical sample image; In the third prediction stage, the global features of the image corresponding to the medical sample image are predicted by the initial pre-trained model to obtain the sentence-level description text corresponding to the medical sample image; The tag-level description text, the phrase-level description text, and the sentence-level description text are used as the predicted description text set corresponding to the medical sample image; The label-level description text includes the abnormality category text corresponding to the abnormality in the medical sample image; the phrase-level description text includes the abnormality location text corresponding to the abnormality in the medical sample image; the sentence-level description text includes the abnormality semantic text corresponding to the abnormality in the medical sample image; one prediction description text in the prediction description text set corresponds to the prediction result of one stage; and the backbone network of the initial pre-trained model includes RadFM. For the multiple sets of training data, the parameters of the initial pre-trained model are adjusted according to the differences between multiple medical description text sets and multiple predicted description text sets until the adjusted model meets the pre-training cutoff condition, and the pre-training ends to obtain a medical multimodal pre-trained model; the medical multimodal pre-trained model is then fine-tuned based on a medical knowledge graph to obtain a fine-tuned medical multimodal pre-trained model. Prior to acquiring the medical multimodal data, the following is also included: Acquire target training data and multiple initial medical images, wherein the target training data includes target medical images and target description text sets corresponding to the target medical images, and the multiple initial medical images are obtained through a generative adversarial network; Based on generative adversarial networks, the multiple initial medical images are processed according to the target medical image and the target descriptive text set to obtain medical sample images corresponding to the multiple initial medical images respectively.

2. The method according to claim 1, characterized in that, The step of processing the plurality of initial medical images based on the target medical image and the target description text set to obtain medical sample images corresponding to the plurality of initial medical images includes: Based on the target description text set, determine the abnormal image features present in the target medical image; The abnormal image features and the multiple initial medical images are fused to obtain medical sample images corresponding to the multiple initial medical images respectively.

3. The method according to claim 1, characterized in that, Each set of training data also includes medical sample questions for the medical sample images. For each set of training data, the image visual features, local image features, and global image features corresponding to the medical sample images are predicted in stages using an initial pre-trained model to obtain a predicted description text set corresponding to the medical sample images, including: For each set of training data, the initial pre-trained model performs staged predictions on the image visual features, image local features, and image global features corresponding to the medical sample image based on the medical sample problem, thereby obtaining the predicted description text set corresponding to the medical sample image.

4. The method according to claim 1, characterized in that, The acquisition of medical multimodal data includes: The target description text set is assigned to multiple medical sample images to obtain multiple sets of training data, and the multiple sets of training data are used as medical multimodal data, wherein the target description text set and a medical sample image constitute a set of training data.

5. A pre-training device for a RadFM-based medical multimodal model, characterized in that, include: A multimodal data acquisition unit is used to acquire medical multimodal data, wherein the medical multimodal data includes multiple sets of training data, and each set of training data includes a medical sample image and a set of medical description texts corresponding to the medical sample image; The image feature extraction unit is used to extract color features, texture features, and shape features from the medical sample image for each set of training data to obtain the image visual features corresponding to the medical sample image; and to extract local features from the medical sample image based on the FAST algorithm to obtain the image local features corresponding to the medical sample image. Global feature extraction is performed on the medical sample image to obtain the global image features corresponding to the medical sample image; The feature-stage prediction unit is used to perform staged predictions on the image visual features, image local features, and image global features corresponding to the medical sample image for each set of training data, using an initial pre-trained model, to obtain a predicted description text set corresponding to the medical sample image, including: In the first prediction stage, the visual features of the medical sample image are predicted by the initial pre-trained model to obtain the label-level descriptive text corresponding to the medical sample image; In the second prediction stage, the local features of the image corresponding to the medical sample image are predicted by the initial pre-trained model to obtain the phrase-level descriptive text corresponding to the medical sample image; In the third prediction stage, the global features of the image corresponding to the medical sample image are predicted by the initial pre-trained model to obtain the sentence-level description text corresponding to the medical sample image; The tag-level description text, the phrase-level description text, and the sentence-level description text are used as the predicted description text set corresponding to the medical sample image; The label-level description text includes the abnormality category text corresponding to the abnormality in the medical sample image; the phrase-level description text includes the abnormality location text corresponding to the abnormality in the medical sample image; the sentence-level description text includes the abnormality semantic text corresponding to the abnormality in the medical sample image; one prediction description text in the prediction description text set corresponds to the prediction result of one stage; and the backbone network of the initial pre-trained model includes RadFM. The pre-trained model acquisition unit is used to adjust the parameters of the initial pre-trained model based on the differences between multiple medical description text sets and multiple predicted description text sets for the multiple sets of training data, until the adjusted model meets the pre-training cutoff condition, and the pre-training ends to obtain a medical multimodal pre-trained model; the medical multimodal pre-trained model is fine-tuned based on the medical knowledge graph to obtain a fine-tuned medical multimodal pre-trained model; Prior to acquiring the medical multimodal data, the following is also included: Acquire target training data and multiple initial medical images, wherein the target training data includes target medical images and target description text sets corresponding to the target medical images, and the multiple initial medical images are obtained through a generative adversarial network; Based on generative adversarial networks, the multiple initial medical images are processed according to the target medical image and the target descriptive text set to obtain medical sample images corresponding to the multiple initial medical images respectively.

6. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the pre-training method for a RadFM-based medical multimodal model as described in any one of claims 1 to 4 when executing the computer program.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the pre-training method for the RadFM-based medical multimodal model as described in any one of claims 1 to 4.

Images