Multi-modal medical feature extraction model training method and device, equipment and medium

Abstract

The invention provides a multi-modal medical feature extraction model training method and device, equipment and a medium. The method comprises the steps that diagnosis image samples and diagnosis text samples corresponding to a target diagnosis object are acquired, and all the diagnosis image samples and all the diagnosis text samples carry multiple thermal tags representing medical semantic features; determining a first semantic similarity between the diagnosis image sample and the diagnosis text sample according to the multiple hot labels; determining a second semantic similarity between the diagnosis image sample and the diagnosis text sample according to the initial multi-modal medical feature extraction model; the initial multi-modal medical feature extraction model is trained with the purpose of minimizing the difference between the first semantic similarity and the second semantic similarity, a multi-modal medical feature extraction model is obtained, and the multi-modal medical feature extraction model is used for outputting image feature representation and text feature representation with semantic alignment. The model obtained by training through the method has the advantages of being accurate in semantic understanding, high in generalization ability and the like.

Description

Technical Field

[0001] This application relates to the field of image and text processing technology, and in particular to a method, apparatus, device and medium for training a multimodal medical feature extraction model. Background Technology

[0002] In the fields of computer vision and natural language processing, large-scale image-text pre-training, represented by Contrastive Language-Image Pre-training (CLIP), has achieved significant success. It enables joint training of image and text representations on large-scale image-text pairs to generate transferable representations and support flexible downstream tasks. In certain medical fields, such as ophthalmology, medical images acquired through image acquisition techniques like color fundus photography and optical coherence tomography are frequently used in clinical practice for disease diagnosis, monitoring, and treatment. With the rapid growth of medical image and response report data, the medical community's need for ophthalmic multimodal feature extraction models (used to extract image and text feature representations) to support clinical decision-making is also increasing. However, the enormous data requirements of CLIP training pose a significant challenge to this task. Thanks to the abundance of widely available, general-type public data on the internet, CLIP, as a general-purpose image-text pre-trained model, was initially trained on an image-text dataset containing 400 million data points. However, the number of publicly available medical images and reports is orders of magnitude smaller, and paired image-text datasets are relatively even fewer, often making it difficult to fully realize the model's potential. Furthermore, the differences in the medical field are more subtle and nuanced compared to general domains, further increasing the difficulty of model training.

[0003] To address the aforementioned issues, existing multimodal feature extraction models (especially ophthalmology multimodal feature extraction models) largely focus on the global and local interactions between medical images and text reports to capture pathological meaning from specific image domains, thereby achieving the differentiation and alignment of subtle semantic differences. However, these methods still fail to address the problem of limited available image-text pairing data. Furthermore, existing training methods focus more on patient-level alignment, which may lead to false negatives (in the model training process, image-text pairs describing the same condition but from different patients are incorrectly treated as mismatched negative samples). In summary, multimodal feature extraction models trained using existing methods suffer from inaccurate semantic understanding and poor generalization ability. Summary of the Invention

[0004] This application provides a training method, apparatus, device, and medium for a multimodal medical feature extraction model, which solves the problems of limited available image-text pairing data and false negatives in the training of multimodal feature extraction models in the prior art. It can ensure that the trained multimodal feature extraction model has the advantages of accurate semantic understanding and strong generalization ability.

[0005] This application provides a method for training a multimodal medical feature extraction model, including the following steps: Obtain diagnostic image samples and diagnostic text samples corresponding to the target diagnostic object. Each diagnostic image sample and each diagnostic text sample carries a multi-hot tag that represents medical semantic features. Based on the multi-hot tags, a first semantic similarity is determined between the diagnostic image sample and the diagnostic text sample; Based on the initial multimodal medical feature extraction model, the second semantic similarity between the diagnostic image samples and the diagnostic text samples is determined; With the goal of minimizing the difference between the first semantic similarity and the second semantic similarity, the initial multimodal medical feature extraction model is trained to obtain a multimodal medical feature extraction model, which is used to output semantically aligned image feature representations and text feature representations.

[0006] According to the multimodal medical feature extraction model training method provided in this application, determining the second semantic similarity between the diagnostic image sample and the diagnostic text sample includes: Feature extraction is performed on the diagnostic image samples to obtain visual feature vectors; Feature extraction is performed on the diagnostic text sample to obtain a text feature vector; The second semantic similarity is determined based on the similarity between the visual feature vector and the text feature vector.

[0007] According to the multimodal medical feature extraction model training method provided in this application, the initial multimodal medical feature extraction model is trained with the objective of minimizing the difference between the first semantic similarity and the second semantic similarity, including: The first probability distribution corresponding to the first semantic similarity and the second probability distribution corresponding to the second semantic similarity are determined by using a flexible maximum value function. The difference between the first probability distribution and the second probability distribution is determined by the cross-entropy loss function; The initial multimodal medical feature extraction model is trained with the goal of minimizing the differences.

[0008] According to the multimodal medical feature extraction model training method provided in this application, the multi-hot tags of the diagnostic text samples are determined through the following steps: Semantic information is extracted from the diagnostic text sample to obtain a semantic feature vector; Based on the similarity between the semantic feature vector and multiple target semantic feature vectors, at least one target semantic feature vector is determined, and one of the target semantic feature vectors corresponds to a medical semantic tag among the multiple hot tags; Based on the at least one target semantic feature vector, generate multi-hot tags for the diagnostic text sample.

[0009] According to the multimodal medical feature extraction model training method provided in this application, the multiple target semantic feature vectors are determined through the following steps: Obtain multiple standard diagnostic text samples; Semantic information is extracted from the multiple standard diagnostic text samples to obtain multiple semantic feature vectors; Clustering is performed on the semantic feature vectors corresponding to multiple standard diagnostic text samples to obtain multiple clusters; The centroid vector of each cluster is determined based on the semantic feature vector contained in each cluster. Based on each of the central point vectors, the plurality of target semantic feature vectors are determined.

[0010] According to the multimodal medical feature extraction model training method provided in this application, the multi-hot labels of the diagnostic image samples are determined through the following steps: The multi-hot tags corresponding to the diagnostic text samples that match the diagnostic image samples are determined as the multi-hot tags of the diagnostic image samples.

[0011] According to the multimodal medical feature extraction model training method provided in this application, diagnostic image samples and diagnostic text samples corresponding to the target diagnostic object are obtained, including: Obtain the diagnostic image sample corresponding to the target diagnostic object from the diagnostic image sample set and / or the diagnostic image and text pairing sample set; Obtain the diagnostic text sample corresponding to the target diagnostic object from the diagnostic text sample set and / or the diagnostic image and text paired sample set; The diagnostic image sample set includes diagnostic image samples that do not have paired diagnostic text, the diagnostic text sample set includes diagnostic text samples that do not have paired diagnostic images, and the diagnostic image and text pairing sample set includes paired diagnostic image samples and diagnostic text samples.

[0012] This application also provides a multimodal medical feature extraction model training device, comprising: The acquisition module is used to acquire diagnostic image samples and diagnostic text samples corresponding to the target diagnostic object. Each diagnostic image sample and each diagnostic text sample carries a multi-hot tag that represents medical semantic features. The first determining module is used to determine a first semantic similarity between the diagnostic image sample and the diagnostic text sample based on the multi-hot tags; The second determining module is used to determine the second semantic similarity between the diagnostic image sample and the diagnostic text sample based on the initial multimodal medical feature extraction model; The training module is used to train the initial multimodal medical feature extraction model with the goal of minimizing the difference between the first semantic similarity and the second semantic similarity, so as to obtain a multimodal medical feature extraction model, which is used to output semantically aligned image feature representations and text feature representations.

[0013] This application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement a multimodal medical feature extraction model training method as described above.

[0014] This application also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements a multimodal medical feature extraction model training method as described above.

[0015] This application also provides a computer program product, including a computer program that, when executed by a processor, implements a multimodal medical feature extraction model training method as described above.

[0016] The method of this application has at least the following technical effects: First, the multi-hot label generation mechanism employed during training enables the processing of unpaired image-text data. Even independent diagnostic image samples or independent diagnostic text samples can participate in model training by generating multi-hot labels. Therefore, the multi-hot label generation mechanism adopted in this application can greatly expand the data scale of effective training samples, significantly improve data utilization efficiency, solve the problem of limited available paired image-text data in existing technologies, and also avoid the waste of unpaired image or text data caused by strict reliance on paired image-text data in existing technologies.

[0017] Second, this application determines whether two samples match solely based on the similarity of the medical content represented by their multi-hot tags, not on whether they come from the same patient. If images and texts from different patients describe the same symptom, their multi-hot tags will have a high similarity (the first semantic similarity score is high). Based on this, the training process aims to minimize the difference between the first and second semantic similarities. Essentially, this forces the model to learn and accept this semantic consistency across patient identities, thus directly correcting erroneous learning signals caused by false negatives. It guides the model to ignore biases related to patient identity and instead focus on generalizable disease features, effectively eliminating the false negative problem. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating a multimodal medical feature extraction model training method according to an embodiment of this application.

[0020] Figure 2 This is a schematic diagram illustrating the training process of a multimodal medical feature extraction model according to an embodiment of this application.

[0021] Figure 3 This is a schematic diagram illustrating a clustering result in one embodiment of this application.

[0022] Figure 4 This is a structural block diagram of a multimodal medical feature extraction model training device shown in one embodiment of this application.

[0023] Figure 5 This is a schematic diagram of the physical structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0025] The following is combined Figures 1-5This application describes the training method for the multimodal medical feature extraction model.

[0026] To address the problems in existing technologies, this application provides a novel method for training a multimodal medical feature extraction model. The method is implemented using a training device.

[0027] Figure 1 This is a flowchart illustrating a multimodal medical feature extraction model training method according to an embodiment of this application. (Refer to...) Figure 1 The training method for the multimodal medical feature extraction model in this application includes the following steps: Step 101: Obtain diagnostic image samples and diagnostic text samples corresponding to the target diagnostic object. Each diagnostic image sample and each diagnostic text sample carries a multi-hot tag that represents medical semantic features.

[0028] The target diagnostic object can be the eye area or the oral cavity area, etc., and the type of target diagnostic object can be set according to actual needs. Any object that relies on medical images and corresponding text reports for diagnosis can be the target diagnostic object in step 101.

[0029] The model training method proposed in this application can be applied to the training of multimodal medical feature extraction models for multiple clinical departments (such as ophthalmology, dentistry, etc.). This application does not specifically limit the type of clinical department. Any clinical department that heavily relies on medical images and corresponding text reports for diagnosis and faces the problem of scarce high-quality paired data can use the model training method provided in this application to train its required multimodal medical feature extraction model.

[0030] In this embodiment, the multi-hot tag is a vector, and each position in the vector corresponds to a predefined single medical concept. For example, position 1 corresponds to glaucoma, position 2 to drusen, and position 3 to macular edema. If the value of a position is 1, it means that the associated sample data (diagnostic image sample or diagnostic text sample) contains the feature of the medical concept corresponding to that position; if the value of a position is 0, it means that the associated sample data does not contain the feature of the medical concept corresponding to that position. Therefore, the medical semantic features represented by the multi-hot tag are essentially used to indicate which medical concept features the sample data to which the multi-hot tag belongs contains and which medical concept features it does not contain.

[0031] For example, the diagnostic image sample X carries a multi-hot label [0, 1, 1, 0, ...]. If position 1 corresponds to glaucoma, position 2 corresponds to drusen, and position 3 corresponds to macular edema, then the multi-hot label indicates that the diagnostic image sample X contains the features of drusen and macular edema, but does not contain the features of glaucoma.

[0032] In this embodiment, each diagnostic image sample and each diagnostic text sample carries multiple hot tags. These hot tags can be obtained in advance in any way. This embodiment does not impose specific restrictions on the method of obtaining the hot tags.

[0033] In step 101, the number of diagnostic image samples and the number of diagnostic text samples can be the same or different. The specific number can be set according to actual needs, and this embodiment does not impose any restrictions on this.

[0034] Step 102: Determine the first semantic similarity between the diagnostic image samples and the diagnostic text samples based on the multi-hot tags.

[0035] In this embodiment, when performing step 102, for each diagnostic image sample, it is necessary to calculate the similarity between its multi-hot tags and the multi-hot tags carried by each other diagnostic text sample, to obtain a semantic similarity matrix (hereinafter referred to as the first semantic similarity matrix) representing semantic similarity (hereinafter referred to as the first semantic similarity). That is, performing step 102, based on the multi-hot tags, determines the first semantic similarity between each pair of diagnostic image samples and diagnostic text samples, and obtains the first semantic similarity matrix. .

[0036] For example, the number of diagnostic image samples is The number of diagnostic text samples is [number]. If there are 1, then the final first semantic similarity matrix is ​​obtained. Total OK Column, each item in the matrix Indicates the first The multi-hot tags of a diagnostic image sample, with the first Semantic similarity between multiple hot tags of a diagnostic text sample.

[0037] Step 103: Determine the second semantic similarity between diagnostic image samples and diagnostic text samples based on the initial multimodal medical feature extraction model.

[0038] In this embodiment, step 103 is executed: based on the initial multimodal medical feature extraction model, for each diagnostic image sample, the similarity between it and every other diagnostic text sample is calculated, resulting in a semantic similarity matrix (hereinafter referred to as the second semantic similarity matrix) representing semantic similarity (hereinafter referred to as the second semantic similarity). That is, step 103, based on the initial multimodal medical feature extraction model, predicts the pairwise semantic similarity between diagnostic image samples and diagnostic text samples, thus obtaining the second semantic similarity matrix. Second semantic similarity matrix Similarly, there are OK The columns, each element of the matrix, and the first semantic similarity matrix. The corresponding element in the text.

[0039] Unlike calculating the first semantic similarity matrix, calculating the second semantic similarity matrix no longer requires multi-hot tags. The value of each element in the second semantic similarity matrix is ​​predicted based on the initial multimodal medical feature extraction model to be trained.

[0040] Step 104: With the goal of minimizing the difference between the first semantic similarity and the second semantic similarity, train the initial multimodal medical feature extraction model to obtain the multimodal medical feature extraction model. The multimodal medical feature extraction model is used to output semantically aligned image feature representations and text feature representations.

[0041] Step 104 involves using the first semantic similarity as a reference standard and training the initial multimodal medical feature extraction model with the goal of minimizing the difference between the first and second semantic similarities until a preset stopping condition is met. Meeting the preset stopping condition can be either a loss value representing the difference being less than a set threshold or the number of training iterations being less than a set number. This embodiment does not impose specific limitations on the preset stopping condition.

[0042] In this embodiment, the multimodal medical feature extraction model is the initial multimodal medical feature extraction model obtained after training. The multimodal medical feature extraction model includes a visual encoder and a text encoder. The former is used to encode image feature representations, and the latter is used to encode text feature representations.

[0043] The multimodal medical feature extraction model trained in this application can transform two completely different forms of data (images and text) into a common mathematical space. Specifically, the visual encoder is responsible for extracting the complex visual information of the input image and outputting it as a condensed digital vector, i.e., the image feature representation. The text encoder is responsible for converting the linguistic meaning of the text into a digital vector with the same dimension and consistent structure, i.e., the text feature representation. Semantic alignment refers to the high degree of consistency achieved between these two independent representations after co-training: if an image and a piece of text describe the same medical concept (e.g., both about macular edema), then their respective generated feature vectors will be very close in position within this shared space; otherwise, they will be far apart. Therefore, this alignment enables downstream tasks to connect vision and text, achieving a deep and unified understanding of diagnostic data across modalities.

[0044] In other words, if the multimodal medical feature extraction model cannot output semantically aligned image and text feature representations, then although the image and text feature vectors may each contain information within the modality, they exist in completely different mathematical spaces. The similarity calculation between the two becomes meaningless, which will directly lead to the complete failure of all downstream functions that rely on cross-modal understanding: the model cannot retrieve relevant medical images based on text descriptions, nor can it use text descriptions to perform zero-shot identification and diagnosis of medical images, let alone achieve comprehensive reasoning on the image and text information of the same case.

[0045] To train the aforementioned multimodal medical feature extraction model capable of outputting semantically aligned image and text feature representations, this application improves upon existing training methods. Compared to existing training methods, the training method of this application possesses at least the following technical advantages: First, the multi-hot label generation mechanism employed during training enables the processing of unpaired image-text data. Even independent diagnostic image samples or independent diagnostic text samples can participate in model training by generating multi-hot labels. Therefore, the multi-hot label generation mechanism adopted in this application can greatly expand the data scale of effective training samples, significantly improve data utilization efficiency, solve the problem of limited available paired image-text data in existing technologies, and also avoid the waste of unpaired image or text data caused by strict reliance on paired image-text data in existing technologies.

[0046] Second, this application determines whether two samples match solely based on the similarity of the medical content represented by their multi-hot tags, not on whether they come from the same patient. If images and texts from different patients describe the same symptom, their multi-hot tags will have a high similarity (the first semantic similarity score is high). Based on this, the training process aims to minimize the difference between the first and second semantic similarities. Essentially, this forces the model to learn and accept this semantic consistency across patient identities, thus directly correcting erroneous learning signals caused by false negatives. It guides the model to ignore biases related to patient identity and instead focus on generalizable disease features, effectively eliminating the false negative problem.

[0047] In conjunction with the above embodiments, in one implementation, step 103 may include: Step 1031: Extract features from the diagnostic image samples to obtain visual feature vectors.

[0048] In this embodiment, since the initial multimodal medical feature extraction model includes a visual encoder and a text encoder, step 1031 can be performed by inputting the diagnostic image sample into the visual encoder and extracting features from the diagnostic image sample through the visual encoder to obtain a visual feature vector.

[0049] Step 1032: Extract features from the diagnostic text sample to obtain the text feature vector.

[0050] Similarly, by executing step 1032, the diagnostic text sample can be input into the text encoder, and the text encoder can extract features from the diagnostic text sample to obtain the text feature vector.

[0051] Step 1033: Determine the second semantic similarity based on the similarity between the visual feature vector and the text feature vector.

[0052] Since the initial multimodal medical feature extraction model is an untrained model, the visual feature vector and text feature vector are the predictions obtained by the visual encoder and text encoder, respectively. Accordingly, the similarity between the calculated visual feature vector and text feature vector is also a prediction result. The similarity between the visual feature vector and text feature vector is the second semantic similarity.

[0053] In this embodiment, the predicted values ​​of the feature vectors of the diagnostic image samples and diagnostic text samples are obtained, and then the second semantic similarity is calculated based on the predicted values, which can ensure the smooth implementation of the entire multimodal medical feature extraction model training method.

[0054] In one implementation, in conjunction with the above embodiments, step 102 may include: Based on the multi-hot tags, the first semantic similarity between the diagnostic image samples and the diagnostic text samples is determined according to the following formula (1).

[0055] (1) in, The first semantic similarity; Multi-thermal tags carried by diagnostic image samples; This indicates the multi-hot tags carried by the diagnostic text samples; The vector norm is used to calculate the length of a vector in space.

[0056] Formula (1) is used to calculate the cosine similarity between the multi-hot tag vector of a diagnostic image sample and the multi-hot tag vector of a diagnostic text sample. In other words, the first semantic similarity matrix... Each element in can be calculated using formula (1).

[0057] In this embodiment, the cosine similarity of the multi-hot tags of the diagnostic image sample and the diagnostic text sample can be calculated using formula (1), thereby quantifying the consistency of the two in medical semantics into a score. This score serves as a soft target in model training and can provide a clear optimization direction for subsequent learning of cross-modal semantic alignment.

[0058] In conjunction with the above embodiments, in one implementation, step 103 may include: Based on the initial multimodal medical feature extraction model, the second semantic similarity between diagnostic image samples and diagnostic text samples is determined by the following formula (2).

[0059] (2) in, Indicates second semantic similarity; Represents visual feature vectors; Represents the text feature vector. This indicates transpose.

[0060] Similarly, formula (2) is used to calculate the cosine similarity between the visual feature vector of a diagnostic image sample and the text feature vector of a diagnostic text sample. In other words, the second semantic similarity matrix... Each element in can be calculated using formula (2).

[0061] In this embodiment, the cosine similarity between the visual feature vector and the text feature vector can be calculated using formula (2), thereby quantifying the model's current understanding of the relationship between the image and the text into a specific score, which can then be used to evaluate the gap between its predicted semantics and the true semantic target, ensuring the smooth training of the model.

[0062] In conjunction with the above embodiments, in one implementation, step 104 may include: Step 1041: Determine the first probability distribution corresponding to the first semantic similarity and the second probability distribution corresponding to the second semantic similarity using the flexible maximum value function.

[0063] Among them, the flexible maximum value function is function.

[0064] Executing step 1041 allows the first similarity matrix to be... pass The function is transformed into a probability distribution matrix, that is, the probability distribution matrix representing the first probability distribution. At the same time, the second similarity matrix... pass The function is transformed into a probability distribution matrix, that is, the probability distribution matrix representing the second probability distribution. .

[0065] Wherein, the probability distribution matrix of the first probability distribution There are two types. The first type is calculated from the image perspective, where each element in the matrix is... The second method is calculated from a textual perspective, where each element in the matrix is... .

[0066] The calculation is performed using the following formula (3). The calculation is performed using the following formula (4): (3) (4) in, This represents the calculated probability value; and This is an index representing the position of a sample within a batch. Used for indexing diagnostic image samples Used for indexing diagnostic text samples Indicates for the first The diagnostic image sample, among all diagnostic text samples in the current batch, is the... Each diagnostic text sample is a probability value of its correct semantic pairing; Indicates for the first The diagnostic text sample, among all diagnostic image samples in the current batch, is the... Each diagnostic image sample is a probability value of its correct semantic pairing; Indicates the first The first diagnostic image sample and the first The score of the first semantic similarity between the diagnostic text samples; Indicates the first The first diagnostic text sample and the first The score of the first semantic similarity between diagnostic image samples; This indicates the batch size, which is the total number of samples contained in the batch.

[0067] Similarly, the probability distribution matrix of the second probability distribution There are two types. The first type is calculated from the image perspective, where each element in the matrix is... The second method is calculated from a textual perspective, where each element in the matrix is... .

[0068] The calculation is performed using the following formula (5). The calculation is performed using the following formula (6): (5) (6) in, This represents the calculated probability value; and This is an index representing the position of a sample within the batch. Used for indexing diagnostic image samples Used for indexing diagnostic text samples; Indicates for the first The diagnostic image sample, among all diagnostic text samples in the current batch, is the... Each diagnostic text sample is a predicted probability value of its correct semantic pairing; Indicates for the first The diagnostic text sample, among all diagnostic image samples in the current batch, is the... Each diagnostic image sample is a predicted probability value of its correct semantic pairing; Indicates the first The first diagnostic image sample and the first The score of second semantic similarity between diagnostic text samples; Indicates the first The first diagnostic text sample and the first The score of the second semantic similarity between diagnostic image samples; Indicates the batch size, i.e., the total number of samples contained in the batch; (The symbol is missing from the original text.) This is a hyperparameter used to adjust the smoothness or sharpness of the probability distribution.

[0069] Step 1042: Determine the difference between the first probability distribution and the second probability distribution using the cross-entropy loss function.

[0070] Specifically, the difference between the first probability distribution and the second probability distribution can be calculated using the following formulas (7)-(9). : (7) (8) (9) in, The cross-entropy loss is calculated from the image perspective. The cross-entropy loss is calculated from a textual perspective; The difference between the first and second probability distributions represents the total loss. , , , , , , The meaning is the same as in the previous text.

[0071] Step 1043: Train the initial multimodal medical feature extraction model with the goal of minimizing the differences.

[0072] In the calculation Then, the parameters in the initial multimodal medical feature extraction model can be updated using the backpropagation algorithm. This embodiment does not impose specific restrictions on the update process.

[0073] In this embodiment, on the one hand, a standardized multi-hot tagging mechanism can be used to accommodate unpaired image-text data, solving the problems of data waste and resource scarcity. On the other hand, a semantically based soft-objective loss function can ensure the correctness and robustness of the model's learning direction. Therefore, the model obtained through the training method of this application can output high-quality feature representations that truly achieve deep semantic alignment between vision and text, providing technical support for subsequent clinical applications.

[0074] In conjunction with the above embodiments, in one implementation, step 101 may include: Step 1011: Obtain the diagnostic image sample corresponding to the target diagnostic object from the diagnostic image sample set and / or the diagnostic image and text pairing sample set.

[0075] Step 1012: Obtain the diagnostic text sample corresponding to the target diagnostic object from the diagnostic text sample set and / or the diagnostic image and text pairing sample set.

[0076] The diagnostic image sample set includes diagnostic image samples that do not have paired diagnostic text, the diagnostic text sample set includes diagnostic text samples that do not have paired diagnostic images, and the diagnostic image and text pairing sample set includes paired diagnostic image samples and diagnostic text samples.

[0077] In this embodiment, there are three sample sets: a diagnostic image sample set, a diagnostic image-text pairing sample set, and a diagnostic text sample set. The diagnostic image samples in the diagnostic image sample set are all samples without paired diagnostic text; the diagnostic text samples in the diagnostic text sample set are all samples without paired diagnostic images; and the diagnostic image-text pairing sample set contains pre-calibrated, correctly matched diagnostic image-text sample pairs.

[0078] When performing step 1011, diagnostic image samples can be randomly selected from the diagnostic image sample set and / or the diagnostic image and text paired sample set according to a pre-set first selection probability. Similarly, when performing step 1012, diagnostic text samples can be randomly selected from the diagnostic text sample set and / or the diagnostic image and text paired sample set according to a pre-set second selection probability. The first and second selection probabilities can be set according to actual needs.

[0079] In this embodiment, the diagnostic image sample set is as follows: Figure 2 The image-only dataset in the image dataset shows a sample set of diagnostic images paired with text, as shown in the image dataset. Figure 2 As shown in the image-text dataset, the diagnostic text sample set is as follows: Figure 2 The text-only dataset is shown in the example. Figure 2 This is a schematic diagram illustrating the training process of a multimodal medical feature extraction model according to an embodiment of this application.

[0080] In this embodiment, three types of sample sets—paired text / image, pure image, and pure text—are used to overcome the limitations of a single data source, thereby training a model with stronger generalization ability and robustness. The large amount of text-only dataset enriches the model's understanding of medical terminology, rare diseases, and diverse linguistic descriptions, significantly enhancing the semantic depth and breadth of the text encoder. Meanwhile, the large amount of image-only data exposes the model to a wide range of visual samples from different patients, devices, and disease stages, significantly improving the robustness of the visual encoder and its generalization ability to unknown new images.

[0081] In conjunction with the above embodiments, in one implementation, the multiple hot tags of the diagnostic text sample are determined in advance through the following steps: Semantic information is extracted from diagnostic text samples to obtain semantic feature vectors; Based on the similarity between the semantic feature vector and multiple target semantic feature vectors, at least one target semantic feature vector is determined, and one target semantic feature vector corresponds to one medical semantic tag among multiple hot tags. Generate multi-hot tags for diagnostic text samples based on at least one target semantic feature vector.

[0082] The multiple target semantic feature vectors are determined through the following steps: Obtain multiple standard diagnostic text samples; Semantic information was extracted from multiple standard diagnostic text samples to obtain multiple semantic feature vectors; Clustering is performed on the semantic feature vectors corresponding to multiple standard diagnostic text samples to obtain multiple clusters; The centroid vector of each cluster is determined based on the semantic feature vector contained in each cluster. Based on the vectors of each center point, multiple target semantic feature vectors are determined.

[0083] In this embodiment, the entire process of generating multi-hot tags can be divided into two stages: the first stage is creating a prototype library, and the second stage is generating multi-hot tags. The following explanation uses a multimodal medical feature extraction model applicable to ophthalmology as an example to illustrate each stage.

[0084] Phase 1: Creating a medical concept prototype library.

[0085] The first phase aims to extract standard mathematical definitions for various ophthalmic diseases and characteristics from high-quality data. This phase specifically includes the following steps: Step 1: Select standard diagnostic text samples. In practice, only diagnostic text samples from the diagnostic image-text pairing sample set are used. This is because the data in this set includes doctor-annotated images and corresponding detailed text reports, making the information more reliable. Step 1 corresponds to... Figure 2 Text data is selected from the graph-text dataset.

[0086] Step 2: Text Feature Extraction and Vectorization. For each diagnostic text sample in the set of paired diagnostic image and text samples, a pre-designed text feature extractor incorporating ophthalmological medical knowledge is used to read the text content and extract key semantic information (such as symptoms, location, severity, etc.). This information is then converted into a high-dimensional mathematical vector, ultimately generating a semantic vector (i.e., semantic feature vector) representing the medical meaning of each diagnostic text sample in the set of paired diagnostic image and text samples. Step 2 corresponds to... Figure 2 The feature extraction steps within the dashed box generated by the text label are embedded as semantic vectors.

[0087] Step 3: Cluster analysis to form concept clusters. All semantic vectors generated in Step 2 are clustered using a clustering algorithm (e.g., K-Means), resulting in multiple distinct clusters. For example, all semantic vectors describing macular edema will cluster in one cluster, and all semantic vectors describing retinal detachment will cluster in another. Step 3 corresponds to... Figure 2 The clustering operation within the dashed box is generated from Chinese text labels.

[0088] Step 4: Determine cluster prototypes. Calculate the centroid of each cluster. This centroid vector represents the most standard mathematical expression of the medical concept associated with its cluster. Finally, form a cluster prototype library from all centroid vectors, where each prototype corresponds to a single medical concept, in this case, a specific ophthalmological concept (e.g., prototype A represents glaucoma, prototype B represents cataracts, etc.). Step 4 corresponds to... Figure 2The clustering prototype is obtained within the dashed box of the Chinese text tags. The clustering result is as follows: Figure 3 As shown. Figure 3 This is a schematic diagram illustrating a clustering result in one embodiment of this application. A centroid vector is a target semantic feature vector.

[0089] Phase 2: Generating multi-hot tags.

[0090] After obtaining the prototype library, the first step is to generate multi-hot tags for the diagnostic text samples in all sample sets. The process of generating multi-hot tags for a single diagnostic text sample includes the following steps: Step 1: Input diagnostic text sample. Obtain a diagnostic text sample T from either a text-only dataset or a graph-text dataset.

[0091] Step 2: Text Vectorization. Using the exact same feature extraction and vectorization methods as in the first stage, the diagnostic text sample T is transformed into a semantic vector.

[0092] Step 3: Compare with the prototype library. Calculate the similarity (e.g., cosine similarity) between the new semantic vector and each prototype in the prototype library, obtaining a series of similarity scores, for example: {Similarity with prototype A (glaucoma): 0.12; Similarity with prototype B (cataract): 0.89; Similarity with prototype C (macular edema): 0.95; ...}. Step 3 corresponds to... Figure 2 The similarity is calculated within the dashed box generated by the text labels.

[0093] Step 4: Generate multi-hot tags. Truncate the prototypes based on the similarity scores from the previous step. For example, set a threshold to retain all prototypes with similarity greater than the threshold; or, take the top-K prototypes with the highest similarity, where K can be set according to actual needs. Finally, create a vector with a length equal to the size of the prototype library based on the selected prototypes. In the vector, mark the positions corresponding to the selected prototypes as 1, and mark all other positions as 0, for example, obtaining a multi-hot tag [0, 1, 1, ...]. This step corresponds to... Figure 2 The text tag generation part of the multi-hot tag.

[0094] In one implementation, since the sample set also includes diagnostic image samples, the multi-hot tags of the diagnostic image samples are determined accordingly through the following steps: The multi-hot tags corresponding to the diagnostic text samples that match the diagnostic image samples are determined as the multi-hot tags of the diagnostic image samples.

[0095] This method is applicable to diagnostic image samples in a set of diagnostic image and text paired sample samples.

[0096] For diagnostic image samples in the diagnostic image sample set, multi-hot tags can be calculated in other ways. For example, one way is to input the diagnostic image sample into a trained image encoder to generate an image feature vector representing the image content, and then calculate the similarity between the image feature vector and each prototype in the prototype library to generate multi-hot tags. The specific principle is the same as steps 3-4 of the second stage mentioned above.

[0097] The first and second stages described above are preliminary steps before model training. Next, during actual model training, step 101 is executed first to obtain diagnostic image samples and diagnostic text samples carrying multiple hot tags corresponding to the target diagnostic object; then step 102 is executed to determine the first semantic similarity between the diagnostic image samples and the diagnostic text samples based on the multiple hot tags. Step 102 corresponds to... Figure 2 The part with the dashed box for semantic similarity calculation is executed directly; step 103 is executed directly, and the second semantic similarity between the diagnostic image sample and the diagnostic text sample is determined according to the initial multimodal medical feature extraction model. Specifically, the visual feature vector is obtained according to the visual encoder, and the text feature vector is obtained according to the text encoder; finally, step 104 is executed, and the second semantic similarity is determined according to the visual feature vector and the text feature vector during multimodal training. Then, the initial multimodal medical feature extraction model is trained with the goal of minimizing the difference between the first semantic similarity and the second semantic similarity.

[0098] In this embodiment, a series of cluster prototypes representing authoritative medical concepts are first extracted from high-quality image-text pairing data using a prototype clustering algorithm, forming a stable semantic dictionary to ensure the professionalism and accuracy of the labeling system. Then, for any input diagnostic text sample, a standard multi-hot label can be automatically generated by calculating the similarity between its feature vector and each prototype in the semantic dictionary. This process not only enables automated labeling of unlabeled data, solving the problem of high manual labeling costs, but also elevates the data matching relationship from the physical source level to the robust semantic content level. Therefore, the multi-hot labeling system in this embodiment can solve the data scarcity problem and eliminate false negative interference.

[0099] The model training method proposed in this application enables the training of a multimodal medical feature extraction model using a small amount of paired image-text data. This application decouples the data in the image-text dataset and replaces the InfoNCE loss with a semantic matching loss based on medical knowledge. This allows for the training of a decoupled image-text contrastive learning model and reduces the impact of false negatives on the model to some extent. Specifically, the training device includes: a visual encoder, a text encoding module, a text label generation module, a semantic similarity matching module, and a multimodal training module. The visual encoder uses a Swin Transformer with ImageNet pre-trained weights. The text encoder employs BioClinicalBERT. The text label generation module is implemented using a feature extractor designed based on medical knowledge, enabling the generation of multiple hot labels for unlabeled text data. The semantic similarity calculation module utilizes the multiple hot labels of image and text data, calculating the semantic matching loss as a training soft loss to achieve semantic unification. The multimodal training module is implemented using a standard CLIP structure, combining semantic soft loss to achieve multimodal contrastive learning.

[0100] The multimodal medical feature extraction model training method of this application is particularly suitable for training ophthalmic multimodal medical feature extraction models. When training the ophthalmic multimodal medical feature extraction model, this application can use unpaired ophthalmic image datasets and ophthalmic report datasets to solve the problem of paired image-text data shortage in the training process of ophthalmic multimodal medical feature extraction models, achieving data-efficient training of ophthalmic multimodal medical feature extraction models with low false negative rates.

[0101] Unlike existing ophthalmic multimodal medical feature extraction models, this application provides a semantic matching loss calculation method. By performing semantic analysis and fine-grained segmentation on text report data, it achieves accurate semantic matching and alignment of ophthalmic images. When calculating the semantic matching loss, the cross-entropy between the cosine similarity between the image and text and the cosine similarity between their multi-hot tags is used as the semantic matching loss, thus fully leveraging the role of semantic information in the image-text matching process. This application can utilize a text feature extractor designed based on ophthalmic medical knowledge to adapt to different forms of ophthalmic reports. Furthermore, this application achieves adaptation to various types of ophthalmic text data from different sources, including image descriptions in textbooks, example descriptions in papers, and publicly available ophthalmic case report datasets.

[0102] Specifically, in training the ophthalmology multimodal medical feature extraction model, the text label generation module extracts features based on ophthalmology medical knowledge and uses prototype clustering to generate multi-hot labels. The visual encoder and text encoder embed data from different modalities. The semantic similarity calculation module processes the multi-hot labels to achieve semantic alignment of data from different modalities. The multimodal training module, combined with semantic matching loss, trains the ophthalmology multimodal medical feature extraction model. During training, to mitigate the potential false negatives in contrastive learning, the cosine similarity between the multi-hot labels corresponding to different modalities is first calculated as a prior constraint in the semantic alignment process. Then, based on the calculated cosine similarity, a softmax function is used to obtain a soft target for subsequent multimodal training. The semantic constraints corresponding to the multi-hot labels refine the granularity of contrastive learning from the patient level to the feature level, effectively mitigating the potential impact of false negatives.

[0103] The multimodal ophthalmology basic model training method proposed in this application utilizes multi-hot tags related to medical semantics and effectively achieves semantic alignment and false negative elimination of different forms of ophthalmology data by calculating semantic soft loss. In the process of training the image-text comparison learning-based multimodal ophthalmology basic model, image-only data and text-only data are introduced, making full use of various forms of ophthalmology medical data. It has the advantages of high data efficiency and low false negative impact, and will provide inspiration for solving the problem of scarce ophthalmology medical data.

[0104] In summary, the model training method proposed in this application can decouple image-text data through multi-hot labels, thereby fully utilizing image-only and text-only data to reduce the need for image-text matching data. This application trains a multimodal medical feature extraction model by calculating the matching loss of semantic labels and combining it with the classic contrastive loss. Unlike existing training methods, this application breaks free from the constraints of patient-level data granularity and aligns image and text data from a semantic perspective. This significantly improves data utilization while reducing the probability of false negatives, providing a new approach for training multimodal medical feature extraction models using limited paired data.

[0105] The following describes a multimodal medical feature extraction model training device provided in this application. The multimodal medical feature extraction model training device described below and the multimodal medical feature extraction model training method described above can be referred to in correspondence.

[0106] Figure 4 This is a structural block diagram of a multimodal medical feature extraction model training device according to an embodiment of this application. (Refer to...) Figure 4 The multimodal medical feature extraction model training device 400 of this application may include: The acquisition module 401 is used to acquire diagnostic image samples and diagnostic text samples corresponding to the target diagnostic object. Each of the diagnostic image samples and each of the diagnostic text samples carries multi-hot tags that characterize medical semantic features. The first determining module 402 is used to determine a first semantic similarity between the diagnostic image sample and the diagnostic text sample based on the multi-hot tags; The second determining module 403 is used to determine the second semantic similarity between the diagnostic image sample and the diagnostic text sample based on the initial multimodal medical feature extraction model; The training module 404 is used to train the initial multimodal medical feature extraction model with the goal of minimizing the difference between the first semantic similarity and the second semantic similarity, so as to obtain a multimodal medical feature extraction model. The multimodal medical feature extraction model is used to output semantically aligned image feature representations and text feature representations.

[0107] According to the multimodal medical feature extraction model training device 400 provided in this application, the second determining module 403 is specifically used for: performing feature extraction on the diagnostic image sample to obtain a visual feature vector; performing feature extraction on the diagnostic text sample to obtain a text feature vector; and determining the second semantic similarity based on the similarity between the visual feature vector and the text feature vector.

[0108] According to the multimodal medical feature extraction model training device 400 provided in this application, the training module 404 is specifically used to: determine the first probability distribution corresponding to the first semantic similarity and the second probability distribution corresponding to the second semantic similarity through a flexible maximum value function; determine the difference between the first probability distribution and the second probability distribution through a cross-entropy loss function; and train the initial multimodal medical feature extraction model with the goal of minimizing the difference.

[0109] According to the multimodal medical feature extraction model training device 400 provided in this application, the multi-hot tags of the diagnostic text sample are determined by the following steps: extracting semantic information from the diagnostic text sample to obtain a semantic feature vector; determining at least one target semantic feature vector based on the similarity between the semantic feature vector and multiple target semantic feature vectors, wherein one target semantic feature vector corresponds to a medical semantic tag in the multi-hot tags; and generating multi-hot tags of the diagnostic text sample based on the at least one target semantic feature vector.

[0110] According to the multimodal medical feature extraction model training device 400 provided in this application, the multiple target semantic feature vectors are determined by the following steps: acquiring multiple standard diagnostic text samples; extracting semantic information from the multiple standard diagnostic text samples to obtain multiple semantic feature vectors; performing clustering processing on the semantic feature vectors corresponding to the multiple standard diagnostic text samples to obtain multiple clusters; determining the centroid vector corresponding to each cluster based on the semantic feature vector contained in each cluster; and determining the multiple target semantic feature vectors based on each centroid vector.

[0111] According to the multimodal medical feature extraction model training device 400 provided in this application, the multi-hot tags of the diagnostic image samples are determined by the following steps: the multi-hot tags corresponding to the diagnostic text samples that match the diagnostic image samples are determined as the multi-hot tags of the diagnostic image samples.

[0112] According to the multimodal medical feature extraction model training device 400 provided in this application, the acquisition module 401 is specifically used for: acquiring diagnostic image samples corresponding to the target diagnostic object from a diagnostic image sample set and / or a diagnostic image and text pairing sample set; acquiring diagnostic text samples corresponding to the target diagnostic object from a diagnostic text sample set and / or a diagnostic image and text pairing sample set; wherein, the diagnostic image sample set includes diagnostic image samples without paired diagnostic text, the diagnostic text sample set includes diagnostic text samples without paired diagnostic images, and the diagnostic image and text pairing sample set includes paired diagnostic image samples and diagnostic text samples.

[0113] Figure 5 This is a schematic diagram of the physical structure of an electronic device according to an embodiment of this application. Figure 5As shown, the electronic device may include a processor 510, a communications interface 520, a memory 530, and a communication bus 540, wherein the processor 510, the communications interface 520, and the memory 530 communicate with each other via the communication bus 540. The processor 510 can call logical instructions in the memory 530 to execute a multimodal medical feature extraction model training method. This method includes: acquiring diagnostic image samples and diagnostic text samples corresponding to a target diagnostic object, each of the diagnostic image samples and each of the diagnostic text samples carrying multi-hot tags representing medical semantic features; determining a first semantic similarity between the diagnostic image samples and the diagnostic text samples based on the multi-hot tags; determining a second semantic similarity between the diagnostic image samples and the diagnostic text samples based on an initial multimodal medical feature extraction model; and training the initial multimodal medical feature extraction model with the objective of minimizing the difference between the first semantic similarity and the second semantic similarity to obtain a multimodal medical feature extraction model, which is used to output semantically aligned image feature representations and text feature representations.

[0114] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0115] On the other hand, this application also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute a multimodal medical feature extraction model training method provided by the above methods. The method includes: acquiring diagnostic image samples and diagnostic text samples corresponding to a target diagnostic object, each of the diagnostic image samples and each of the diagnostic text samples carrying multi-hot tags representing medical semantic features; determining a first semantic similarity between the diagnostic image samples and the diagnostic text samples based on the multi-hot tags; determining a second semantic similarity between the diagnostic image samples and the diagnostic text samples based on an initial multimodal medical feature extraction model; and training the initial multimodal medical feature extraction model with the goal of minimizing the difference between the first semantic similarity and the second semantic similarity to obtain a multimodal medical feature extraction model, wherein the multimodal medical feature extraction model is used to output semantically aligned image feature representations and text feature representations.

[0116] In another aspect, this application also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements a multimodal medical feature extraction model training method provided by the methods described above. This method includes: acquiring diagnostic image samples and diagnostic text samples corresponding to a target diagnostic object, each of the diagnostic image samples and each of the diagnostic text samples carrying multi-hot tags representing medical semantic features; determining a first semantic similarity between the diagnostic image samples and the diagnostic text samples based on the multi-hot tags; determining a second semantic similarity between the diagnostic image samples and the diagnostic text samples based on an initial multimodal medical feature extraction model; and training the initial multimodal medical feature extraction model with the objective of minimizing the difference between the first semantic similarity and the second semantic similarity to obtain a multimodal medical feature extraction model, wherein the multimodal medical feature extraction model is used to output semantically aligned image feature representations and text feature representations.

[0117] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0118] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0119] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for training a multimodal medical feature extraction model, characterized in that, include: Obtain diagnostic image samples and diagnostic text samples corresponding to the target diagnostic object. Each diagnostic image sample and each diagnostic text sample carries a multi-hot tag that represents medical semantic features. Based on the multi-hot tags, a first semantic similarity is determined between the diagnostic image sample and the diagnostic text sample; Based on the initial multimodal medical feature extraction model, the second semantic similarity between the diagnostic image samples and the diagnostic text samples is determined; With the goal of minimizing the difference between the first semantic similarity and the second semantic similarity, the initial multimodal medical feature extraction model is trained to obtain a multimodal medical feature extraction model, which is used to output semantically aligned image feature representations and text feature representations.

2. The method according to claim 1, characterized in that, Determining the second semantic similarity between the diagnostic image sample and the diagnostic text sample includes: Feature extraction is performed on the diagnostic image samples to obtain visual feature vectors; Feature extraction is performed on the diagnostic text sample to obtain a text feature vector; The second semantic similarity is determined based on the similarity between the visual feature vector and the text feature vector.

3. The method according to claim 1, characterized in that, The step of training the initial multimodal medical feature extraction model with the objective of minimizing the difference between the first semantic similarity and the second semantic similarity includes: The first probability distribution corresponding to the first semantic similarity and the second probability distribution corresponding to the second semantic similarity are determined by using a flexible maximum value function. The difference between the first probability distribution and the second probability distribution is determined by the cross-entropy loss function; The initial multimodal medical feature extraction model is trained with the goal of minimizing the differences.

4. The method according to claim 1, characterized in that, The multi-hot tags of the diagnostic text samples were determined through the following steps: Semantic information is extracted from the diagnostic text sample to obtain a semantic feature vector; Based on the similarity between the semantic feature vector and multiple target semantic feature vectors, at least one target semantic feature vector is determined, and one of the target semantic feature vectors corresponds to a medical semantic tag among the multiple hot tags; Based on the at least one target semantic feature vector, generate multi-hot tags for the diagnostic text sample.

5. The method according to claim 4, characterized in that, The multiple target semantic feature vectors are determined through the following steps: Obtain multiple standard diagnostic text samples; Semantic information is extracted from the multiple standard diagnostic text samples to obtain multiple semantic feature vectors; Clustering is performed on the semantic feature vectors corresponding to multiple standard diagnostic text samples to obtain multiple clusters; The centroid vector of each cluster is determined based on the semantic feature vector contained in each cluster. Based on each of the central point vectors, the plurality of target semantic feature vectors are determined.

6. The method according to claim 1, characterized in that, The multiple hot tags of the diagnostic image samples are determined through the following steps: The multi-hot tags corresponding to the diagnostic text samples that match the diagnostic image samples are determined as the multi-hot tags of the diagnostic image samples.

7. The method according to any one of claims 1-6, characterized in that, Obtain diagnostic image samples and diagnostic text samples corresponding to the target diagnostic object, including: Obtain the diagnostic image sample corresponding to the target diagnostic object from the diagnostic image sample set and / or the diagnostic image and text pairing sample set; Obtain the diagnostic text sample corresponding to the target diagnostic object from the diagnostic text sample set and / or the diagnostic image and text paired sample set; The diagnostic image sample set includes diagnostic image samples that do not have paired diagnostic text, the diagnostic text sample set includes diagnostic text samples that do not have paired diagnostic images, and the diagnostic image and text pairing sample set includes paired diagnostic image samples and diagnostic text samples.

8. A multimodal medical feature extraction model training device, characterized in that, include: The acquisition module is used to acquire diagnostic image samples and diagnostic text samples corresponding to the target diagnostic object. Each diagnostic image sample and each diagnostic text sample carries a multi-hot tag that represents medical semantic features. The first determining module is used to determine a first semantic similarity between the diagnostic image sample and the diagnostic text sample based on the multi-hot tags; The second determining module is used to determine the second semantic similarity between the diagnostic image sample and the diagnostic text sample based on the initial multimodal medical feature extraction model; The training module is used to train the initial multimodal medical feature extraction model with the goal of minimizing the difference between the first semantic similarity and the second semantic similarity, so as to obtain a multimodal medical feature extraction model, which is used to output semantically aligned image feature representations and text feature representations.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements a multimodal medical feature extraction model training method as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements a multimodal medical feature extraction model training method as described in any one of claims 1 to 7.

Images