Data enhancement method and device based on multi-modal language alignment

Abstract

The invention discloses a data enhancement method and device based on multi-modal language alignment, and the method comprises the steps: collecting basic information of a patient in an electronic medical record system, a diagnosis report during an observation window, and various types of image data, and carrying out the preprocessing of the basic information, the diagnosis report and the various types of image data; respectively carrying out image data coding and text data coding on the preprocessed image and the preprocessed diagnosis report to obtain coded image embedding representation and text embedding representation; performing modal projection fusion on the encoded image embedding representation and text embedding representation, constructing a loss function through a comparative learning method, and aligning the image embedding representation and the text embedding representation through the loss function of minimizing comparative learning; image-text pair data is generated through a decoding method based on language similarity image-text pair generation. According to the invention, images, texts and images of different modalities can be distinguished; and data of a specified mode can be generated, so that the problem of lack of data volume in multi-mode large model research is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal data enhancement, and in particular to a data enhancement method and device based on multimodal language alignment. Background Art

[0002] With the development of artificial intelligence technology, the study and mining of multimodal data has become an increasingly important research field. Multimodal data refers to a data set that contains multiple types of information, such as images, text, audio or video. The application range of multimodal data is wide, covering many fields such as medical diagnosis, autonomous driving, virtual reality, augmented reality, etc. Multimodal data provides information from different senses or sources, which helps the model understand the data from multiple perspectives, so that it has stronger generalization ability and robustness when facing new data. But at the same time, when mining multimodal data, it is necessary to overcome the inconsistency of modal semantics, mismatch of feature dimensions, different strengths of correlation between different modalities, and possible missing modal data.

[0003] The goal of multimodal data augmentation is to generate richer and more diverse training samples by processing these different types of data, thereby improving the generalization and robustness of the model. With the rise of Large Language Model (LLM), it has become increasingly popular in academia and industry due to its powerful language understanding and reasoning capabilities. The progress of LLM has also inspired researchers to use LLM as an interface for multimodal tasks, such as visual language learning, audio and speech recognition, video understanding, etc., so the Multimodal Large Language Model (MLLM) has also attracted the attention of researchers. Currently, multimodal language alignment training is performed using techniques such as large model fine-tuning and Vision Transformer (ViT), with the goal of training a general multimodal large model. This method requires a large amount of manually annotated image-text pair data as basic data for training. In addition, these models and data are targeted at general data, do not contain domain knowledge, and can only perform simple question and answer.

[0004] With the continuous development of technology, medical imaging data mainly includes CT (Computed Tomography, electronic computer tomography) images, ultrasound (Ultrasound, US) images, electrocardiogram (Electrocardiogram, ECG), magnetic resonance (Magnetic Resonance, MR) images, nuclear medicine (SPECT, PET) images, etc. Among them, CT images are suitable for imaging hard tissues of the human body such as bones and metals, which have a higher grayscale level. Ultrasound images are suitable for soft tissues such as prostate and blood vessels, and have obvious speckle noise. Magnetic resonance images have rich grayscale information, not only for hard tissues such as bones, but also for soft tissues such as white matter and gray matter. Magnetic resonance is often used for brain detection. PET images are positron concentration images, which are highly sensitive to early cancers, but the images themselves are relatively blurred, with low spatial resolution, and cannot clearly reflect the geometric topological information inside the human body. The electrocardiogram uses an electrocardiograph to record the changes in electrical activity generated by each cardiac cycle of the heart from the body surface. These multimodal data provide a huge reference for doctors' diagnosis, but there are huge differences between different images, and between images and graphics. In reality, most patients do not have or need to undergo all examinations, and it is common for some modal data to be missing. How to unify the multimodal data representation, mine the associations between different modalities, and effectively fill in the missing modal data is an urgent problem to be solved. Therefore, if different modalities can be encoded and the missing modalities can be effectively supplemented through multimodal data enhancement methods, it can not only reduce the cost of patient treatment, but also improve the efficiency and accuracy of doctors' diagnosis.

[0005] For multimodal medical imaging data, the present invention proposes a data enhancement method that can directly generate different modality-text pairs and use the similarity between the generated texts for modality alignment or modality data interpolation. This method is intuitively generated based on language similarity, has strong interpretability, and does not require manual pre-labeling of text pairs. It can generate different modality-text pairs to solve the problem of lack of data in current multimodal large model research, and can support downstream task research of different modalities, such as auxiliary diagnosis, disease prediction, complication warning, etc. Summary of the invention

[0006] The purpose of the present invention is to provide a data enhancement method and device based on multimodal language alignment to address the deficiencies of the prior art.

[0007] The objective of the present invention is achieved through the following technical solutions: In a first aspect, an embodiment of the present invention provides a data enhancement method based on multimodal language alignment, comprising the following steps: (1) Collecting multimodal medical data of patients in the electronic medical record system; wherein the multimodal medical data includes basic information, diagnostic reports during the observation window, and various types of image data; (2) preprocessing the multimodal medical data collected in step (1) to reset the sizes of various images to the same fixed value and to clean and standardize the diagnostic reports corresponding to the images; (3) Performing image data encoding and text data encoding on the preprocessed image and its corresponding diagnostic report, respectively, to obtain encoded image embedding representation and text embedding representation; (4) Perform modal projection fusion on the encoded image embedding representation and the encoded text embedding representation, learn positive and negative samples by using the contrastive learning method to construct a contrastive learning loss function; determine the aligned image embedding representation and text embedding representation with the goal of minimizing the contrastive learning loss function; (5) Based on the aligned image embedding representation and text embedding representation determined in step (4), a corresponding diagnosis report is generated for the image, and a corresponding image is generated for the diagnosis report to obtain an aligned image-text pair.

[0008] Furthermore, in step (1), the basic information includes age and gender; and the various types of image data include CT images, ultrasound images, electrocardiograms, and magnetic resonance images.

[0009] Furthermore, in step (3), encoding the preprocessed image data to obtain an encoded image embedding representation specifically includes: (3.1) Slicing the preprocessed image to obtain multiple slice images corresponding to the image; (3.2) Slice embedding via linear mapping to convolve each slice image into a d-dimensional embedding vector; (3.3) For the embedding vector of each slice image obtained in step (3.2), add a learnable vector , and then splice the position code and modality code identifier corresponding to the slice image into the embedding vector to obtain the spliced ​​embedding vector; (3.4) The concatenated embedding vector is processed through multiple Transformer encoder layers to finally obtain the encoded image embedding representation.

[0010] Furthermore, the Transformer encoder layer includes a multi-head attention mechanism, a feedforward neural network and a normalization layer, and its input and output process is expressed as:

[0011] Among them, MSA stands for multi-head attention mechanism, LN stands for normalization layer, MLP stands for feedforward neural network, also stands for multi-layer perceptron; represents the output of the h-1th Transformer encoder layer, represents the intermediate vector of the h-th Transformer encoder layer, represents the output of the h-th Transformer encoder layer; , H indicates that there are H Transformer encoder layers in total; y indicates the prediction vector output by the last Transformer encoder layer.

[0012] Furthermore, in step (3), the preprocessed diagnosis report is encoded with text data to obtain an encoded text embedding representation, which specifically includes: First, the preprocessed diagnosis report is converted into a binary encoding vector; then it and its modality identifier are concatenated and input into a bidirectional Transformer encoder. After being processed by multiple bidirectional Transformer encoders, the encoded text embedding representation is obtained.

[0013] Furthermore, the bidirectional Transformer encoder includes a bidirectional attention mechanism, a feedforward neural network and a normalization layer, and its input and output process is expressed as:

[0014] Among them, Bi-A represents the bidirectional attention mechanism. represents the output of the w-1th bidirectional Transformer encoder, represents the intermediate vector of the w-th bidirectional Transformer encoder, represents the output of the w-th bidirectional Transformer encoder, , W means there are W bidirectional Transformer encoders, Represents the encoded text embedding representation output by the last bidirectional Transformer encoder.

[0015] Furthermore, in step (4), the loss function of the contrastive learning is expressed as:

[0016] in, represents the loss function of contrastive learning, N is the total number of samples, represents the image embedding representation of the i-th sample, represents the text embedding representation of the jth sample, express and The cosine similarity between express and The cosine similarity between is a hyperparameter used to tune the probability distribution.

[0017] Furthermore, in step (5), generating a corresponding diagnosis report for the image specifically includes: firstly encoding the image data of the given image to obtain an encoded image embedding representation; then calculating the cosine similarity between the image embedding representation and all text embedding representations determined in step (4), and selecting the text embedding representation corresponding to the maximum cosine similarity as the diagnosis report text corresponding to the image; The method of generating a corresponding image for a diagnostic report specifically includes: firstly encoding the text data of a given diagnostic report to obtain an encoded text embedding representation; then calculating the cosine similarity between the text embedding representation and all image embedding representations determined in step (4), and selecting the image embedding representation corresponding to the maximum cosine similarity as the image corresponding to the diagnostic report.

[0018] Furthermore, after step (5), the method further includes: Different post-processing processes are designed according to different downstream tasks, and the corresponding downstream tasks are performed based on the aligned image-text pairs obtained in step (5).

[0019] A second aspect of an embodiment of the present invention provides a data enhancement device based on multimodal language alignment, which is used to implement the above-mentioned data enhancement method based on multimodal language alignment, including: A data acquisition module, used to collect multimodal medical data of patients in the electronic medical record system; wherein the multimodal medical data includes basic information, diagnostic reports during the observation window, and various types of image data; A data preprocessing module is used to preprocess the multimodal medical data collected by the data acquisition module to reset the sizes of various types of images to the same fixed value and clean and standardize the diagnostic reports corresponding to the images; Each modality data encoding module includes an image data encoding module and a text data encoding module, wherein the image data encoding module is used to perform image data encoding on the preprocessed image to obtain an encoded image embedding representation; the text data encoding module is used to perform text data encoding on the preprocessed diagnosis report to obtain an encoded text embedding representation; A projection module, for performing modality projection fusion on the encoded image embedding representation and the encoded text embedding representation, learning positive and negative samples by using a contrastive learning method to construct a loss function for contrastive learning; determining the aligned image embedding representation and text embedding representation with the goal of minimizing the loss function for contrastive learning; and A decoding module is used to generate a corresponding diagnosis report for the image and a corresponding image for the diagnosis report according to the aligned image embedding representation and text embedding representation determined by the projection module, so as to obtain an aligned image-text pair.

[0020] The beneficial effect of the present invention is that the present invention encodes different image modalities and texts through a multimodal encoding method, which can not only distinguish the modality between images and texts, but also distinguish images of different modalities; the present invention aligns multimodal medical imaging data and text data at the language level, and then proposes a decoding method for generating image-text pairs based on language similarity, and conducts research on multimodal large models based on the generated image-text pairs. For research tasks with less single-modal data, images of specified modalities can be generated, which solves the problem of lack of data volume in the research of multimodal large models and helps the research of single-modal downstream tasks, such as auxiliary diagnosis, disease prediction, and complication warning. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 is a flow chart of a data enhancement method based on multimodal language alignment of the present invention; Figure 2 It is a framework flow chart of each modal data encoding module in the present invention; Figure 3 It is a framework flow chart of the projection module in the present invention; Figure 4 It is a framework flow chart of the decoding module in the present invention; Figure 5 It is a structural schematic diagram of a data enhancement device based on multimodal language alignment of the present invention. DETAILED DESCRIPTION

[0022] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of devices and methods consistent with some aspects of the present invention as detailed in the appended claims.

[0023] The terms used in the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The singular forms "a", "the" and "the" used in the present invention and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0024] It should be understood that although the terms first, second, third, etc. may be used in the present invention to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present invention, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0025] The present invention is described in detail below in conjunction with the accompanying drawings. In the absence of conflict, the features of the following embodiments and implementations can be combined with each other.

[0026] See also Figure 1 The data enhancement method based on multimodal language alignment of the present invention specifically includes the following steps: (1) Collect the multimodal medical data of patients in the electronic medical record system; the multimodal medical data includes basic information, diagnostic reports during the observation window, and various types of image data.

[0027] Furthermore, the basic information includes age and gender, and various types of image data include CT images, ultrasound images, electrocardiograms, and magnetic resonance imaging images.

[0028] (2) Preprocessing the multimodal medical data collected in step (1) to reset the sizes of various types of images to the same fixed value and clean and standardize the diagnostic reports corresponding to the images.

[0029] Specifically, the sizes of all types of images collected in step (1) are reset to the same fixed value, and the diagnostic reports corresponding to the images are cleaned and standardized to unify the representation of fixed meanings. For example, different doctors may abbreviate cancer into CA, Ca, ca, etc., and they are unified as CA during data preprocessing. The patient set is represented by V, the patient's image data is represented by I, the patient's diagnostic report data is represented by T, and n represents the maximum index of the corresponding category data of the patient. The value of n will change with the changes of different categories of medical data. Therefore, the patient can be formally represented as ,in, represents the image data of the patient, , , , and They represent CT images, ultrasound images, electrocardiogram images, magnetic resonance images, and nuclear medicine images, respectively. Indicates the diagnostic report data corresponding to the patient, , , , and Respectively represent the diagnostic report data corresponding to CT images, ultrasound images, electrocardiogram images, magnetic resonance images and nuclear medicine images. The i-th patient can be represented as , for medical data of different modalities, data missing is allowed, i.e., patients Value May be empty, replaced by 0, and the same applies to other modes.

[0030] (3) Perform image data encoding and text data encoding on the preprocessed image and its corresponding diagnostic report, respectively, to obtain the encoded image embedding representation and text embedding representation, as shown in Figure 2 shown.

[0031] Furthermore, the preprocessed image is encoded to obtain the encoded image embedding representation, such as Figure 2 As shown, specifically including: (3.1) Slice the preprocessed image to obtain multiple slice images corresponding to the image.

[0032] Specifically, the size of the preprocessed image is set to H*W, and it is sliced ​​to obtain multiple slice images, each of which has a size of N*N. After the slicing is completed, P=(H / N)*(W / N) slice images can be finally obtained, which can be adjusted according to actual needs. In this embodiment, N is set to 16, that is, the original image 224*224*3 is cut into a sequence image consisting of 14*14 slice images of a size of 16*16*3.

[0033] (3.2) Patch Embedding is performed through linear mapping to convolve each slice image into a d-dimensional embedding vector.

[0034] Specifically, slice embedding is performed through linear mapping to convolve each 3-dimensional 16*16 slice image into a d-dimensional embedding vector, that is, the original 16*16*3 slice image is convolved into an embedding vector of length 768, and the entire image becomes a dimension of 14*14*768. Flattening it becomes a two-dimensional matrix of 196*768, so the entire image can be represented by a two-dimensional matrix.

[0035] (3.3) For the embedding vector of each slice image obtained in step (3.2), add a learnable vector , and then splice the position code and modality code identifier corresponding to the slice image into the embedded vector to obtain the spliced ​​embedded vector.

[0036] Specifically, in order to represent the global information of the matrix, the Bert (Bidirectional Encoder Representations from Transformers) model adds a learnable vector to the embedding vector [CLS] . Again, the positional encoding (PE) and modal encoding identifier (MI) of the slice image are concatenated into the embedding vector, and finally the concatenated embedding vector Z is obtained. As shown in formula (1), the concatenated embedding vector Z contains the original slice image information after slice embedding, the learnable vector , position encoding PE and modality encoding identification MI.

[0037] (1) Among them, Z represents the concatenated embedding vector, I represents the patient's image data, is the slice function, is a linear mapping function, is a learnable vector [CLS], is the position code, is the modal coding identifier, d represents the total dimension of the position coding of the word vector (i.e., the embedding vector corresponding to the current slice image), p represents the pth word in the sentence (i.e., the pth slice image in the whole image), and i=[1,d / 2] represents the dimension index of the position coding vector. Represents a learnable modal vector, according to which the learnable modal vector Perform modal coding identification, and the subscript m represents the mth mode.

[0038] (3.4) The concatenated embedding vector is processed through multiple Transformer encoder layers to finally obtain the encoded image embedding representation; each Transformer encoder layer includes components such as a multi-head attention mechanism, a feedforward neural network, and a normalization layer. The input and output process is expressed by formula (2): (2) Among them, MSA stands for multi-head attention mechanism, LN stands for normalization layer, MLP stands for feedforward neural network, also stands for multi-layer perceptron; represents the output of the h-1th Transformer encoder layer, represents the intermediate vector of the h-th Transformer encoder layer, represents the output of the h-th Transformer encoder layer; , H indicates that there are H Transformer encoder layers; y indicates the prediction vector output by the last Transformer encoder layer. By combining the prediction vectors corresponding to all slice images, we can get the image embedding representation Y corresponding to the entire image.

[0039] It should be understood that In the image, the position before 0 is a learnable category vector. After encoding, the information at this position represents the overall representation of the image, and the other positions are the encoded representations of the corresponding images.

[0040] Furthermore, the preprocessed diagnostic report is encoded for text data to obtain the encoded text embedding representation, such as Figure 2 As shown, it specifically includes: first converting the preprocessed diagnostic report into a binary encoding vector; then concatenating it and its modality identifier and inputting it into the bidirectional Transformer encoder, and after being processed by multiple bidirectional Transformer encoders, the encoded text embedding representation is obtained. Among them, each bidirectional Transformer encoder includes components such as a bidirectional attention mechanism, a feedforward neural network, and a normalization layer. The input and output process is shown in formula (3): (3) Among them, Bi-A represents the bidirectional attention mechanism. represents the output of the w-1th bidirectional Transformer encoder, represents the intermediate vector of the w-th bidirectional Transformer encoder, represents the output of the w-th bidirectional Transformer encoder, , W means there are W bidirectional Transformer encoders, Represents the encoded text embedding representation output by the last bidirectional Transformer encoder.

[0041] It should be understood that by using a bidirectional Transformer encoder, contextual information in text data can be captured and text-based diagnostic reports can be encoded. First, the text is converted into a binary encoding vector. Then it and its modality identifier are input into the encoder. After the bidirectional attention mechanism, feedforward neural network and normalization operation, the encoding vector of the text is obtained, that is, the encoded text embedding representation .

[0042] (4) The encoded image embedding representation and the encoded text embedding representation are modally projected and fused, and the positive and negative samples are learned by using the contrastive learning method to construct a contrastive learning loss function; with the goal of minimizing the contrastive learning loss function, the aligned image embedding representation and text embedding representation are determined.

[0043] Specifically, the encoded image is embedded in Y and the encoded text is embedded in Perform modal projection fusion, the result is as follows Figure 3 As shown in the figure, the blue diagonal line is the positive sample with consistent image and text, and the others are negative samples. The contrastive learning method is used to learn the positive and negative samples of the two modalities to maximize the similarity between each image and its corresponding text description, while minimizing its similarity with other text descriptions. To this end, the contrastive learning method defines the loss function of contrastive learning as As shown in formula (4), the loss function of contrastive learning includes two items. The first item is the contrastive loss from image to text (i.e., the first log), and the second item is the contrastive loss from text to image (i.e., the second log). The average of the contrastive losses in these two directions is the loss function of contrastive learning. With the goal of minimizing the loss function of contrastive learning, the aligned image embedding representation and text embedding representation can be determined for subsequent reasoning tasks.

[0044] (4) in, represents the loss function of contrastive learning, N is the total number of samples, represents the image embedding representation of the i-th sample, represents the text embedding representation of the jth sample, express and The cosine similarity between (i.e. the cosine similarity between positive sample pairs), express and The cosine similarity between (i.e. the cosine similarity between positive and negative samples), is a hyperparameter used to tune the probability distribution.

[0045] (5) Based on the aligned image embedding representation and text embedding representation determined in step (4), a corresponding diagnosis report is generated for the image, and a corresponding image is generated for the diagnosis report, so as to obtain aligned image-text pairs, which helps to reduce the cost of manually labeling image-text pairs.

[0046] Furthermore, if Figure 4As shown, generating a corresponding diagnosis report for an image specifically includes: firstly encoding the image data of a given image to obtain an encoded image embedding representation; then calculating the cosine similarity between the image embedding representation and all text embedding representations determined in step (4), and selecting the text embedding representation corresponding to the maximum cosine similarity as the diagnosis report text corresponding to the image.

[0047] Furthermore, a corresponding image is generated for the diagnosis report, specifically comprising: firstly encoding the text data of a given diagnosis report to obtain an encoded text embedding representation; then calculating the cosine similarity between the text embedding representation and all image embedding representations determined in step (4), and selecting the image embedding representation corresponding to the maximum cosine similarity as the image corresponding to the diagnosis report.

[0048] Furthermore, after step (5), it also includes: designing different post-processing processes according to different downstream tasks, which is the reasoning application of the whole method, and executing corresponding downstream tasks based on the aligned image-text pairs obtained in step (5), such as auxiliary diagnosis, disease prediction, complication warning and other downstream tasks.

[0049] It is worth mentioning that the embodiment of the present invention also provides a data enhancement device based on multimodal language alignment, such as Figure 5 As shown, the device includes a data acquisition module, a data preprocessing module, each modality data encoding module, a projection module and a decoding module; wherein each modality data encoding module includes an image data encoding module and a text data encoding module.

[0050] In this embodiment, the data acquisition module is used to collect multimodal medical data of patients in the electronic medical record system; wherein the multimodal medical data includes basic information, diagnostic reports during the observation window, and various types of image data.

[0051] In this embodiment, the data preprocessing module is used to preprocess the multimodal medical data collected by the data acquisition module to reset the size of each type of image to the same fixed value and clean and standardize the diagnostic report corresponding to the image.

[0052] In this embodiment, the image data encoding module is used to encode the image data after preprocessing to obtain the encoded image embedding representation. The text data encoding module is used to encode the text data of the preprocessed diagnosis report to obtain the encoded text embedding representation.

[0053] In this embodiment, the projection module is used to perform modal projection fusion on the encoded image embedding representation and the encoded text embedding representation, and to construct a loss function of contrastive learning by learning positive and negative samples using a contrastive learning method; with the goal of minimizing the loss function of contrastive learning, the aligned image embedding representation and text embedding representation are determined.

[0054] In this embodiment, the decoding module is used to generate a corresponding diagnosis report for the image and a corresponding image for the diagnosis report according to the aligned image embedding representation and text embedding representation determined by the projection module, so as to obtain an aligned image-text pair.

[0055] The data enhancement method and device based on multimodal language alignment of the present invention are described in detail below according to the embodiments, and the purpose and effect of the present invention will become more obvious.

[0056] The data enhancement method specifically includes: S1. The data acquisition module collects the basic information of 10,000 patients in a medical institution, the diagnosis reports during the observation window, and various image data.

[0057] S2. The data preprocessing module preprocesses the data collected by the data acquisition module, resets the image size of each modality to 224*224*3, cleans and standardizes the diagnostic reports corresponding to the images, and unifies the fixed meaning (for example, different doctors may abbreviate cancer into CA, Ca, ca, etc., which are unified as CA during data preprocessing). The patient set is represented by V, the patient's image information is represented by I, the patient's diagnostic information is represented by T, and N represents the maximum index of the corresponding category data of the patient. The N value will change with the change of different categories of data. Patients can be formally represented as ,in Represents CT images , ultrasound image , ECG image , MRI and nuclear medicine images . Represents the corresponding diagnostic information. For patient i, it can be expressed as: , for data of different modalities, missing data are allowed, that is, the value of patient i May be empty, as are other modals.

[0058] S3, the image data encoding module and the text data encoding module of the multimodal data encoding module encode the preprocessed medical images and diagnostic reports respectively. The modality is identified and encoded according to the medical image data of different modalities. First, the image is sliced. Here, the slice size is set to 16, that is, the original image 224*224*3 is cut into 14*14 sequence images composed of 16*16*3 small slices. Secondly, each 3D small slice is convolved into a d-dimensional vector through the linear mapping of Patch Embedding. The original 16*16*3 slice becomes a vector Z with a length of 768, and the entire image becomes a dimension of 14*14*768, which is then flattened into a 196*768 two-dimensional matrix. In addition, a learnable vector [CLS] is concatenated in the vector head. Then, the position code and modality code identifier of the image are concatenated into the vector Z, which is input into the Transformer encoder layer for encoding. At the same time, the standardized diagnostic report is converted into a binary identifier, and then the modality identifier is added, which is input into the BERT model for text encoding to obtain the encoding vector .

[0059] S4, the projection module embeds the image representation Z and the text embedding representation Z obtained by the multimodal data encoding module Perform modal projection fusion. Use contrastive learning to learn the positive and negative pairs of two modalities. The loss function of contrastive learning is , by minimizing the loss function , maximizing the similarity between each image and its corresponding text description while minimizing the similarity with other text descriptions.

[0060] S5. In the decoding module, a new image-text pair is generated, the image with missing diagnosis report is encoded, and then the similarity between the image and text embedding is calculated, and the most similar ultrasound report is output. For image data with a small amount of single-modality data, the given ultrasound diagnosis image is generated by encoding the text and then calculating and outputting the most similar image.

[0061] The above embodiments are only used to illustrate the design ideas and features of the present invention, and their purpose is to enable those skilled in the art to understand the content of the present invention and implement it accordingly. The protection scope of the present invention is not limited to the above embodiments. Therefore, any equivalent changes or modifications made based on the principles and design ideas disclosed by the present invention are within the protection scope of the present invention.

Claims

1. A data enhancement method based on multimodal language alignment, characterized in that: The following steps are involved: (1) Collecting multimodal medical data of patients in the electronic medical record system; wherein the multimodal medical data includes basic information, diagnostic reports during the observation window, and various types of image data; (2) preprocessing the multimodal medical data collected in step (1) to reset the sizes of various images to the same fixed value and to clean and standardize the diagnostic reports corresponding to the images; (3) Performing image data encoding and text data encoding on the preprocessed image and its corresponding diagnostic report, respectively, to obtain encoded image embedding representation and text embedding representation; (4) Perform modal projection fusion on the encoded image embedding representation and the encoded text embedding representation, learn positive and negative samples by using the contrastive learning method to construct a contrastive learning loss function; determine the aligned image embedding representation and text embedding representation with the goal of minimizing the contrastive learning loss function; (5) Based on the aligned image embedding representation and text embedding representation determined in step (4), a corresponding diagnosis report is generated for the image, and a corresponding image is generated for the diagnosis report to obtain an aligned image-text pair.

2. The data enhancement method based on multimodal language alignment according to claim 1, characterized in that: In step (1), the basic information includes age and gender; and the various types of image data include CT images, ultrasound images, electrocardiograms, and magnetic resonance images.

3. The data enhancement method based on multimodal language alignment according to claim 1, characterized in that: In the step (3), encoding the preprocessed image data to obtain an encoded image embedding representation specifically includes: (3.1) Slicing the preprocessed image to obtain multiple slice images corresponding to the image; (3.2) Slice embedding via linear mapping to convolve each slice image into a d-dimensional embedding vector; (3.3) For the embedding vector of each slice image obtained in step (3.2), add a learnable vector , and then splice the position code and modality code identifier corresponding to the slice image into the embedding vector to obtain the spliced ​​embedding vector; (3.4) The concatenated embedding vector is processed through multiple Transformer encoder layers to finally obtain the encoded image embedding representation.

4. The data enhancement method based on multimodal language alignment according to claim 3 is characterized in that: The Transformer encoder layer includes a multi-head attention mechanism, a feedforward neural network, and a normalization layer. The input and output process is expressed as: ； Among them, MSA stands for multi-head attention mechanism, LN stands for normalization layer, MLP stands for feedforward neural network, also stands for multi-layer perceptron; represents the output of the h-1th Transformer encoder layer, represents the intermediate vector of the h-th Transformer encoder layer, represents the output of the h-th Transformer encoder layer; , H indicates that there are H Transformer encoder layers in total; y indicates the prediction vector output by the last Transformer encoder layer.

5. The data enhancement method based on multimodal language alignment according to claim 1, characterized in that: In the step (3), the preprocessed diagnosis report is encoded with text data to obtain the encoded text embedding representation, which specifically includes: First, the preprocessed diagnosis report is converted into a binary encoding vector; then it and its modality identifier are concatenated and input into a bidirectional Transformer encoder. After being processed by multiple bidirectional Transformer encoders, the encoded text embedding representation is obtained.

6. The data enhancement method based on multimodal language alignment according to claim 5, characterized in that: The bidirectional Transformer encoder includes a bidirectional attention mechanism, a feedforward neural network and a normalization layer, and its input and output process is expressed as: ； Among them, Bi-A represents the bidirectional attention mechanism. represents the output of the w-1th bidirectional Transformer encoder, represents the intermediate vector of the w-th bidirectional Transformer encoder, represents the output of the w-th bidirectional Transformer encoder, , W means there are W bidirectional Transformer encoders, Represents the encoded text embedding representation output by the last bidirectional Transformer encoder.

7. The data enhancement method based on multimodal language alignment according to claim 1, characterized in that: In step (4), the loss function of contrastive learning is expressed as: ； in, represents the loss function of contrastive learning, N is the total number of samples, represents the image embedding representation of the i-th sample, represents the text embedding representation of the jth sample, express and The cosine similarity between express and The cosine similarity between is a hyperparameter used to tune the probability distribution.

8. The data enhancement method based on multimodal language alignment according to claim 1, characterized in that: In the step (5), generating a corresponding diagnosis report for the image specifically comprises: firstly encoding the image data of the given image to obtain an encoded image embedding representation; then calculating the cosine similarity between the image embedding representation and all text embedding representations determined in step (4), and selecting the text embedding representation corresponding to the maximum cosine similarity as the diagnosis report text corresponding to the image; The method of generating a corresponding image for a diagnostic report specifically includes: firstly encoding the text data of a given diagnostic report to obtain an encoded text embedding representation; then calculating the cosine similarity between the text embedding representation and all image embedding representations determined in step (4), and selecting the image embedding representation corresponding to the maximum cosine similarity as the image corresponding to the diagnostic report.

9. The data enhancement method based on multimodal language alignment according to claim 1, characterized in that: After step (5), the method further includes: Different post-processing processes are designed according to different downstream tasks, and the corresponding downstream tasks are performed based on the aligned image-text pairs obtained in step (5).

10. A data enhancement device based on multimodal language alignment, used to implement the data enhancement method based on multimodal language alignment according to any one of claims 1 to 9, characterized in that: include: A data acquisition module, used to collect multimodal medical data of patients in the electronic medical record system; wherein the multimodal medical data includes basic information, diagnostic reports during the observation window, and various types of image data; A data preprocessing module is used to preprocess the multimodal medical data collected by the data acquisition module to reset the sizes of various types of images to the same fixed value and clean and standardize the diagnostic reports corresponding to the images; Each modality data encoding module includes an image data encoding module and a text data encoding module, wherein the image data encoding module is used to perform image data encoding on the preprocessed image to obtain an encoded image embedding representation; the text data encoding module is used to perform text data encoding on the preprocessed diagnosis report to obtain an encoded text embedding representation; A projection module, for performing modality projection fusion on the encoded image embedding representation and the encoded text embedding representation, learning positive and negative samples by using a contrastive learning method to construct a loss function for contrastive learning; determining the aligned image embedding representation and text embedding representation with the goal of minimizing the loss function for contrastive learning; and A decoding module is used to generate a corresponding diagnosis report for the image and a corresponding image for the diagnosis report according to the aligned image embedding representation and text embedding representation determined by the projection module, so as to obtain an aligned image-text pair.

Images