Text Embedding Representation Method and System for Medical Vision-Language Multimodal Tasks

Abstract

This invention discloses a text embedding representation method for medical vision-language multimodal tasks, comprising: Step 1, acquiring a medical image dataset with category labels, preprocessing it, and dividing it into a test set and a training set; Step 2, pre-training a neural network model based on Transformer block contextual information features and performing optimization training; Step 3, acquiring a dataset with medical image-text pairs, standardizing it, performing multi-class prediction, and obtaining image-label-text triple pairs; Step 4, performing predetermined operations on the triple pairs to obtain the text feature representation of the image label and the feature representation of the text in the image-text pair; Step 5, integrating the text feature representation of the image label into the feature representation of the text in the image-text pair to obtain a joint embedding feature representation; Step 6, using the t-SNE algorithm to reduce the dimensionality of the joint embedding feature representation to a two-dimensional space, thereby observing the feature distribution in the implicit embedding space.

Description

Technical Field

[0001] This invention relates to the field of multimodal medicine in computer vision and natural language processing, specifically to a text embedding representation method and system for medical vision-language multimodal tasks. Background Technology

[0002] Medical vision-language multimodal tasks typically include visual question answering, image description, image-text retrieval, and image reasoning. These tasks are cross-modal, high-order cognitive tasks. In visual question answering (VQA), the computer takes an image and a question as input, and through understanding and reasoning on both the visual (image) and natural language (question) modalities, outputs a natural language answer as the corresponding response. Medical visual question answering is a branch of visual question answering tasks. It outputs natural language to answer a series of medical or clinical questions based on the visual content of medical images. Medical visual question answering can achieve intelligent diagnosis, provide greater convenience for doctors, and address issues such as insufficient medical resources and difficulty in accessing medical care, making it of great significance. Similarly, medical image description can describe medical images, which also greatly facilitates medical imaging diagnosis.

[0003] However, in these tasks, there is a lack of explicit alignment information between images and text, making alignment modeling a weakly supervised learning task. Cross-modal feature alignment is one of the challenges in cross-modal tasks. Text itself is a high-level abstraction of the real world, while images have a lower level of abstraction. Although images contain rich information, they also contain more noise, and as high-dimensional data, they are more difficult for computers to "understand." Therefore, due to the heterogeneity of text and image data, cross-modal semantic alignment often becomes difficult in cross-modal feature fusion. Secondly, in medical multimodal tasks, the content features contained in images are monotonous, and medical language also has monotonous features, which leads to greater inference difficulty.

[0004] In general vision-language tasks, images often contain rich semantic information. The Faster R-CNN object detector can extract feature maps of dozens or even hundreds of objects from common-sense images, and multiple labels also provide guidance for text feature encoding, greatly aiding cross-modal tasks. However, medical images lack the rich features of common-sense images, offering more global features such as lesions and organs. Therefore, fully utilizing medical image features is crucial for medical multimodal tasks. Secondly, medical language is often more difficult and harder to understand than common-sense language, posing greater challenges to text semantic understanding and comprehension.

[0005] The Oscar method designs novel anchor points as a shared space for images and text in the pre-training task, improving the semantic space of images and text, and learning the spatial relationship between each image and text separately. End-to-end visual language pre-training (E2E-VLP) incorporates a learning-to-attend mechanism to address semantic barriers. These methods, however, require large-scale unsupervised pre-training support, necessitating significant computational resources and time investment. Summary of the Invention

[0006] This invention is made to solve the above-mentioned problems, and aims to provide a text embedding representation method and system for medical vision-language multimodal tasks.

[0007] This invention provides a text embedding representation method for medical vision-language multimodal tasks, characterized by the following steps: Step 1, acquiring a medical image dataset with category labels, preprocessing the images in the medical image dataset, and dividing the medical image dataset into a test set and a training set; Step 2, pre-training a neural network model for medical image classification based on contextual information features of Transformer coding blocks, optimizing the neural network model based on the training set to obtain the trained neural network model; Step 3, acquiring a dataset with medical image-text pairs, and standardizing the images and text... Step 4: Perform predefined operations on the image-label-text triplet pairs to obtain the text feature representation of the image label and the feature representation of the text in the image-text pair. Step 5: Integrate the text feature representation of the image label into the feature representation of the text in the image-text pair to obtain the joint embedding feature representation, and input it into the Transformer. Step 6: Use the t-SNE algorithm to reduce the dimensionality of the joint embedding feature representation to a two-dimensional space, thereby observing the feature distribution in the implicit embedding space.

[0008] The text embedding representation method for medical vision-language multimodal tasks provided by this invention may also have the following features: wherein, in step 1, the preprocessing includes: adjusting the input image to a uniform size of 224*224 and performing standardization preprocessing.

[0009] The text embedding representation method for medical vision-language multimodal tasks provided by this invention may also have the following features: In step 2, the neural network model is a neural network model with dynamic attention and contextual information association, the Transformer encoding block is a Transformer encoding block with dynamic attention model used on a two-dimensional feature map, and the optimization training process is as follows: calculate the classification cross-entropy loss through the cross-entropy loss function, construct an image feature classifier based on the backpropagation algorithm and using a stochastic gradient descent optimizer, optimize the network parameters, iteratively update the parameters according to the set learning rate, and evaluate the accuracy. When the accuracy is greater than 95%, the neural network model training is completed.

[0010] The text embedding representation method for medical vision-language multimodal tasks provided by this invention may also have the following features: In step 3, the image standardization process is to adjust the input image to a uniform size of 224*224, and the text standardization process is to process the maximum text length to 35.

[0011] The text embedding representation method for medical vision-language multimodal tasks provided by this invention may also have the following features: In step 4, the predetermined operation is as follows: Initialize the embedding layer of the BERT text language model. First, calculate the sum of word vector embedding, position embedding, and token embedding of the triplet pair of image-label-text to obtain the sum of text embeddings. Then, normalize the sum of text embeddings through layer normalization. Finally, reduce the noise of the samples and further standardize the data through dropout operation, thereby successfully encoding the text of the image label and obtaining the text feature representation of the image label. Repeat the predetermined operation to obtain the feature representation of the text in the image-text pair.

[0012] The text embedding representation method for medical vision-language multimodal tasks provided by this invention may also have the following feature: in step 5, the text feature representation of the image label is processed by one or more operations of linear transformation and dimensional transformation, and associated with the feature representation of the Chinese text in the image text pair.

[0013] The text embedding representation method for medical vision-language multimodal tasks provided by this invention may also have the following feature: In step 5, the process of integrating the text feature representation of the image label into the feature representation of the image-text pair is as follows: Heterogeneous image type information is integrated into the text features. Using the maximum length of the text encoding as a benchmark, the encoded label text features are reconstructed to the maximum length of the text encoding, that is, the low-dimensional features are averaged across each dimension of the token embedding, as shown in the following formula:

[0014]

[0015] in, The representation of the embedding of image type labels. The representation of text embedding, n tp and n q These represent the lengths of the type point marker and the question marker, respectively, and ME indicates the computation length of n. tp -D average value, and in n q - The average value is extended in the -D dimension, that is, the representation of the image type label embedding is used as a guide to shift the text embedding itself in the embedding space, and then the transformed E tp Add to E q In order to generate new embedded E TQ .

[0016] The text embedding representation method for medical vision-language multimodal tasks provided by this invention may also have the following feature: In step 5, after inputting the joint embedding feature representation into the Transformer, a normalization layer is added before the feedforward neural network and the multi-head attention layer using the GeLU activation function on the basis of the Transformer.

[0017] The text embedding representation method for medical vision-language multimodal tasks provided by this invention may also have the following feature: In step 6, the neural network model learns implicit knowledge of image types in advance when initializing vector features. When different types of features have independent spaces in the low-dimensional space, it proves that the neural network model incorporates type knowledge during initialization.

[0018] This invention provides a text embedding representation system for medical vision-language multimodal tasks, characterized by: a medical image pre-training module, which acquires a medical image dataset with category labels, preprocesses the images in the medical image dataset, divides the medical image dataset into a test set and a training set, pre-trains a neural network model for medical image classification based on contextual information features of Transformer encoding blocks, and optimizes the neural network model based on the training set to obtain a trained neural network model; and an automatic type knowledge reasoning module, which acquires a dataset with medical image-text pairs, standardizes the images and text, and performs automatic reasoning based on the test set. The trained neural network model performs multi-class prediction, predicting image type knowledge and obtaining image-label-text triplet pairs. The spatial embedding representation calculation module includes a type knowledge fusion unit. The spatial embedding representation calculation module performs predetermined operations on the image-label-text triplet pairs to obtain the text feature representation of the image label and the feature representation of the text in the image-text pair. The text feature representation of the image label is integrated into the feature representation of the text in the image-text pair using the type knowledge fusion unit to obtain the joint embedding feature representation, which is then input into the Transformer. The t-SNE algorithm is used to reduce the dimensionality of the joint embedding feature representation to a two-dimensional space, thereby observing the feature distribution in the implicit embedding space.

[0019] The role and effect of invention

[0020] According to the present invention, a text embedding representation method for medical vision-language multimodal tasks comprises the following steps: Step 1, acquiring a medical image dataset with category labels, preprocessing the images in the medical image dataset, and dividing the medical image dataset into a test set and a training set; Step 2, pre-training a neural network model for medical image classification based on contextual information features of Transformer coding blocks, optimizing the neural network model based on the training set to obtain a trained neural network model; Step 3, acquiring a dataset with medical image-text pairs, and standardizing the images and text. The algorithm first performs multi-class prediction based on the test set and the trained neural network model to predict image type knowledge, obtaining image-label-text triplet pairs; second, it performs predetermined operations on the image-label-text triplet pairs to obtain the text feature representation of the image label and the feature representation of the text in the image-text pair; third, it integrates the text feature representation of the image label into the feature representation of the text in the image-text pair to obtain the joint embedding feature representation, and inputs it into the Transformer; fourth, it uses the t-SNE algorithm to reduce the dimensionality of the joint embedding feature representation to a two-dimensional space, thereby observing the feature distribution in the implicit embedding space.

[0021] Therefore, the general text embedding representation method for medical vision-language multimodal tasks provided by this invention takes into account the characteristics of medical problems and images, and does not use the pre-training methods in traditional vision-language tasks. Instead, it utilizes the type knowledge contained in the medical images themselves, and proposes a new method to improve the cross-modal feature alignment problem in medical vision-language multimodal tasks through cross-modal transformation, and optimizes the embedding feature relationship in the embedding space. This embedding representation can be widely applied to vision-language multimodal tasks in various medical scenarios.

[0022] Furthermore, unlike general word embeddings, the TQ embeddings proposed in this invention fully incorporate type knowledge from the image in the implicit embedding space, resulting in a more reasonable distribution of text representations in the space. Specifically, different types of text representations have independent embedding spaces, meaning that different types of text representations are farther apart in the embedding space, while text representations of the same type are closer together. Experiments show that the positional offset in the text space further affects the representation of image features, thereby improving the model's ability to align features across modalities and bringing significant performance improvements to medical cross-modal tasks. Attached Figure Description

[0023] Figure 1 This is a flowchart of a text embedding representation method for medical vision-language multimodal tasks in an embodiment of the present invention;

[0024] Figure 2 This is a graph composed of image-type label pairs in the dataset in an embodiment of the present invention;

[0025] Figure 3 This is a structural diagram of the Transformer coding block in an embodiment of the present invention;

[0026] Figure 4 This is a process diagram illustrating the embedded changes in an embodiment of the present invention;

[0027] Figure 5 This is a visualization of the spatial representation before and after TQ embedding in an embodiment of the present invention. Detailed Implementation

[0028] To make the technical means, creative features, objectives and effects of the present invention easy to understand, the following embodiments, in conjunction with the accompanying drawings, specifically illustrate a text embedding representation method and system for medical vision-language multimodal tasks.

[0029] In this embodiment, a text embedding representation method and system for medical vision-language multimodal tasks are provided.

[0030] Figure 1 This is an algorithm flowchart of the text embedding representation method for medical vision-language multimodal tasks in this embodiment.

[0031] like Figure 1 As shown, the text embedding representation method for medical vision-language multimodal tasks involved in this embodiment includes the following steps:

[0032] Step S1: Obtain a medical image dataset with category labels, preprocess the images in the medical image dataset, and divide the medical image dataset into a test set and a training set.

[0033] In this embodiment, 727 medical images with type labels were crawled from public websites using a web crawler. The main image types included chest, abdomen, and head. The images were then standardized, cropped to 224*224 pixels, and converted into feature vectors for use by the model.

[0034] Step S2: Pre-train a neural network model for medical image classification based on contextual information features of Transformer coding blocks, and optimize the neural network model based on the training set to obtain the trained neural network model.

[0035] In this embodiment, the neural network model is a neural network model with dynamic attention and context information association, and the Transformer encoding block is a Transformer encoding block with a dynamic attention model used on a two-dimensional feature map.

[0036] The specific process is as follows: A neural network model for medical image classification based on contextual information features from Transformer encoding blocks is pre-trained. The network structure is CoTNet152. This network mainly replaces the 3*3 convolutional blocks of the traditional ResNet residual network with encoding blocks based on the Transformer structure. The query (size H*W*C) and key (size H*W*C) calculation results are concatenated to (H*W*2C). Then, 1*1 convolutional blocks are used to reduce the number of channels. The dimensionality-reduced features are weighted with the values ​​in the Transformer to obtain the fused features. Finally, the fully connected layer structure is modified to make the network a multi-classification network.

[0037] After initializing the network, the preprocessed image dataset is input into the network for optimization training. The batch size is set to 64, the learning rate to 0.2, the momentum to 0.9, the number of training epochs to 15, and the number of classes to 3 (chest, abdomen, and head). Then, the classification cross-entropy loss L is calculated using the cross-entropy loss function, as shown in the following formula:

[0038]

[0039] Where y ic Let p represent the sign function, which takes the value 1 if the true class of sample i is equal to c, and 0 otherwise. ic This represents the probability that observed sample i belongs to category c, and N represents the number of samples.

[0040] In addition, an image feature classifier is constructed based on the backpropagation algorithm and using a stochastic gradient descent optimizer to optimize network parameters. The parameters are then iteratively updated, and the evaluation metric is accuracy. When the accuracy is greater than 95%, the network training is complete.

[0041] Step S3: Obtain a dataset with medical image-text pairs, standardize the images and text, adjust the input images to a uniform size of 224*224, and reduce the maximum text length to 35.

[0042] Based on the test set, a trained neural network model performs multi-class classification prediction to predict image type knowledge, obtaining image-label-text triplet pairs. In this embodiment, the images in the dataset are used as keys, and the network performs multi-class classification prediction to obtain the type knowledge label corresponding to each image, which is then used as the value. These key-value pairs are stored in a dictionary. This results in the image-label-text triplet pair for each sample. The embedding layer is initialized as a text encoder, as shown below:

[0043] The BERT text language model's embedding layer is initialized by calculating the sum of word embeddings, position embeddings, and token embeddings for each tag in a triplet. The word embeddings have an input dimension of 30522 and an output dimension equal to the hidden layer size (768). The position embeddings have an input dimension of 512 and an output dimension of 768. The token embeddings have an input of 2 and an output of 768. Then, layer normalization is applied to normalize the sum of the text embeddings. Finally, dropout is used to reduce noise in the samples and further standardize the data, resulting in the embedded knowledge after tag encoding, i.e., the feature representation of the tag.

[0044] Image features are encoded using a pre-trained image model as the backbone network, resulting in a 2048-dimensional feature vector. This feature vector is then downsampled to a hidden size of 768-D using a 1x1 convolutional kernel. The dimensionality-reduced features are used for image feature embedding, i.e., Ei. I .

[0045] Step S4: Perform a predetermined operation on the triplet pair of image-label-text to obtain the text feature representation of the image label and the feature representation of the text in the image-text pair.

[0046] The predetermined operation is as follows: reinitialize the embedding layer of the BERT text language model, then extract the text part of the triplet pair separately, and then calculate the sum of the word vector embedding, position embedding and token embedding of the text. Similarly, normalize the sum of the text embeddings through the layer normalization operation to obtain the feature representation of the text in the image text pair.

[0047] Step S5: The text feature representation of the image label is integrated into the feature representation of the Chinese text in the image text pair to obtain a joint embedding feature representation, which is then input into the Transformer. After inputting the joint embedding feature representation into the Transformer, a normalization layer is added before the feedforward neural network and the multi-head attention layer using the GeLU activation function.

[0048] Specifically, the text feature representation of the image label is processed through one or more operations, including linear transformation and dimensionality transformation, and associated with the feature representation of the Chinese text in the image text pair.

[0049] In this embodiment, the type embedding is remodeled as follows: the type knowledge fusion part integrates heterogeneous image type information into text features, and reconstructs the encoded label text features to the maximum text encoding length based on the maximum text encoding length, that is, the low-dimensional features are averaged and distributed to each token embedding.

[0050]

[0051] in, The representation of the embedding of image type labels. Representation of text embedding. n tp and n q These represent the lengths of the type dot marker and the text marker, respectively. ME indicates the calculation of n. tp -D average value, and in n q-The average value is extended in the -D dimension. In other words, this embodiment uses the representation of the image type label embedding as a guide to offset the text embedding itself in the embedding space.

[0052] Figure 2 This is the image-type label pair composition graph in the dataset of this embodiment.

[0053] Finally, the converted E tp Add to E q In order to generate new embedded E TQ This offset process is as follows: Figure 2 As shown.

[0054] E = {E CLS E I E SEP E TQ E SEP}

[0055] By fusing all the above features using the above structure, a fused joint embedding is obtained, where the image feature embedding is E. I E TQ This represents the combined embedding after text embedding and tag embedding transformation. The [CLS] token and [SEP] token are encoded as E. CLS and E SEP .

[0056] Figure 3 This is a structural diagram of the Transformer coding block in this embodiment.

[0057] Then the joint embedding feature representation E is input into the Transformer, such as... Figure 3 As shown. The feedforward neural network layer, or FFN for short, has the following structure:

[0058] FFN(h i ) = GeLU(h i W 1 +b 1 W 2 +b 2

[0059] Among them, h i It is the vector of the hidden layer. W 1 b 1 W 2 b 2 These are the parameters of FFN, and GeLU is the activation function. For example... Figure 3 As shown, in order to suppress overfitting in downstream tasks, a normalization operation is added before the multi-head attention layer and the feedforward neural network layer.

[0060] Step S6: Use the t-SNE algorithm to reduce the dimensionality of the joint embedding feature representation to a two-dimensional space, thereby observing the feature distribution in the implicit embedding space.

[0061] Figure 4 This is a process diagram illustrating the embedded changes in an embodiment of the present invention.

[0062] Figure 5 This is a visualization of the spatial representation before and after TQ embedding in an embodiment of the present invention.

[0063] like Figure 5 As shown in the figure, the embedding distribution in two-dimensional space is illustrated. The left side represents the distribution representation before fusion, and the right side represents the distribution representation after fusion.

[0064] like Figure 5 As shown in (a), without using TQ embeddings that incorporate type knowledge, and only using text representations as features, the embeddings are irregularly distributed before being fused with the image. Furthermore, the results after fusion with image features also exhibit an irregular distribution, clearly showing no distributional pattern among the embeddings. Conversely, when using TQ embeddings, as... Figure 5 As shown in (b), the type points and questions are encoded into text representations that are spatially correlated with image embeddings of different types and have approximately equal distances. This demonstrates that combining the encoding of questions and type points as text representations can make the embedding distribution more regular. More importantly, the embeddings after fusion with image features also exhibit a regular distribution, and different types of embeddings have class relationships in space. This also shows that the neural network model can learn implicit knowledge of image types in advance when initializing vector features. Therefore, considering the characteristics of medical images, this novel feature representation method provides excellent assistance for multimodal tasks.

[0065] The success of machine learning algorithms depends on appropriate data representation, which can effectively enhance the semantic relationships (semantic alignment) of cross-modal data in cross-modal tasks. Observing the visualized spatial representation reveals that when different types of features have independent spaces in a low-dimensional space, it can be assumed that type knowledge is well incorporated during model initialization. Specifically, text representations of different types are farther apart in the embedding space, while text representations of the same type are closer together, further improving the model's ability to align cross-modal features. This establishes a relative relationship between the semantics of features from different modalities in space, resulting in a significant performance improvement for medical image-text cross-modal tasks.

[0066] This embodiment also provides a text embedding representation system for medical vision-language multimodal tasks, including:

[0067] The medical image pre-training module uses the methods in steps S1 to S2 of this embodiment to perform medical image pre-training.

[0068] The automatic type knowledge reasoning module performs automatic type knowledge reasoning using the method in step S3 of this embodiment.

[0069] The spatial embedding representation calculation module includes a type knowledge fusion unit, which performs spatial embedding representation calculation using the methods in steps S4 to S6 of this embodiment.

[0070] The role and effect of the embodiments

[0071] According to the text embedding representation method for medical vision-language multimodal tasks involved in this embodiment, the specific steps are as follows: Step 1, obtain a medical image dataset with category labels, preprocess the images in the medical image dataset, and divide the medical image dataset into a test set and a training set; Step 2, pre-train a neural network model for medical image classification based on the contextual information features of Transformer coding blocks, optimize the neural network model based on the training set, and obtain the trained neural network model; Step 3, obtain a dataset with medical image-text pairs, and standardize the images and text. The process involves processing the data and performing multi-class predictions based on the trained neural network model on the test set to predict image type knowledge and obtain image-label-text triplet pairs. Step 4 involves performing predetermined operations on the image-label-text triplet pairs to obtain the text feature representation of the image label and the feature representation of the text in the image-text pair. Step 5 involves integrating the text feature representation of the image label into the feature representation of the text in the image-text pair to obtain the joint embedding feature representation, which is then input into the Transformer. Step 6 involves using the t-SNE algorithm to reduce the dimensionality of the joint embedding feature representation to a two-dimensional space, thereby observing the feature distribution in the implicit embedding space.

[0072] Therefore, the general text embedding representation method for medical vision-language multimodal tasks provided in this embodiment takes into account the characteristics of medical problems and images, and does not use the pre-training methods in traditional vision-language tasks. Instead, it utilizes the type knowledge contained in the medical images themselves, and proposes a new method to improve the cross-modal feature alignment problem in medical vision-language multimodal tasks through cross-modal transformation, and optimizes the embedding feature relationship in the embedding space. This embedding representation can be widely applied to vision-language multimodal tasks in various medical scenarios.

[0073] Furthermore, unlike general word embeddings, the TQ embeddings proposed in this embodiment fully incorporate type knowledge from the image in the implicit embedding space, resulting in a more reasonable distribution of text representations in the space. Specifically, different types of text representations have independent embedding spaces, meaning that different types of text representations are farther apart in the embedding space, while text representations of the same type are closer together. Experiments show that the positional offset in the text space further affects the representation of image features, thereby improving the model's ability to align cross-modal features and bringing significant performance improvements to medical cross-modal tasks.

[0074] The above embodiments are preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention.

Claims

1. A method for text embedding representation for medical vision-language multimodal tasks, characterized in that, The method comprises the following steps: Step 1, obtaining a medical image dataset with category labels, preprocessing the images in the medical image dataset, and dividing the medical image dataset into a test set and a training set; Step 2, pre-training a neural network model for medical image classification based on a Transformer encoding block, optimizing and training the neural network model based on the training set to obtain a trained neural network model; Step 3, obtaining a dataset with medical image-text pairs, standardizing the image and text, and performing multi-classification prediction based on the test set using the trained neural network model to predict image type knowledge and obtain image-label-text triplets; Step 4, performing a predetermined operation on the image-label-text triplets to obtain text feature representation of image labels and feature representation of text in image-text pairs; Step 5, integrating the text feature representation of the image labels into the feature representation of the text in the image-text pairs to obtain joint embedding feature representation and inputting it into the Transformer; Step 6, using the t-SNE algorithm to reduce the joint embedding feature representation to a two-dimensional space to observe the feature distribution in the implicit embedding space; In step 4, the predetermined operation is: Initializing the embedding layer of the Bert text language model, first calculating the sum of the word vector embedding, position embedding and token embedding of the image-label-text triplet to obtain the sum of the text embedding, then normalizing the sum of the text embedding through layer normalization operation, and finally reducing the noise of the sample through dropout operation and further standardizing the data to successfully encode the text of the image label to obtain the text feature representation of the image label. Repeat the predetermined operation to obtain the feature representation of the text in the image-text pair, In step 5, the process of integrating the text feature representation of the image labels into the feature representation of the text in the image-text pairs is: Integrate heterogeneous image type information into text features, and reconstruct the encoded label text features to the maximum length of text encoding as a benchmark, that is, average the low-dimensional features to each dimension token embedding, the formula is as follows: wherein, a representation of an embedding of an image type label, a representation of an embedding of text, n tp and n q denote the length of the type point mark and the question mark, respectively, ME denotes the mean of n tp -D, and the mean is expanded in the n q -D dimensional space, i.e. using the representation of the image type label embedding as a guide, the text embedding is shifted in the embedding space, and the transformed E tp is added to E q to generate a new embedding E TQ .

2. The text embedding representation method for medical visual-linguistic multi-modal tasks according to claim 1, wherein: wherein In step 1, the preprocessing includes adjusting the input image to a uniform size of 224*224 and performing standardization preprocessing.

3. The text embedding representation method for medical visual-linguistic multi-modal tasks according to claim 1, wherein: wherein In step 2, the neural network model is a neural network model with dynamic attention and context information correlation, the Transformer encoding block is a Transformer encoding block with dynamic attention model used on a two-dimensional feature map, and the optimization training process is: The classification cross-entropy loss is calculated by a cross-entropy loss function, an image feature classifier is constructed based on a back propagation algorithm and using a stochastic gradient descent optimizer, network parameters are optimized, parameter iterative updating is performed according to a set learning rate, and the accuracy rate is an evaluation index, and the neural network model training is completed when the accuracy rate is greater than 95%.

4. The text embedding representation method for medical visual-language multi-modal tasks according to claim 1, wherein: wherein In step 3, the standardization processing of the image is to adjust the input image to a uniform size of 224*224, and the standardization processing of the text is to process the maximum length of the text to 35.

5. The text embedding representation method for medical visual-language multi-modal tasks according to claim 1, wherein: wherein In step 5, the text feature representation of the image label is processed by one or more operations of linear transformation and dimension conversion, and is associated with the text feature representation of the image text pair.

6. The text embedding representation method for medical visual-language multi-modal tasks according to claim 1, wherein: wherein In step 5, the joint embedding feature representation is input into a Transformer, and on the basis of the Transformer, a layer normalization is added before a feed-forward neural network and a multi-head attention layer using a GeLU activation function.

7. A text embedding representation system for medical vision-language multimodal tasks, characterized in that, Comprise: a medical image pre-training module, which acquires a medical image dataset with category labels, pre-processes images in the medical image dataset, divides the medical image dataset into a test set and a training set, pre-trains a neural network model for medical image classification based on context information features of a Transformer encoding block, optimizes and trains the neural network model based on the training set, and obtains a trained neural network model; a type knowledge automatic reasoning module, which acquires a dataset with medical image-text pairs, standardizes images and texts, performs multi-classification prediction based on the test set and the trained neural network model, predicts image type knowledge, and obtains a triple of image-label-text; a spatial embedding representation calculation module, which comprises a type knowledge fusion unit, obtains text feature representation of image labels and text feature representation of texts in image text pairs by performing predetermined operations on the triple of image-label-text, integrates the text feature representation of image labels into the text feature representation of texts in image text pairs using the type knowledge fusion unit, obtains joint embedding feature representation, inputs the joint embedding feature representation into a Transformer, and uses a t-SNE algorithm to reduce the dimensionality of the joint embedding feature representation to a two-dimensional space, so as to observe feature distribution in an implicit embedding space, wherein the predetermined operations are: The embedding layer of the initialized Bert text language model is obtained by calculating the sum of the word vector embedding, the position embedding and the token embedding of the image-label-text triple pair, normalizing the sum of the text embedding by layer normalization operation, and reducing the noise of the sample by dropout operation and further standardizing the data, so as to successfully encode the text of the image label, obtain the text feature representation of the image label, repeat the predetermined operation, and obtain the text feature representation of the image text pair, The process of fusing the text feature representation of the image label into the text feature representation of the image text pair is: The heterogeneous image type information is fused into the text feature, and the encoded label text feature is reconstructed to the maximum length of the text encoding according to the maximum length of the text encoding as the benchmark, that is, the low-dimensional feature is averaged to each dimensional token embedding, and the formula is as follows: wherein, a representation of an embedding of an image type label, a representation of an embedding of text, n tp and n q denote the length of the type point mark and the question mark, respectively, ME denotes the mean of n tp -D, and the mean is expanded in n q -D dimensions, i.e. using the representation of the image type label embedding as a guide, the text embedding is shifted in the embedding space, and the transformed E tp is added to E q to generate a new embedding E TQ .

Images