A cross-modal retrieval method based on fetal congenital heart disease ultrasound image-diagnosis report

Abstract

The present invention discloses a cross-modal retrieval method based on fetal congenital heart disease ultrasound image-diagnosis report, including: pre-processing a cross-modal fetal congenital heart disease ultrasound image text data set; using a feature extraction network to extract image features, text features and semantic features during training; inputting the features of each modality into a parameter-shared cross-modal attention layer to obtain features of each modality with semantic association information; calculating the contrast center loss of image features and text features with semantic features respectively; inputting feature vectors into a parameter-shared label prediction layer, generating predicted labels, and calculating label prediction losses; using a trained optimal model to generate low-dimensional features from multi-modal data in a database during retrieval; for query data, first generating low-dimensional features, and then calculating its similarity with other modal features; sorting by similarity, and returning cross-modal instances that meet the conditions. The present invention effectively improves the accuracy of cross-modal multi-label ultrasound image retrieval.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence cross-modal ultrasound image text retrieval, and more specifically, relates to a cross-modal retrieval method based on fetal congenital heart disease ultrasound image-diagnosis report. Background Art

[0002] With the development of medical technology, hospitals generate a large number of medical scan images every day. In the medical field, ultrasound images are of great significance for modern clinical diagnosis. For example, ultrasound, X-ray, MRI and other ultrasound images provide doctors with a large amount of pathological information. Doctors analyze the information in ultrasound images and form ultrasound image diagnosis reports. In practice, providing doctors with different mode search results (i.e., X-ray, MRI, PET and corresponding medical imaging reports) allows doctors to obtain more comprehensive disease information, which is of great significance for assisting clinical diagnosis and multimodal medical information management. However, there are large cross-modal differences between ultrasound images and texts. In addition, a diagnostic report and its corresponding ultrasound image often show multiple diseases. Medical data is often multi-label data. The semantics corresponding to ultrasound images and texts are relatively complex. How to reduce the differences between modalities while learning the semantic relevance of images and texts is a difficult point. Summary of the invention

[0003] To address the above issues, the cross-modal retrieval method based on ultrasound images and diagnostic reports of fetal congenital heart disease can make full use of the semantic labels of image-text pairs. A semantic feature subnetwork with label information as input generates semantic features, which are combined with the predicted labels generated by the label prediction layer to achieve simultaneous supervision of the model in feature space and semantic space to generate effective image and text features. In addition, a cross-modal attention layer is used to further extract the correlation information between the image and text modalities. Ultimately, the cross-modal retrieval accuracy of multi-label ultrasound image text data can be effectively improved.

[0004] The present invention aims to solve the above problems of the prior art and proposes a cross-modal retrieval method based on fetal congenital heart disease ultrasound image-diagnosis report, which specifically includes the following steps:

[0005] 1) Preprocess the cross-modal ultrasound image text dataset, use the word segmentation tool to segment the medical text, and then train the word vector through the bag-of-words model to convert the label information into a one-hot vector. Each sample in the cross-modal medical dataset includes an ultrasound image text pair and its corresponding one-hot vector.

[0006] 2) Setting up a model, which includes an image feature extraction network, a text feature extraction network, a semantic feature extraction network, a cross-modal attention module, a label prediction layer, and a loss function.

[0007] The image feature extraction network is composed of a convolutional neural network. Through convolution, pooling, activation function, and fully connected layer, the original image is output as an image feature vector. The text feature extraction network first converts the text into a vector form through a pre-built dictionary. Each number in the vector corresponds to a 300-dimensional word vector. Through word vector embedding, the vector representing the text is converted into a p×300 matrix, where p is the length of the text, and then the matrix is ​​input into the convolutional neural network to obtain a text feature vector. The semantic feature extraction network is composed of a fully connected layer and an activation function. The one-hot vector representing the semantic label is output as a semantic feature vector with the same dimension as the image or text feature. The cross-modal attention module is composed of a fully connected layer and a Sigmoid activation function. The previously obtained features are input into the cross-modal attention module. After learning the feature weights through the fully connected layer and the Sigmoid activation function, the features of each modality are multiplied to obtain the attention features of each modality. Then, after an identity mapping, the features of each modality are input into a parameter-sharing fully connected layer, and the final feature vector is output to a common feature space. The label prediction layer is composed of a fully connected layer, which maps image features, text features and semantic features into vectors with the same dimension as the label vector. The loss function includes contrast center loss in feature space and label prediction loss in label space.

[0008] 3) Model training: Use the feature extraction network to extract image features, text features and semantic features; input the image-text pairs and the one-hot vector representing the semantic label into the corresponding feature extraction network to obtain the image features, text features and semantic features of the intermediate layer; input the features of the intermediate layer into the cross-modal attention layer, learn the feature weights through the fully connected layer and the Sigmoid activation function, and then multiply the features of each modality to obtain the attention features of each modality, and then input the features of each modality into a parameter-shared fully connected layer after an identity mapping, and the final feature vector is output to a common feature space; calculate the contrast center loss of the obtained image features and text features with the semantic features respectively; input the features of each modality into the parameter-shared label prediction layer, generate the predicted label, and then calculate the label prediction loss with the true label respectively; update the model parameters using the contrast center loss in the feature space and the label prediction loss in the label space; when the model parameters converge, save the optimal model.

[0009] 4) Using the model trained in step 3), map the images and text data in the dataset into low-dimensional feature vectors.

[0010] 5) For the query data, first generate a low-dimensional feature vector according to step 4), then calculate the Euclidean distance between the low-dimensional feature vector and the feature vectors of other modal data as the similarity, sort by similarity, and return the cross-modal instances that meet the conditions.

[0011] The advantages and beneficial effects of the present invention are as follows:

[0012] The present invention constructs a cross-modal ultrasound image text retrieval framework. In actual scenarios, an ultrasound image or a diagnostic report often shows multiple symptoms, so these image-text pairs generally correspond to multiple semantic categories. Cross-modal retrieval methods based on feature embedding often use semantic supervision information to supervise the model to generate features with inter-class discriminative properties, and then calculate feature similarity for retrieval. Some existing methods usually use triple loss or contrast loss to construct inter-class relationships in the feature space. In order to further utilize multiple semantic labels, the present invention generates high-dimensional semantic information in the feature space through a semantic label network, and supervises the generation of multimodal features by using semantic information in the feature space. In addition, the inter-class differences between ultrasound images are generally only reflected in a certain diseased area in the image, such as Figure 3 The difference between the diagnostic reports corresponding to the images is generally only reflected in the keywords describing the diseased area. Therefore, in order to extract the joint information of the diseased area in the image and the keywords in the diagnostic report, the present invention uses a cross-modal attention layer to jointly learn the high-level semantic information in the image features and text features with semantic features, referring to the residual attention mechanism proposed in . Finally, the entire model is optimized by the cross-modal contrast center loss in the feature space and the label prediction loss in the label space.

[0013] The present invention constructs a cross-modal retrieval framework of ultrasound images and diagnostic reports based on fetal congenital heart disease, which makes full use of the multiple semantic labels in medical data. A semantic feature network with label information as input generates semantic features in the feature space through a cross-modal contrast center loss supervision model to generate image and text features with cross-modal consistency and cross-modal inter-class discriminability. At the same time, a predicted label is generated through a label prediction layer, and a label prediction loss supervision model is calculated in the label space to generate features with high-level semantic information. In order to further extract the correlation information between multiple modalities, a shared cross-modal attention layer is used to output the final feature vector. Such features are semantically distinct and similar between different modalities. The accuracy of cross-modal multi-label ultrasound image retrieval is effectively improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 The present invention is an ultrasound image-diagnosis report cross-modal retrieval framework based on fetal congenital heart disease;

[0015] Figure 2 is a cross-modal attention layer;

[0016] Figure 3 It is a schematic diagram of the association of ultrasound image text labels. DETAILED DESCRIPTION

[0017] The following will describe the technical solutions in the embodiments of the present invention in detail in conjunction with the accompanying drawings in the embodiments of the present invention. The described embodiments are only part of the embodiments of the present invention.

[0018] like Figure 1 As shown in the figure, the cross-modal retrieval method based on fetal congenital heart disease ultrasound image-diagnosis report includes:

[0019] 1) Preprocess the cross-modal ultrasound image text dataset; each instance sample of the cross-modal ultrasound image text data contains an image-text pair and the corresponding semantic label. The text data is segmented using a word segmentation tool to obtain unique words, and then the word vector is trained through the CBOW model; the text data is segmented to retain words with a frequency of more than 5 times to form a dictionary, and word vector training is performed to obtain a word vector with a dimension of 300; each image is resized to the same size and data normalization is performed; the label information is converted into a one-hot vector. Each sample in the cross-modal ultrasound medical dataset includes an ultrasound image-text pair and its corresponding one-hot vector. Each image text instance is assigned a one-hot vector represents the dimension, where c is the number of categories. If the i-th instance belongs to the j-th category, y ij =1, otherwise y ij = 0, each image text instance can belong to a single semantic category or multiple semantic categories. The image sample is defined as A text sample is defined as and the semantic tag is defined as

[0020] 2) Figure 1 It is the model framework designed in the present invention, which includes: an image feature extraction network, defined as: where v′ i represents the image features, θ v represents the trainable parameters of the image modality sub-network, d represents the dimension of the common feature space, Represents the i-th ultrasound image sample. The network is a ResNet model pre-trained on the ImageNet dataset, which consists of a convolutional neural network. Through convolution, pooling, activation function, and fully connected layers, the original image is output as a 2048-dimensional image feature vector. The text feature extraction network is defined as: where t i ′ represents the text feature, θ t are the trainable parameters of the text modality sub-network, Represents the i-th ultrasound image sample. The word vector embedding matrix is ​​N*300, where N is the dictionary size. First, the text is converted into a vector form through a pre-built dictionary. Each number in the vector corresponds to a 300-dimensional word vector. Through word vector embedding, the vector representing the text is converted into a p×300 matrix, where p is the length of the text. The matrix is ​​then input into the convolutional neural network and then passed through a 2048-dimensional fully connected layer to obtain the text feature vector. The semantic feature extraction network is defined as: where s i ′ represents the semantic features of the middle layer, y i Represents the one-hot vector corresponding to the semantic label, θ y It is a trainable parameter of the semantic label sub-network, which consists of 2048- and 1024-dimensional fully connected layers and Relu activation functions. The one-hot vector representing the semantic label is output as a semantic feature vector with the same dimension as the image (text) feature. Figure 2 The figure shows the cross-modal attention module, which consists of a 1024-dimensional fully connected layer and a Sigmoid activation function. The previously obtained features are input into the cross-modal attention module, and the feature weights are learned through the fully connected layer and the Sigmoid activation function, which are defined as: M i (v) = f att (v i ′)M i (t) = f att (t i ′)M i (s) = f att (s i ′). i (v) M i (t), M i (s) represent the image feature weight, text feature weight and semantic feature weight extracted by the cross-modal attention layer, and the value range is (0,1], f att Represents a fully connected mapping layer. Then the weights are multiplied by the features of each modality to obtain the attention features of each modality, and then the features of each modality are obtained after an identity mapping: H i (v) = (1 + M i (v))·v′,H i (t) = (1 + M i (t))·t′,H i (s)=(1+M i (s))·s′, where v′, t′, s′ represent the intermediate layer image, text and semantic features extracted by the feature extraction network respectively. i (v) H i (t) and H i(s) Input a parameter-shared 1024-dimensional fully connected layer and output image features, text features, and semantic features v i ,t i and i into a common feature space; the label prediction layer consists of a c-dimensional fully connected layer, where c is the number of semantic labels in the dataset and the image feature v i , text features t i and semantic features i After the label prediction layer P generates the predicted label: Y v =P(V,θ p ), Y t =P(T,θ p ), Y s =P(S,θ p ),in, represents the image feature matrix from the kth image modality, represents the text feature matrix, represents the text feature matrix, Y v , Y t and Y s Denote the predicted label matrices corresponding to V, T, and S, respectively. P is a fully connected layer with c hidden units, where c is the number of categories in the dataset. Loss function: The loss function includes the contrastive center loss in the feature space and the label prediction loss in the label space.

[0021] 3) During model training, the feature extraction network of each modality extracts features, and after obtaining the final feature vector through the cross-modal attention module, the semantic feature center μ of each category in this training batch is calculated. j , then calculate the cross-modal contrast center loss L1, input the features of each modality into the label prediction layer, and obtain the predicted label Y v , Y t , Y s , and then calculate the label prediction loss L2 with the true label vector Y. Combining L1 and L2, and adding constraints in the label space and semantic space, we get the objective function of this method: L = L1 + α • L2. Among them, the hyperparameter α controls the contribution ratio of these two items. Through back propagation, the parameters of the model are updated using the stochastic gradient descent method. The accuracy evaluation indicator of the model is the mAP score. When the model parameters converge and the retrieval accuracy on the test data reaches the optimal, save the optimal model at this time.

[0022] The cross-modal contrast center loss L1 is calculated as follows:

[0023]

[0024] in represents the semantic feature center of the jth category, v i ,t i Respectively represent the image feature representation and text feature representation in the low-dimensional common feature space, n b represents the number of training samples in a mini-batch, c represents the number of semantic categories, and y ij =1 means that the i-th image-text pair belongs to the j-th semantic category, δ is a constant used to prevent the denominator from being equal to 0, v i , t i ,s i , respectively represent the i-th image feature, text feature and semantic feature.

[0025] The calculation formula of the label prediction loss L2 is as follows:

[0026]

[0027] where Y v , Y t and Y s They represent the predicted label matrix output by the label prediction layer, represents the true label matrix.

[0028] 4) Using the model trained in step 3), the images and text data in the dataset are mapped into low-dimensional feature vectors, in preparation for the next step of real-valued cross-modal retrieval.

[0029] 5) Retrieval stage: For the query data, first generate a low-dimensional feature vector according to step 4), then calculate the Euclidean distance between the low-dimensional feature vector and the feature vectors of other modal data as the similarity, sort by similarity, and return the cross-modal instances that meet the conditions.

[0030] The Euclidean distance is calculated as: x A , x B ,y A ,y B Represents the values ​​of the first and second dimensions of vector A and vector B respectively.

[0031] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0032] The above embodiments should be understood to be only used to illustrate the present invention and not to limit the protection scope of the present invention. After reading the contents of the present invention, technicians can make various changes or modifications to the present invention, and these equivalent changes and modifications also fall within the scope defined by the claims of the present invention.

Claims

1. A cross-modal retrieval method based on fetal congenital heart disease ultrasound image-diagnosis report, characterized in that: The following steps are involved: 1) Preprocess the cross-modal ultrasound image text dataset, use the word segmentation tool to segment the medical text, and then train the word vector through the bag-of-words model to convert the label information into a one-hot vector. Each sample in the cross-modal medical dataset includes an ultrasound image text pair and its corresponding one-hot vector. 2) Setting a model, which includes an image feature extraction network, a text feature extraction network, a semantic feature extraction network, a cross-modal attention module, a label prediction layer, and a loss function; The output of the cross-modal attention module is: H i (v) = (1 + M i (v))v′,H i (t) = (1 + M i (t))·t′,H i (s)=(1+M i (s))·s′, where v′, t′, s′ represent the intermediate layer image, text and semantic features extracted by the feature extraction network, respectively, and M i (v) M i (t), M i (s) represents the feature weights learned in the cross-modal attention module, H i (v) H i (t), H i (s) represent the image, text and semantic features after identity mapping respectively; 3) Model training: Use the feature extraction network to extract image features, text features, and semantic features; input the image-text pairs and the one-hot vector representing the semantic label into the corresponding feature extraction network to obtain the image features, text features, and semantic features of the intermediate layer; The features of the middle layer are input into the cross-modal attention layer. After learning the feature weights through the fully connected layer and the Sigmoid activation function, the features of each modality are multiplied to obtain the attention features of each modality. Then, after the identity mapping, the features of each modality are input into a parameter-shared fully connected layer, and the final feature vector is output to a common feature space. The obtained image features and text features are respectively calculated with the semantic features for the contrast center loss. The features of each modality are input into the parameter-shared label prediction layer. After the predicted labels are generated, the label prediction losses are calculated with the true labels respectively. The model parameters are updated using the contrast center loss in the feature space and the label prediction loss in the label space. When the model parameters converge, the optimal model is saved. Calculate the semantic feature center μ j , then calculate the cross-modal contrast center loss L1, input the features of each modality into the label prediction layer, and obtain the predicted label Y v , Y t , Y s , and then calculate the label prediction loss L2 with the true label vector Y, combine L1 and L2, and add constraints in the label space and semantic space at the same time, and get the objective function: L = L1 + αL2, α is a hyperparameter; The cross-modal contrast center loss L1 is calculated as follows: in represents the semantic feature center of the jth category, u z represents the semantic feature center of the z-th category, v i ,t i Respectively represent the image feature representation and text feature representation in the low-dimensional common feature space, n b represents the number of training samples in a mini-batch, c represents the number of semantic categories, and y ij =1 means that the i-th image-text pair belongs to the j-th semantic category, δ is a constant used to prevent the denominator from being equal to 0, v i , t i ,s i , respectively represent the i-th image feature, text feature and semantic feature; The calculation formula of the label prediction loss L2 is as follows: where Y v , Y t and Y s They represent the predicted label matrix output by the label prediction layer, Y = [y1, y2, ..., y nb ] represents the true label matrix; 4) Using the model trained in step 3), map the image and text data in the dataset into low-dimensional feature vectors; 5) For the query data, first generate a low-dimensional feature vector according to step 4), then calculate the Euclidean distance between the low-dimensional feature vector and the feature vectors of other modal data as the similarity, sort by similarity, and return the cross-modal instances that meet the conditions.

2. The cross-modal retrieval method based on fetal congenital heart disease ultrasound image-diagnosis report according to claim 1 is characterized by: The step 1) specifically includes: each instance sample of the cross-modal ultrasound image text data contains an image-text pair and a corresponding semantic label; among the unique words obtained after word segmentation of the text data, words with a frequency of occurrence of more than 5 times are retained to form a dictionary, and word vector training is performed to obtain a word vector; each image-text instance is assigned a semantic label, which is converted into a one-hot vector where c is the number of categories, Represents the vector dimension. If the i-th instance belongs to the j-th category, y ij =1, otherwise y ij =0, each image text instance can belong to a single semantic category or multiple semantic categories.

3. The cross-modal retrieval method based on fetal congenital heart disease ultrasound image-diagnosis report according to claim 1 is characterized by: The image feature extraction network in step 2) is composed of a convolutional neural network. Through convolution, pooling, activation function, and fully connected layers, the original image is output as an image feature vector; The text feature extraction network first converts the text into a vector form through a pre-built dictionary. Each number in the vector corresponds to a 300-dimensional word vector. Through word vector embedding, the vector representing the text is converted into a p×300 matrix, where p is the length of the text. The matrix is ​​then input into a convolutional neural network to obtain a text feature vector. The semantic feature extraction network is composed of a fully connected layer and an activation function, and the one-hot vector representing the semantic label is output as a semantic feature vector with the same dimension as the image or text feature; The cross-modal attention module is composed of a fully connected layer and a Sigmoid activation function. The previously obtained features are input into the cross-modal attention module. After the feature weights are learned through the fully connected layer and the Sigmoid activation function, the features of each modality are multiplied to obtain the attention features of each modality. Then, after an identity mapping, the features of each modality are input into a parameter-sharing fully connected layer, and the final feature vector is output to a common feature space. The label prediction layer is composed of a fully connected layer, which maps image features, text features and semantic features into vectors with the same dimension as the label vector; The loss function includes contrast center loss in feature space and label prediction loss in label space.

4. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed, the cross-modal retrieval method based on fetal congenital heart disease ultrasound image-diagnosis report as described in any one of claims 1 to 3 can be implemented.

Images