An efficient multi-modal contrastive deep hashing retrieval method for medical big data

Abstract

The application discloses a kind of efficient multi-modal contrast depth hash retrieval method for medical big data, it is related to artificial intelligence technical field, "search by image" function can let researcher search all similar image cases in database, not be influenced by previous expert diagnosis conclusion, provide possibility for further enrich and correct database.The "text search" function of cross-modal is more suitable for primary doctors or researchers.The method occupies less storage space, can realize fast search across modalities, by utilizing the potential correlation between medical report and its corresponding x-ray image, an efficient multi-modal medical data retrieval model is developed, the storage space is reduced, the efficiency of medical big data retrieval is improved, so that doctors can better learn, research and clinical diagnosis.

Description

TECHNICAL FIELD

[0001] The new type of the present application relates to the field of artificial intelligence technology, in particular to a high-efficiency multi-modal contrast deep hash retrieval method for medical big data, which combines large-scale image processing, artificial neural network and multi-modal data retrieval, and is mainly aimed at the problems of high medical image annotation cost, low retrieval efficiency and long time consumption. The present application mainly improves the retrieval efficiency by mining the internal advanced semantic information of medical data through contrast learning, and reduces the search time and storage space by compressing high-dimensional features into two-dimensional hash features. BACKGROUND

[0002] The progress of medical imaging technology has completely changed the healthcare practice and improved the diagnosis results of patients. However, in recent years, the increasing storage of image data has brought a huge burden to radiologists, affecting the quality and speed of clinical decision-making. The deep cross-modal hash retrieval method provides a promising solution for automatic analysis of medical images. Current hash medical image retrieval technologies include the following: single-modal hash medical image retrieval technology: single-modal hash medical image retrieval technology can be divided into supervised and unsupervised. Supervised single-modal hash medical image retrieval technology uses label information to supervise the training of the hash model, and has high retrieval efficiency, but the cost of manual annotation is high, and it is not suitable for large-scale data. Unsupervised single-modal hash medical image retrieval technology uses the inherent properties of medical images to train the hash model, without the need for manual annotation, but the retrieval efficiency is poor. Traditional single-modal hash medical image retrieval technology only uses image data to train the model, which has great limitations in the era of coexistence of a large number of multimedia heterogeneous data. Multi-modal hash medical image retrieval technology: uses medical reports as a supervision signal to guide the model to learn multi-modal representation and realize cross-modal retrieval, but the retrieval performance is poor and cannot meet the demand of large data retrieval.

[0003] The existing single-modal hash medical image retrieval technology uses a neural network to extract medical image features and transfer them to the Hamming space for similarity calculation to train the discriminability of the model. The structure is simple, but it is not suitable for multi-modal data and cannot meet the current medical needs. Although the existing multi-modal hash medical data retrieval technology solves the above problems, it simply migrates the traditional multi-modal method and cannot obtain the advanced semantic information in medical images, resulting in low retrieval efficiency. The purpose of the new type of the present application is to establish a multi-modal medical data hash retrieval model based on contrast learning, which reduces the search time and storage space while improving the cross-modal search efficiency. SUMMARY

[0004] To solve the above problems, the application provides a kind of efficient multi-modal contrast deep hash retrieval method for medical big data, "search by image" function can let researcher search all similar image cases in database, not affected by previous expert diagnosis conclusion, provide possibility for further enriching and correcting database;Cross-modal "text search" function is more suitable for primary doctors or researchers.

[0005] The technical solutions adopted by the application are as follows:

[0006] A kind of efficient multi-modal contrast deep hash retrieval method for medical big data, comprising the following steps:

[0007] S1.Obtain the multi-modal data set comprising medical x-ray image and corresponding radiology report for model training.

[0008] S2.Using original feature encoding model and momentum encoding model, respectively extract the original features and momentum features of medical multi-modal data, then pass through hash layer, convert into original hash features and momentum hash features.

[0009] S3.The momentum hash feature is further subjected to clustering operation and is converted into a cluster hash feature, and then is subjected to contrast learning training with the original hash feature to explore the intra-class and inter-class distinction of multi-modal hash feature.

[0010] S4.The cluster center is used as a pseudo label to guide the hash generation network to filter out a large amount of noise in the data.

[0011] S5.The original feature representation of image mode and text mode is subjected to contrast learning at the same time, and the inter-class similarity of multi-modal data is further explored.

[0012] S6.In order to be more suitable for multi-modal retrieval task, cross-modal similarity learning is also needed.

[0013] S7.The contrast training framework established by S1-S6 is used in the training of multi-modal contrast hash model to guide the model learning process and help the model to realize more efficient multi-modal medical data retrieval.

[0014] Based on the above scheme, each step can be realized by the above method;

[0015] Further, in step S1, the training data set contains a plurality of medical x-ray images I train And its corresponding medical report T train .

[0016] Further, in step S2, the original feature encoding model and the momentum encoding model are used to extract the original features and the momentum features of the medical multi-modal data, which specifically includes the following sub-steps:

[0017] S21. Obtain two image hash feature encoders ImgNet o and ImgNet m with network parameters θ o and θ m respectively. The image hash feature encoders include a vision transformer encoder E v and a hash layer H i .

[0018] S22. Obtain two text hash feature encoders TexNet o and TexNet m with network parameters θ and θ respectively. The text hash feature encoders include a transformer encoder E T and a hash layer H t .

[0019] S23. For each medical x-ray image I train in the training dataset, first pass it through ImgNet o and ImgNet m to generate original image features and momentum image features respectively, and then pass it through the hash layer H i to generate original hash features and momentum hash features and

[0020] S24. For each medical report T train in the training dataset, first pass it through TexNet o and TexNet m to generate original text features and momentum text features respectively, and then pass it through the hash layer H t to generate original hash features and momentum hash features and

[0021]

[0022]

[0023]

[0024]

[0025] Further, in step S3, the momentum hash feature is further clustered into a cluster hash feature, and then compared with the original hash feature for contrastive learning training to explore the intra-class and inter-class distinction of the multi-modal hash feature, including the following sub-steps:

[0026] S31. The image momentum hash feature is clustered by a K-means clustering algorithm to obtain a cluster center and the cluster center is stored in a dynamic queue for contrastive learning with the text hash feature .

[0027] S32. The image momentum hash feature is clustered by a K-means clustering algorithm to obtain a cluster center and the cluster center is stored in a dynamic queue for contrastive learning with the text hash feature .

[0028] S33. The model is trained using the contrastive loss L h to expand the intra-class and inter-class distinction of different diseases in medical data:

[0029]

[0030] wherein < > represents matrix multiplication, τ is a temperature hyperparameter, and λ is a balance hyperparameter. The similarity between different hash points is measured by dot product. Here, the first query comes from the image modality, while the key values and come from the text modality. Conversely, the second query comes from the text modality, while the key values and come from the image modality, wherein the key values come from the momentum queue of the corresponding modality.

[0031] Further, in step S5, in order to further explore the inter-class similarity of multi-modal data, the original feature representation of the image modality and the text modality is subjected to contrastive learning.

[0032]

[0033] wherein sim(f x , f y ) = f xT f y / ||f x |||f y ||, and k = x takes a value of 1. f x , f yRepresent the original features of image modality and text modality respectively.

[0034] Furthermore, in step S6, in order to better adapt to the multimodal retrieval task, cross-modal similarity learning is required, and the cross-modal similarity loss is:

[0035]

[0036] Among them, S pair Represents x i with y i semantic similarity.

[0037] S pair (b x , b y )=b xT b y / ||b x ||||b y ||, in a multi-label setting, two instances (x i and y j ) is annotated with multiple labels. Therefore, if x i and y j Share at least one tag, then define otherwise, v represents the distance boundary, and N represents the number of samples for this training.

[0038] Furthermore, in step S7, the comparative training framework established by steps S1 to S6 is used in the training of the multimodal comparative hashing model to guide the model learning process and help the model achieve more efficient multimodal medical data retrieval. Specifically, the following steps are included:

[0039] S71. Set the total loss function of the medical multimodal comparative hashing model (where α and β are both balanced hyperparameters):

[0040] L c =β(L h +L d )+(1-β)L f

[0041] L=αL s +(1-α)L c

[0042] S72. Use the SGD optimization method and the backpropagation algorithm to train the medical multimodal comparative hashing model through the loss function L until the loss function converges.

[0043] On the other hand, the present invention proposes a multimodal comparative deep hash retrieval method for medical big data, which uses the method described in the above scheme to obtain the trained model ImgNet oand TexNet o , which can achieve efficient multimodal retrieval.

[0044] Its advantages are:

[0045] Compared with existing multimodal medical hash models, the multimodal comparative hash model for medical data retrieval of the present invention has the following more beneficial effects:

[0046] First, the dual comparison structure designed in the present invention can effectively mine medical semantic information in multimodal data and improve the accuracy of the model.

[0047] Then, the present invention adds distillation training to remove useless information in the data and improve the generalization of the model.

[0048] Finally, the proposed multimodal retrieval model is applicable to different multimodal medical databases and has achieved significant performance improvements on mainstream large-scale multimodal medical datasets. It is a multimodal retrieval model that is easy to promote. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 This is a training flowchart of the multimodal comparative hashing model of the present invention.

[0050] Figure 2 This is the overall structural diagram of the multimodal comparative hashing model of the present invention.

[0051] Figure 3 Schematic diagram of the multimodal retrieval process.

[0052] Figure 4 for Figure 2 A partial enlarged view of the first part.

[0053] Figure 5 for Figure 2 A partial enlarged view of the second part.

[0054] Figure 6 for Figure 2 A partial enlarged view of the third part. DETAILED DESCRIPTION

[0055] Figure 1 The following is a training flow chart of the multimodal comparative hashing model of the present invention. The specific training steps of the model are described in detail below. The model training specifically includes the following process:

[0056] S1. Obtain a multimodal dataset for model training, including medical X-ray images and corresponding radiology reports.

[0057] In this embodiment, the training data set in step S1 includes several medical x-ray images Itrain and its corresponding medical report T train .

[0058] The goal of training the medical multimodal hash comparison model is to achieve efficient mutual retrieval of X-ray images and medical reports, making the search results as comprehensive and accurate as possible. It should be noted that the present invention is applicable to X-ray images of various parts of the human body and their corresponding diagnostic reports.

[0059] S2. Use the original feature encoding model and momentum encoding model to extract the original features and momentum features of medical multimodal data respectively, and then convert them into original hash features and momentum hash features through the hash layer.

[0060] In this embodiment, in the above step S2, the original feature encoding model and the momentum encoding model are used to extract the original features and momentum features of the medical multimodal data respectively, and then the original features and momentum features are converted into original hash features and momentum hash features through the hash layer. Specifically, the following steps are included:

[0061] S21. Get two image hash feature encoders ImgNet o and ImgNet m , and its network parameters are θ o and θ m , the image hash feature encoder includes the vision transformer encoder E v and a hash layer H i .

[0062] S22. Get two text hash feature encoders TexNet o and TexNet m , and its network parameters are and The text hash feature encoder includes the transformer encoder E T and a hash layer H t .

[0063] S23. For each medical x-ray image I in the training dataset train , which first passes through ImgNet o and ImgNet m , respectively generate original image features and momentum image features Then pass through the hash layer H i , respectively generate the original hash features and momentum hashing features

[0064] S24. For each medical report T in the training datasettrain , which is first passed through TexNet o and TexNet m , respectively, to generate original image features and momentum image features Then, the hash layer H t is applied to generate original hash features and momentum hash features

[0065]

[0066]

[0067]

[0068]

[0069] S3. The momentum hash features are further subjected to clustering operation to transform into cluster hash features, and then comparative learning training is performed with the original hash features to explore the intra-class and inter-class distinguishability of the multi-modal hash features.

[0070] In the above step S3, the clustering operation is further performed to transform into cluster hash features, and then comparative learning training is performed with the original hash features to explore the intra-class and inter-class distinguishability of the multi-modal hash features, which specifically includes the following sub-steps:

[0071] S31. The image momentum hash features are subjected to K-means clustering algorithm to obtain the cluster centers and the cluster centers are stored into a dynamic queue for comparative learning with the text hash features .

[0072] S32. The image momentum hash features are subjected to K-means clustering algorithm to obtain the cluster centers and the cluster centers are stored into a dynamic queue for comparative learning with the text hash features .

[0073] S33. The contrastive loss L h is used to train the model to expand the intra-class and inter-class distinguishability of different diseases in the medical data:

[0074]

[0075] where < > denotes matrix multiplication, and τ is a temperature hyper-parameter. λ is a balance hyper-parameter, and the similarity between different hash points is measured by the dot product. Here, the first query comes from the image modality, while the key-value and come from the text modality. In contrast, the second query comes from the text modality, while the key-value and come from the image modality, where the key-value pairs are all from the momentum queue of the corresponding modality.

[0076] S5. Simultaneously, the original feature representations of the image and text modalities are subjected to contrastive learning, further exploring the inter-class similarity of multi-modal data.

[0077]

[0078] where sim(f x , f y ) = f xT f y / ||f x |||f y |||, and k = x takes the value of 1. f x , f y represent the original features of the image and text modalities, respectively.

[0079] S6. To better adapt to the multi-modal retrieval task, cross-modal similarity learning is also required:

[0080]

[0081] where S pair denotes the semantic similarity between x i and y i . S pair (b x , b y ) = b xT b y / ||b x |||b y |||, in the multi-label setting, two instances (x i and y j ) are annotated with multiple labels. Therefore, if x i and y j share at least one label, then define otherwise, v represents the distance from the boundary, and N represents the number of samples trained.

[0082] S7. The contrastive training framework established in the above steps S1-S6 guides the model learning process, helping the model to achieve more efficient multi-modal medical data retrieval.

[0083] In the present embodiment, in the above step S6, the contrastive training framework established by S1-S5 guides the model learning process, helping the model to achieve more efficient multi-modal medical data retrieval, specifically including the following sub-steps:

[0084] S71. Set the total loss function of the medical multi-modal contrastive hash model (wherein α, β are balance hyperparameters):

[0085] L c =β(L h +L d )+(1-β)L f

[0086] L=αL s +(1-α)L c

[0087] S72. Use the SGD (stochastic gradient descent) optimization method and the back propagation algorithm to train the medical multi-modal contrastive hash model through the loss function L until the loss function converges.

[0088] Finally, when performing a specific cross-modal retrieval task, it utilizes the method described in the foregoing scheme to obtain the trained model ImgNet o and TexNet o can achieve efficient multi-modal retrieval.

[0089] The following will apply the above multi-modal contrastive hash model for efficient medical data retrieval to specific data set instances, which has shown the technical effects it can achieve.

[0090] The implementation method of the present example is as described in the foregoing S1-S7, and the specific steps will not be elaborated, and the effects of the present application will be demonstrated only for the case data, which is implemented on a multi-modal data set with real annotations, respectively:

[0091] CheXpert 5x200 data set: This multi-class classification data set has 64740 front images specially for CheXpert competition tasks: cardiac enlargement, edema, pleural effusion, etc., including 64540 training images, 200 test images and their corresponding report forms.

[0092] The overall structure of the model is as shown in Figure 2 , Figure 2The middle blue color is a momentum coding module for generating momentum features, the light yellow color is an original feature coding module for generating image and text features, the light purple color is a clustering level momentum queue for participating in contrast learning training, and the dark yellow color is a hash library for storing generated hash codes for retrieval.

[0093] The multi-modal medical data specific retrieval process is shown in Figure 3 A modal data (text / image) is defined, which is first converted into a hash code by a hash model, and then similarity calculation is performed with the stored image / text modal hash codes in the hash library to find one or more most similar hash codes, and finally the corresponding image / text is found to output the query result.

[0094] To compare the effectiveness of the method, the performance of the model proposed in the application is compared with three of the most advanced models, including DSVE, VSE++, and ConVIRT (note that these three methods are multi-modal medical retrieval models and do not involve hash methods, and there is currently no medical retrieval model that uses hash methods for multi-modal data searching).

[0095] The detection accuracy of the detection results of this example is shown in the following table, and top-K accuracy is mainly used to measure retrieval performance, which is also an important indicator of multi-modal retrieval. The larger the value, the better the retrieval performance.

[0096] Method Prec@5 Prec@10 Prec@100 DSVE 40.60 32.77 24.74 VSE++ 44.28 36.81 26.89 ConVIRT 66.98 63.06 49.03 ours 70.84 68.22 60.69

[0097] As shown in the above table, in this example, the three most advanced methods (DSVE, VSE++, and ConVIRT) are compared with ours, which represents the model proposed in the application. It can be seen that the method of the application has a significant improvement in performance, and most importantly, the application stores short and effective hash codes in the database, occupies less storage space, ensures that the retrieval operation can be quickly implemented in the computer memory, and greatly improves the search speed.

[0098] Through the above technical solutions, a multi-modal contrast deep hash retrieval method for medical big data is invented based on deep learning technology. The solution proposed in the application fully utilizes the contrast learning and multi-modal hash method, and can significantly improve the retrieval performance and speed of medical multi-modal data.

[0099] The method has the advantages of small storage space occupation, and can realize cross-modal quick search, including two query modes: graph search and keyword search. The method comprises the following steps: using a transformer (text feature encoding converter) and a vision transformer (image feature encoding converter) to construct an original feature encoding model and a momentum encoding model, and using them to extract original features and momentum features of text modal and image modal, and convert them into hash features and momentum hash features, clustering the momentum hash features, obtaining the cluster center of each class, replacing the momentum hash features with the cluster center, and storing them in a momentum queue. Subsequently, the original hash features and the cluster center in the momentum queue are trained by contrast learning, and the intra-class and inter-class distinguishability of the multi-modal hash features is obtained. In order to better utilize the semantic information of the original data, the original features of the image and text modal are subjected to contrast learning, and the inter-class distinguishability of the multi-modal data is further increased. In addition, the momentum encoding model is only used for training and momentum updating during training, and the original feature encoding model is used for realizing multi-modal quick search during test application. The quick search method proposed by the application fully utilizes the contrast learning to mine the inherent distinguishability of large-scale multi-modal medical data, develops an efficient multi-modal medical data retrieval model by utilizing the potential correlation between the medical report and the corresponding x-ray image, reduces the storage space, improves the efficiency of medical big data retrieval, and enables doctors to better learn, research and clinically diagnose (given an image / text as input query, the most relevant several text / images can be found, helping doctors to search similar cases and shorten the clinical diagnosis time).

Claims

1. An efficient multi-modal contrastive deep hashing retrieval method for medical big data, characterized in that The method comprises the following steps: S1. Obtain a multi-modal data set comprising medical x-ray images and corresponding radiology reports for model training; S2. Extract original features and momentum features of the medical multi-modal data using an original feature encoding model and a momentum encoding model, respectively, and then process them through a hash layer to convert them into original hash features and momentum hash features; S3. Perform clustering operation on the momentum hash features to convert them into cluster hash features, and then perform contrastive learning optimization with the original hash features to mine the intra-class and inter-class differences of the multi-modal hash features; S4. Use the cluster centers as pseudo labels to guide the hash generation network to filter out a large amount of noise in the data; S5. Perform contrastive learning on the original feature representations of the image and text modalities to further mine the inter-class similarity of the multi-modal data; S6. In order to better adapt to the multi-modal retrieval task, cross-modal similarity learning is also needed; S7. Use the contrastive training framework established by S1-S6 to guide the model learning process in the training of the multi-modal contrastive hash model, which helps the model to achieve more efficient multi-modal medical data retrieval; In step S2, the original feature encoding model and the momentum encoding model are used to extract the original features and the momentum features of the medical multi-modal data, which comprises the following sub-steps: S21. Obtain two image hash feature encoders ImgNet o and ImgNet m , whose network parameters are θ o and θ m , respectively, and the image hash feature encoder comprises a vision transformer encoder E v and a hash layer H i ; S22. Obtain two text hash feature encoders TexNet o and TexNet m with network parameters and respectively. The text hash feature encoders include a transformer encoder E T and a hash layer H t ; S23. For each medical x-ray image I in the training dataset train , which is first passed through the encoder ImgNet o , and ImgNet m , respectively, to generate the original image features and the momentum image features , and then passed through the hashing layer H i , respectively, to generate the original hash features and the momentum hash features and S24. For each medical report T in the training dataset train , which first encodes its TexNet o and TexNet m , respectively generate original text features f o t and momentum text features Then pass through the hash layer H t , respectively generate original hash features and momentum hash features and The mathematical expression is: 2.The efficient multi-modal contrastive deep hashing retrieval method for medical big data of claim 1, wherein The method comprises the following steps: In step S1, the training data set contains several medical x-ray images I train and their corresponding medical reports T train . 3.The efficient multi-modal contrastive deep hashing retrieval method for medical big data of claim 1, wherein The method comprises the following steps: In step S3, the momentum hash features are clustered to convert them into cluster hash features, and then contrastive learning training is performed with the original hash features to mine the intra-class and inter-class differences of the multi-modal hash features, which comprises the following sub-steps: S31. The image momentum hashing features are obtained by K-means clustering algorithm and the clustering centers are stored into a dynamic queue for contrastive learning with the text hashing features ; S32. The image momentum hashing features are obtained by K-means clustering algorithm and the clustering centers are stored into a dynamic queue for contrastive learning with the text hashing features ​ S33. Adopting a contrastive loss L h Training the model to enlarge the intra-class and inter-class discriminativeness of different disease data in medical data: where: < > denotes matrix multiplication, τ is a temperature hyperparameter; λ is a balance hyperparameter, and the similarity between different hash points is measured by the dot product; here, the first query comes from the image modality, while the key value and come from the text modality; in contrast, the second query comes from the text modality, while the key value and come from the image modality, where the key values all come from the momentum queue of the corresponding modality. 4.The efficient multi-modal contrastive deep hashing retrieval method for medical big data of claim 1, wherein The method comprises the following steps: In step S4, in order to further explore the inter-class similarity of the multi-modal data, contrastive learning is performed on the original feature representations of the image and text modalities; The feature-level contrastive loss is represented as: where sim(f x ,f y ) = f xT f y / ||f x ||||f y ||, and 1 when k = x; fxand fydenote the original features of the image modality and the text modality, respectively. 5.The efficient multi-modal contrastive deep hashing retrieval method for medical big data of claim 1, wherein The method comprises the following steps: In step S5, in order to better adapt to the multi-modal retrieval task, cross-modal similarity learning is also needed, and the cross-modal similarity loss is: where S pair represents the semantic similarity between x i and y i ; S pair (b x ,b y ) = b xT b y / |b x |||b y ||, in a multi-label setting, two instances x i and y j are annotated with multiple labels; thus, if x i and y j share at least one label, then define otherwise, v represents the distance to the boundary, and N represents the number of samples trained for this. 6.The efficient multi-modal contrastive deep hashing retrieval method for medical big data of claim 1, wherein The method comprises the following steps: In step S7, the contrastive training framework established by S1-S6 is used to guide the model learning process in the training of the multi-modal contrastive hash model, which comprises the following steps: S71. Set the total loss function of the medical multi-modal contrastive hash model (wherein α and β are balance hyperparameters): L c = β(L h + L d ) + (1 - β)L f L = aL S + (1 - a)L c S72. Use the SGD optimization method and the back propagation algorithm to train the medical multi-modal contrastive hash model through the loss function L.

Images