Cross-modal medical data hash retrieval method

Through the hash retrieval method of cross-modal medical data, the problems of video memory resource dependence, neglect of information interaction and insufficient manifold structure in the existing technology are solved, and efficient cross-modal medical data retrieval is achieved, which improves the search accuracy and efficiency , providing more precise support for diagnosis and treatment.

CN118939847APending Publication Date: 2024-11-12CHONGQING UNIV OF POSTS & TELECOMM
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202410988751.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-23
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

The existing deep cross-modal hashing methods have problems such as memory resource dependence in medical data retrieval, modal feature extraction ignores information interaction, and insufficient consideration of the underlying manifold structure, resulting in inefficient retrieval efficiency and affecting the diagnosis and treatment process.

Method used

The hash retrieval method of cross-modal medical data, including preprocessing medical images and diagnostic reports, training modal feature extraction networks, learning hash functions, generating hash codes through feature extraction networks, and cross-modal through Hamming distance Retrieve, combining manifold similarity matrix and alternating learning strategies, optimize the quality and performance of hash codes.

Benefits of technology

Improves the flexibility of modal feature extraction and the learning efficiency of hash functions, promotes information exchange between modals, significantly improves the accuracy and efficiency of cross-modal retrieval, and helps doctors quickly access and understand information from different modalities. Provide patients with more accurate and personalized diagnosis and treatment plans.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118939847A_ABST
    Figure CN118939847A_ABST
Patent Text Reader

Abstract

The invention relates to a cross-modal medical data hash retrieval method, and belongs to the field of artificial intelligence information retrieval. The method comprises the steps of performing data cleaning and preprocessing on a medical ultrasonic image and an ultrasonic report, and unifying the size of the ultrasonic image and the format of the ultrasonic report; an image encoder and a text encoder are customized for the medical image and the diagnostic report based on a Transform encoder, and features of the medical image and the diagnostic report are extracted by means of combined learning of three pre-training tasks; manifold similar matrixes of different modes are constructed, and balance constraints and linear classification network constraints between Hash codes are introduced to learn a Hash function; hash codes of the ultrasonic picture and the ultrasonic report are obtained through the Hash function, and cross-modal Hash retrieval is carried out. According to the method, semantic similarity retrieval of medical cases among different modes can be realized, for example, related ultrasonic images are quickly positioned through ultrasonic reports, or corresponding ultrasonic reports are retrieved and matched by reversely utilizing the ultrasonic images.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence information retrieval and relates to a cross-modal medical data hash retrieval method. Background Art

[0002] Medical cross-modal retrieval is a key research direction in the field of medical information retrieval. It not only focuses on the retrieval between medical images produced by different imaging technologies (such as CT, MRI, X-ray, etc.), but also covers the mutual retrieval between medical images and related diagnostic reports. This cross-modal retrieval capability is crucial to improving the quality and efficiency of healthcare. It enables doctors and clinical researchers to quickly access and understand relevant information in one modality through data in another modality, thereby providing patients with more accurate and personalized diagnosis and treatment plans.

[0003] With the continuous advancement of computer vision and medical imaging technology, the amount of medical image data is growing exponentially. These images are crucial for medical research, disease diagnosis and treatment. However, the diversity and complexity of data pose severe challenges to retrieval technology, not only because of the huge amount of data, but also because of the heterogeneity between different modal data, uneven image quality, and the prevalence of noise and artifacts. Although existing deep cross-modal hashing methods have made some progress, they still have the following shortcomings:

[0004] (1) Its end-to-end training method leads to dependence on a large amount of video memory resources, which limits the scalability and practicality of the algorithm in practical applications.

[0005] (2) The information interaction between modalities is ignored in the modal feature extraction stage.

[0006] (3) Due to the significant semantic differences and subtle visual differences between categories in medical datasets, current methods do not adequately consider the underlying manifold structure between different modal data.

[0007] These problems have a significant impact in actual clinical applications. When doctors perform cross-modal data retrieval, they often suffer from low retrieval efficiency, which leads to delays in diagnosis and treatment, which is particularly serious in emergency situations. Summary of the invention

[0008] In view of this, an object of the present invention is to provide a cross-modal medical data hash retrieval method.

[0009] In order to achieve the above object, the present invention provides the following technical solutions:

[0010] A cross-modal medical data hash retrieval method, the method comprising the following steps:

[0011] S1: Preprocess medical images and diagnostic reports;

[0012] S2: training modality feature extraction network;

[0013] S3: Learning hash functions of different modalities;

[0014] S4: Get the hash code through the feature extraction network hash function;

[0015] S5: Cross-modal hash retrieval via generated hash codes.

[0016] Furthermore, in S1, the medical image and the diagnosis report are preprocessed, specifically including the following steps:

[0017] S11: Unify the resolution of medical images to 224×224×3, perform data augmentation by random cropping and flipping to increase the diversity of training samples, and combine the patient’s diagnostic keywords and patient information into a unified structure to form a diagnostic description;

[0018] S12: The medical image and diagnosis report data are each copied three times, and one of the diagnosis reports is subjected to random masking.

[0019] Further, in S2, training the modality feature extraction network includes the following steps:

[0020] S21: Initialize the medical image encoder using the 12-layer Vit-B / 16 parameters based on ImageNet-1k pre-trained weights; the input image I of the image encoder is encoded into a series of embeddings: {v 0 ,v 1 ,v 2 ,…,v p},in It is an extra [CLS] token added at the head of the sequence where the image is divided into patches to represent the global features of the image, and p is the sequence length of the medical image;

[0021] S22: Use the first 6 layers of the BERT-base model’s parameters to initialize the unimodal text encoder, and use the last 6 layers of the BERT-base model’s parameters to initialize the multimodal encoder; the text encoder converts the input diagnosis report T into an embedding sequence {w 0 ,w 1 ,w 2 ,…,w q},in It is an additional [CLS] token added at the head of the text sequence to represent the global features of the text, and q is the sequence length of the diagnosis report;

[0022] S23: Through three pre-training tasks, namely,

[0023] Image-Text Contrastive Learning (ITC) on Medical Image Encoder and Diagnosis Report Text Encoder,

[0024] The multimodal encoder performs masked language modeling (MTM) of the diagnosis report,

[0025] The joint learning of ITM, a matching learning task of medical image and diagnosis report pairs, is implemented on the multimodal encoder to extract the features of each modality.

[0026] Furthermore, in S3, learning hash functions of different modes specifically includes:

[0027] S31: The medical image and the diagnosis report are respectively processed by the medical image encoder and the diagnosis report encoder to generate the feature vector of the medical image and the feature vector of the diagnostic report

[0028] S32: Design manifold similarity matrix S ma and calculate the Hamming distance D between hash codes h , construct the weighted cumulative Hamming distance metric F S ;

[0029] S33: applying the feature vector X of the medical image and the feature vector Y of the diagnosis report to the corresponding hash functions to obtain the medical image feature F and the diagnosis report feature G;

[0030] S34: The medical image feature F and the diagnosis report feature G are subjected to a symbolic function to obtain a medical image hash code B x = sign(F) and diagnostic report hash code B y =sign(G);

[0031] S35: In order to reduce the feature representations F and G and the mapped hash code B x and B y The gap between them is approximated by the 2-norm loss constraint to obtain the quantized loss

[0032] S36: To constrain the semantic information to be evenly distributed across all bits, add bit balancing loss is a vector whose elements are all 1;

[0033] S37: In order to improve the discriminative ability of the learned hash code, a classifier with two fully connected layers is introduced to predict the original label of the learned hash code to obtain the semantic embedding constraint W c Represents the weight parameter of the classifier;

[0034] S38: weighted cumulative Hamming distance metric F S , quantization loss F q , balance loss F b and the semantic embedding constraint F L Add together the total goal F of hash function learning for medical images and diagnostic reports H =F S +aF q +bF b +lF L , α, β, and λ are introduced hyperparameters to balance the weights between different constraints;

[0035] S39: Using an alternating learning strategy to learn the hash function f for medical image modalities (2) (·) and the hash function g of the diagnostic report modality (2) (·), as well as the hash code B and the classification network parameters W c ; Through these two hash functions, the corresponding modal features are obtained respectively: and During the optimization process, only one parameter is optimized at a time, and other parameters are kept unchanged. The parameters of each network are updated through alternating cycles.

[0036] Further, in S4, a hash code is obtained by using a feature extraction network hash function, which specifically includes the following steps:

[0037] S41: Given a medical image I q First, the medical image I is obtained through the medical image encoder in the modality feature extraction stage q The eigenvector of Then the medical image I is obtained through the forward propagation algorithm of the medical image hash layer. q The hash code

[0038] S42: Given a diagnostic report T q First, the diagnostic report T is obtained through the diagnostic report encoder in the modal feature extraction stage q The eigenvector of Then the diagnostic report T is obtained through the forward propagation algorithm of the diagnostic report hash layer. q The hash code

[0039] Furthermore, in S5, cross-modal hash retrieval is performed using the generated hash code, specifically including the following steps:

[0040] S51: Calculate the Hamming distance between the query modality data hash code and the retrieval library modality hash code;

[0041] S52: The sample with the smallest Hamming distance is given through the Hamming distance sorting to complete the query function.

[0042] The beneficial effects of the present invention are: first, the method uses a segmented training strategy, which greatly enhances the flexible extraction capability of modal features and the learning efficiency of hash functions; second, by cleverly integrating the joint learning framework of three pre-training tasks, we successfully promoted effective information exchange between modalities in the feature extraction stage and achieved deep semantic alignment between modal features; finally, the constructed manifold similarity matrix effectively captures the complex heterogeneous relationships between cross-modalities with its accuracy, thereby significantly improving the quality and performance of generated hash codes. A large model architecture is used to deeply mine the semantic information within the modality, and a lightweight model architecture is used to efficiently learn hash functions to improve the accuracy and efficiency of retrieval between different medical case modalities. This helps doctors and clinical researchers to quickly access and understand relevant information of another modality through data from one modality, thereby providing patients with more accurate and personalized diagnosis and treatment plans.

[0043] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0045] Figure 1 A flow chart of a cross-modal medical data hash retrieval method of the present invention;

[0046] Figure 2 This is the overall architecture diagram of the cross-modal medical data hash retrieval method;

[0047] Figure 3 It is the module diagram of modal feature extraction;

[0048] Figure 4 Specific implementation diagram for each module. DETAILED DESCRIPTION

[0049] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0050] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on the present invention. In order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0051] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "front", "rear", etc. indicate the orientation or position relationship, they are based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation. Therefore, the terms describing the position relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0052] See also Figure 1 to Figure 4 , Figure 1 The figure shows a flow chart of a cross-modal medical data hash retrieval method provided by the present invention, which includes the following steps:

[0053] Step 1: pre-process the medical images and diagnostic reports, including:

[0054] 11) The resolution of medical images was unified to 224×224×3, and data enhancement was performed through random cropping and flipping to increase the diversity of training samples. The patient's diagnostic keywords and patient information were organized into a unified structure to form a diagnostic description;

[0055] 12) The medical image and diagnosis report data are each copied three times, and one of the diagnosis reports is randomly masked;

[0056] Figure 2 This is the overall architecture diagram of the cross-modal medical data hash retrieval method. The left part shows the diagnostic report and image modality data visualization examples of the input model.

[0057] Step 2: training the medical image and diagnostic report modality feature extraction network, including:

[0058] 21) Figure 3 Shown are the designed modal feature extraction networks, which are the medical image encoder, diagnosis report text encoder and image-text cross-modal encoder designed based on the components of the Transform model. Figure 3 The following is a diagnostic report of the input model and an example of visualization of image modality data. The specific implementation of each component is as follows Figure 4 As shown in Figure 2. The medical image encoder is initialized using 12 layers of Vit-B / 16 parameters based on ImageNet-1k pre-trained weights. The input image I of the image encoder is encoded into a series of embeddings: {v 0 ,v 1 ,v 2 ,…,v p},in It is an extra [CLS] token added at the head of the sequence where the image is divided into patches to represent the global features of the image, and p is the sequence length of the medical image;

[0059] 22) Use the first 6 layers of the BERT-base model parameters to initialize the unimodal text encoder, and use the last 6 layers of the BERT-base model parameters to initialize the multimodal encoder. The text encoder converts the input diagnosis report T into an embedding sequence {w 0 ,w 1 ,w 2 ,…,w q},in It is an additional [CLS] token added at the head of the text sequence to represent the global features of the text, and q is the sequence length of the diagnosis report;

[0060] 23) Extract the features of each modality through joint learning of three pre-training tasks (image-text contrastive learning (ITC) on the medical image encoder and the diagnosis report text encoder, masked language modeling (MTM) of the diagnosis report performed by the multimodal encoder, and matching learning task (ITM) of the medical image and diagnosis report pairs implemented on the multimodal encoder). The joint learning of the three pre-training tasks to extract the features of each modality includes the following steps:

[0061] 231) The ITC loss function promotes semantic alignment between the feature representations of medical images and diagnostic reports by maximizing the similarity of positive sample pairs and minimizing the similarity of negative sample pairs. For each sample pair in the dataset, the image-text contrastive learning loss on the medical image encoder and the diagnostic report text encoder is calculated by its cross entropy loss H: Where S(I,T) represents the binary similarity between image I and text T. i and T j With the same label S(I i ,T j )=1, otherwise S(I i ,T j )=0. is the training data set, where N represents the number of sample instance pairs in the data set, and D i = {I i ,T i ,L i} describes the i-th instance. i ,T i Represents the i-th medical image and medical diagnosis report text respectively, L i ={0,1} c×1 Indicates D i The label vector of D, c represents the number of categories. S (I i ,T j ) is image I i and text T j The cosine similarity after the encoder extracts the feature representation: f (1) (·) and g (1) (·) are the unimodal encoder network functions for medical images and diagnostic reports, respectively, and are the parameters of their corresponding networks respectively;

[0062] 232) ITM loss promotes semantic alignment between medical images and diagnostic reports by predicting whether they are positive matches or negative matches. It aims to learn image-text multimodal representations to capture fine-grained alignment between vision and language. ITM loss is a binary classification task. The [CLS] token output by the multimodal encoder is passed through a fully connected layer and then SoftMax is performed to obtain the prediction probability p of the two categories. IT In order to obtain more informative negative samples, a hard negative mining strategy is adopted, in which negative samples with higher contrast similarity in the batch are more likely to be selected to calculate the loss. Then the ITM loss F is obtained by its cross entropy ITM =E D H(S(I,T),P IT (I,T));

[0063] 233) Randomly mask about 15% of the words in the input medical diagnosis report sequence and replace them with special tokens [mask], allowing the model to use medical images and context information to predict these masked words. The MTM loss is obtained by calculating the difference between the output probability distribution and the true label m represents the number of mask words, v represents the size of the vocabulary, is a one-hat vector representing the true label of the jth masked word in the i-th image-text pair, with a value of 1 at position k and 0 at the rest. ijk |I i ,T i ′) indicates that the model is i and the masked text T i ′, predict the probability distribution of the masked words, w ijk is the predicted probability of the jth masked word at position k in the i-th image-text pair;

[0064] 234) Add ITC loss, ITM loss and MTM loss to get the total loss function F F =F ITC +F ITM +F MTM To train the modal feature extraction network;

[0065] 24) Input the medical images and diagnostic reports into Figure 3 The modal feature extraction network shown in the figure calculates ITC, MTM and ITM losses respectively, and adds them together for back propagation to update the model parameters. During training, the AdamW optimizer is used, with the weight decay set to 0.02, the batch size set to 256, the initial learning rate set to 0.01, and the strategy of decaying to 0.0001 every 10 iterations is adopted in 1000 iterations.

[0066] Step 3, using the modality features of the medical images and diagnosis reports to train the medical image and diagnosis report hash functions, specifically including:

[0067] 31) Figure 3 As shown, the medical image and the diagnosis report are respectively passed through the trained medical image encoder and the diagnosis report encoder to obtain the feature vector of the medical image and the feature vector of the diagnostic report

[0068] 32) Design manifold similarity matrix S ma and calculate the Hamming distance D between hash codes h , construct the weighted cumulative Hamming distance metric F S , including the following steps:

[0069] 321) The manifold similarity between medical images and diagnostic reports is abstracted into a hypergraph Gra=(V,S ma ), the hypergraph consists of a vertex set V and an edge set S ma Among them, V contains data instances of two modes, S ma is the manifold similarity between two instances;

[0070] 322) For any vertex V i and V j For example, if two vertices have the same label, they are considered to be neighbors. Define N(a) to represent the neighbor set of vertex a, then construct the intra-modal similarity and and the similarity between modalities x i and j Respectively represent the i-th medical image feature vector and the j-th diagnosis report feature vector;

[0071] 323) By defining the intra-modal similarity and and inter-modal similarity Constructing medical imagesx i and diagnostic report j Manifold similarity matrix represents element-wise multiplication;

[0072] 324) Let the hash codes of the medical image and the diagnosis report obtained by the hash function be B x ∈{-1,1} and B y ∈{-1,1}, the hash code can be realized by the inner product of the hash code matrix when calculating the Hamming distance: k represents the length of the hash code, and They are represented as the i-th medical image hash code and the j-th diagnosis report hash code, respectively. Represents a hash code and The inner product of

[0073] 325) Using the defined manifold similarity and Hamming distance, construct the medical image x i and diagnostic report j The weighted cumulative Hamming distance metric

[0074] 33) The feature vector X of the medical image and the feature vector Y of the diagnosis report are respectively obtained by corresponding hash functions to obtain the medical image feature F and the diagnosis report feature G;

[0075] 34) The medical image feature F and the diagnosis report feature G are passed through the symbolic function to obtain the medical image hash code B x = sign(F) and diagnostic report hash code B y =sign(G);

[0076] 35) In order to reduce the feature representations F and G and the mapped hash code B x and B y The gap between them is approximated by the 2-norm loss constraint to obtain the quantized loss

[0077] 36) In order to constrain the semantic information to be evenly distributed on all bits, a bit balance loss is added is a vector whose elements are all 1;

[0078] 37) In order to improve the discriminative ability of the learned hash codes, a classifier with two fully connected layers is introduced to predict the original labels of the learned hash codes to obtain the semantic embedding constraints W c Represents the weight parameter of the classifier.

[0079] 38) The weighted cumulative Hamming distance metric F S , quantization loss F q , balance loss F b and the semantic embedding constraint F L Add together the total goal F of hash function learning for medical images and diagnostic reports H =F S +aF q +bF b +lF L , α, β, and λ are introduced hyperparameters to balance the weights between different constraints.

[0080] 39) Adopting alternating learning strategy to learn hash function f of medical image modality (2) (·) and the hash function g of the diagnostic report modality (2) (·), as well as the hash code B and the classification network parameters W c Through these two hash functions, the corresponding modal features can be obtained respectively: and During the optimization process, only one parameter is optimized at a time, and other parameters are kept unchanged. The parameters of each network are updated through alternating cycles.

[0081] Step 4, obtaining a hash code through the modality feature extraction network and hash function of the trained medical image and diagnosis report, including the following steps:

[0082] 41) Given a medical image Iq First, the medical image I is obtained through the medical image encoder in the modality feature extraction stage q The eigenvector of Then the medical image I is obtained through the forward propagation algorithm of the medical image hash layer. q The hash code

[0083] 42) Given a diagnostic report T q First, the diagnostic report T is obtained through the diagnostic report encoder in the modal feature extraction stage q The eigenvector of Then the diagnostic report T is obtained through the forward propagation algorithm of the diagnostic report hash layer. q The hash code

[0084] Step 5, performing cross-modal hash retrieval using the generated hash code specifically includes the following steps:

[0085] 51) Establishing a hash code database through hash codes of medical images and diagnosis reports;

[0086] 52) Retrieving relevant diagnostic report modality samples through medical image modality samples, including the following specific steps:

[0087] 521) performing data preprocessing on the medical image query samples;

[0088] 522) obtaining a medical image query sample feature vector through a medical image modality feature extraction network;

[0089] 523) obtaining a hash code of a medical image query sample through a medical image hash network;

[0090] 524) Calculate the Hamming distance between the hash code of the medical image query sample and the hash code in the diagnosis report retrieval library;

[0091] 525) returning the diagnostic report sample with the smallest Hamming distance according to the Hamming distance sorting, thus completing the query function;

[0092] 53) Retrieving relevant medical image modality samples through diagnostic report modality samples includes the following specific steps:

[0093] 531) performing data preprocessing on the diagnostic report query sample;

[0094] 532) obtaining a diagnostic report query sample feature vector through a diagnostic report modality feature extraction network;

[0095] 533) obtaining a diagnostic report query sample hash code through a diagnostic report hash network;

[0096] 534) Calculate the Hamming distance between the hash code of the diagnosis report query sample and the hash code in the medical image retrieval library;

[0097] 535) returning the medical image sample with the smallest Hamming distance according to the Hamming distance sorting to complete the query function;

[0098] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A cross-modal medical data hash retrieval method, characterized by: The method comprises the following steps: S1: Preprocess medical images and diagnostic reports; S2: training modality feature extraction network; S3: Learning hash functions of different modalities; S4: Get the hash code through the feature extraction network hash function; S5: Cross-modal hash retrieval via generated hash codes.

2. A cross-modal medical data hash retrieval method according to claim 1, characterized in that: In S1, the medical image and the diagnosis report are preprocessed, specifically including the following steps: S11: Unify the resolution of medical images to 224×224×3, perform data augmentation by random cropping and flipping to increase the diversity of training samples, and combine the patient’s diagnostic keywords and patient information into a unified structure to form a diagnostic description; S12: The medical image and diagnosis report data are each copied three times, and one of the diagnosis reports is subjected to random masking.

3. The cross-modal medical data hash retrieval method according to claim 1, characterized in that: In S2, training the modality feature extraction network includes the following steps: S21: Initialize the medical image encoder using 12 layers of Vit-B / 16 parameters based on ImageNet-1k pre-trained weights; the input image I of the image encoder is encoded into a series of embeddings: {v 0 ,v 1 ,v 2 ,…,v p }, where v 1 to v p is the sequence vector encoded by the image, v 0 It is an additional token added to the head of the sequence vector of the image to represent the global features of the image, and p is the sequence length of the medical image; S22: Use the first 6 layers of parameters of the text language model BERT to initialize the unimodal text encoder, and use the last 6 layers of parameters of the language model BERT to initialize the multimodal encoder; the text encoder converts the input diagnosis report T into an embedding sequence {w 0 ,w 1 ,w 2 ,…,w q }, where w 1 to w q is the sequence vector encoded by the text, w 0 It is an extra token added at the head of the text sequence to represent the global features of the text, and q is the sequence length of the diagnosis report; S23: Through three pre-training tasks, namely, Image-Text Contrastive Learning (ITC) on Medical Image Encoder and Diagnosis Report Text Encoder, The multimodal encoder performs masked language modeling (MTM) of the diagnosis report, The joint learning of ITM, a matching learning task of medical image and diagnosis report pairs, is implemented on the multimodal encoder to extract the features of each modality.

4. The cross-modal medical data hash retrieval method according to claim 1, characterized in that: In S3, learning hash functions of different modes specifically includes: S31: passing the medical image and the diagnosis report through a medical image encoder and a diagnosis report encoder to generate a feature vector X of the medical image and a feature vector Y of the diagnosis report respectively; S32: Design manifold similarity matrix S ma and calculate the Hamming distance D between hash codes h , construct the weighted cumulative Hamming distance metric F S ; S33: applying the feature vector X of the medical image and the feature vector Y of the diagnosis report to the corresponding hash functions to obtain the medical image feature F and the diagnosis report feature G; S34: The medical image feature F and the diagnosis report feature G are subjected to a symbolic function to obtain a medical image hash code B x = sign(F) and diagnostic report hash code B y =sign(G), where the sign function is defined as: S35: In order to reduce the feature representations F and G and the mapped hash code B x and B y The gap between them is approximated by the two-norm loss constraint to obtain the quantized loss The two-norm is defined as: S36: To constrain the semantic information to be evenly distributed across all bits, add bit balancing loss The unit vector I is a vector whose elements are all 1; S37: In order to improve the discriminative ability of the learned hash code, a classifier with two fully connected layers is introduced to predict the original label of the learned hash code to obtain the semantic embedding constraint Where L c is the label matrix of medical images and diagnostic reports, x is the image modality, y is the diagnostic report text modality, N is the number of instances of medical image and diagnostic report pairs, B (x,y) is the hash code corresponding to the medical image and the diagnosis report, W c is the weight parameter of the classifier; S38: weighted cumulative Hamming distance metric F S , quantization loss F q , balance loss F b and the semantic embedding constraint F L Add together the total goal F of hash function learning for medical images and diagnostic reports H =F S +aF q +bF b +lF L , α, β, and λ are introduced hyperparameters to balance the weights between different constraints; S39: Using an alternating learning strategy to learn the hash function f for medical image modalities (2) (·) and the hash function g of the diagnostic report modality (2) (·), as well as the hash code B and the classification network parameters W c ; Through these two hash functions, the corresponding modal features are obtained respectively: and in and are the network parameters of the corresponding functions respectively; during the optimization process, only one parameter is optimized each time, the other parameters are fixed unchanged, and the parameters of each network are updated through alternating cycles.

5. The cross-modal medical data hash retrieval method according to claim 1, characterized in that: In S4, a hash code is obtained by using a feature extraction network hash function, which specifically includes the following steps: S41: Given a medical image I q , first through the medical image feature extraction network f in the modality feature extraction stage (1) (·), obtain the medical image I q The eigenvector of in f (1) (·) The corresponding network parameters; then the medical image I is obtained through the forward propagation algorithm of the medical image hash layer q The hash code S42: Given a diagnostic report T q , first through the diagnostic report feature extraction network g in the modal feature extraction stage (1) (·), get the diagnostic report T q The eigenvector of in g (1) (·) The corresponding network parameters; then the diagnostic report T is obtained through the forward propagation algorithm of the diagnostic report hash layer. q The hash code 6. A cross-modal medical data hash retrieval method according to claim 1, characterized in that: In S5, cross-modal hash retrieval is performed using the generated hash code, specifically including the following steps: S51: Calculate the Hamming distance between the query modality data hash code and the retrieval library modality hash code; S52: The sample with the smallest Hamming distance is given through the Hamming distance sorting to complete the query function.

Citation Information

Cited By

  • Interhospital collaborative cross-modal medical data aggregation system based on cross attention

    CN120748757A

  • Inter-hospital collaborative cross-modal medical data aggregation system based on cross-attention

    CN120748757B