Micromedical cross-modal hash retrieval method based on lifelong learning

Abstract

The invention discloses a micromedical cross-modal Hash retrieval method based on lifelong learning, and the method comprises the steps: carrying out the coding of basic medical data through a double-tower Transform, and inputting the coded features into a Hash learning module; then, a semantic alignment module is used for constraining distance and label correlation in and across modals in the medical data, so that multi-modal data with the same or different semantics are close to or farther than one another when mapped to a Hamming space, and when incremental medical data of a new category arrives, a lifelong Hash learning module is designed; incremental data is directly mapped into a hash code instead of training a hash function by using accumulated medical data, so that catastrophic forgetting of hash learning is effectively avoided. According to the method, the training time and computing resources are saved from a medical stream scene, and continuous and efficient retrieval of clinical stream data is realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of image and text cross-modal processing, and particularly relates to a differentiable medical cross-modal hashing retrieval method based on lifelong learning. Background Art

[0002] With the gradual popularization of medical acquisition devices and the gradual improvement of medical information storage standards, medical data has grown explosively. Medical data presents in multimodal forms, such as diagnostic reports, X-rays, computed tomography, magnetic resonance imaging, and ultrasound imaging. Moreover, these multimodal data often appear simultaneously and complement each other, which are the main sources for disease screening and diagnosis. When radiologists make diagnoses, they usually judge based on the case experience they have observed before. Due to reasons such as large amounts of data and limited experience, misdiagnosis and other situations will inevitably occur, leaving a great hidden danger to the accurate treatment of patients. The multimodal medical retrieval technology can help doctors retrieve semantically similar multimodal case information, thereby improving the efficiency and accuracy of clinical diagnosis and treatment decisions. In addition, the integrated medical resources are convenient for novice doctors and students to learn. For the same patient, retrieving past reports and images to trace the condition can provide a comprehensive consideration for subsequent diagnoses. Furthermore, the medical cross-modal retrieval algorithm provides a new perspective for promoting computer-aided diagnosis systems. Therefore, realizing cross-modal data retrieval has important clinical significance in the medical field.

[0003] The hashing-based cross-modal retrieval method is an important branch of the cross-modal retrieval task. Its basic process is as Figure 1 shown. The cross-modal hashing method first extracts the features of multimodal data and uses a hashing network to map the high-dimensional data to a low-dimensional Hamming space. Then, by constraining the Hamming distance of similar (dissimilar) samples to be smaller (larger), the semantic similarity of the data is maintained. Finally, through the exclusive OR operation, fast and accurate retrieval is achieved. As a retrieval method with high efficiency and low storage, cross-modal hashing conforms to the actual application scenarios of hospitals facing large-scale data and high real-time requirements.

[0004] However, in actual clinical scenarios, as time goes by, multimedia medical data often arrives continuously in the form of a stream. For example, radiologists in a certain affiliated hospital of a university process an average of 300 medical images and diagnostic reports per person per day. It can be seen that clinical medical data presents the characteristics of large scale and streaming arrival. After new data arrives, since the traditional medical cross-modal hashing retrieval method learns the hashing function based on the batch processing mode, all the data must be stored, and then the hashing function is retrained on the accumulated data, which greatly increases the computational complexity and decreases the retrieval efficiency.

[0005] Recently, a few online hashing methods have alleviated the catastrophic forgetting problem that occurs when new tasks arrive. This is manifested as the model, after learning new tasks, disrupting the knowledge obtained from old tasks due to the drift of network parameters, resulting in a sharp decline in retrieval performance. These online hashing methods can not only efficiently retrieve old data but also handle new data search tasks with ease. However, these methods are all based on discrete optimization strategies and simultaneously ignore the problem of mismatched information between text and image data, resulting in suboptimal retrieval effects. In addition, due to the characteristics of medical images such as high similarity, low resolution, small lesion areas, and large edge noise, these methods are only applicable to natural flow scenarios and do not solve the catastrophic forgetting of streaming medical data in clinical scenarios. According to research, there is currently no online cross-modal hashing method specifically designed for medical dynamic flow data.

[0006] In summary, achieving intelligent cross-modal search and collaborative auxiliary diagnosis of "medical images - diagnostic reports" is of great significance for clinical condition analysis. By combining artificial intelligence and medical big data, performing efficient, fast, intelligent, and accurate dynamic incremental cross-media retrieval on medical data, further realizing precise auxiliary diagnosis, providing auxiliary diagnosis basis for inexperienced doctors, reducing the workload of elderly doctors, assisting doctors in quickly and efficiently implementing accurate disease assessment and providing effective treatment plans, and reducing patient waiting time for medical treatment and improving hospital operation efficiency, it has great practical application value. Therefore, it is of great significance to study the lifelong medical cross-modal hashing method to achieve fast and efficient retrieval on large-scale streaming multimedia data sets. Summary of the Invention

[0007] The present invention provides a differentiable medical cross-modal hashing retrieval method based on lifelong learning, and the technical problems to be solved are: realizing the retrieval of continuous flow medical data in the process of medical cross-modal retrieval, solving the discrete optimization NP-hard problem and the cross-modal data semantic gap problem, maintaining good retrieval performance, and having less computational resource overhead.

[0008] To solve the above technical problems, the present invention provides a differentiable medical cross-modal hashing retrieval method based on lifelong learning, including the steps of:

[0009] Obtain a basic data set, where the basic data set contains sample pairs, and each sample pair contains a medical image I and a report text T;

[0010] For each sample pair, use a two-tower Transformer model to extract the features of the medical image I and the report text T, denoted as the image visual feature X and the text feature Y respectively;

[0011] Adopt a continuously differentiable hash code generation method to generate hash codes for the image visual feature X and the text feature Y;

[0012] Using a basic dataset, determine the semantic alignment loss, where the semantic alignment loss includes inter-modal loss and intra-modal loss. The intra-modal loss includes the image-image intra-modal loss and the text-text intra-modal loss. The inter-modal loss refers to the alignment loss between images and texts, and the intra-modal loss refers to the alignment loss between samples of the same modality;

[0013] Using the semantic alignment loss, by the gradient descent method, keep the distance between similar samples closer and the distance between dissimilar samples farther within and between modalities;

[0014] Obtain incremental data, which includes new medical images I' and new report texts T'. The category of the incremental data is different from the category of the samples in the basic dataset;

[0015] Use a two-tower Transformer model to extract the features of the new medical images I' and the new report texts T', denoted as image visual features X' and text features Y';

[0016] Construct a lifelong hashing learning loss, where the lifelong hashing learning loss includes a lifelong image hashing loss and a lifelong text hashing loss;

[0017] By minimizing the lifelong hashing learning loss, update the hashing function to learn the hashing codes of the incremental data, and then perform cross-modal retrieval.

[0018] Furthermore, the two-tower Transformer model encodes images and texts respectively through a multi-layer attention mechanism and a multi-layer perceptron to extract image visual features and text features.

[0019] Furthermore, a continuously differentiable hashing code generation method is used to generate the hashing codes of the image visual features X and the text features Y, including:

[0020] Through linear layer processing, project the image visual features X and the text features Y into a two-dimensional space, and denote the output of each linear layer as represents the hashing position of the hashing code, B represents the length of the hashing code, and the number of linear layers is the same as the length of the hashing code;

[0021] Through softmax layer processing, is converted into a probability distribution for selecting binary hashing codes The number of softmax layers is the same as the length of the hashing code;

[0022] By selecting the index of the maximum value in, generate the hashing code corresponding to the image visual feature and the hashing code corresponding to the text feature.

[0023] Furthermore, the method for determining the semantic alignment loss is:

[0024] Connect the probability distribution corresponding to the visual feature and the probability distribution corresponding to the text feature together;

[0025] The feature probability distribution of the i-th image sample and the feature probability distribution of the j-th text sample are respectively denoted as and

[0026] Determine the modality distance between image and text according to the cosine distance The modality distance between image and image and the modality distance between text and text

[0027] Determine the correlation between images and texts through the sample labels The correlation between images and images and the correlation between texts and texts

[0028] According to and Determine the semantic alignment loss.

[0029] Furthermore, the inter-modal loss is determined by the modality distance between image and text and the correlation between images and texts Specifically, the inter-modal loss J xy is equal to the inter-modal relevant sample loss minus the inter-modal irrelevant sample loss wherein,

[0030] The inter-modal relevant sample loss is:[[]]

[0031] The inter-modal irrelevant sample loss is:[[]]

[0032] wherein, σ 1 is a balance term used to adjust the inter-modal relevant sample loss and the inter-modal irrelevant sample loss, and N represents the number of sample pairs in the basic dataset.

[0033] 6. The differentiable medical cross-modal hashing retrieval method based on lifelong learning according to claim 4, wherein the intra-modal loss of the image modality is determined by the modality distance between image and image and the correlation between images and images Specifically, the intra-modal loss J xx is equal to the image relevant sample loss minus the image irrelevant sample loss

[0034] The loss of image-related samples is as follows:

[0035] The loss of image-unrelated samples is as follows:

[0036] σ 2 is a balancing term used to adjust the loss of image-related samples and the loss of image-unrelated samples, and N represents the number of sample pairs in the basic dataset.

[0037] Furthermore, the loss within the text modality is determined by the modality distance between text-text images and the correlation between texts Specifically, the loss J yy within the text modality is equal to the loss of text-related samples minus the loss of text-unrelated samples σ 3 is a balancing term used to adjust the loss of text-related samples and the loss of text-unrelated samples.

[0038] Furthermore, constructing the lifelong hashing learning loss includes:

[0039] Calculating the similarity matrix S of the basic dataset and the similarity matrix S' of the incremental data;

[0040] Constructing the lifelong image hashing loss J ll (H' x ,H Qx ,θ x ) and the lifelong text hashing loss J ll (H' y ,H Qy ,θ y where the lifelong image hashing loss is used to maintain the similarity between the basic image data and the query data, and the similarity between the incremental image data and the query data,

[0041]

[0042] The lifelong text hashing loss is used to maintain the similarity between the basic text data and the query text data, and the similarity between the incremental text data and the query text data,

[0043]

[0044] where H x 、H' x and H Qx respectively represent the hash codes of the basic image data, the incremental image data, and the query image data, and H y 、H' y and H QyRespectively represent the hash codes of the base text data, incremental text data, and query text data; θ x and θ y Respectively represent the parameters of the image hash function and the parameters of the text hash function; H xi and H yi Respectively represent the i-th bit of the hash code of the base image data and the base text data; H' xi and H' yi Respectively represent the i-th bit of the hash code of the incremental image data and the incremental text data; H Qxi and H Qyi Respectively represent the i-th bit of the hash code of the query image and the query text; K represents a diagonal matrix, and the elements on its main diagonal are the hash code lengths;

[0045] Use the lifelong image hash loss J ll (H' x , H Qx , θ x ) and the lifelong text hash loss J ll (H' y , H Qy , θ y ) to generate the lifelong hash learning loss.

[0046] Furthermore, the method for determining the similarity matrix S of the base dataset is:

[0047]

[0048] where,

[0049] The superscript T represents matrix transpose,

[0050] V x and V y Respectively represent the Hamming space of the base image data and the Hamming space of the base text data;

[0051] W 1 and W 2 Respectively represent the weight matrix of the hash code of the base image data and the weight matrix of the hash code of the base text data;

[0052] H x and H y Respectively represent the hash code of the base image data and the hash code of the base text data in the base dataset.

[0053] Furthermore, the method for determining the similarity matrix S' of the incremental dataset is:

[0054]

[0055] where, The superscript T represents the matrix transpose;

[0056] V x ' and V y ' respectively represent the Hamming space of the incremental image data and the Hamming space of the incremental text data;

[0057] W 3 and W 4 respectively represent the weight matrix of the hash code of the incremental image data and the weight matrix of the hash code of the incremental text data;

[0058] H' x and H' y respectively represent the hash code of the incremental image data and the hash code of the incremental text data in the incremental data set.

[0059] The beneficial effects of the present invention are:

[0060] The differentiable medical cross-modal hashing retrieval method based on lifelong learning provided by the present invention aims at the discontinuity, discrete optimization difficulty and semantic alignment problems of medical cross-modal hashing. Combining the lifelong learning paradigm, differentiable selection mechanism and semantic alignment module, a lifelong differentiable hashing learning method is invented to achieve efficient cross-modal retrieval of streaming medical data. Specifically, for the discontinuity problem, a lifelong learning paradigm is designed to achieve large-scale continuous streaming medical cross-modal retrieval. When incremental medical data with new categories arrives, the incremental data is directly mapped into the binary Hamming space, rather than retraining the hash function with the accumulated data, effectively avoiding the catastrophic forgetting of hash learning, saving training time and computing resources, and realizing continuous retrieval of clinical streaming medical data; for the problem of difficult discrete optimization of hash codes, a differentiable and continuous hash selection mechanism is constructed in the hash learning stage to generate highly discriminative hash codes, and through gradient descent optimization, the quantization error caused by relaxation or approximation is avoided; for the problem of unaligned data semantics, combining feature correlation and label correlation, the information similarity between and within modalities is constrained to solve the problems of semantic imbalance and gap, and improve the retrieval performance..

[0061] Other advantages, objectives and features of the present invention will be described to some extent in the subsequent description, and to some extent, will be obvious to those skilled in the art based on the study of the following text, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the drawings, wherein:

[0063] Figure 1It is the architecture diagram of lifelong differentiable hashing learning provided by the embodiments of the present invention;

[0064] Figures 2(a), 2(b) and 2(c) are the performance degradation result diagrams based on medical data provided by the embodiments of the present invention;

[0065] Figures 3(a) and 3(b) are the retrieval visualization result diagrams based on fundus images provided by the embodiments of the present invention. Detailed implementation manners

[0066] The preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the preferred embodiments are only for illustrating the present invention rather than limiting the protection scope of the present invention.

[0067] Combined with Figure 1 , a differentiable medical cross-modal hashing retrieval method based on lifelong learning proposed by the present invention includes the following steps:

[0068] Obtain a basic data set, which contains multiple sample pairs, and each sample pair contains a medical image I and a report text T;

[0069] For each sample pair, use a two-tower Transformer model to extract the features of the medical image I and the report text T, denoted as the image visual feature X and the text feature Y respectively;

[0070] Adopt a continuously differentiable hashing code generation method to generate hashing codes for the image visual feature X and the text feature Y;

[0071] Use the basic data set to determine the semantic alignment loss. Among them, the semantic alignment loss includes the inter-modal loss and the intra-modal loss. The intra-modal loss includes the intra-modal loss between images and the intra-modal loss between texts. The inter-modal loss refers to the alignment loss between images and texts, and the intra-modal loss refers to the alignment loss between samples of the same modality;

[0072] Use the semantic alignment loss and, through the gradient descent method, keep the distance between similar samples within and between modalities closer and the distance between dissimilar samples farther;

[0073] Obtain incremental data, which contains new medical images I' and new report texts T'. The categories of the incremental data are different from those of the samples in the basic data set. Among them, different categories mean that there are labels that have not appeared before in the incremental data, that is, new labels;

[0074] Use the two-tower Transformer model to extract the features of the new medical image I' and the new report text T', denoted as the image visual feature X' and the text feature Y';

[0075] Construct a lifelong hash learning loss, where the lifelong hash learning loss includes a lifelong image hash loss and a lifelong text hash loss;

[0076] By minimizing the lifelong hash learning loss, the hash function is updated to learn the hash codes of incremental data, and then cross-modal retrieval is performed.

[0077] The above method is described in detail below.

[0078] In some embodiments, the two-tower Transformer model encodes images (e.g., all medical images I) and texts (e.g., all report texts T) respectively through a multi-layer attention mechanism and a multi-layer perceptron, so as to extract image visual features and text features.

[0079] The process of the two-tower Transformer extracting medical image and diagnostic report features is as follows:

[0080] Divide the given medical image into several patches, and then use a multi-layer attention mechanism and a multi-layer perceptron to learn the features of image and text data, which are respectively expressed as:

[0081] X = MLP(MSA(I)),

[0082] Y = MLP(MSA(T)),

[0083] where MSA represents the multi-layer attention mechanism and MLP represents the multi-layer perceptron.

[0084] After extracting the image visual features and text features, these features are input into the hash learning module, and the learned feature mapping is converted into discrete hash codes H ∈ {H x , H y}, where i represents the hash position of the hash code, and B is the length of the hash code. H x and H y represent the hash codes of the image and the text respectively, and h xi and h yi represent the i-th bit hash codes of the image (corresponding to the subscript x) and the text (corresponding to the subscript y) respectively.

[0085] Specifically, the hash learning module includes multiple linear layers and multiple activation function layers (such as the softmax layer), the number of linear layers is the same as the length of the hash code, and the process of generating the hash code includes:

[0086] Through B-layer linear layer processing, project the image visual feature X and the text feature Y into a two-dimensional space, and denote the output of each linear layer as Linear represents the processing of the linear layer, and | indicates that the objects of the linear layer processing are X and Y, and the processing methods are the same; i ∈ {1, 2,..., B}, representing the hash position of the hash code, and B represents the length of the hash code;

[0087] Then, the real-valued vector is converted into a probability distribution through the softmax function, so that the value of each vector represents the corresponding probability, that is, through the processing of the softmax layer, is converted into a probability distribution for selecting the binary hash code That is

[0088]

[0089] By selecting the index of the maximum value in, the hash code corresponding to the visual feature of the image and the hash code corresponding to the text feature are generated, that is

[0090]

[0091] where f i s [0] represents the value corresponding to the part with index 0 in; f i s [1] represents the value corresponding to the part with index 1 in.

[0092] where H ∈ {H x , H y}, H x and H y respectively represent the hash codes of the image and text data, h xi and h yi respectively represent the i-th bit hash codes corresponding to the image and text data, and the h i obtained by processing X is h xi , and the h i obtained by processing Y is h yi . The task of generating the hash code is naturally converted into the task of classifying and selecting the hash value. Since the index of the two-dimensional feature vector can only be 0 or 1, therefore, the hash code is generated by selecting the index of the maximum value in.

[0093] Such a process is differentiable, and the task of generating the hash code can be converted into the task of classifying and selecting the hash value, so as to generate a highly discriminative hash code. Optimize through gradient descent to avoid quantization errors caused by relaxation or approximation.

[0094] Next, by designing a semantic alignment module, calculate the semantic alignment loss, that is, through intra-modal loss and inter-modal loss, measure the relationships between samples (including the relationships between images and texts, images and images, texts and texts), and alleviate semantic imbalance and the gap.

[0095] The method for determining the semantic alignment loss is as follows:

[0096] Connect the probability distribution corresponding to the visual features and the probability distribution corresponding to the text features together;

[0097] Connect the feature probability distribution of the i-th image and the feature probability distribution of the j-th text, and denote them as and

[0098] According to the cosine distance cos(·,·), determine the modality distance between image and text the modality distance between image and image and the modality distance between text and text That is

[0099]

[0100] Although the modalities and dimensions of images and texts are different, they correspond to the same label information.

[0101] Through the sample labels, determine the correlation between images and texts the correlation between images and images and the correlation between texts and texts

[0102] Among them,

[0103] c i and c j respectively represent the labels corresponding to the i-th and j-th samples,

[0104] When calculating That is, R i,j represents at this time, c i and c j respectively represent the labels corresponding to the i-th image sample and the j-th text sample;

[0105] When calculating That is, R i,j represents at this time, c i and c j respectively represent the labels corresponding to the i-th image sample and the j-th image sample;

[0106] When calculating i.e., R i,j denotes when, c i and c j respectively denote the labels corresponding to the i-th text sample and the j-th text sample.

[0107] Then, the semantic alignment loss can be determined according to and the semantic alignment loss is determined.

[0108] The semantic alignment loss includes the inter-modal loss and the intra-modal loss.

[0109] Specifically, the inter-modal loss is determined by the modal distance between the image and the text and the correlation between the image and the text The inter-modal loss consists of the losses of inter-modal relevant samples and irrelevant samples, making different modal data with the same semantics close to each other when mapped to the Hamming space. In the dataset, an image corresponds to several labels, and a text also corresponds to several labels. If a pair of image and text contains more than one common label, it is a relevant sample, otherwise it is an irrelevant sample.

[0110] Specifically, the inter-modal loss J xy is equal to the loss of inter-modal relevant samples minus the loss of inter-modal irrelevant samples That is:

[0111]

[0112] Among them,

[0113] the loss of inter-modal relevant samples is:

[0114] the loss of inter-modal irrelevant samples is:

[0115] Among them, N represents the number of sample pairs in the basic dataset, and σ 1 is a balancing term used to adjust the loss of inter-modal relevant samples and the loss of inter-modal irrelevant samples. Because usually the number of irrelevant samples is more than that of relevant samples. By minimizing the inter-modal loss function, the images and texts corresponding to relevant labels are as close as possible in the Hamming space, while the images and texts corresponding to irrelevant labels are as far away as possible in the Hamming space. Therefore, it can be reasonably explained to maintain the similarity between samples through the inter-modal loss.

[0116] Next, by using the within-modal loss to embed the relevant and irrelevant losses of image-image (text-text) in different modalities, ensuring that samples in each modality are mapped to a common space to preserve semantic relationships, the within-image modality loss and the within-text modality loss are designed respectively.

[0117] The within-image modality loss is determined by the modal distance between images-image and the correlation between images Specifically, the within-image modality loss J xx is equal to the loss of image-related samples minus the loss of image-unrelated samples

[0118] The loss of image-related samples is:

[0119] The loss of image-unrelated samples is:

[0120] σ 2 is a balancing term used to adjust the loss of image-related samples and the loss of image-unrelated samples.

[0121] The within-text modality loss is determined by the modal distance between text-text and the correlation between texts Specifically, the within-text modality loss J yy is equal to the loss of text-related samples minus the loss of text-unrelated samples σ 3 is a balancing term used to adjust the loss of text-related samples and the loss of text-unrelated samples. and are defined similarly to the loss of image-related samples and the loss of image-unrelated samples within the image modality, and will not be elaborated here.

[0122] By combining the inter-modal loss and the intra-modal loss into the semantic alignment module, the distance between similar samples can be shortened, and the distance between dissimilar samples between modalities and within modalities can be increased, further improving the accuracy.

[0123] The present invention realizes large-scale continuous-flow medical cross-modal retrieval through the lifelong hashing module. When incremental medical data with new categories arrives, the incremental data is directly mapped into the binary Hamming space, rather than retraining the hashing function with the accumulated data, effectively avoiding the catastrophic forgetting of hashing learning, saving training time and computing resources, and realizing the continuous retrieval of clinical flow medical data. The working process of the lifelong hashing learning module is as follows:

[0124] By minimizing the inner product of the query hash code and the base hash code using the F-norm and minimizing the inner product of the newly added hash code and the query hash code (i.e., minimizing the lifelong hash learning loss), the correlation between the base dataset and the incremental data is maintained. Here, the query hash code refers to the hash code of the query data, the base hash code refers to the hash code corresponding to the data in the base dataset, and the newly added hash code refers to the hash code of the incremental data;

[0125] Constructing the lifelong hash learning loss specifically includes the following steps:

[0126] Encode the multimodal medical data in the Hamming space \(V\in\{V x ,V y \}\). By calculating the similarity of the samples, the similarity matrix \(S\) of the base dataset and the similarity matrix \(S'\) of the incremental data can be obtained;

[0127] Specifically,

[0128] where, The superscript \(T\) represents the matrix transpose,

[0129] \(V x \) and \(V y \) respectively represent the Hamming space of the base image data and the Hamming space of the base text data;

[0130] \(W 1 \) and \(W 2 \) respectively represent the weight matrix of the base image data and the weight matrix of the base text data;

[0131] \(H x \) and \(H y \) respectively represent the hash code of the base image data and the hash code of the base text data in the base dataset.

[0132] Next, define the similarity preservation in the latent space after Transformer encoding and hash learning, which can be expressed as:

[0133]

[0134] where, \(J(H x ,H y ,\theta x ,\theta y )\) represents the similarity preservation loss after feature encoding;

[0135] \(\theta x \) and \(\theta y \) respectively represent the parameters of the image hash function and the text hash function;

[0136] By treating the expression of J(Hx, Hy, θx, θy) as a bilinear regression function, B can be directly solved 1 , that is:

[0137]

[0138] According to the similarity preservation and optimization objectives, it can be defined:

[0139]

[0140] The solution gives The superscript T represents the matrix transpose;

[0141] V x ' and V y ' respectively represent the Hamming space of the incremental image data and the Hamming space of the incremental text data;

[0142] W 3 and W 4 respectively represent the weight matrix of the hash code of the incremental image data and the weight matrix of the hash code of the incremental text data;

[0143] H' x and H' y respectively represent the hash code of the incremental image data and the hash code of the incremental text data in the incremental dataset.

[0144] At this time, calculate the supervision information S of the basic dataset and the supervision information S' of the incremental dataset to measure whether the image and text information still maintain semantic similarity after passing through the hash function and hash learning.

[0145] To maintain the plasticity of new knowledge and the stability of old knowledge, a lifelong hash loss is designed, including constructing a lifelong image hash loss J ll (H' x , H Qx , θ x ) and a lifelong text hash loss J ll (H' y , H Qy , θ y ), where the lifelong image hash loss is used to maintain the similarity between the basic image data and the query data, as well as the similarity between the incremental image data and the query data, and the lifelong text hash loss is used to maintain the similarity between the basic text data and the query text data, and the incremental text data and the query text data. H x 、H' x and H Qx respectively represent the hash codes of the basic image data, the incremental image data and the query image data, H y 、H' y and HQy respectively represent the hash codes of the base text data, incremental text data, and query text data; θ x and θ y respectively represent the parameters of the image hash function and the text hash function.

[0146]

[0147] Where B is the length of the hash code, K represents a diagonal matrix whose main diagonal elements are the hash code lengths, H xi represents the i-th bit of the hash code of the base image data, H Qxi represents the i-th bit of the hash code of the query image, H' xi represents the i-th bit of the hash code of the incremental image data, H yi represents the i-th bit of the hash code of the base text data, H Qyi represents the i-th bit of the hash code of the query text, H' yi represents the i-th bit of the hash code of the incremental text data.

[0148] Using the lifelong image hash loss J ll (H' x ,H Qx ,θ x ) and the lifelong text hash loss J ll (H' y ,H Qy ,θ y ) to generate the lifelong hash learning loss J ll , that is:

[0149] J ll = J ll (H' x ,H Qx ,θ x ) + J ll (H' y ,H Qy ,θ y ).

[0150] When the incremental data arrives, by minimizing the lifelong hash loss, the hash function is updated without repeating the training of all accumulated data.

[0151] Figures 2(a), 2(b), and 2(c) respectively show the performance degradation results of the present invention on fundus data, colorectal adenocarcinoma pathological section data, and lung X-ray data; Figures 3(a) and 3(b) show the visual retrieval results of the present invention on fundus data. Among them, the performance degradation experiment aims to verify whether lifelong hashing learning can adapt to the environment of continuous growth of clinical streaming data and alleviate the catastrophic forgetting caused by adding new data categories in the data stream. Among them, the fundus data is paired data of the left and right eyes of 3500 patients, and the dataset provides 7 types of labels, namely normal, diabetes, glaucoma, cataract, age-related macular degeneration, hypertension, myopia, and other diseases / abnormalities; the colorectal adenocarcinoma data is divided into 9 categories by pathologists manually dividing the tissue regions in the whole slide image.

[0152] The lung X-ray data is a dataset containing 7,470 frontal and lateral chest X-ray images proposed by Indiana University, accompanied by 3,955 corresponding diagnostic reports. Compared with nine methods, namely DCMH ("Deep cross-modal hashing" published by Jiang et al. in CVPR 2017), DCHMT ("Differentiable cross-modal hashing via multimodal transformers" published by Tu et al. in ACM MM 2022), DESAH ("Deep Enhanced-Similarity Attention Cross-modal Hashing Learning" published by Ge et al. in ACM MM 2023), MESDCH ("Multi-label enhancement based self-supervised deep cross-modal hashing" published by Zou et al. in Neurocomputing 2022), MIAN ("Modality-invariant asymmetric networks for cross-modal hashing" published by Zhang et al. in TKDE 2022), SSAH ("Self-supervised adversarial hashing networks for cross-modal retrieval" published by Li et al. in CVPR 2018), AGAH ("Adversary-guided asymmetric hashing for cross-modal retrieval" published by Gu et al. in ACM MM 2019), PEPFCH ("Privacy-Enhanced Prototype-Based Federated Cross-Modal Hashing for Cross-Modal Retrieval" published by Zou et al. in ACM TOMM 2024) and GCDH ("Graph convolutional network discrete hashing for cross-modal retrieval" published by Bai et al. in NNLS 2024), the present invention can adapt to streaming data and has good retrieval performance.

[0153] Figure 2(a) corresponds to lung X-ray data; Figure 2(b) corresponds to pathological section data of colorectal adenocarcinoma; Figure 2(c) corresponds to fundus data. "Image → Text" above the image indicates retrieving text with the image, and "Text → Image" indicates retrieving the image with the text; bit represents hash bits of different lengths. For example, 128bit means there are 128 hash codes. Among them, the abscissa represents the change of tasks. In the category setting, for example, "5 / 3" means the number of categories in the basic dataset is 5, and the number of newly added categories in the incremental dataset is 3, and these 3 categories are not included in the basic dataset; the ordinate represents the retrieval accuracy. It can be seen from Figure 2(a), Figure 2(b) and Figure 2(c) that the retrieval accuracy of the method of the present invention is higher.

[0154] To verify the effectiveness of the present invention from multiple perspectives, the retrieved medical data results were visualized as shown in Figure 3. Only the results of the present invention on the fundus dataset are shown here. The cosine distance is used as the similarity metric to compare sample instances. The visualization results respectively show the top 5 and 4 retrieval results of the "Image → Text" and "Text → Image" tasks with 64 hash bits.

[0155] Among them, Figure 3(a) is the top 5 retrieval visualization for the "Text → Image" task. The three boxes in red, blue, yellow, and green indicate that the retrieval results share 0, 1, 2, and 3 labels with the query. The higher the proportion of the red box, the worse the retrieval performance; the higher the proportion of the green box, the higher the correlation of the retrieved samples. Figure 3(b) is the top 4 retrieval visualization for the "Image → Text" task. The numbers in parentheses indicate the number of labels shared between the retrieval and the query. The larger the number, the higher the correlation, and zero indicates no correlation.

[0156] It can be seen from Figure 3(a) that for a given query, the proportion of relevant samples in the retrieval results of the present invention is much larger than that of the comparison method. In addition, the number of green boxes is more than that of other comparison retrieval methods, indicating that the present invention has a strong retrieval ability for highly relevant samples. As shown in Figure 3(b), compared with other methods, the retrieval results of the present invention contain more similar and relevant samples. The lifelong learning module in the present invention can flexibly process streaming medical data, learn to adapt to new tasks, and has good adaptability to the dynamic clinical environment.

[0157] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present invention can be modified or equivalently replaced without departing from the purpose and scope of the present technical solution, and they should all be covered by the scope of the claims of the present invention.

Claims

1. A differentiable medical cross-modal hash retrieval method based on lifelong learning, characterized in that: include: Obtain a basic data set, which contains sample pairs, each of which contains a medical image I and a report text T; For each sample pair, the dual-tower Transformer model is used to extract the features of the medical image I and the report text T, which are recorded as image visual features X and text features Y respectively; A continuously differentiable hash code generation method is used to generate hash codes of image visual features X and text features Y; Using the basic data set, determine the semantic alignment loss, where the semantic alignment loss includes inter-modality loss and intra-modality loss. The intra-modality loss includes image-image intra-modality loss and text-text intra-modality loss. The inter-modality loss refers to the alignment loss between image and text, and the intra-modality loss refers to the alignment loss between samples of the same modality. Using semantic alignment loss, we use gradient descent to keep similar samples closer and dissimilar samples farther apart within and between modalities. Acquire incremental data, where the incremental data includes new medical images I' and new report texts T', and the categories of the incremental data are different from those of the samples in the basic data set; The dual-tower Transformer model is used to extract the features of the new medical image I' and the new report text T', which are recorded as image visual features X' and text features Y'; Constructing a lifelong hash learning loss, wherein the lifelong hash learning loss includes a lifelong image hash loss and a lifelong text hash loss; By minimizing the lifelong hash learning loss, the hash function is updated to learn the hash code of incremental data for cross-modal retrieval.

2. The differentiable medical cross-modal hash retrieval method based on lifelong learning according to claim 1 is characterized in that: The dual-tower Transformer model encodes images and texts respectively through a multi-layer attention mechanism and a multi-layer perceptron, thereby extracting image visual features and text features.

3. The differentiable medical cross-modal hash retrieval method based on lifelong learning according to claim 1 is characterized in that: A continuously differentiable hash code generation method is used to generate hash codes of image visual features X and text features Y, including: Through linear layer processing, the image visual features X and text features Y are projected into two-dimensional space, and the output of each linear layer is recorded as represents the hash position of the hash code, B represents the length of the hash code, and the number of linear layers is the same as the length of the hash code; After processing by the softmax layer, Convert to a probability distribution for choosing binary hash codes The number of softmax layers is the same as the length of the hash code; By selecting The index of the maximum value in generates the hash code corresponding to the image visual feature and the hash code corresponding to the text feature.

4. The differentiable medical cross-modal hash retrieval method based on lifelong learning according to claim 1 is characterized in that: The semantic alignment loss is determined as follows: The probability distribution corresponding to the visual features and the probability distribution corresponding to the text features Connect The feature probability distribution of the i-th image sample and the feature probability distribution of the j-th text sample are respectively recorded as and Determine the modal distance between image and text based on the cosine distance Image-to-image modal distance and the modal distance between text and text Determine the correlation between images and text through sample labels Correlation between images and the correlation between text and text according to and Determine the semantic alignment loss.

5. The differentiable medical cross-modal hash retrieval method based on lifelong learning according to claim 4 is characterized in that: Inter-modality loss via image-text modality distance and the correlation between images and text To determine, specifically, the inter-modal loss J xy Equal to the inter-modal correlation sample loss Subtract the loss of irrelevant samples between modalities in, The inter-modality correlation sample loss is: The loss of uncorrelated samples between modalities is: Among them, σ1 is a balancing term used to adjust the loss of inter-modality correlated samples and inter-modality uncorrelated sample losses, and N represents the number of sample pairs in the base dataset.

6. The differentiable medical cross-modal hash retrieval method based on lifelong learning according to claim 4 is characterized in that: Image intra-modality loss via image-to-image modality distance and the correlation between images To determine, specifically, the image intra-modality loss J xx Equal to image-related sample loss Subtract the loss of irrelevant samples from the image The image-related sample loss is: The loss of image-irrelevant samples is: σ2 is a balancing term used to adjust the loss of image-related samples and the loss of image-irrelevant samples, and N represents the number of sample pairs in the base dataset.

7. The differentiable medical cross-modal hash retrieval method based on lifelong learning according to claim 4 is characterized in that: The intra-modality loss of text is calculated by the modality distance between text and text images. and the correlation between text and text To determine, specifically, the textual intra-modal loss J yy Equal to the text-related sample loss Subtract the loss of text irrelevant samples σ3 is a balancing term used to adjust the loss of text-related samples and the loss of text-irrelevant samples.

8. The differentiable medical cross-modal hash retrieval method based on lifelong learning according to claim 1 is characterized in that: Constructing the lifelong hash learning loss involves: Calculate the similarity matrix S of the basic data set and the similarity matrix S' of the incremental data; Constructing Lifetime Image Hashing Loss J ll (H' x ,H Qx ,θ x ) and lifetime text hash loss J ll (H' y ,H Qy ,θ y ), where the lifetime image hash loss is used to maintain the similarity between the base image data and the query data, as well as the similarity between the incremental image data and the query data, Lifetime text hash loss is used to maintain the similarity between base text data and query text data, incremental text data and query text data. Among them, H x , H' x and H Qx Represent the hash codes of the basic image data, incremental image data, and query image data, respectively. y , H' y and H Qy Represent the hash codes of basic text data, incremental text data and query text data respectively; θ x and θ y They represent the parameters of the image hash function and the parameters of the text hash function respectively; H xi and H yi Represents the i-th bit of the hash code of the basic image data and the basic text data respectively; H' xi and H' yi represents the i-th bit of the hash code of the incremental image data and the incremental text data respectively; H Qxi and H Qyi represents the i-th bit of the hash code of the query image and query text respectively; K represents a diagonal matrix, and the elements of its main diagonal are the length of the hash code; Exploiting Lifetime Image Hashing LossJ ll (H' x ,H Qx ,θ x ) and lifetime text hash loss J ll (H' y ,H Qy ,θ y ) Generate lifelong hash learning loss.

9. The differentiable medical cross-modal hash retrieval method based on lifelong learning according to claim 8 is characterized in that: The similarity matrix S of the basic data set is determined as follows: in, The superscript T indicates the matrix transpose. V x and V y Respectively represent the Hamming space of the basic image data and the Hamming space of the basic text data; W1 and W2 represent the weight matrix of the basic image data hash code and the weight matrix of the basic text data hash code respectively; H x and H y They respectively represent the hash code of the basic image data and the hash code of the basic text data in the basic dataset.

10. The differentiable medical cross-modal hash retrieval method based on lifelong learning according to claim 8, characterized in that: The method for determining the similarity matrix S' of the incremental data set is: in, The superscript T indicates matrix transpose; V x ' and V y 'represent the Hamming space of incremental image data and the Hamming space of incremental text data respectively; W3 and W4 represent the weight matrix of the incremental image data hash code and the weight matrix of the incremental text data hash code, respectively; H' x and H' y They respectively represent the hash codes of the incremental image data and the incremental text data in the incremental dataset.

Images