Training method of cross-modal Hash retrieval model, cross-modal Hash retrieval method, device and equipment

Abstract

The invention provides a cross-modal Hash retrieval model training method, a cross-modal Hash retrieval method, a cross-modal Hash retrieval device and cross-modal Hash retrieval equipment. The cross-modal Hash retrieval model training method comprises the following steps: acquiring at least one training sample pair comprising sample image data and a sample diagnosis text; determining a first type of Hash real value feature of each sample image data, and determining a second type of Hash real value feature of each sample diagnosis text; determining an adjacent matrix between the diagnosis tags in the diagnosis tag set of all the training sample pairs based on the reference diagnosis tags in the knowledge graph; for each training sample pair, based on the adjacent matrix and the training sample pair, determining a third-class hash real value feature corresponding to the training sample pair; and training the initial cross-modal Hash retrieval model based on each first type of Hash real value feature, each second type of Hash real value feature and each third type of Hash real value feature to obtain a cross-modal Hash retrieval model. According to the method, the accuracy of the cross-modal hash retrieval model can be improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a training method for a cross-modal hash retrieval model, a cross-modal hash retrieval method, a device and equipment. Background Art

[0002] With the continuous diversification of medical testing methods and the continuous advancement of medical information technology, the accumulation of multimodal patient data is becoming increasingly abundant. Cross-modal retrieval for multimodal patient data has become a research hotspot in academia and industry in recent years.

[0003] At present, deep cross-modal hashing has become a relatively popular supervised cross-modal hashing method and has been applied to the cross-modal retrieval task of patient data. In the prior art, the diagnosis results and symptoms annotated by doctors are usually used as multiple labels, and their pairwise similarity is determined based on whether the patient instances of different modalities have the same labels, and then the training process of the cross-modal hash retrieval model is supervised to complete the training of the cross-modal hash retrieval model.

[0004] However, in the above training process, only the information of the label itself is used, so that the accuracy of the cross-modal hash retrieval model finally trained is not high. Summary of the invention

[0005] The present invention provides a training method for a cross-modal hash retrieval model, a cross-modal hash retrieval method, an apparatus and a device, so as to solve the defect that the accuracy of the cross-modal hash retrieval model obtained by training in the prior art is not high, and achieve the purpose of improving the accuracy of the cross-modal hash retrieval model.

[0006] The present invention provides a training method for a cross-modal hash retrieval model, comprising: Acquire at least one training sample pair, wherein the training sample pair includes sample image data and sample diagnosis text; Determine a first type of hash real-valued feature corresponding to each of the sample image data, and determine a second type of hash real-valued feature corresponding to each of the sample diagnosis texts; Based on each reference diagnostic label in the knowledge graph, determining an adjacency matrix between diagnostic labels in a diagnostic label set corresponding to all the training sample pairs; For each of the training sample pairs, based on the adjacency matrix and the training sample pair, determine a third type of hash real-valued feature corresponding to the training sample pair; Based on each of the first-category hash real-valued features, each of the second-category hash real-valued features, and each of the third-category hash real-valued features, an initial cross-modal hash retrieval model is trained to obtain a cross-modal hash retrieval model.

[0007] According to a training method for a cross-modal hash retrieval model provided by the present invention, the method of determining the third type of hash real-valued features corresponding to the training sample pairs based on the adjacency matrix and the training sample pairs includes: Based on the diagnostic label set and at least one target diagnostic label corresponding to the training sample pair, determining an initial multi-label vector of the training sample pair, wherein the element value in the initial multi-label vector is used to indicate whether the diagnostic label set includes the target diagnostic label; Obtaining initial features corresponding to each of the target diagnostic labels; For each of the target diagnostic labels, based on the adjacency matrix, the initial multi-label vector and the initial features corresponding to the target diagnostic label, determine the class hash feature corresponding to the target diagnostic label; Based on the class hash features corresponding to each of the target diagnostic labels, a third class hash real-valued feature corresponding to the training sample pair is determined.

[0008] According to a training method for a cross-modal hash retrieval model provided by the present invention, the initial cross-modal hash retrieval model is trained based on each of the first-category hash real-valued features, each of the second-category hash real-valued features, and each of the third-category hash real-valued features to obtain a cross-modal hash retrieval model, including: For each of the training sample pairs, based on the first type of hash real-valued features, the second type of hash real-valued features and the third type of hash real-valued features corresponding to the training sample pairs, and the cross-modal attention weights between the modal data corresponding to the target class hash real-valued features and the modal data corresponding to the other types of hash real-valued features, an updated target class hash real-valued feature is determined; the target class hash real-valued feature includes at least one of the following: the first type of hash real-valued features, the second type of hash real-valued features and the third type of hash real-valued features; Based on the updated target class hash real-valued features, the initial cross-modal hash retrieval model is trained to obtain a cross-modal hash retrieval model.

[0009] According to a training method for a cross-modal hash retrieval model provided by the present invention, the method further includes: In the case where the target class hash real-valued features include first-class hash real-valued features, for each of the training sample pairs, based on the first-class hash real-valued features and the third-class hash real-valued features corresponding to the training sample pairs, determine a first cross-modal attention weight between the sample image data and the corresponding target diagnosis label in the training sample pair, and based on the first-class hash real-valued features and the second-class hash real-valued features corresponding to the training sample pairs, determine a cross-modal attention weight between the sample image data and the corresponding sample diagnosis text in the training sample pair; In the case where the target class hash real-valued features include second-class hash real-valued features, for each of the training sample pairs, based on the second-class hash real-valued features and the third-class hash real-valued features corresponding to the training sample pairs, the cross-modal attention weight between the sample diagnostic text and the corresponding target diagnostic label in the training sample pair is determined.

[0010] According to a training method for a cross-modal hash retrieval model provided by the present invention, the target class hash real-valued features include the first class hash real-valued features, the second class hash real-valued features and the third class hash real-valued features; the initial cross-modal hash retrieval model is trained based on the updated target class hash real-valued features to obtain the cross-modal hash retrieval model, including: Based on each updated first-category hash real-valued feature, each updated second-category hash real-valued feature, and each updated third-category hash real-valued feature, respectively determine an inter-modality loss and an intra-modality loss; Based on the inter-modality loss and the intra-modality loss, the initial cross-modal hash retrieval model is trained to obtain the cross-modal hash retrieval model.

[0011] According to a training method for a cross-modal hash retrieval model provided by the present invention, the initial cross-modal hash retrieval model is trained based on the inter-modal loss and the intra-modal loss to obtain a cross-modal hash retrieval model, comprising: Binarizing the first type of hash real-valued features corresponding to the training sample pair to obtain a first hash code, and binarizing the second type of hash real-valued features corresponding to the training sample pair to obtain a second hash code; Determine a quantization loss based on the updated first type of hash real-valued features corresponding to the training sample pair, the first hash code, the updated second type of hash real-valued features corresponding to the training sample pair, and the second hash code; Based on the inter-modality loss, the intra-modality loss and the quantization loss, the initial cross-modal hash retrieval model is trained to obtain the cross-modal hash retrieval model.

[0012] According to a training method for a cross-modal hash retrieval model provided by the present invention, the initial cross-modal hash retrieval model is trained based on each of the first-category hash real-valued features, each of the second-category hash real-valued features, and each of the third-category hash real-valued features to obtain a cross-modal hash retrieval model, including: Based on each of the first-category hash real-valued features, each of the second-category hash real-valued features, and each of the third-category hash real-valued features, the network parameters of the image encoder, the network parameters of the text encoder, and the network parameters of the label encoder in the initial cross-modal hash retrieval model are iteratively adjusted to obtain the cross-modal hash retrieval model.

[0013] The present invention also provides a cross-modal hash retrieval method, comprising: Obtain the data to be retrieved of the first mode; Inputting the to-be-retrieved data of the first modality into a cross-modal hash retrieval model to obtain target data of the second modality output by the cross-modal hash retrieval model; the cross-modal hash retrieval model is trained based on any of the above methods; Among them, when the data to be retrieved in the first modality includes image data, the target data in the second modality includes diagnostic text related to the image data; when the data to be retrieved in the first modality includes diagnostic text, the target data in the second modality includes image data related to the diagnostic text.

[0014] The present invention also provides a training device for a cross-modal hash retrieval model, comprising: An acquisition module, used to acquire at least one training sample pair, wherein the training sample pair includes sample image data and sample diagnosis text; A determination module, used to determine the first type of hash real-valued features corresponding to each of the sample image data, and to determine the second type of hash real-valued features corresponding to each of the sample diagnosis texts; The determination module is further used to determine the adjacency matrix between the diagnostic labels in the diagnostic label set corresponding to all the training sample pairs based on the reference diagnostic labels in the knowledge graph; The determination module is further configured to determine, for each of the training sample pairs, a third type of hash real-valued feature corresponding to the training sample pair based on the adjacency matrix and the training sample pair; The training module is also used to train the initial cross-modal hash retrieval model based on each of the first-category hash real-valued features, each of the second-category hash real-valued features, and each of the third-category hash real-valued features to obtain a cross-modal hash retrieval model.

[0015] The present invention also provides a training device for a cross-modal hash retrieval model, comprising: An acquisition module, used for acquiring the data to be retrieved in the first mode; An input module, used for inputting the to-be-retrieved data of the first modality into a cross-modal hash retrieval model to obtain target data of the second modality output by the cross-modal hash retrieval model; the cross-modal hash retrieval model is trained based on any of the above methods; Among them, when the data to be retrieved in the first modality includes image data, the target data in the second modality includes diagnostic text related to the image data; when the data to be retrieved in the first modality includes diagnostic text, the target data in the second modality includes image data related to the diagnostic text.

[0016] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements a training method for a cross-modal hash retrieval model as described in any one of the above, or implements a cross-modal hash retrieval method as described in any one of the above.

[0017] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the computer program implements a training method for a cross-modal hash retrieval model as described in any one of the above, or implements a cross-modal hash retrieval method as described in any one of the above.

[0018] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements a training method for a cross-modal hash retrieval model as described in any one of the above, or implements a cross-modal hash retrieval method as described in any one of the above.

[0019] The training method, cross-modal hash retrieval method, device and equipment of the cross-modal hash retrieval model provided by the present invention obtain at least one training sample pair, which includes sample image data and sample diagnosis text, determines the first type of hash real-valued features corresponding to each sample image data, and determines the second type of hash real-valued features corresponding to each sample diagnosis text, based on each reference diagnosis label in the knowledge graph, determines the adjacency matrix between diagnostic labels in the diagnostic label set corresponding to all training sample pairs, for each training sample pair, based on the adjacency matrix and the training sample pair, determines the third type of hash real-valued features corresponding to the training sample pair, and based on each first type of hash real-valued features, each second type of hash real-valued features, and each third type of hash real-valued features, trains the initial cross-modal hash retrieval model to obtain a cross-modal hash retrieval model. Since the knowledge graph can characterize the association between reference diagnostic labels, the adjacency matrix determined based on the knowledge graph is added to the training process of the cross-modal hash retrieval model, so that the association between different diagnostic labels is considered during model training. Therefore, the medical significance contained in the association can be added to the model, which increases the information content of the hash code and improves the accuracy of the cross-modal hash retrieval model. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1 A flowchart of a method for training a cross-modal hash retrieval model provided in an embodiment of the present invention.

[0022] Figure 2 A schematic diagram of a label encoder provided in accordance with an embodiment of the present invention.

[0023] Figure 3 A flowchart of a method for training a cross-modal hash retrieval model provided in an embodiment of the present invention.

[0024] Figure 4 A schematic diagram of the flow of a cross-modal hash retrieval method provided in an embodiment of the present invention.

[0025] Figure 5 A schematic diagram of the structure of a training device for a cross-modal hash retrieval model provided in an embodiment of the present invention.

[0026] Figure 6 A schematic diagram of the structure of a cross-modal hash retrieval device provided in an embodiment of the present invention.

[0027] Figure 7 A schematic diagram of the physical structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0029] With the continuous diversification of medical testing methods and the continuous progress of medical information technology, the accumulation of multimodal patient data is becoming increasingly abundant, including imaging data and diagnostic texts. Cross-modal retrieval for multimodal patient data has become a research hotspot in academia and industry in recent years. Cross-modal retrieval technology can help doctors retrieve semantically similar multimodal case information as a reference, so that doctors can more comprehensively evaluate the patient's current condition and make clinical diagnosis and treatment decisions accurately and efficiently. It has important research significance and clinical application value. Existing cross-modal retrieval methods, such as subspace learning models, topic models, and deep learning models, usually map data features of different modalities to the same common real-valued subspace, and use metrics such as Euclidean distance and cosine distance to calculate the semantic similarity between different modal data. However, with the rapid expansion of the scale of multimodal patient data, real-valued methods have gradually become difficult to meet the requirements of low storage and high efficiency retrieval, so hashing methods have gradually become a popular paradigm in the field of cross-modal retrieval of patient data.

[0030] Cross-modal hashing methods can use hash functions to map high-dimensional original features of multimodal data to low-dimensional Hamming space and represent them in the form of binary compressed hash codes. Therefore, the correlation between instances can be obtained through simple XOR calculations, which significantly improves retrieval efficiency and reduces storage costs. Cross-modal hashing methods are mainly divided into two categories: unsupervised methods and supervised methods. Unsupervised methods do not require explicit labels and learn hash functions by retaining the structural information of the original data, but the retrieval accuracy is not high in practical applications. Supervised methods use the supervised information of labels to maintain the semantic similarity between cross-modal data during the hash coding process, which has higher retrieval accuracy and is also used more in practical tasks.

[0031] With the rise of deep learning models, deep cross-modal hashing has become the most popular supervised cross-modal hashing method and has been applied to cross-modal retrieval tasks of patient data. These methods focus on cross-modal retrieval between patient medical images and diagnostic reports, taking the diagnoses and symptoms annotated by doctors as multi-labels, and determining their pairwise similarities based on whether patient instances in different modalities have the same labels, thereby supervising the model training process. However, this multi-label utilization method only utilizes the semantic information contained in the medical data labels themselves, and fails to fully utilize the associations between multiple labels. Since the associations between medical data labels usually have important medical significance, ignoring the associations between labels will reduce the similarity discrimination ability of hash encoding, resulting in low accuracy of the trained cross-modal hash retrieval model.

[0032] In response to the above problems, an embodiment of the present invention provides a training method for a cross-modal hash retrieval model. In the method, a knowledge graph is provided, wherein the knowledge graph includes at least two nodes and edges connecting the nodes, wherein the nodes are used to represent reference diagnostic labels, and the edges are used to represent the association relationship between the corresponding two reference diagnostic labels. Through the knowledge graph, the adjacency matrix between the diagnostic labels in the diagnostic label set corresponding to all training samples can be determined, and the element values ​​in the adjacency matrix can represent whether there is an association between the corresponding two diagnostic labels. The adjacency matrix is ​​added to the training process of the cross-modal hash retrieval model, so that the association relationship between different diagnostic labels is taken into account during model training. Therefore, the medical significance contained in the association relationship can be added to the model, thereby increasing the information content of the hash code, thereby improving the accuracy of the cross-modal hash retrieval model.

[0033] Combine the following Figures 1 to 3 The training method of the cross-modal hash retrieval model provided by an embodiment of the present invention is described. The executor of the method can be an electronic device such as a terminal device, a computer or a server, or a smart device specially designed for training a cross-modal hash retrieval model, or a training device for a cross-modal hash retrieval model arranged in the electronic device or smart device, and the training device for the cross-modal hash retrieval model can be implemented by software, hardware or a combination of both. The method can be applied in the medical field to retrieve semantically similar multimodal cases through cross-model retrieval.

[0034] Figure 1 A flow chart of a method for training a cross-modal hash retrieval model provided by an embodiment of the present invention is shown in FIG. Figure 1 As shown, the method includes: Step 101: Obtain at least one training sample pair, where the training sample pair includes sample image data and sample diagnosis text.

[0035] In this step, the sample image data includes X-rays, computed tomography (CT) scans, and magnetic resonance imaging (MRI) images. The sample diagnosis text includes the test results and diagnosis information corresponding to the sample image data. Among them, a training sample pair can correspond to a patient instance, that is, the training sample pair includes the sample image data and sample diagnosis text corresponding to the patient instance.

[0036] Step 102: Determine the first type of hash real-valued features corresponding to each sample image data, and determine the second type of hash real-valued features corresponding to each sample diagnosis text.

[0037] In this step, for the sample image data in each training sample pair, the sample image data can be input into the image encoder of the cross-modal hash retrieval model, so that features can be extracted from the original sample image data to obtain the first type of hash real-valued features output by the image encoder. Specifically, the first type of hash real-valued features can be determined by the following formula (1): (1) in, represents the image encoder, represents the trainable parameters of the image encoder, represents sample image data, represents the matrix composed of all first-class hashed real-valued features, It represents the first-class hash real-valued feature of the training sample pair corresponding to the i-th patient instance, N represents the total number of patient instances, that is, the total number of training sample pairs, and K represents the hash code length.

[0038] In addition, the second type of hash real-valued features of the sample diagnostic text corresponding to each sample image data can also be determined. Specifically, in order to extract features from the sample diagnostic text, the frequency of occurrence of each word in the sample diagnostic text of all training sample pairs can be counted first, and the frequency of occurrence is sorted in order from high to low, and the first preset number of target words are selected. Of course, target words with an occurrence frequency greater than a preset threshold can also be selected. Furthermore, for each sample diagnostic text, by determining whether the sample diagnostic text contains these words, combined with word embedding technology, the sample diagnostic text can be converted into a vector of fixed dimension as the original text data. For example, suppose that 1000 target words with high frequency of occurrence are selected from all sample diagnostic texts, and the original text data corresponding to each sample diagnostic text is set is a 1000-dimensional vector, in which an element value corresponds to a target word. For a sample diagnostic text, if the sample diagnostic text contains a target word, the original text data corresponding to the sample diagnostic text The element value corresponding to the target word in is 1. If the sample diagnostic text does not contain a target word, the original text data corresponding to the sample diagnostic text is The element value corresponding to the target word in is 0.

[0039] After determining the original text data corresponding to each sample diagnostic text After that, Transformer can be used to extract features, such as outputting the second-class hash real-valued features corresponding to the sample diagnostic text through a text encoder. Specifically, the second-class hash real-valued features can be determined by the following formula (2): (2) in, represents a text encoder, represents the trainable parameters of the text encoder, Indicates a sample diagnostic text. represents the matrix composed of all second-class hashed real-valued features, It represents the second type of hash real-valued feature of the training sample pair corresponding to the i-th patient instance, N represents the total number of patient instances, that is, the total number of training sample pairs, and K represents the hash code length.

[0040] Step 103: Based on each reference diagnostic label in the knowledge graph, determine the adjacency matrix between diagnostic labels in the diagnostic label set corresponding to all training sample pairs.

[0041] Among them, the element values ​​in the adjacency matrix are used to represent whether there is a correlation between the corresponding two diagnostic labels. The knowledge graph includes at least two nodes and edges connecting the nodes. The nodes are used to represent the reference diagnostic labels, and the edges are used to represent the correlation relationship between the corresponding two reference diagnostic labels.

[0042] In this step, in order to effectively mine and utilize the medical associations between multiple labels of patient instances, in this embodiment, multi-label features can be extracted by integrating the label encoder of the knowledge graph. Among them, the knowledge graph is pre-established based on whether there is an association between multiple reference diagnostic labels and each reference diagnostic label. The nodes of the knowledge graph represent reference diagnostic labels, and the edges between the nodes represent the association between the reference diagnostic labels. If there is an association between two reference diagnostic labels, an edge will be established between the two reference diagnostic labels. In the medical field, the association between two reference diagnostic labels can be understood as the existence of similar symptoms or signs between the two reference diagnostic labels, or the same cause of the two reference diagnostic labels, or the relationship between the two reference diagnostic labels, and so on.

[0043] Furthermore, each diagnostic label in the diagnostic label set corresponding to all training sample pairs is matched with the reference diagnostic label in the knowledge graph, and the reference diagnostic labels corresponding to these diagnostic labels are searched in the knowledge graph to find out whether there is an edge, that is, whether there is an association relationship, so as to obtain the adjacency matrix between the diagnostic labels in the diagnostic label set. , where C represents the total number of diagnostic labels contained in the diagnostic label set. If diagnostic label i and diagnostic label j are connected in the knowledge graph, the element value in the corresponding adjacency matrix is is 1, otherwise, is 0. Therefore, the determined value of each element in the adjacency matrix A can be used to characterize whether the corresponding two diagnostic labels are connected in the knowledge graph, and can also be understood as characterizing whether the corresponding two diagnostic labels are associated. If the value is 1, it means that the corresponding two diagnostic labels are associated, and if the value is 0, it means that there is no association between the corresponding two diagnostic labels. In addition, whether the above two diagnostic labels are connected in the knowledge graph includes whether they are directly connected or indirectly connected. If there is an edge between node a and node b, and there is an edge between node b and node c, then node a and node c are indirectly connected.

[0044] Step 104: for each training sample pair, based on the adjacency matrix and the training sample pair, determine the third type of hash real-valued feature corresponding to the training sample pair.

[0045] In this step, the class hash real-valued features corresponding to each diagnostic label can be determined based on the adjacency matrix and the initial feature matrix corresponding to each diagnostic label. Since the training sample pairs corresponding to the same patient instance may correspond to multiple diagnostic labels, it is also necessary to determine the third class hash real-valued features corresponding to each training sample pair based on the class hash real-valued features corresponding to each diagnostic label, that is, the third class hash real-valued features corresponding to each patient instance.

[0046] Step 105: Based on each first-category hash real-valued feature, each second-category hash real-valued feature, and each third-category hash real-valued feature, an initial cross-modal hash retrieval model is trained to obtain a cross-modal hash retrieval model.

[0047] In this step, after obtaining each first-class hash real-valued feature, each second-class hash real-valued feature, and each third-class hash real-valued feature, the loss information between different modalities and the loss information between the same modalities can be constructed based on these class hash real-valued features, so as to iteratively train the initial cross-modal hash retrieval model based on these loss information to adjust the model parameters of each encoder in the initial cross-modal hash retrieval model. The above process is repeated continuously until the obtained model converges or the number of repetitions exceeds the preset number of times, and the final model is determined as the cross-modal hash retrieval model.

[0048] The training method of the cross-modal hash retrieval model provided in the present embodiment obtains at least one training sample pair, the training sample pair includes sample image data and sample diagnosis text, determines the first type of hash real-valued features corresponding to each sample image data, and determines the second type of hash real-valued features corresponding to each sample diagnosis text, based on each reference diagnosis label in the knowledge graph, determines the adjacency matrix between the diagnostic labels in the diagnostic label set corresponding to all training sample pairs, for each training sample pair, based on the adjacency matrix and the training sample pair, determines the third type of hash real-valued features corresponding to the training sample pair, and trains the initial cross-modal hash retrieval model based on each first type of hash real-valued features, each second type of hash real-valued features, and each third type of hash real-valued features to obtain a cross-modal hash retrieval model. Since the knowledge graph can characterize the association relationship between the reference diagnosis labels, the adjacency matrix determined based on the knowledge graph is added to the training process of the cross-modal hash retrieval model, so that the association relationship between different diagnostic labels is considered during model training, and therefore, the medical significance contained in the association relationship can be added to the model, increasing the information content of the hash code and improving the accuracy of the cross-modal hash retrieval model.

[0049] Exemplarily, based on the above embodiment, when determining the third type of hash real-valued features corresponding to the training sample pairs based on the adjacency matrix and the training sample pairs, it can be performed in the following manner: Based on the diagnostic label set and at least one target diagnostic label corresponding to the training sample pair, an initial multi-label vector of the training sample pair is determined, the element value in the initial multi-label vector is used to characterize whether the diagnostic label set contains the target diagnostic label, and the initial features corresponding to each target diagnostic label are obtained. For each target diagnostic label, based on the adjacency matrix, the initial multi-label vector and the initial features corresponding to the target diagnostic label, the class hash feature corresponding to the target diagnostic label is determined. Based on the class hash features corresponding to each target diagnostic label, the third class hash real-valued feature corresponding to the training sample pair is determined.

[0050] Specifically, Figure 2 A schematic diagram of a tag encoder provided by an embodiment of the present invention, such as Figure 2 As shown in , for a training sample pair of a patient instance, the initial multi-label vector of the training sample pair can be determined based on whether each target diagnostic label corresponding to the training sample pair is included in the diagnostic label set corresponding to all training sample pairs. Figure 2In , assuming that the diagnostic label set corresponding to all training sample pairs includes c diagnostic labels, the initial multi-label vector of each training sample pair can be set to a c-dimensional vector. For the training sample pair corresponding to patient instance 1, if it includes diagnostic label 1, diagnostic label 4 and diagnostic label c in the diagnostic label set, the initial multi-label vector of the training sample pair corresponding to patient instance 1 can be set to [1, 0, 0, 1, ..., 1]. For the training sample pair corresponding to patient instance 2, if it includes diagnostic label 1, diagnostic label 2, diagnostic label 3 and diagnostic label c in the diagnostic label set, the initial multi-label vector of the training sample pair corresponding to patient instance 2 can be set to [1, 0, 0, 1, ..., 1] and so on, so that the initial multi-label vector matrix can be determined. ,in, represents the initial multi-label vector of the i-th training sample pair, and N represents the total number of training sample pairs. It can be seen that the element value in the initial multi-label vector can represent whether the diagnostic label set contains the target diagnostic label corresponding to the training sample pair, and the initial multi-label vector is in the form of a multi-hot vector.

[0051] In addition, the initial features of each target diagnostic label corresponding to the training sample pair can be obtained. The initial features of all target diagnostic labels can form an initial feature matrix , the initial feature matrix is a The identity matrix of , that is, the initial features of each target diagnostic label are regarded as a one-hot vector.

[0052] The adjacency matrix A constructed based on the knowledge graph, the obtained initial multi-label vector matrix L, and the initial feature matrix Y are input into the label encoder to obtain the class hash feature corresponding to each target diagnostic label. The label encoder can be, for example, a graph attention network (GAT). The class hash feature corresponding to each target diagnostic label can form the class hash feature matrix of the diagnostic label. .

[0053] Furthermore, there are often cases where a training sample pair of a patient instance has multiple target diagnostic labels. Therefore, the weighted average of the multiple target diagnostic labels corresponding to the training sample pair can be performed based on the following formula (3) to obtain the third type of hash real-valued features corresponding to the training sample pair: (3) in, represents the third type of hash real-valued feature of the training sample pair corresponding to the i-th patient instance, C represents the total number of diagnostic labels in the diagnostic label set, represents the kth target diagnostic label of the training sample pair corresponding to the i-th patient instance. If the kth target diagnostic label is included in the diagnostic label set, then The value is 1 if yes, otherwise it is 0.

[0054] In summary, the extraction process of the third type of hash real-valued features can be expressed by the following formula (4): (4) in, represents a tag encoder, represents the trainable parameters of the label encoder, represents the matrix composed of all third-class hashed real-valued features, It represents the third type of hash real-valued feature corresponding to the training sample pair corresponding to the i-th patient instance, N represents the total number of patient instances, that is, the total number of training sample pairs, and K represents the hash code length.

[0055] In this embodiment, the initial multi-label vector of the training sample pair is determined based on the diagnostic label set and at least one target diagnostic label corresponding to the training sample pair, so that the relationship between the target diagnostic labels and the diagnostic label sets of different training sample pairs can be established. When the class hash feature corresponding to the target diagnostic label is determined based on the adjacency matrix, the initial multi-label vector and the initial features corresponding to the target diagnostic label, the association between different diagnostic labels is also maintained in the third class hash feature finally obtained, thereby increasing the information in the third class hash feature. Subsequently, when the cross-modal hash retrieval model is trained based on the third class hash feature, the accuracy and robustness of the cross-modal hash retrieval model can be improved.

[0056] Exemplarily, based on the above embodiments, when the initial cross-modal hash retrieval model is trained based on each first-class hash real-valued feature, each second-class hash real-valued feature, and each third-class hash real-valued feature to obtain the cross-modal hash retrieval model, it can be performed in the following manner: For each training sample pair, based on the first type of hash real-valued features, the second type of hash real-valued features and the third type of hash real-valued features corresponding to the training sample pair, and the cross-modal attention weights between the modal data corresponding to the target class hash real-valued features and the modal data corresponding to other types of hash real-valued features, an updated target class hash real-valued feature is determined, and the target class hash real-valued feature includes at least one of the following: the first type of hash real-valued features, the second type of hash real-valued features and the third type of hash real-valued features; based on the updated target class hash real-valued features, an initial cross-modal hash retrieval model is trained to obtain a cross-modal hash retrieval model.

[0057] Specifically, in this embodiment, the hash-like real-valued features of the image data, diagnostic text, and multiple diagnostic labels can be fused through the attention mechanism, which not only promotes feature alignment but also transfers the semantic information contained in the data of each modality to each other, thereby enriching the amount of information contained in the hash code.

[0058] When the target class hash real-valued feature is the first class hash real-valued feature, the first class hash real-valued feature of the training sample pair corresponding to the updated i-th patient instance can be determined based on the following formula (5): : (5) in, represents the first cross-modal attention weight between the sample image data and the target diagnosis label in the training sample pair of the i-th patient instance, Represents the second cross-modal attention weight between the sample image data and the sample diagnosis text in the training sample pair of the i-th patient instance.

[0059] After determining the updated first-class hash real-valued features corresponding to each training sample pair After that, we can get the updated first-class hash real-valued feature matrix .

[0060] In addition, when the target class hash real-valued feature is the second class hash real-valued feature, the second class hash real-valued feature of the training sample pair corresponding to the updated i-th patient instance can be determined based on the following formula (6): : (6) in, represents the third cross-modal attention weight between the sample diagnosis text and the target diagnosis label in the i-th training sample pair.

[0061] After determining the updated second type of hash real-valued features corresponding to each training sample pair After that, we can get the updated second-class hash real-valued feature matrix .

[0062] When the target class hash real-valued feature is the third class hash real-valued feature, the third class hash real-valued feature of the training sample pair corresponding to the updated i-th patient instance can be determined based on the following formula (7): : (7) After determining the updated third-class hash real-valued feature corresponding to each training sample pair After that, we can get the updated third-class hash real-valued feature matrix .

[0063] Furthermore, the initial cross-modal hash retrieval model can be trained based on the determined updated first-category hash real-valued feature matrix, the updated second-category hash real-valued feature matrix, and the updated third-category hash real-valued features to obtain a cross-modal hash retrieval model.

[0064] In this embodiment, at least one of the first type of hash real-valued features, the second type of hash real-valued features, and the third type of hash real-valued features are updated respectively through the cross-modal attention weights between different modalities, so that the image data, diagnostic text, and the respective hash real-valued features of multiple diagnostic labels of different modalities can be fused, which can promote feature alignment and reduce the distribution differences between features of different modalities. It can also realize the mutual transmission of semantic information contained in the data of each modality, thereby enriching the information content of the updated first type of hash real-valued features, the updated second type of hash real-valued features, and the updated third type of hash real-valued features, thereby improving the robustness of the cross-modal hash retrieval model finally trained.

[0065] Exemplarily, based on the above embodiments, the cross-modal attention weight can be determined in the following manner: when the target class hash real-valued features include first-class hash real-valued features, for each training sample pair, based on the first-class hash real-valued features and the third-class hash real-valued features corresponding to the training sample pair, the cross-modal attention weight between the sample image data and the corresponding target diagnostic label in the training sample pair is determined, and based on the first-class hash real-valued features and the second-class hash real-valued features corresponding to the training sample pair, the cross-modal attention weight between the sample image data and the corresponding sample diagnostic text in the training sample pair is determined; when the target class hash real-valued features include second-class hash real-valued features, for each training sample pair, based on the second-class hash real-valued features and the third-class hash real-valued features corresponding to the training sample pair, the cross-modal attention weight between the sample diagnostic text and the corresponding target diagnostic label in the training sample pair is determined.

[0066] Specifically, for the training sample pair of the i-th patient instance, the cross-modal attention weights between the sample image data and the corresponding target diagnosis label can be determined based on the following formula (8): , the cross-modal attention weights between sample image data and corresponding sample diagnostic text and the cross-modal attention weights between the sample diagnosis text and the corresponding target diagnosis label : (8) In this embodiment, by adopting hash-like real-valued features of different modalities, the cross-modal attention weights between different modalities are determined respectively. In this way, when different hash-like real-valued features are fused based on the cross-modal attention weights between different modalities, the contribution between different modalities can be dynamically adjusted, thereby improving the accuracy and flexibility of feature fusion.

[0067] Exemplarily, on the basis of the above embodiments, when the initial cross-modal hash retrieval model is trained based on the updated target class hash real-valued features to obtain the cross-modal hash retrieval model, since in the patient instances with the same diagnostic label, even if the modalities of the data are different, the class hash real-valued features should maintain semantic similarity, therefore, it is necessary to consider the loss information between different modalities. In addition, the similarity between the same modality data should also be maintained, therefore, it is also necessary to consider the loss information between the same modalities. Among them, when the target class hash real-valued features include the first class hash real-valued features, the second class hash real-valued features and the third class hash real-valued features, the inter-modal loss and the intra-modal loss can be determined based on each updated first class hash real-valued feature, each updated second class hash real-valued feature and each updated third class hash real-valued feature, respectively, and based on the inter-modal loss and the intra-modal loss, the initial cross-modal hash retrieval model is trained to obtain the cross-modal hash retrieval model.

[0068] Specifically, the first inter-modal sub-loss can be determined based on each updated first-class hash real-valued feature and each updated second-class hash real-valued feature, the second inter-modal sub-loss can be determined based on each updated first-class hash real-valued feature and each updated third-class hash real-valued feature, and the third inter-modal sub-loss can be determined based on each updated second-class hash real-valued feature and each updated third-class hash real-valued feature; the inter-modal loss is determined based on the first inter-modal sub-loss, the second inter-modal sub-loss, and the third inter-modal sub-loss; the first intra-modal sub-loss is determined based on each updated first-class hash real-valued feature, the second intra-modal sub-loss is determined based on each updated second-class hash real-valued feature, and the third intra-modal sub-loss is determined based on each updated third-class hash real-valued feature; the intra-modal loss is determined based on the first intra-modal sub-loss, the second intra-modal sub-loss, and the third intra-modal sub-loss, and the initial cross-modal hash retrieval model is trained based on the inter-modal loss and the intra-modal loss to obtain a cross-modal hash retrieval model.

[0069] Since the hash code generated in the end is the result of binarization of the sign function, and the gradient of the sign function at non-zero values ​​is always 0, it will cause the gradient vanishing problem during training, so the hash code cannot be directly used as the input of the loss function. To address this problem, in this embodiment, the deep hashing method usually adopts a continuous relaxation strategy, using hash-like real-valued features instead of hash codes to input the loss function.

[0070] Therefore, the first inter-modal loss can be determined based on the following formula (9): : (9) Where S represents the pairwise similarity set of patient instances. If there is at least one identical diagnostic label in the training sample pairs corresponding to the i-th patient instance and the j-th patient instance, then is 1, otherwise, is 0.

[0071] The second modal inter-sub-loss can be determined based on the following formula (10): : (10) The third modal inter-sub-loss is determined based on the following formula (11): : (11) Therefore, the inter-modal loss can be determined based on the following formula (12): : (12) By determining the similarity loss function between different modalities , semantically similar data of different modalities will be closer in the feature space, and dissimilar data will be pushed away in the feature space, which can effectively promote the feature alignment between the modal data. At the same time, passing the label association information based on the knowledge graph fused by the diagnostic label encoder to the hash coding training process of the image data and diagnostic text can greatly enrich the semantic information of the hash coding.

[0072] Furthermore, the first mode internal loss can be determined based on the following formula (13): : (13) The second mode internal loss is determined based on the following formula (14): : (14) The third mode internal loss is determined based on the following formula (15): : (15) Therefore, the intra-modal loss can be determined based on the following formula (16): : (16) By determining the similarity loss function between the same modalities , which can achieve the purpose of maintaining the similarity between the same modality data.

[0073] After determining the inter-modal loss and the intra-modal loss, these two losses can be used together as a loss function to train the initial cross-modal hash retrieval model to obtain the final cross-modal hash retrieval model.

[0074] In this embodiment, by determining the inter-modal loss, semantically similar data of different modalities will be brought closer in the feature space, and dissimilar data will be pushed further away in the feature space, thereby effectively promoting feature alignment between modal data. By determining the intra-modal loss, the similarity between the same modal data can also be maintained. When the initial cross-modal hash retrieval model is trained by the inter-modal loss and the intra-modal loss, the accuracy of the final cross-modal hash retrieval model can be improved.

[0075] Exemplarily, on the basis of the above embodiments, when the initial cross-modal hash retrieval model is trained based on the inter-modal loss and the intra-modal loss to obtain the cross-modal hash retrieval model, the first type of hash real-valued features corresponding to the training sample pairs can be binarized to obtain the first hash code, and the second type of hash real-valued features corresponding to the training sample pairs can be binarized to obtain the second hash code, and the quantization loss is determined based on the updated first type of hash real-valued features corresponding to the training sample pairs, the first hash code, the updated second type of hash real-valued features corresponding to the training sample pairs, and the second hash code, and the initial cross-modal hash retrieval model is trained based on the inter-modal loss, the intra-modal loss and the quantization loss to obtain the cross-modal hash retrieval model.

[0076] Specifically, the symbol function can be used to binarize the first type of hash real-valued features corresponding to each training sample pair, so as to obtain the first hash code corresponding to the image data mode. ,in, , Represents the first hash code of the training sample pair corresponding to the i-th patient instance.

[0077] In addition, the symbol function can also be used to binarize the second type of hash real-valued features corresponding to each training sample pair, so as to obtain the second hash code corresponding to the diagnostic text mode. ,in, , Represents the second hash code of the training sample pair corresponding to the i-th patient instance.

[0078] In order to reduce the information loss caused by the binary conversion of the hash-like real-valued features, in this embodiment, the quantization loss can also be determined based on formula (17): : (17) Furthermore, the final similarity loss function can be determined based on formula (18): : (18) in, and It is an adjustable parameter.

[0079] When training the model, the final similarity loss function can be determined based on , iteratively adjust the model parameters in the initial cross-modal hash retrieval model until the number of iterations reaches a preset number or the final model converges, and determine the final model as the cross-modal hash retrieval model.

[0080] In this embodiment, the quantization loss determined by the updated first-class hash real-valued features, the first hash codes, the updated second-class hash real-valued features, and the second hash codes corresponding to each training sample pair can reduce the information loss caused by binarization of the class hash real-valued features, thereby improving the accuracy of the cross-modal hash retrieval model after the final training.

[0081] Exemplarily, based on the above embodiments, the first type of hash real-valued features are obtained by inputting the sample image data corresponding to the training sample pairs into the image encoder of the initial cross-modal hash retrieval model, the second type of hash real-valued features are obtained by inputting the sample diagnostic text corresponding to the training sample pairs into the text encoder of the initial cross-modal hash retrieval model, and the third type of hash real-valued features are obtained by inputting the adjacency matrix and the initial multi-label vector of the training sample pairs into the text label encoder of the initial cross-modal hash retrieval model, and the original text data is determined based on the vocabulary that appears more than a preset number of times in the sample diagnostic texts corresponding to all training sample pairs.

[0082] An initial cross-modal hash retrieval model is trained based on each first-class hash real-valued feature, each second-class hash real-valued feature, and each third-class hash real-valued feature. When the cross-modal hash retrieval model is obtained, the network parameters of the image encoder, the network parameters of the text encoder, and the network parameters of the label encoder are iteratively adjusted based on each first-class hash real-valued feature, each second-class hash real-valued feature, and each third-class hash real-valued feature to obtain the cross-modal hash retrieval model.

[0083] Specifically, Figure 3 A flowchart of a method for training a cross-modal hash retrieval model provided by an embodiment of the present invention is shown in FIG. Figure 3As shown, the initial cross-modal hash retrieval model includes an image encoder, a text encoder, and a label encoder. The sample image data in the training sample pair corresponding to the patient instance is input into the image encoder to obtain the first type of hash real-valued features output by the image encoder. The sample image data may include, for example, medical images of the patient, and the image encoder may include, for example, a CNN.

[0084] The sample diagnosis text in the training sample pair corresponding to the patient instance is input into the text encoder to obtain the second type of hash real-valued features output by the text encoder, wherein the sample diagnosis text may include, for example, the patient's diagnosis report, and the text encoder may include, for example, a Transformer.

[0085] In addition, by inputting the adjacency matrix determined based on the knowledge graph and the initial multi-label vector of the training sample pair into the label encoder, the third type of hash real-valued features output by the label encoder can be obtained, wherein the knowledge graph can be, for example, a medical knowledge graph, and the label encoder can include GAT.

[0086] When training the model, a cross-modal attention module is introduced. Through the attention mechanism, the features of image data, diagnostic text and multi-label are integrated. In addition, the intra-modal and inter-modal correlation complementary similarity loss functions are introduced. For example, the inter-modal loss between medical image features and multi-label embedding features and text embedding features, as well as the inter-modal loss between multi-label embedding features and text embedding features can be determined, where the medical image features are the first type of hash real-valued features, the text embedding features are the second type of hash real-valued features, and the multi-label embedding features are the third type of hash real-valued features. It is also necessary to determine the intra-modal loss of medical image features, multi-label embedding features and text embedding features respectively. In addition, in order to reduce the information loss generated when the hash-like real-valued features are binarized, a quantization loss is added. The quantization loss can be determined based on the first type of hash real-valued features and the first hash code quantized by the symbolic function, and the first type of hash real-valued features and the second hash code quantized by the symbolic function.

[0087] Therefore, based on the inter-modal loss, intra-modal loss and quantization loss, the network parameters of the image encoder, the network parameters of the text encoder and the network parameters of the label encoder can be iteratively adjusted until the final cross-modal hash retrieval model is obtained.

[0088] The cross-modal hash retrieval model obtained through the above model training process takes into account inter-modal loss, intra-modal loss and quantization loss, which not only brings semantically similar data of different modalities closer in feature space and pushes dissimilar data further apart in feature space, but also reduces the loss caused by binarization, making the final cross-modal hash retrieval model more robust and accurate. In addition, the above training method can strengthen the collaborative modeling of data of each modality during the training process, help reduce the distribution differences between features, thereby narrowing the semantic gap between different modalities and improving the similarity discrimination ability of hash coding.

[0089] Figure 4 A schematic diagram of a cross-modal hash retrieval method provided by an embodiment of the present invention is shown in FIG. Figure 4 As shown, the method includes: Step 401: Obtain the data to be retrieved in the first mode.

[0090] Step 402: input the to-be-retrieved data of the first modality into the cross-modal hash retrieval model to obtain the target data of the second modality output by the cross-modal hash retrieval model.

[0091] The cross-modal hash retrieval model is trained based on the method described in any of the above embodiments.

[0092] Among them, when the data to be retrieved in the first modality includes image data, the target data in the second modality includes diagnostic text related to the image data; when the data to be retrieved in the first modality includes diagnostic text, the target data in the second modality includes image data related to the diagnostic text.

[0093] Specifically, if the data to be retrieved in the first modality includes image data, the cross-modal hash retrieval model will analyze the image data and output diagnostic text related to the image data. If the data to be retrieved in the first modality includes diagnostic text, the cross-modal hash retrieval model will analyze the diagnostic text and output image data related to the diagnostic text.

[0094] The cross-modal hash retrieval method provided by the embodiment of the present invention obtains the target data of the second modality output by the cross-modal hash retrieval model by acquiring the data to be retrieved of the first modality and inputting the data to be retrieved of the first modality into the cross-modal hash retrieval model. When training the cross-modal hash retrieval model, since the adjacency matrix is ​​added to the training process of the cross-modal hash retrieval model, the association relationship between different diagnostic labels is taken into account. Therefore, the medical significance contained in the association relationship can be added to the model, which increases the information content of the hash code and improves the accuracy of the cross-modal hash retrieval model. When performing cross-modal retrieval based on the cross-modal hash retrieval model, the accuracy of the retrieval can be improved.

[0095] The training device for the cross-modal hash retrieval model provided by the present invention is described below. The training device for the cross-modal hash retrieval model described below and the training method for the cross-modal hash retrieval model described above can refer to each other.

[0096] Figure 5 A schematic diagram of the structure of a training device for a cross-modal hash retrieval model provided by an embodiment of the present invention, referring to Figure 5 As shown, the training device 500 of the cross-modal hash retrieval model includes: An acquisition module 11 is used to acquire at least one training sample pair, wherein the training sample pair includes sample image data and sample diagnosis text; A determination module 12, used to determine the first type of hash real-valued features corresponding to each of the sample image data, and to determine the second type of hash real-valued features corresponding to each of the sample diagnosis texts; The determination module 12 is further used to determine the adjacency matrix between the diagnostic labels in the diagnostic label set corresponding to all the training sample pairs based on the reference diagnostic labels in the knowledge graph; The determination module 12 is further configured to determine, for each of the training sample pairs, a third type of hash real-valued feature corresponding to the training sample pair based on the adjacency matrix and the training sample pair; The training module 13 is also used to train the initial cross-modal hash retrieval model based on each of the first-category hash real-valued features, each of the second-category hash real-valued features, and each of the third-category hash real-valued features to obtain a cross-modal hash retrieval model.

[0097] In an exemplary embodiment, the determining module 12 is specifically configured to: Based on the diagnostic label set and at least one target diagnostic label corresponding to the training sample pair, determining an initial multi-label vector of the training sample pair, wherein the element value in the initial multi-label vector is used to indicate whether the diagnostic label set includes the target diagnostic label; Obtaining initial features corresponding to each of the target diagnostic labels; For each of the target diagnostic labels, based on the adjacency matrix, the initial multi-label vector and the initial features corresponding to the target diagnostic label, determine the class hash feature corresponding to the target diagnostic label; Based on the class hash features corresponding to each of the target diagnostic labels, a third class hash real-valued feature corresponding to the training sample pair is determined.

[0098] In an exemplary embodiment, the training module 13 is specifically configured to: For each of the training sample pairs, based on the first type of hash real-valued features, the second type of hash real-valued features, and the third type of hash real-valued features corresponding to the training sample pairs, The cross-modal attention weight between the modal data corresponding to the target class hash real-valued feature and the modal data corresponding to the other class hash real-valued features is used to determine the updated target class hash real-valued feature; the target class hash real-valued feature includes at least one of the following: the first class hash real-valued feature, the second class hash real-valued feature and the third class hash real-valued feature; Based on the updated target class hash real-valued features, the initial cross-modal hash retrieval model is trained to obtain a cross-modal hash retrieval model.

[0099] In an exemplary embodiment, the determination module 12 is further used to determine, for each of the training sample pairs, a cross-modal attention weight between the sample image data and the corresponding target diagnosis label in the training sample pair based on the first class hash real-valued feature and the third class hash real-valued feature corresponding to the training sample pair, when the target class hash real-valued feature includes the first class hash real-valued feature, and determine a cross-modal attention weight between the sample image data and the corresponding sample diagnosis text in the training sample pair based on the first class hash real-valued feature and the second class hash real-valued feature corresponding to the training sample pair; The determination module 12 is also used to determine, for each of the training sample pairs, the cross-modal attention weight between the sample diagnostic text and the corresponding target diagnostic label in the training sample pair based on the second class hash real-valued features and the third class hash real-valued features corresponding to the training sample pair, when the target class hash real-valued features include the second class hash real-valued features.

[0100] In an exemplary embodiment, the target class hash real-valued features include the first class hash real-valued features, the second class hash real-valued features, and the third class hash real-valued features; The training module 13 is specifically used for: Based on each updated first-category hash real-valued feature, each updated second-category hash real-valued feature, and each updated third-category hash real-valued feature, respectively determine an inter-modality loss and an intra-modality loss; Based on the inter-modality loss and the intra-modality loss, the initial cross-modal hash retrieval model is trained to obtain the cross-modal hash retrieval model.

[0101] In an exemplary embodiment, the training module 13 is specifically configured to: Binarizing the first type of hash real-valued features corresponding to the training sample pair to obtain a first hash code, and binarizing the second type of hash real-valued features corresponding to the training sample pair to obtain a second hash code; Determine a quantization loss based on the updated first type of hash real-valued features corresponding to the training sample pair, the first hash code, the updated second type of hash real-valued features corresponding to the training sample pair, and the second hash code; Based on the inter-modality loss, the intra-modality loss and the quantization loss, the initial cross-modal hash retrieval model is trained to obtain the cross-modal hash retrieval model.

[0102] In an exemplary embodiment, the training module 13 is specifically configured to: Based on each of the first-category hash real-valued features, each of the second-category hash real-valued features, and each of the third-category hash real-valued features, the network parameters of the image encoder, the network parameters of the text encoder, and the network parameters of the label encoder in the initial cross-modal hash retrieval model are iteratively adjusted to obtain the cross-modal hash retrieval model.

[0103] The device of this embodiment can be used to execute the method of any embodiment in the training method side embodiment of the cross-modal hash retrieval model. Its specific implementation process and technical effects are similar to those in the training method side embodiment of the cross-modal hash retrieval model. For details, please refer to the detailed introduction in the training method side embodiment of the cross-modal hash retrieval model, which will not be repeated here.

[0104] Figure 6 A schematic diagram of the structure of a cross-modal hash retrieval device provided by an embodiment of the present invention, referring to Figure 6 As shown, the cross-modal hash retrieval device 600 includes: An acquisition module 21 is used to acquire the data to be retrieved in the first mode; An input module 22, used for inputting the to-be-retrieved data of the first modality into a cross-modal hash retrieval model to obtain target data of the second modality output by the cross-modal hash retrieval model; the cross-modal hash retrieval model is trained based on any of the above methods; Among them, when the data to be retrieved in the first modality includes image data, the target data in the second modality includes diagnostic text related to the image data; when the data to be retrieved in the first modality includes diagnostic text, the target data in the second modality includes image data related to the diagnostic text.

[0105] The device of this embodiment can be used to execute the method of any embodiment in the cross-modal hash retrieval method side embodiment. Its specific implementation process and technical effects are similar to those in the cross-modal hash retrieval method side embodiment. For details, please refer to the detailed introduction in the cross-modal hash retrieval method side embodiment, which will not be repeated here.

[0106] Figure 7A schematic diagram of the physical structure of an electronic device provided by an embodiment of the present invention, such as Figure 7 As shown, the electronic device may include: a processor (processor) 710 , a communication interface (Communications Interface) 720 , a memory (memory) 730 and a communication bus 740 , wherein the processor 710 , the communication interface 720 , and the memory 730 communicate with each other through the communication bus 740 . The processor 710 can call the logic instructions in the memory 730 to execute a training method for a cross-modal hash retrieval model, which method includes: obtaining at least one training sample pair, the training sample pair including sample image data and sample diagnosis text; determining the first type of hash real-valued features corresponding to each of the sample image data, and determining the second type of hash real-valued features corresponding to each of the sample diagnosis texts; based on each reference diagnosis label in the knowledge graph, determining the adjacency matrix between the diagnostic labels in the diagnostic label set corresponding to all the training sample pairs; for each of the training sample pairs, based on the adjacency matrix and the training sample pair, determining the third type of hash real-valued features corresponding to the training sample pair; based on each of the first type of hash real-valued features, each of the second type of hash real-valued features, and each of the third type of hash real-valued features, training the initial cross-modal hash retrieval model to obtain a cross-modal hash retrieval model.

[0107] The processor 710 can call the logic instructions in the memory 730 to execute a cross-modal hash retrieval method, which includes: obtaining data to be retrieved in a first modality; inputting the data to be retrieved in the first modality into a cross-modal hash retrieval model to obtain target data of a second modality output by the cross-modal hash retrieval model; the cross-modal hash retrieval model is trained based on the method described in any of the above embodiments; wherein, when the data to be retrieved in the first modality includes image data, the target data of the second modality includes diagnostic text related to the image data; when the data to be retrieved in the first modality includes diagnostic text, the target data of the second modality includes image data related to the diagnostic text.

[0108] In addition, the logic instructions in the above-mentioned memory 730 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0109] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the training method of the cross-modal hash retrieval model provided by the above-mentioned methods, and the method includes: obtaining at least one training sample pair, wherein the training sample pair includes sample image data and sample diagnostic text; determining a first type of hash real-valued feature corresponding to each of the sample image data, and determining a second type of hash real-valued feature corresponding to each of the sample diagnostic texts; based on each reference diagnostic label in the knowledge graph, determining an adjacency matrix between diagnostic labels in a diagnostic label set corresponding to all of the training sample pairs; for each of the training sample pairs, determining a third type of hash real-valued feature corresponding to the training sample pair based on the adjacency matrix and the training sample pair; based on each of the first type of hash real-valued feature, each of the second type of hash real-valued feature, and each of the third type of hash real-valued feature, training an initial cross-modal hash retrieval model to obtain a cross-modal hash retrieval model.

[0110] When the computer program is executed by the processor, the computer can execute the cross-modal hash retrieval method provided by the above methods, which method includes: obtaining the data to be retrieved in the first modality; inputting the data to be retrieved in the first modality into the cross-modal hash retrieval model to obtain the target data of the second modality output by the cross-modal hash retrieval model; the cross-modal hash retrieval model is trained based on the method described in any of the above embodiments; wherein, when the data to be retrieved in the first modality includes image data, the target data of the second modality includes diagnostic text related to the image data; when the data to be retrieved in the first modality includes diagnostic text, the target data of the second modality includes image data related to the diagnostic text.

[0111] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the training method of the cross-modal hash retrieval model provided by the above-mentioned methods, the method comprising: obtaining at least one training sample pair, the training sample pair comprising sample image data and sample diagnosis text; determining a first type of hash real-valued feature corresponding to each of the sample image data, and determining a second type of hash real-valued feature corresponding to each of the sample diagnosis texts; based on each reference diagnosis label in the knowledge graph, determining an adjacency matrix between diagnostic labels in a diagnostic label set corresponding to all of the training sample pairs; for each of the training sample pairs, determining a third type of hash real-valued feature corresponding to the training sample pair based on the adjacency matrix and the training sample pair; based on each of the first type of hash real-valued feature, each of the second type of hash real-valued feature, and each of the third type of hash real-valued feature, training an initial cross-modal hash retrieval model to obtain a cross-modal hash retrieval model.

[0112] When the computer program is executed by the processor, it is implemented to execute the cross-modal hash retrieval method provided by the above methods, and the method includes: obtaining the data to be retrieved in the first modality; inputting the data to be retrieved in the first modality into the cross-modal hash retrieval model to obtain the target data of the second modality output by the cross-modal hash retrieval model; the cross-modal hash retrieval model is trained based on the method described in any of the above embodiments; wherein, when the data to be retrieved in the first modality includes image data, the target data of the second modality includes diagnostic text related to the image data; when the data to be retrieved in the first modality includes diagnostic text, the target data of the second modality includes image data related to the diagnostic text.

[0113] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0114] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0115] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A training method for a cross-modal hash retrieval model, characterized in that: include: Acquire at least one training sample pair, wherein the training sample pair includes sample image data and sample diagnosis text; Determine a first type of hash real-valued feature corresponding to each of the sample image data, and determine a second type of hash real-valued feature corresponding to each of the sample diagnosis texts; Based on each reference diagnostic label in the knowledge graph, determining an adjacency matrix between diagnostic labels in a diagnostic label set corresponding to all the training sample pairs; For each of the training sample pairs, based on the adjacency matrix and the training sample pair, determine a third type of hash real-valued feature corresponding to the training sample pair; Based on each of the first-category hash real-valued features, each of the second-category hash real-valued features, and each of the third-category hash real-valued features, an initial cross-modal hash retrieval model is trained to obtain a cross-modal hash retrieval model.

2. The training method of the cross-modal hash retrieval model according to claim 1 is characterized in that: The determining, based on the adjacency matrix and the training sample pairs, a third type of hash real-valued feature corresponding to the training sample pairs includes: Based on the diagnostic label set and at least one target diagnostic label corresponding to the training sample pair, determining an initial multi-label vector of the training sample pair, wherein the element value in the initial multi-label vector is used to indicate whether the diagnostic label set includes the target diagnostic label; Obtaining initial features corresponding to each of the target diagnostic labels; For each of the target diagnostic labels, based on the adjacency matrix, the initial multi-label vector and the initial features corresponding to the target diagnostic label, determine the class hash feature corresponding to the target diagnostic label; Based on the class hash features corresponding to each of the target diagnostic labels, a third class hash real-valued feature corresponding to the training sample pair is determined.

3. The training method of the cross-modal hash retrieval model according to claim 1 is characterized in that: The initial cross-modal hash retrieval model is trained based on each of the first-type hash real-valued features, each of the second-type hash real-valued features, and each of the third-type hash real-valued features to obtain a cross-modal hash retrieval model, including: For each of the training sample pairs, based on the first type of hash real-valued features, the second type of hash real-valued features and the third type of hash real-valued features corresponding to the training sample pairs, and the cross-modal attention weights between the modal data corresponding to the target class hash real-valued features and the modal data corresponding to the other types of hash real-valued features, an updated target class hash real-valued feature is determined; the target class hash real-valued feature includes at least one of the following: the first type of hash real-valued features, the second type of hash real-valued features and the third type of hash real-valued features; Based on the updated target class hash real-valued features, the initial cross-modal hash retrieval model is trained to obtain a cross-modal hash retrieval model.

4. The training method of the cross-modal hash retrieval model according to claim 3 is characterized in that: The method further comprises: In the case where the target class hash real-valued features include first-class hash real-valued features, for each of the training sample pairs, based on the first-class hash real-valued features and the third-class hash real-valued features corresponding to the training sample pairs, determine the cross-modal attention weights between the sample image data and the corresponding target diagnosis labels in the training sample pairs, and based on the first-class hash real-valued features and the second-class hash real-valued features corresponding to the training sample pairs, determine the cross-modal attention weights between the sample image data and the corresponding sample diagnosis texts in the training sample pairs; In the case where the target class hash real-valued features include second-class hash real-valued features, for each of the training sample pairs, based on the second-class hash real-valued features and the third-class hash real-valued features corresponding to the training sample pairs, the cross-modal attention weight between the sample diagnostic text and the corresponding target diagnostic label in the training sample pair is determined.

5. The training method of the cross-modal hash retrieval model according to claim 3 is characterized in that: The target class hash real-valued features include the first class hash real-valued features, the second class hash real-valued features and the third class hash real-valued features; The initial cross-modal hash retrieval model is trained based on the updated target class hash real-valued feature to obtain the cross-modal hash retrieval model, including: Based on each updated first-category hash real-valued feature, each updated second-category hash real-valued feature, and each updated third-category hash real-valued feature, respectively determine an inter-modality loss and an intra-modality loss; Based on the inter-modality loss and the intra-modality loss, the initial cross-modal hash retrieval model is trained to obtain the cross-modal hash retrieval model.

6. The training method of the cross-modal hash retrieval model according to claim 5, characterized in that: The initial cross-modal hash retrieval model is trained based on the inter-modal loss and the intra-modal loss to obtain a cross-modal hash retrieval model, including: Binarizing the first type of hash real-valued features corresponding to the training sample pair to obtain a first hash code, and binarizing the second type of hash real-valued features corresponding to the training sample pair to obtain a second hash code; Determine a quantization loss based on the updated first type of hash real-valued features corresponding to the training sample pair, the first hash code, the updated second type of hash real-valued features corresponding to the training sample pair, and the second hash code; Based on the inter-modality loss, the intra-modality loss and the quantization loss, the initial cross-modal hash retrieval model is trained to obtain the cross-modal hash retrieval model.

7. The training method of the cross-modal hash retrieval model according to any one of claims 1 to 6, characterized in that: The initial cross-modal hash retrieval model is trained based on each of the first-type hash real-valued features, each of the second-type hash real-valued features, and each of the third-type hash real-valued features to obtain a cross-modal hash retrieval model, including: Based on each of the first-category hash real-valued features, each of the second-category hash real-valued features, and each of the third-category hash real-valued features, the network parameters of the image encoder, the network parameters of the text encoder, and the network parameters of the label encoder in the initial cross-modal hash retrieval model are iteratively adjusted to obtain the cross-modal hash retrieval model.

8. A cross-modal hash retrieval method, characterized in that: include: Obtain the data to be retrieved of the first mode; Inputting the to-be-retrieved data of the first modality into a cross-modal hash retrieval model to obtain target data of the second modality output by the cross-modal hash retrieval model; The cross-modal hash retrieval model is trained based on the method according to any one of claims 1 to 7; Among them, when the data to be retrieved in the first modality includes image data, the target data in the second modality includes diagnostic text related to the image data; when the data to be retrieved in the first modality includes diagnostic text, the target data in the second modality includes image data related to the diagnostic text.

9. A training device for a cross-modal hash retrieval model, characterized in that: include: An acquisition module, used to acquire at least one training sample pair, wherein the training sample pair includes sample image data and sample diagnosis text; A determination module, used to determine the first type of hash real-valued features corresponding to each of the sample image data, and to determine the second type of hash real-valued features corresponding to each of the sample diagnosis texts; The determination module is further used to determine the adjacency matrix between the diagnostic labels in the diagnostic label set corresponding to all the training sample pairs based on the reference diagnostic labels in the knowledge graph; The determination module is further configured to determine, for each of the training sample pairs, a third type of hash real-valued feature corresponding to the training sample pair based on the adjacency matrix and the training sample pair; The training module is also used to train the initial cross-modal hash retrieval model based on each of the first-category hash real-valued features, each of the second-category hash real-valued features, and each of the third-category hash real-valued features to obtain a cross-modal hash retrieval model.

10. A cross-modal hash retrieval device, characterized in that: include: An acquisition module, used for acquiring the data to be retrieved in the first mode; An input module, used for inputting the to-be-retrieved data of the first modality into the cross-modal hash retrieval model to obtain the target data of the second modality output by the cross-modal hash retrieval model; The cross-modal hash retrieval model is trained based on the method according to any one of claims 1 to 7; Among them, when the data to be retrieved in the first modality includes image data, the target data in the second modality includes diagnostic text related to the image data; when the data to be retrieved in the first modality includes diagnostic text, the target data in the second modality includes image data related to the diagnostic text.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, it implements the training method of the cross-modal hash retrieval model as described in any one of claims 1 to 7, or implements the cross-modal hash retrieval method as described in claim 8.

12. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the training method of the cross-modal hash retrieval model as described in any one of claims 1 to 7, or implements the cross-modal hash retrieval method as described in claim 8.

13. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, it implements the training method of the cross-modal hash retrieval model as described in any one of claims 1 to 7, or implements the cross-modal hash retrieval method as described in claim 8.

Images