Large language model assisted cross-modal medical data hash retrieval method

Abstract

The invention relates to a cross-modal medical data hash retrieval method assisted by a large language model, and belongs to the field of artificial intelligence information retrieval in an intelligent medical system. Designing an image description type prompt template, a diagnosis report summary type prompt template and a combined modal reasoning type prompt template in combination with the semantic generalization ability of the large language model to perform modal data enhancement, and realizing deep semantic mining of medical image and diagnosis report data; a structured coding layer is utilized to ensure that the features of the image and the text can be accurately matched in a unified embedding space; finely adjusting the large language model by adopting a mode of combining soft prompt and hard prompt to realize alignment of image and text features; a Gaussian binary restricted Boltzmann machine is introduced to carry out probabilistic Hash mapping, data structure information is effectively reserved, and Hash codes are generated to carry out cross-modal Hash retrieval. Semantic similarity retrieval of medical cases among different modes can be realized, high retrieval efficiency is realized, and accurate and personalized diagnosis and treatment schemes are provided for patients.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence information retrieval in smart medical systems, and relates to a cross-modal medical data hash retrieval method assisted by a large language model. Background Art

[0002] With the rapid development of medical imaging technology and the widespread use of electronic medical records, massive amounts of medical images and diagnostic reports have been stored and utilized, providing rich data support for disease diagnosis, clinical decision-making, and precision medicine. However, the heterogeneity and complexity of medical data make effective information retrieval a very challenging task. Medical images primarily reflect the anatomical structure or functional status of lesions, while diagnostic reports contain rich text descriptions, including pathological analysis, clinical decision-making, and treatment plans. The two differ significantly in modality, data format, and semantic expression. This difference makes it difficult for traditional single-modal retrieval methods to fully utilize information from different modalities, limiting the sharing and application efficiency of medical data. Therefore, how to effectively integrate medical images and text information to achieve efficient and accurate cross-modal medical data retrieval has become an important research topic in the field of intelligent medicine.

[0003] Cross-modal medical data retrieval has broad application value in clinical diagnosis, medical education, disease research, and decision support. For example, in clinical auxiliary diagnosis, doctors can quickly retrieve similar case reports from imaging data to improve diagnostic accuracy. In medical education and research, students and researchers can search images and pathology reports to gain a deeper understanding of disease progression. In telemedicine, cross-modal retrieval can help doctors efficiently match relevant cases to improve consultation efficiency. Furthermore, cross-modal retrieval technology can be used to construct medical knowledge graphs, promoting the intelligent application of medical big data. However, the heterogeneity of medical data, particularly the semantic gap between medical images and diagnostic reports, remains a major challenge for cross-modal retrieval. Visual information such as lesion morphology and location in medical images is difficult to directly correspond to and match with the textual descriptions in diagnostic reports at the semantic level. Furthermore, pathological changes in medical images are often not completely independent and may collectively suggest the presence of the same disease. Therefore, developing a unified cross-modal representation space within which similarity can be calculated remains a key research issue.

[0004] There are currently some invention patents to solve the problems of the semantic gap between medical images and diagnostic reports, the relatively rough feature alignment methods, and the inadequacy of hash coding methods in modeling structural information. For example: Patent No. CN118939847A discloses a cross-modal medical data hash retrieval method, which customizes image encoders and text encoders for medical images and diagnostic reports based on the Transformer encoder, and extracts features of medical images and diagnostic reports with the help of joint learning of three pre-training tasks. This method constructs manifold similarity matrices of different modalities, and introduces balance constraints between hash codes and linear classification network constraints to learn hash functions. However, this method has room for optimization in terms of the segmented training strategy, and it is difficult to fully improve the ability to extract modal features and the learning efficiency of hash functions. Patent No. CN116431847A discloses a cross-modal hash retrieval method based on multiple comparisons and two-way adversarial learning. This method can perform cross-modal semantic fusion of image information from the image modality and text information from the text modality, and performs inter-modal, intra-modal, and global-local multiple comparison hash learning and two-way adversarial learning on image and text inference features to obtain binary hash codes. However, in practical applications, the distribution of cross-modal data may be unbalanced or irrelevant, affecting the generalization ability and retrieval performance of the hash code. It is necessary to introduce more positive examples or semantic information in the multiple comparison learning and optimize the loss function in the two-way adversarial learning to better align the semantic distribution of hash codes from different modalities.

[0005] Currently, deep learning methods have made breakthroughs in the field of cross-modal retrieval. Through end-to-end learning, they can automatically extract high-level features of images and text and establish a mapping relationship between the two in a shared embedding space. However, existing deep learning methods still face the following challenges:

[0006] 1. Existing methods fail to adequately model the deep semantic relationships between medical images and diagnostic reports, limiting retrieval accuracy. Most existing deep learning methods rely on independent visual encoders and text encoders, using simple linear projections or attention mechanisms to align image and text features, failing to fully exploit the underlying semantic relationships between the two.

[0007] 2. Feature alignment methods are relatively crude and struggle to capture the complex semantic relationships of medical data. Many existing methods only use shared space or simple attention mechanisms for cross-modal feature alignment, ignoring the structural differences between medical images and text.

[0008] 3. Hash coding methods have shortcomings in modeling structural information, resulting in insufficient retrieval efficiency and stability. Traditional hashing methods mostly rely on fully connected layers or norm-based optimization methods for hash mapping, failing to fully utilize the structural information of the data, resulting in limited retrieval performance.

[0009] 4. There is room for optimization in the segmented training strategy of existing methods, which makes it difficult to fully improve the ability to extract modal features and the learning efficiency of hash functions.

[0010] 5. In practical applications, the distribution of cross-modal data may be unbalanced or uncorrelated, affecting the generalization and retrieval performance of hash codes. This requires introducing more positive examples or semantic information in multi-contrast learning and optimizing the loss function in two-way adversarial learning to better align the semantic distributions of hash codes across modalities. Summary of the Invention

[0011] In view of this, the purpose of the present invention is to provide an efficient cross-modal hash retrieval method for medical data, which adopts a large language model to assist in medical data enhancement, and optimizes cross-modal feature learning through prompt templates and soft prompt mechanisms to ensure that medical images and text can be efficiently matched in a unified embedding space. Through the probabilistic modeling capability of the restricted Boltzmann machine, hash coding can better preserve the structural information of the data, thereby improving the stability and computational efficiency of the retrieval.

[0012] In order to achieve the above object, the present invention provides the following technical solutions:

[0013] 1. A large language model-assisted cross-modal medical data hash retrieval method, comprising the following steps:

[0014] S1. Clean and preprocess medical images and diagnostic reports;

[0015] S2, modality data enhancement using three prompt templates based on a large language model;

[0016] S3, using structured coding layer to align modal features;

[0017] S4, using the prompt template that combines soft prompts and hard prompts to fine-tune the prompt instruction model;

[0018] S5. Using Gaussian binary restricted Boltzmann machine to learn hash function to generate hash code;

[0019] S6. Perform cross-modal hash retrieval using the hash code.

[0020] Furthermore, the S1 specifically includes the following steps:

[0021] S11. Adjusting the patient's medical image to a uniform resolution; composing the patient's diagnostic keywords and patient information into a uniform structure for a diagnosis description;

[0022] S12. Clean the medical reports, remove report pairs with missing information, and remove repeated redundant descriptions.

[0023] Furthermore, the step S2 specifically includes the following steps:

[0024] S21. Using medical images I and related labels L to create a set of medical image description prompt templates Q I ;

[0025] S22. Use the diagnostic report T and related labels L to create a set of diagnostic report summary prompt templates Q T ;

[0026] S23, combining the medical image I and the corresponding diagnosis report T and the related labels L to create a set of joint modality reasoning prompt templates Q IT ;

[0027] S24. For each medical image, from Q I Randomly select a question from Q and use GPT-4 to generate a detailed description; for each diagnosis report, T Randomly select a question, use GPT-4 to summarize the information, and extract the core content of the report; use Q IT , enabling GPT-4 to generate subsequent development and diagnostic recommendations for the case.

[0028] Furthermore, the step S3 specifically includes the following steps:

[0029] S31, converting the input medical image I into a visual feature representation Z=g(I) through a nonlinear mapping function g, where

[0030] S32, visual feature Z and adaptive weighted weight matrix Perform dimension transformation, where p represents the sequence length of the visual features, and d represents the dimension of each feature;

[0031] S33, the features after the above dimension transformation are passed through the hyperbolic tangent activation function and then combined with the feature matrix Multiply to adjust the length of the visual feature sequence, where L is the sequence length;

[0032] S34, calculate the flexible maximum value of the above features and multiply them with the visual feature Z, and embed the target semantics into the weight matrix of the space Multiply them and finally get the semantic embedding X,d model Represents the embedding dimension size of the target sequence.

[0033] Furthermore, the S4 specifically includes the following steps:

[0034] S41. Randomly initialize a vector as a soft prompt, and use the soft prompt as a parameter that can be optimized during the training process;

[0035] S42. During training, the hint template of the data enhancement part is used as the relevant prefix, and the hard hint does not participate in the gradient update;

[0036] S43. In each training step, the soft prompt vector is combined with the hard prompt as the input of the model.

[0037] Furthermore, the S5 specifically includes the following steps:

[0038] S51. Pass the high-dimensional features as input to the Gaussian binary restricted Boltzmann machine;

[0039] S52, constructing a weighted cumulative Hamming distance metric using the manifold similarity matrix and the Hamming distance between hash codes;

[0040] S53, constructing constraints for linear classification networks through cross entropy loss function;

[0041] S54, combining the Hamming distance accumulation on the manifold similarity matrix, the energy function of the Gaussian binary restricted Boltzmann machine and the hash code classification constraint as the total constraint for training the hash function;

[0042] S55. In a Gaussian binary restricted Boltzmann machine, use the visible layer to learn a low-dimensional latent representation of input features to generate hash codes.

[0043] Furthermore, the S52 specifically includes the following steps:

[0044] S521, setting the manifold similarity between the medical image and the diagnosis report to a hypergraph, where the hypergraph is composed of a vertex set and an edge set;

[0045] S522: If any two vertices have the same label, the two vertices are considered to be neighbor nodes.

[0046] S523, constructing a manifold similarity matrix of the medical image and the diagnosis report by using the defined intra-modality similarity and inter-modality similarity;

[0047] S524, subjecting the medical image and the diagnosis report to a hash function to obtain a hash code;

[0048] S525. Using the defined manifold similarity and Hamming distance, construct a weighted cumulative Hamming distance metric for the medical image and the diagnosis report.

[0049] Furthermore, the step S6 specifically includes the following steps:

[0050] S61. Establishing a hash code database using hash codes of medical images and diagnosis reports;

[0051] S62. Retrieving relevant diagnostic report modality samples through the medical image modality samples;

[0052] S63. Retrieve relevant medical image modality samples through the diagnosis report modality samples.

[0053] Furthermore, the step S62 specifically includes the following steps:

[0054] The specific steps include:

[0055] S621, performing data preprocessing on the medical image query sample;

[0056] S622. Obtaining a medical image query sample feature vector through a medical image modality feature extraction network;

[0057] S623, obtaining a hash code of a medical image query sample through a medical image hash network;

[0058] S624, calculating the Hamming distance between the hash code of the medical image query sample and the hash code in the diagnosis report retrieval database;

[0059] S625. Return the diagnosis report sample with the smallest Hamming distance according to the sorting of the Hamming distances, and complete the query.

[0060] Furthermore, the step S63 specifically includes the following steps:

[0061] S631, performing data preprocessing on the diagnostic report query sample;

[0062] S632. Obtaining a diagnostic report query sample feature vector through a diagnostic report modality feature extraction network;

[0063] S633. Obtain a diagnostic report query sample hash code through a diagnostic report hash network;

[0064] S634, calculating the Hamming distance between the hash code of the diagnosis report query sample and the hash code in the medical image retrieval database;

[0065] S635. Return the medical image sample with the smallest Hamming distance according to the sorting of the Hamming distances, thus completing the query.

[0066] The beneficial effects of the present invention are:

[0067] 1. By using a large language model to assist in medical data augmentation and utilizing prompt templates and soft prompt mechanisms to optimize cross-modal feature learning, it can effectively mine the deep semantic relationships between medical images and diagnostic reports, improving the accuracy of cross-modal retrieval.

[0068] 2. The designed structured encoding layer can capture the structural differences between medical images and text, perform efficient cross-modal feature alignment, fully utilize information from different modalities, and improve retrieval accuracy;

[0069] 3. Based on the probabilistic modeling capabilities of the Gaussian binary restricted Boltzmann machine, by introducing a manifold similarity matrix and hash code classification constraints, hash coding can better preserve the structural information of the data, improving retrieval stability and computational efficiency;

[0070] 4. Fine-tuning the model through a combination of soft and hard prompts can fully utilize the prior knowledge of the large language model and improve the efficiency of modal feature extraction and hash function learning;

[0071] 5. The proposed cross-modal hash retrieval method can effectively integrate medical images and diagnostic report information, provide efficient and accurate retrieval support for clinical auxiliary diagnosis, medical education, disease research and decision support, and promote the intelligent application of medical big data.

[0072] Other advantages, objects, and features of the present invention will be described in part in the following description and, in part, will be apparent to those skilled in the art upon examination of the following description or may be learned from practice of the present invention. The objects and other advantages of the present invention may be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below with reference to the accompanying drawings, in which:

[0074] Figure 1 This is a flowchart of a large language model-assisted cross-modal medical data hash retrieval method according to an embodiment of the present invention;

[0075] Figure 2 This is an overall architecture diagram of a large language model-assisted cross-modal medical data hash retrieval method according to an embodiment of the present invention;

[0076] Figure 3 This is a diagram illustrating a medical image description and a diagnostic report summary according to an embodiment of the present invention;

[0077] Figure 4 This is an example diagram of data enhancement of an example data set according to an embodiment of the present invention;

[0078] Figure 5 This is a network architecture diagram of a large language model-assisted cross-modal medical data hash retrieval method according to an embodiment of the present invention;

[0079] Figure 6 This is an example diagram of a prompt template according to an embodiment of the present invention. DETAILED DESCRIPTION

[0080] The following describes the embodiments of the present invention by means of specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0081] Among them, the accompanying drawings are only for illustrative purposes and represent only schematic diagrams rather than actual pictures, and should not be understood as limiting the present invention. In order to better illustrate the embodiments of the present invention, some parts of the accompanying drawings may be omitted, enlarged or reduced, and do not represent the dimensions of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions may be omitted in the accompanying drawings.

[0082] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if there are terms such as "upper", "lower", "left", "right", "front", "back", etc. indicating directions or positional relationships, they are based on the directions or positional relationships shown in the drawings. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific direction, be constructed and operate in a specific direction. Therefore, the terms describing the positional relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0083] See also Figure 1 , which is a flow chart of a large language model-assisted cross-modal medical data hash retrieval method according to an embodiment of the present invention; please refer to Figure 2 , which is an overall architecture diagram of a large language model-assisted cross-modal medical data hash retrieval method according to an embodiment of the present invention;

[0084] The method proposed by the present invention comprises the following steps:

[0085] S1: Data cleaning and preprocessing of medical images and diagnostic reports, including:

[0086] S11. Unify the resolution of the patient's medical images, preferably adjusting the resolution to 224 pixels × 224 pixels × 3 channels; and compose a diagnosis description of the patient's diagnostic keywords and patient information in a unified structure.

[0087] S12. Clean the medical reports, remove report pairs with missing information, and remove some repeated redundant descriptions;

[0088] S2: Based on the large language model, modal data is enhanced using three prompt templates. This step specifically includes:

[0089] S21. Using medical images I and related labels L to create a set of medical image description prompt templates Q I ; see Figure 3 , which is a diagram illustrating a medical image description and a diagnosis report summary according to an embodiment of the present invention; Figure 3 As shown, a medical image description prompt template Q is organized on the left I ;

[0090] S22. Use the diagnostic report T and related labels L to create a set of diagnostic report summary prompt templates Q T , specifically Figure 3 As shown, a medical image description prompt template Q is organized on the right T ;

[0091] S23, combining the medical image I and the corresponding diagnosis report T and the related labels L to create a set of joint modality reasoning prompt templates Q IT

[0092] S24, for each medical image, from the medical image description prompt template Q I A question is randomly selected and a detailed description is generated using GPT-4. For each diagnostic report, a summary prompt template Q T A question is randomly selected from the list so that GPT-4 can summarize the information and extract the core content of the report. In order to make GPT-4 pay more attention to the multi-dimensionality of information, the joint modal reasoning prompt template Q is used. IT , prompting GPT-4 to conduct deeper reasoning and generate possible subsequent developments and diagnostic recommendations for the case.

[0093] See also Figure 4 , which is an example diagram of data enhancement of an example data set according to an embodiment of the present invention.

[0094] S25, such as Figure 4 As shown in the figure, an example of data augmentation is given in a dataset. For the content generated by GPT-4, we first removed redundant information that often appears at the beginning and rhetorical questions that appear at the end. Secondly, we standardized the format of the generated content to ensure that the output description is concise, clear, and meets the requirements of subsequent retrieval and alignment.

[0095] See also Figure 5, which is a network architecture diagram of a large language model-assisted cross-modal medical data hash retrieval method according to an embodiment of the present invention.

[0096] S3: Modal feature alignment is performed through the designed structured encoding layer, specifically including:

[0097] S31, such as Figure 5 As shown in Figure 1, a 12-layer Vit-B / 16 model based on ImageNet pre-trained weights is used as a medical image encoder. The medical image encoder converts the input medical image I into a visual feature representation Z = g(I) through a nonlinear mapping function g, where

[0098] S32, visual feature Z and adaptive weighted weight matrix Perform dimension transformation, where In this representation, p represents the sequence length of visual features and d represents the dimension size of each feature.

[0099] S33, the features after the above dimension transformation are passed through the hyperbolic tangent activation function (tanh) and then combined with the feature matrix Multiply to adjust the length of the visual feature sequence, where L is the sequence length.

[0100] S34, perform softmax on the features obtained above and multiply them with the visual features Z, and embed the target semantics into the weight matrix of the space Multiply them and finally get the semantic embedding X,d model Represents the embedding dimension size of the target sequence.

[0101] S35, the implementation process of the entire structured coding layer is:

[0102] X=(softmax(W2tanh(W1Z T ))Z)W3

[0103] Where, tanh(x)=(e x -e -x ) / (e x +e -x ),

[0104] S4: Fine-tune the prompt instruction model using a prompt template that combines soft prompts and hard prompts, including the following steps:

[0105] S41. Randomly initialize a vector as a soft prompt. The soft prompt vector [Soft Prompt] is inserted into the template as a task prefix and used as an optimizable parameter during training.

[0106] S42, use the hint template in the data enhancement part as the relevant prefix. The hard hint is fixed during the training process and does not participate in the gradient update. Please refer to Figure 6 , which is an example diagram of a prompt template according to an embodiment of the present invention. The specific structure of the prompt template is as follows Figure 6 shown.

[0107] S43. In each training step, the soft hint vector is combined with the hard hint vector as the model input and optimized during the training process. <eof>As a delimiter for definitions.

[0108] S5: Learn the hash function through the Gaussian binary restricted Boltzmann machine to generate the hash code, which includes the following steps:

[0109] S51. Pass the high-dimensional features as input to the Gaussian Binary Restricted Boltzmann Machine. The core of the Gaussian Binary Restricted Boltzmann Machine is its energy function, which is used to define the probability distribution of the network. Our goal is to minimize the energy function so that, given the input data of the visible layer, the network can produce output that conforms to the true data distribution. The energy function can be expressed as:

[0110]

[0111] Where, is the continuous value of the i-th neuron in the visible layer corresponding to the modality m∈{x,y}; is the bias of the i-th neuron in the visible layer corresponding to modality m; is the standard deviation of the corresponding mode m associated with the visible unit i; is the state of the jth neuron in the hidden layer corresponding to mode m; is the bias of the jth neuron in the hidden layer corresponding to mode m; is the weight between the visible layer and the hidden layer corresponding to the modality m. In the Gaussian binary restricted Boltzmann machine, given the hidden layer h, the conditional probability distribution of the visible layer is Gaussian. The conditional probability of each visible unit is:

[0112]

[0113] Where, represents a normal distribution with mean b i +∑ j w ij h j , the variance is Given the visible layer v, the conditional probability of the hidden layer is Bernoulli distribution, expressed as:

[0114]

[0115] Among them, sig(x)=1 / (1+exp(-x)) is the sigmoid activation function, which represents the probability of activation of the hidden layer unit.

[0116] S52. Design manifold similarity matrix S ma and calculate the Hamming distance D between hash codes h , construct the weighted cumulative Hamming distance metric F S , including the following steps:

[0117] S521, abstract the manifold similarity between medical images and diagnosis reports into a hypergraph Gra=(V,S ma ), the hypergraph consists of a vertex set V and an edge set S ma Among them, V contains two modal data instances, S ma is the manifold similarity between two instances;

[0118] S522. For any vertex V i and V j For example, if there are the same labels between two vertices, they are considered to be neighbor nodes. Define N(a) to represent the neighbor set of vertex a, then construct the similarity within the modality and and the similarity between modalities x i and y j Represent the i-th medical image feature vector and the j-th diagnosis report feature vector respectively;

[0119] S523, by defining the intra-modal similarity and and inter-modal similarity Constructing medical image x i and diagnostic report j Manifold similarity matrix represents element-wise multiplication;

[0120] S524, let the hash codes obtained by the hash function of the medical image and the diagnosis report be B x ∈{-1,1} and B y ∈{-1,1}, the hash code can be implemented by the inner product of the hash code matrix when calculating the Hamming distance: k represents the length of the hash code, and They are represented as the i-th medical image hash code and the j-th diagnosis report hash code, Represents a hash code and The inner product of

[0121] S525, using the defined manifold similarity and Hamming distance to construct the medical image x i and diagnostic report j Weighted cumulative Hamming distance metric

[0122] S53. The constraints for constructing a linear classification network are defined by the cross entropy loss function H: F L =H(L i ,W c B), where W c Represents the weight parameter of the classifier.

[0123] S54. Combine the Hamming distance accumulation on the manifold similarity matrix, the energy function of the Gaussian binary restricted Boltzmann machine, and the hash code classification constraint as the total constraint to train the hash function of the Gaussian binary restricted Boltzmann machine. The total loss function is:

[0124]

[0125] Where α and β are introduced hyperparameters to balance the weights between different constraints.

[0126] S55. In the Gaussian binary restricted Boltzmann machine, the visible layer is responsible for learning the low-dimensional potential representation of the input features and ultimately generating a hash code.

[0127] S6: Perform cross-modal hash retrieval using the generated hash code. This step specifically includes the following steps:

[0128] S61. Establishing a hash code database using hash codes of medical images and diagnosis reports;

[0129] S62, retrieving relevant diagnostic report modality samples through medical image modality samples, including the following specific steps:

[0130] S621, performing data preprocessing on the medical image query sample;

[0131] S622. Obtaining a medical image query sample feature vector through a medical image modality feature extraction network;

[0132] S623, obtaining a hash code of a medical image query sample through a medical image hash network;

[0133] S624, calculating the Hamming distance between the hash code of the medical image query sample and the hash code in the diagnosis report retrieval database;

[0134] S625, sorting by Hamming distance and returning the diagnostic report sample with the smallest Hamming distance, completing the query function;

[0135] S63, retrieving relevant medical image modality samples through the diagnosis report modality samples, including the following specific steps:

[0136] S631, performing data preprocessing on the diagnostic report query sample;

[0137] S632. Obtaining a diagnostic report query sample feature vector through a diagnostic report modality feature extraction network;

[0138] S633. Obtain a diagnostic report query sample hash code through a diagnostic report hash network;

[0139] S634, calculating the Hamming distance between the hash code of the diagnosis report query sample and the hash code in the medical image retrieval database;

[0140] S635. Return the medical image sample with the smallest Hamming distance according to the sorting of the Hamming distances, thereby completing the query function.

[0141] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solutions, which should all be included in the scope of the claims of the present invention.< / eof>

Claims

1. A large language model-assisted cross-modal medical data hash retrieval method, characterized by: The following steps are involved: S1. Clean and preprocess medical images and diagnostic reports; S2, modality data enhancement using three prompt templates based on a large language model; S3, using structured coding layer to align modal features; S4, using the prompt template that combines soft prompts and hard prompts to fine-tune the prompt instruction model; S5. Using Gaussian binary restricted Boltzmann machine to learn hash function to generate hash code; S6. Perform cross-modal hash retrieval using the hash code.

2. The cross-modal medical data hash retrieval method assisted by a large language model according to claim 1 is characterized by: The S1 specifically includes the following steps: S11. Adjusting the patient's medical image to a uniform resolution; composing the patient's diagnostic keywords and patient information into a uniform structure for a diagnosis description; S12. Clean the medical reports, remove report pairs with missing information, and remove repeated redundant descriptions.

3. The cross-modal medical data hash retrieval method assisted by a large language model according to claim 1 is characterized by: The S2 specifically includes the following steps: S21. Using medical images I and related labels L to create a set of medical image description prompt templates Q I ; S22. Use the diagnostic report T and related labels L to create a set of diagnostic report summary prompt templates Q T ; S23, combining the medical image I and the corresponding diagnosis report T and the related labels L to create a set of joint modality reasoning prompt templates Q IT ; S24. For each medical image, from Q I Randomly select a question from Q and use GPT-4 to generate a detailed description; for each diagnosis report, T Randomly select a question, use GPT-4 to summarize the information, and extract the core content of the report; use Q IT , enabling GPT-4 to generate subsequent development and diagnostic recommendations for the case.

4. The cross-modal medical data hash retrieval method assisted by a large language model according to claim 1, characterized in that: The S3 specifically includes the following steps: S31, converting the input medical image I into a visual feature representation Z=g(I) through a nonlinear mapping function g, where S32, visual feature Z and adaptive weighted weight matrix Perform dimension transformation, where p represents the sequence length of the visual features, and d represents the dimension of each feature; S33, the features after the above dimension transformation are passed through the hyperbolic tangent activation function and then combined with the feature matrix Multiply to adjust the length of the visual feature sequence, where L is the sequence length; S34, calculate the flexible maximum value of the above features and multiply them with the visual feature Z, and embed the target semantics into the weight matrix of the space Multiply them and finally get the semantic embedding X,d model Represents the embedding dimension size of the target sequence.

5. The cross-modal medical data hash retrieval method assisted by a large language model according to claim 1, characterized in that: The S4 specifically includes the following steps: S41. Randomly initialize a vector as a soft prompt, and use the soft prompt as a parameter that can be optimized during the training process; S42. During training, the hint template of the data enhancement part is used as the relevant prefix, and the hard hint does not participate in the gradient update; S43. In each training step, the soft prompt vector is combined with the hard prompt as the input of the model.

6. The cross-modal medical data hash retrieval method assisted by a large language model according to claim 1, characterized in that: The S5 specifically includes the following steps: S51. Pass the high-dimensional features as input to the Gaussian binary restricted Boltzmann machine; S52, constructing a weighted cumulative Hamming distance metric using the manifold similarity matrix and the Hamming distance between hash codes; S53, constructing constraints for linear classification networks through cross entropy loss function; S54, combining the Hamming distance accumulation on the manifold similarity matrix, the energy function of the Gaussian binary restricted Boltzmann machine and the hash code classification constraint as the total constraint for training the hash function; S55. In a Gaussian binary restricted Boltzmann machine, use the visible layer to learn a low-dimensional latent representation of input features to generate hash codes.

7. The cross-modal medical data hash retrieval method assisted by a large language model according to claim 6, characterized in that: The S52 specifically includes the following steps: S521, setting the manifold similarity between the medical image and the diagnosis report to a hypergraph, where the hypergraph is composed of a vertex set and an edge set; S522: If any two vertices have the same label, the two vertices are considered to be neighbor nodes. S523, constructing a manifold similarity matrix of the medical image and the diagnosis report by using the defined intra-modality similarity and inter-modality similarity; S524, subjecting the medical image and the diagnosis report to a hash function to obtain a hash code; S525. Using the defined manifold similarity and Hamming distance, construct a weighted cumulative Hamming distance metric for the medical image and the diagnosis report.

8. The cross-modal medical data hash retrieval method assisted by a large language model according to claim 1, characterized in that: The S6 specifically includes the following steps: S61. Establishing a hash code database using hash codes of medical images and diagnosis reports; S62. Retrieving relevant diagnostic report modality samples through the medical image modality samples; S63. Retrieve relevant medical image modality samples through the diagnosis report modality samples.

9. The large language model-assisted cross-modal medical data hash retrieval method according to claim 8, characterized in that: The S62 specifically includes the following steps: The specific steps include: S621, performing data preprocessing on the medical image query sample; S622. Obtaining a medical image query sample feature vector through a medical image modality feature extraction network; S623, obtaining a hash code of a medical image query sample through a medical image hash network; S624, calculating the Hamming distance between the hash code of the medical image query sample and the hash code in the diagnosis report retrieval database; S625. Return the diagnosis report sample with the smallest Hamming distance according to the sorting of the Hamming distances, and complete the query.

10. The cross-modal medical data hash retrieval method assisted by a large language model according to claim 8, characterized in that: The S63 specifically includes the following steps: S631, performing data preprocessing on the diagnostic report query sample; S632. Obtaining a diagnostic report query sample feature vector through a diagnostic report modality feature extraction network; S633. Obtain a diagnostic report query sample hash code through a diagnostic report hash network; S634, calculating the Hamming distance between the hash code of the diagnosis report query sample and the hash code in the medical image retrieval database; S635. Return the medical image sample with the smallest Hamming distance according to the sorting of the Hamming distances, thus completing the query.

Images