Medical information retrieval method and system based on multi-modal vectorization and storage medium

Abstract

The invention discloses a medical information retrieval method and system based on multi-modal vectorization and a storage medium, and relates to the field of intelligent retrieval. The method comprises the following steps: constructing a corpus and performing field self-adaptive fine tuning to obtain a field optimization recall model; generating a mixed modal vector through a cross-modal attention mechanism and a gating fusion mechanism according to the domain optimization recall model; carrying out slicing processing on the retrieval original text; and after to-be-retrieved information is rewritten and vectorized, retrieval is performed according to the mixed modal vector and the father-child node slice structure. According to the retrieval method, professional corpus semantic optimization, multi-modal information fusion, intelligent document slicing and query rewriting can be combined, and the problem of semantic fracture caused by insufficient multi-modal information fusion and document slicing in a medical RAG system is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of intelligent retrieval, in particular to a medical information retrieval method and system based on multi-modal vectorization and a storage medium. BACKGROUND

[0002] Traditional medical information retrieval systems mainly rely on keyword matching, rule engines or single-modal text representation models to achieve information retrieval. For text data, TF-IDF, BM25 or vector representation methods based on general pre-trained language models are commonly used; for medical images, convolutional neural networks are often used to extract features, but they are usually processed independently of text information; structured data is usually input into the retrieval system through simple normalization or linear mapping. In terms of document processing, traditional systems often use fixed-length slicing or simple segmentation methods, which lack semantic-oriented intelligent summarization and slicing mechanisms. In addition, user queries are usually directly input, lacking standardization of medical terminology, anaphora resolution and multi-round follow-up processing mechanisms.

[0003] Due to the lack of unified fusion mechanisms in traditional technologies for text, image and structured data processing, multi-modal information cannot be fully utilized, resulting in insufficient coverage of key medical information in the retrieval results; fixed slicing and simple segmentation methods can easily cause semantic fragmentation of documents, making the retrieval results lack contextual coherence; in addition, abbreviations, pronouns and multi-round follow-up in user queries are not fully processed, which can easily cause semantic misunderstandings or retrieval deviations. SUMMARY

[0004] To address the deficiencies in the prior art, the present application solves the technical problem of providing a retrieval method that can combine professional corpus semantic optimization, multi-modal information fusion, intelligent document slicing and query rewriting, thereby improving the accuracy, semantic integrity and user response efficiency of medical information retrieval.

[0005] To achieve the above purpose, in a first aspect, the embodiments of the present application provide a medical information retrieval method based on multi-modal vectorization, which comprises the following steps: After constructing a corpus and performing domain adaptation fine-tuning, a domain optimization recall model is obtained; After encoding and processing different types of data according to the domain optimization recall model, corresponding feature vectors are obtained, and the feature vectors are converted into hybrid modal vectors through cross-modal attention mechanisms and gate fusion mechanisms; Slicing the original retrieval text: generating a structured summary and keywords of the original retrieval text; constructing a parent-child node slicing structure, the parent node storing the structured summary and keywords as global semantic anchors, and the child node slicing the original retrieval text and associating with the parent node ID; Rewriting the information to be retrieved: rewriting, disambiguating and judging the multi-round follow-up of the information to be retrieved; The rewritten information to be searched is vectorized, and searching is performed according to the mixed modal vector and the parent-child node slice structure.

[0006] In combination with the first aspect, in an implementation, the fine-tuning target function of the domain adaptive fine-tuning is a double-task loss function: the main task is a contrastive learning loss, and the auxiliary task is a triple loss. The contrastive learning loss The calculation formula is as follows:

[0007] Wherein is a cosine similarity function, is a positive sample pair vector, is a negative sample vector. The triple loss The calculation formula is as follows:

[0008] Wherein is an anchor point vector, is a positive sample vector, is a negative sample vector, is a boundary parameter and takes a value of 0.5. The total loss function L is: , is a weight coefficient.

[0009] In combination with the first aspect, in an implementation, the process of encoding different types of data includes: Text information is encoded by using a domain optimization recall model; Image information is feature extracted; The numerical data x is standardized to obtain :

[0010] Wherein is a mean value, is a standard.

[0011] In combination with the first aspect, in an implementation, the process of generating a mixed modal vector by a cross-modal attention mechanism and a gate fusion mechanism includes: After layer normalization processing of the feature vector, a standardized feature vector is obtained; According to the standardized feature vector, an attention weight matrix reflecting the importance of the features is calculated; The dynamic weight coefficient of the standardized feature vector is calculated; The fusion weight is determined according to the attention weight matrix and the dynamic weight coefficient, and the hybrid modality vector is generated according to the fusion weight and the corresponding normalized vector.

[0012] In combination with the first aspect, in an implementation, the feature vector includes a text feature vector , an image feature vector , and a structured feature vector , and the corresponding normalized feature vector obtained after the normalization processing is , and the calculation formula of the normalization processing is:

[0013] wherein is a feature mean value, is a feature standard deviation, is a smoothing coefficient, and ; The attention weight matrix is:

[0014]

[0015]

[0016]

[0017] wherein , , is a trainable parameter matrix, is an attention dimension; The dynamic weight coefficient is calculated by a gating unit :

[0018] wherein is a query vector, ; the dynamic weight coefficient , , ; The calculation formula of the fusion weight is:

[0019] The calculation formula of the hybrid modality vector is: .

[0020] In combination with the first aspect, in an implementation, the keyword generation process includes: calculating word weights :

[0021] in For word frequency, , Total number of documents This represents the number of documents containing the word. Calculate word scores :

[0022] in The damping coefficient is 0.85. For words and The co-occurrence weights.

[0023] In conjunction with the first aspect, in one implementation, the construction process of the parent-child node slice structure includes: Calculate sentence similarity:

[0024] s m ,s j This indicates two sentences, v m , v j They are s m ,s j The corresponding vector representation; A preset similarity threshold is used. When the similarity of a specified number of sentences is below the threshold, the sentence is sliced ​​to generate child nodes.

[0025] In conjunction with the first aspect, in one implementation, the process of rewriting and completing the information to be retrieved includes: Calculate the edit distance between the query string s to be retrieved and the term string t that matches s in the knowledge graph, and then calculate the edit distance. Calculate the similarity between s and t. :

[0026] in Indicates taking and The maximum value of both lengths; judge If the specified numerical condition is met, replace s with t; otherwise, express s in a fuzzy manner. The process of performing the aforementioned referential disambiguation on the retrieved information includes: For each entity e in the entity construction list in the session history, calculate the referential probability P of the word pron that needs to be replaced:

[0027] denotes the mth entity in the entity construction list; pron vec , are the vector representations of pron and respectively; when the maximum value of P is above the specified substitution threshold, pron is replaced by the corresponding entity; The process of the multi-round follow-up judgment on the information to be retrieved includes: identifying the frequency f of the conjunction word in the information to be retrieved, and the vector representation q of the current information to be retrieved curr,vec the relevance R of the vector representation q of the previous round of query prev,vec ;

[0028] when f is above the frequency threshold and R is above the relevance threshold, it is determined that the current information to be retrieved is a follow-up to the previous query, and the rewritten information to be retrieved is generated by splicing the current information to be retrieved, the context information related to the current information to be retrieved, and the summary of the previous round of query results.

[0029] In a second aspect, the embodiments of the present application provide a medical information retrieval system based on multi-modal vectorization, which comprises: The recall model fine-tuning module is configured to implement the process of obtaining a domain-optimized recall model after the corpus construction and domain adaptation fine-tuning provided in the first aspect; The multi-modal and hybrid-modal vectorization module is configured to implement the process of obtaining a hybrid-modal vector by encoding and processing different types of data according to the domain-optimized recall model provided in the first aspect, and then generating the hybrid-modal vector through a cross-modal attention mechanism and a gating fusion mechanism; The document slicing module is configured to implement the slicing processing of the retrieval original text provided in the first aspect: generating a structured summary and keywords of the retrieval original text; constructing a parent-child node slicing structure, the parent node storing the structured summary and the keywords as global semantic anchors, and the child node slicing the retrieval original text and associating with the parent node ID; The search term intelligent rewriting module is configured to implement the rewriting of the information to be retrieved provided in the first aspect: rewriting and completing the information to be retrieved, resolving the pronoun, and judging the multi-round follow-up; and vectorizing the rewritten information to be retrieved, and searching according to the hybrid-modal vector and the parent-child node slicing structure.

[0030] In a third aspect, the embodiments of the present application provide a computer readable storage medium, which stores a medical information retrieval program based on multi-modal vectorization, and the medical information retrieval program based on multi-modal vectorization, when executed, implements the steps of the method provided in the first aspect.

[0031] Compared with the prior art, the present application has the following advantages: The present application helps the model to learn the semantic features of the professional corpus by performing field self-adaptive fine-tuning on the professional corpus, so that the query text and the medical document can be more accurately matched.

[0032] Further, after the encoded feature vectors are generated into unified hybrid modal vectors through the cross-modal attention mechanism and the gating fusion mechanism, such standardization processing can ensure that the features of different modalities are fused in the same scale, thereby enhancing the comprehensive utilization of multi-modal information and improving the vector retrieval capability.

[0033] On this basis, the present application ensures the standardization of user queries and optimizes semantic expression through two-stage intelligent slicing and three-layer query rewriting mechanism, thereby realizing the context integrity and efficient response in multi-modal information fusion retrieval.

[0034] In summary, the present application proposes a retrieval method that can combine professional corpus semantic optimization, multi-modal information fusion, intelligent document slicing, and query rewriting, solves the problem of insufficient multi-modal information fusion and semantic fragmentation caused by document slicing in the medical RAG system, improves the accuracy and context coherence of medical knowledge retrieval, and further improves the accuracy, semantic integrity, and user response efficiency of medical information retrieval. BRIEF DESCRIPTION OF DRAWINGS

[0035] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0036] Figure 1 The flowchart of S2 in the embodiments of the present application is shown in the figure; Figure 2 The flowchart of S3 in the embodiments of the present application is shown in the figure; Figure 3 The timing diagram of the medical information retrieval method based on multi-modal vectorization in the embodiments of the present application is shown in the figure; Figure 4 The flowchart of three-layer rewriting in the embodiments of the present application is shown in the figure; Figure 5A hardware structure schematic diagram of a medical information retrieval device based on multi-modal vectorization involved in an embodiment of the present application. DETAILED DESCRIPTION

[0037] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the following will be combined with the accompanying drawings to make a clear and complete description of the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0038] The flowchart shown in the accompanying drawings is only an example and does not necessarily include all the contents and operations / steps, nor does it necessarily be executed in the described order. For example, some operations / steps can be further decomposed, combined, or partially merged, so that the actual execution order can be changed according to the actual situation.

[0039] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the following will be combined with the accompanying drawings to make a clear and complete description of the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0040] In a first aspect, the embodiments of the present application provide a medical information retrieval method based on multi-modal vectorization, comprising the following steps: S1: Construct a professional corpus (in this embodiment, a tumor field), select Sentence-BERT as a basic model, and obtain a field-optimized recall model after field self-adaptive fine-tuning of the professional corpus. In an embodiment, the corpus in S1 contains clinical guidelines, authoritative journal papers, drug instruction manuals, and real-world cases, and the corpus scale is not less than 500,000.

[0041] In an embodiment, the fine-tuning target function of the field self-adaptive fine-tuning in S1 is a double-task loss function: the main task is a contrastive learning loss to maximize the vector similarity of positive sample pairs; and the auxiliary task is a triple loss to narrow the distance between related medical concepts and push away irrelevant concepts.

[0042] Specifically, the calculation formula of the contrastive learning loss is as follows:

[0043] wherein is a cosine similarity function, is a positive sample pair vector, is a negative sample vector; The calculation formula of the triple loss is as follows:

[0044] in For anchor vectors, For positive sample vectors, For negative sample vectors, It is a boundary parameter and takes the value 0.5; In this case, the total loss function L is: , These are the weighting coefficients.

[0045] The principle described above is that the final training objective of the total loss function is a mixture consisting of two parts: and , It's a weighting system that determines which task is given more importance, based on experience. Start by trying 0.5 (both tasks are equally important), and then use a grid search to try multiple values ​​within a range (e.g., 0.1, 0.2, ..., 0.9). Values, using different Train multiple models, test their performance on an independent validation set, and finally select the one that performs best on the validation set. The value is used as the final parameter, that is 0.7.

[0046] The domain-optimized recall model for semantic matching of medical texts was obtained by conducting 10 rounds of iterative training using the above method. The model improved the average cosine similarity on the validation set by ≥15%.

[0047] S2: Based on the domain-optimized recall model obtained in S1, after encoding different types of data in the same case, the corresponding feature vectors are obtained. The feature vectors are then used to generate a unified hybrid modality vector through cross-modal attention mechanism and gating fusion mechanism, and then stored in a vector database that supports multimodal indexing. The performance goal of this step is that in subsequent retrieval tasks, the system's vector retrieval accuracy (i.e. the accuracy of query results) is ≥92%.

[0048] In one embodiment, the process of encoding different types of data to obtain feature vectors in S2 includes: See Figure 1 As shown, after performing corresponding encoding and data standardization on different types of data, corresponding feature vectors are obtained. The encoding process includes: After text information is encoded by a text encoder using a domain-optimized recall model, a 768-dimensional text feature vector is output. This optimization of the text ensures that semantic information is effectively preserved. The image information (medical image) is extracted by the image encoder using ResNet-50, and a 2048-dimensional image feature vector is output through a global average pooling layer to retain the key visual features of the medical image. The numerical data (medical data) x is standardized by the structured data encoder, and a structured feature vector is output. The standardization formula is:

[0049] wherein is the mean, is the standard deviation, and the principle is: The numerical ranges of different indicators in the medical data are very different (for example: age 0-150 years vs. blood sugar 3.9-6.1 mmol / L); if not processed, the model will pay excessive attention to the features with large values, and ignore the truly important information. The standardized values are embedded into a 768-dimensional vector space through a 3-layer fully connected network.

[0050] In an embodiment, referring to FIG. 2, the process of generating a unified mixed modality vector by the cross-modal attention mechanism and the gating fusion mechanism in S2 includes: Figure 1 S201: After layer normalization processing of the feature vector, a standardized feature vector is obtained.

[0051] In an embodiment, the feature vector in S201 includes a text feature vector , an image feature vector , and a structured feature vector , and the corresponding standardized feature vectors after normalization processing are , and the calculation formula of the normalization processing is:

[0052] wherein is the feature mean, is the feature standard deviation, is a smoothing coefficient, and .

[0053] S202: The attention weight matrix reflecting the importance of the features is calculated according to the standardized feature vector through the self-attention module.

[0054] In an embodiment, the attention weight matrix is:

[0055]

[0056]

[0057] ​

[0058] wherein , , is a trainable parameter matrix, is an attention dimension and takes a value of 64.

[0059] S203: Calculate the dynamic weight coefficient of the normalized feature vector.

[0060] In an embodiment, S203 introduces a gating unit Calculate the dynamic weight coefficient:

[0061] wherein is a query vector, ; The softmax ensures that the output dynamic weight coefficient , , , and satisfies After such processing, the weights of different modalities can be adaptively allocated at the same scale, avoiding the problem that the weight sum is not 1 caused by the independent output of sigmoid.

[0062] S204: Determine the fusion weight according to the attention weight matrix and the dynamic weight coefficient, and generate the mixed modality vector according to the fusion weight and the corresponding normalized vector.

[0063] In an embodiment, the fusion weight in S204 is calculated according to the following formula:

[0064] The calculation formula of the mixed modality vector is as follows:

[0065] Through the above dynamic weight adjustment, the multi-modal fusion can be adaptively optimized according to the query intent and data characteristics, solving the problem of key information loss caused by fixed weight fusion.

[0066] S3: Perform two-stage intelligent slicing processing on the document original text (medical document, i.e. the original text to be retrieved): First stage: generate a structured summary and keywords of the document original text; Second stage: build a parent-child node slicing structure, the parent node stores the structured summary and keywords as global semantic anchors, and the child node cuts the document original text and associates the parent node ID; during retrieval, the child node is used for retrieval, and the parent node information is automatically carried when the result is returned.

[0067] In an embodiment, the flow of S3 includes: First stage: using a large language model to generate a structured summary, the length of the summary being 15%-20% of the original text of the document, and using a TF-IDF algorithm to extract core keywords, setting the word frequency threshold to be ≥5 and the inverse document frequency to be ≥0.3; The second stage constructs a parent-child node slicing structure, the parent node stores the structured summary and the core keywords; the child nodes are cut according to semantic paragraphs, a sliding window method is used, the window size is set to 500±50 words, and when the cosine similarity of three consecutive sentences is all ≤0.6, slicing is performed, each child node is associated with a parent node ID; the child nodes are individually vectorized and stored in a vector database, and when searching, the child node results automatically carry the corresponding parent node information through the associated ID, the semantic coherence score is ≥85 points, and the semantic coherence score is a performance standard that the system needs to achieve, which is used to measure the slicing effect.

[0068] In an embodiment, referring to Figure 2 , the specific implementation flow of S3 includes: S301: In the first stage, after inputting the medical document into the large language model, a structured summary is generated by prompting words, the ROUGE-L index is used to evaluate the quality of the summary, and the threshold is set to be ≥0.65; the flow of extracting core keywords (i.e., the keyword generation flow in S3) includes: The TF-IDF algorithm is used to calculate the word weight:

[0069] Among them, is the word frequency, , is the total number of documents, is the number of documents containing the word, and the product of and is used to evaluate the importance of a word to a document by comprehensively considering the frequency of the word in the document and the universality of the word in the document set.

[0070] The TextRank algorithm is used to calculate the word score:

[0071] Among them, is the damping coefficient and takes a value of 0.85, is the co-occurrence weight of the word and , which reflects the relationship between the word and ​The degree of co-occurrence is calculated based on the topological structure of the graph. By iteratively calculating the scores of words, key information in the document is mined. Combined with the TF-IDF algorithm, core keywords with weights ≥ 0.5 are extracted.

[0072] S302: In the second stage, the semantic window sliding method is used to cut the molecular nodes and calculate the sentence similarity:

[0073] s m ,s j This indicates two sentences, v m , v j They are s m ,s j The corresponding vector representation, cos(v) m , v j This indicates that the cosine similarity between two vectors is calculated, with values ​​ranging from [...]. Between 1 and 1, the closer the value is to 1, the more similar the two sentences are semantically.

[0074] A preset similarity threshold is used. When the similarity of a specified number of sentences (3 consecutive sentences) is below the threshold, the text is sliced ​​to generate child nodes. Each child node is stored at its starting position in the original text. and the end position ,satisfy character; S303: Establish a bidirectional index table ,in The parent node ID, The child node ID is implemented through hash mapping. Index lookup complexity; during retrieval, when a child node is hit The system through Retrieve the structured summary and core keywords of the parent node p, and return them together with the content of the child nodes, ensuring that the contextual integrity is ≥90%.

[0075] The above processing method solves the semantic fragmentation problem caused by traditional document slicing.

[0076] S4: See also Figure 3 As shown, after the user inputs the information to be retrieved, the information is rewritten through a three-layer rewriting mechanism. See [link to documentation]. Figure 4 As shown, the three layers are rewritten as follows: First layer: Rewrite and complete the information to be retrieved using a medical knowledge graph; The second layer: disambiguation of the information to be retrieved is performed using a substitution disambiguation model; The third layer: After multiple rounds of questioning and judgment of the information to be retrieved, the rewritten information to be retrieved is obtained; Specifically, this embodiment utilizes a medical knowledge graph for abbreviation completion, with a matching threshold set to 0.85; it employs a BERT-based pronoun resolution model, achieving a pronoun recognition accuracy of ≥90%; and it uses a RoBERTa-Base classifier to determine multiple rounds of follow-up questions, with a semantic relevance threshold set to 0.7.

[0077] After vectorizing the information to be retrieved, the retrieval is performed based on the mixed modality vector in S2 and the parent-child node slice structure in S3, and the retrieval results that integrate multimodal information and complete context are returned.

[0078] In one embodiment, the rewrite and completion process of the first layer in S4 includes: Calculate the edit distance (i.e., the distance between the query string s to be retrieved and the term string t that matches s in the UMLS knowledge graph) of the query string s to be retrieved. Convert to (Minimum number of editing operations (insert, delete, replace) required), based on edit distance Calculate the similarity between s and t. :

[0079] in Indicates taking and The maximum of the two lengths is used to normalize the edit distance.

[0080] judge Does it meet the specified numerical conditions? If the above applies, replace s with t; otherwise, express s in a vague way.

[0081] The process of fuzzy expression includes: building a query template library through retrieval history, and calculating the information to be retrieved. Match degree M with each template in the template library:

[0082] in It is the first one in the template The weight of each element is used to measure the importance of different elements in the template. It is the first one in the template One element, Information to be retrieved; represent and The matching degree is the sum of the weighted results of all elements, which is the template matching degree. .

[0083] Selecting templates whose matching value (0.7) is above the specified matching value to process the information to be searched, and in this embodiment, the first three templates of are selected to generate candidate queries to assist in clarifying the user demand.

[0084] In one embodiment, the process of anaphora resolution of the first layer in S4 includes: In the anaphora resolution stage, entities in the session history are extracted to build a list, and the referential probability P of each entity e in the entity list and the word pron that needs to be replaced is calculated:

[0085] represents the mth entity in the entity list; pron vec , are the vector representations of pron and vec , respectively; sin (pron curr,vec , e) is a certain similarity between pron and prev,vec , and the similarity is converted into a probability distribution through a softmax function.

[0086] If the maximum value of P is above the specified replacement threshold (0.7), pron is replaced with the corresponding entity; otherwise, a follow-up question is generated to eliminate the ambiguity of pron.

[0087] In one embodiment, the process of multi-round follow-up judgment of the third layer in S4 includes: The occurrence frequency f of the connecting word in the information to be searched and the relevance R between the vector representation q curr,vec of the current information to be searched and the vector representation q prev,vec of the previous query are identified:

[0088] cos (q curr,vec , q prev,vec ) is the cosine similarity between q and q

[0001] , which measures the semantic relevance between the two.

[0089] If f is above the frequency threshold and R is above the relevance threshold (0.7 in this embodiment), it is determined that it is a follow-up question to the previous query, and the result summary of the previous query is concatenated to the current query; otherwise, it is determined to be a new query.

[0090] ​​​​​When it is determined that the current query is a follow-up question to the previous query, the current information to be searched, the context information related to the current information to be searched, and the summary of the previous query result are spliced to generate more rich and targeted extended information to be searched, so that the subsequent required information can be more accurately obtained.

[0091] Through the cooperative execution of the above steps, the following effects are achieved: 1. By performing domain adaptation fine-tuning on the professional corpus, the model is helped to learn the semantic features of the professional corpus, so that the query text and the medical document can be more accurately matched.

[0092] 2. After the feature vector obtained by encoding is generated into a unified mixed modal vector through the cross-modal attention mechanism and the gating fusion mechanism, the standardized processing can ensure that the features of different modalities are fused in the same scale, thereby enhancing the comprehensive utilization of multi-modal information and improving the vector retrieval capability.

[0093] 3. Through the two-stage intelligent slicing (corresponding to S3) and the three-layer query rewriting mechanism (corresponding to S4), the user query standardization and the optimization of semantic expression are ensured, so that the context integrity and efficient response in the multi-modal information fusion retrieval are realized.

[0094] In summary, the present application proposes a retrieval method capable of combining professional corpus semantic optimization, multi-modal information fusion, intelligent document slicing, and query rewriting, solves the problems of insufficient multi-modal information fusion and semantic fragmentation caused by document slicing in the medical RAG system, improves the accuracy and context coherence of medical knowledge retrieval, and further improves the accuracy, semantic integrity, and user response efficiency of medical information retrieval.

[0095] In a second aspect, the embodiments of the present application also provide a medical information retrieval system based on multi-modal vectorization, which comprises: The recall model fine-tuning module is configured to implement the process of obtaining the domain-optimized recall model after constructing the corpus and performing domain adaptation fine-tuning in S1; The multi-modal and mixed modal vectorization module is configured to implement the process of generating the mixed modal vector by encoding the different types of data according to the domain-optimized recall model to obtain the corresponding feature vector, and then passing the feature vector through the cross-modal attention mechanism and the gating fusion mechanism in S2; The document slicing module is configured to implement the slicing process of the retrieval original text in S3: generating the structured summary and keywords of the retrieval original text; constructing the parent-child node slicing structure, the parent node storing the structured summary and keywords as global semantic anchors, and the child node slicing the retrieval original text and associating the parent node ID; The search word intelligent rewriting module is configured to: rewrite the information to be searched in S4; complete the rewriting of the information to be searched, disambiguate the anaphora, and determine the multiple rounds of follow-up questions; and vectorize the rewritten information to be searched, and search according to the mixed mode vector and the parent-child node slice structure.

[0096] The system solves the semantic breakage problem caused by the split of multi-modal information and the document slice in the medical field question and answer system through the cooperation of the above modules, and improves the accuracy of the search and the quality of the answers.

[0097] The functions of the modules in the medical information retrieval device based on multi-modal vectorization correspond to the steps in the medical information retrieval method based on multi-modal vectorization, and the functions and implementation processes will not be described here.

[0098] In a third aspect, the embodiments of the present application also provide a computer readable storage medium.

[0099] The computer readable storage medium of the present application stores a medical information retrieval program based on multi-modal vectorization, wherein the medical information retrieval program based on multi-modal vectorization is executed by a processor to implement the steps of the medical information retrieval method based on multi-modal vectorization.

[0100] The method implemented by the medical information retrieval program based on multi-modal vectorization can refer to the embodiments of the medical information retrieval method based on multi-modal vectorization of the present application, and will not be described here.

[0101] It should be noted that the above sequence numbers of the embodiments of the present application are only for description, and do not represent the advantages and disadvantages of the embodiments.

[0102] The terms "include" and "have" and any variations thereof in the specification and claims of the present application and the above drawings are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device including a series of steps or units is not limited to the listed steps or units, but can optionally include steps or units not listed, or can optionally include other steps or units inherent to the process, method, product or device. The terms "first", "second" and "third" and the like descriptions are used to distinguish different objects, and do not represent the order or limit the types of "first", "second" and "third".

[0103] In the description of the present embodiments, "exemplary," "for example," or "e.g." is used to indicate an example, an implementation, or the like. Any embodiment or design presented as "exemplary," "for example," or "e.g." in the present embodiments should not be construed as preferred or advantageous over other embodiments or designs. Rather, use of "exemplary," "for example," or "e.g." is merely intended to present concepts in a concrete manner.

[0104] In the description of the present embodiments, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in the text merely indicates a relationship between associated objects, which means that there can be three relationships, for example, A and / or B, which means that there are three cases of A alone, A and B together, and B alone. In addition, in the description of the present embodiments, "multiple" means two or more than two.

[0105] In some of the processes described in the present embodiments, a plurality of operations or steps are included in a specific order, but it should be understood that these operations or steps can be executed or performed in parallel or in an order different from that in which they appear in the present embodiments. The serial number of the operation is only used to distinguish different operations, and the serial number itself does not represent any execution order. In addition, these processes can include more or fewer operations, and these operations or steps can be executed in sequence or in parallel, and these operations or steps can be combined.

[0106] From the above description of the embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment method can be realized by means of software and a necessary general hardware platform, of course, it can also be realized by hardware, but in many cases the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a storage medium (such as a ROM / RAM, a magnetic disk, an optical disk) as described above, and includes a plurality of instructions for causing a terminal device to execute the methods described in the various embodiments of the present application.

[0107] The above is merely a specific implementation of the present embodiments, but the protection scope of the present embodiments is not limited thereto. Any person skilled in the art can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present embodiments, and these modifications or replacements should be covered within the protection scope of the present embodiments. Therefore, the protection scope of the present embodiments should be subject to the protection scope of the claims.

Claims

1. A medical information retrieval method based on multimodal vectorization, characterized in that, The method includes the following steps: After constructing a corpus and performing domain-adaptive fine-tuning, a domain-optimized recall model is obtained. After encoding different types of data according to the domain-optimized recall model, the corresponding feature vectors are obtained. The feature vectors are then used to generate hybrid modality vectors through cross-modal attention mechanism and gating fusion mechanism. The retrieved text is sliced: a structured summary and keywords are generated; a parent-child node slicing structure is constructed, where the parent node stores the structured summary and keywords as global semantic anchors, and the child nodes slice the retrieved text and associate it with the parent node ID; Rewriting the retrieved information: Rewriting and completing the retrieved information, disambiguating pronouns, and conducting multiple rounds of follow-up questions and judgments; After the rewritten information to be retrieved is vectorized, the retrieval is performed based on the hybrid modality vector and the parent-child node slice structure.

2. The medical information retrieval method based on multimodal vectorization as described in claim 1, characterized in that: The fine-tuning objective function for the domain adaptive fine-tuning is a dual-task loss function: the main task is the contrastive learning loss, and the auxiliary task is the triplet loss. Comparative learning loss The calculation formula is: in The cosine similarity function is used. For positive sample pairs vectors, For negative sample vectors; Triple loss The calculation formula is: in For anchor vectors, For positive sample vectors, For negative sample vectors, It is a boundary parameter and takes the value 0.5; The total loss function L is: , These are the weighting coefficients.

3. The medical information retrieval method based on multimodal vectorization as described in claim 1, characterized in that: The process of encoding different types of data includes: A domain-optimized recall model is used to encode text information. Feature extraction from image information; Standardizing numerical data x yields : in The mean, As the standard.

4. The medical information retrieval method based on multimodal vectorization as described in claim 1, characterized in that, The process of generating hybrid modality vectors from feature vectors through cross-modal attention and gating fusion mechanisms includes: After performing layer normalization on the feature vectors, we obtain standardized feature vectors; Calculate the attention weight matrix reflecting the importance of features based on the standardized feature vectors; Calculate the dynamic weight coefficients of the standardized feature vector; The fusion weights are determined based on the attention weight matrix and dynamic weight coefficients, and the hybrid modality vector is generated based on the fusion weights and the corresponding normalized vectors.

5. The medical information retrieval method based on multimodal vectorization as described in claim 4, characterized in that: The feature vector includes text feature vectors. Image feature vectors and structured feature vectors The normalization process yields the corresponding standardized feature vector as follows: The normalization calculation formula is as follows: in The characteristic mean, The characteristic standard deviation, The smoothing coefficient and ; The attention weight matrix is ​​as follows: in , , For a trainable parameter matrix, For the attention dimension; The dynamic weighting coefficient is controlled by a gating unit. calculate: in For query vector, Dynamic weighting coefficients , , ; The fusion weight The calculation formula is: The mixed mode vector The calculation formula is: 。 6. The medical information retrieval method based on multimodal vectorization as described in claim 1, characterized in that, The keyword generation process includes: Calculate word weight : in For word frequency, , Total number of documents This represents the number of documents containing the word. Calculate word scores : in The damping coefficient is 0.

85. For words and The co-occurrence weights.

7. The medical information retrieval method based on multimodal vectorization as described in claim 1, characterized in that, The construction process of the parent-child node slice structure includes: Calculate sentence similarity: s m ,s j This indicates two sentences, v m , v j They are s m ,s j The corresponding vector representation; A preset similarity threshold is used. When the similarity of a specified number of sentences is below the threshold, the sentence is sliced ​​to generate child nodes.

8. The medical information retrieval method based on multimodal vectorization as described in any one of claims 1 to 7, characterized in that: The process of rewriting and completing the retrieved information includes: Calculate the edit distance between the query string s to be retrieved and the term string t that matches s in the knowledge graph, and then calculate the edit distance. Calculate the similarity between s and t. : in Indicates taking and The maximum value of both lengths; judge If the specified numerical condition is met, replace s with t; otherwise, express s in a fuzzy manner. The process of performing the aforementioned referential disambiguation on the retrieved information includes: For each entity e in the entity construction list in the session history, calculate the referential probability P of the word pron that needs to be replaced: Represents the m-th entity in the entity construction list; pron vec , They are pron and Vector representation of; When P reaches its maximum value When the specified substitution threshold is exceeded, pron is replaced with The corresponding entity; The process of conducting multiple rounds of follow-up questions to determine the retrieved information includes: Identify the frequency f of conjunctions in the information to be retrieved, and the vector representation q of the current information to be retrieved. curr,vec The vector representation q of the previous query prev,vec Relevance R: When f is above the frequency threshold and R is above the relevance threshold, the current information to be retrieved is determined to be a follow-up query to the previous query. The current information to be retrieved, the context information related to the current information to be retrieved, and the summary of the previous query results are concatenated to generate the rewritten information to be retrieved.

9. A medical information retrieval system based on multimodal vectorization, characterized in that, The system includes: The recall model fine-tuning module is used to: implement the process of constructing a corpus and performing domain adaptive fine-tuning as described in any one of claims 1 to 8 to obtain a domain-optimized recall model; The multimodal and hybrid modal vectorization module is used to: implement the process described in any one of claims 1 to 8, which encodes different types of data according to the domain-optimized recall model to obtain corresponding feature vectors, and generates hybrid modal vectors from the feature vectors through a cross-modal attention mechanism and a gating fusion mechanism; The document slicing module is used to: implement the slicing process of the retrieved original text as described in any one of claims 1 to 8: generate a structured summary and keywords of the retrieved original text; construct a parent-child node slicing structure, wherein the parent node stores the structured summary and keywords as global semantic anchors, and the child node segments the retrieved original text and associates it with the parent node ID; The intelligent search term rewriting module is used to: rewrite the information to be retrieved as described in any one of claims 1 to 8; rewrite and complete the information to be retrieved, perform disambiguation of pronouns, and perform multi-round follow-up questioning judgment; vectorize the rewritten information to be retrieved, and then perform retrieval based on the hybrid modality vector and the parent-child node slice structure.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a medical information retrieval program based on multimodal vectorization, wherein when the medical information retrieval program based on multimodal vectorization is executed, it implements the steps of the medical information retrieval method based on multimodal vectorization as described in any one of claims 1 to 8.

Images