A urology question-answering method based on clinical data and magnetic resonance imaging analysis

Abstract

The present invention provides a urology question-answering method based on clinical data and nuclear magnetic resonance imaging analysis, which relates to the field of natural language processing. The present application vectorizes the clinical analysis data related to urology to construct a urology clinical analysis data vector library; vectorizes the summary of the urology clinical analysis data to obtain a summary vector; matches the nuclear magnetic resonance imaging provided to the patient with the nuclear magnetic resonance imaging analysis report through comparative learning, and textualizes the patient's condition reflected by the nuclear magnetic resonance imaging; combines the textualized content with the patient's clinical test data and vectorizes it to obtain a query reference vector; the first system prompt word control language model generates a query based on the summary vector and the query reference vector; screens the urology clinical analysis data with the largest similarity score to the query from the urology clinical analysis data vector library; and the second system prompt word control language model generates a corresponding query response result based on the query and the matched urology clinical analysis data.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a urology question-answering method based on clinical data and nuclear magnetic resonance imaging analysis. Background Art

[0002] During the medical treatment of urinary system diseases, doctors will make detailed records of patients' clinical data and analysis, which are generally used as references for patients' long-term treatment. These data are extremely valuable resources. With the development of artificial intelligence in the field of natural language processing, such as the emergence of large language models such as GPT Llama BERT, it has become possible to use language models to process and analyze clinical data and analyze data. At present, the application of language models in the medical field is multifaceted, such as: extracting important information such as diagnosis, treatment plan and patient medical history from doctors' notes, case reports and clinical records. Compressing long medical documents into concise summaries helps doctors quickly understand the patient's condition. Using natural language processing technology, patients' symptom information is collected in the form of dialogue to provide patients with preliminary diagnosis suggestions and health guidance. Among them, preliminary diagnosis and health guidance have the greatest value. However, training language models dedicated to the medical field based on medical data is costly and difficult, and it is not practical for hospitals to implement it. Therefore, generally based on general language models, enhanced retrieval is performed to achieve accurate medical questions and answers, but existing enhanced retrieval is difficult to process patients' image data. Summary of the invention

[0003] In order to solve the above technical problems or at least partially solve the above technical problems, the present invention provides a urology question-answering method based on clinical data and magnetic resonance imaging analysis.

[0004] In a first aspect, the present invention provides a urology question-answering method based on clinical data and magnetic resonance imaging analysis, comprising:

[0005] Collect clinical analysis data related to urology, vectorize the tables and texts, and build a vector library of clinical analysis data for urology;

[0006] The urology clinical analysis data is summarized through the language model, and the summary is vectorized to obtain a summary vector that the language model can understand.

[0007] In the application process, the MRI provided for the patient is matched with the corresponding MRI analysis report through comparative learning, and the patient's condition reflected by the MRI is textualized;

[0008] After the patient condition reflected by the MRI is textualized, the textualized content is combined with the patient's clinical test data to obtain the patient's clinical data, and the combined patient clinical data is vectorized to obtain a query reference vector that can be understood by the language model;

[0009] The first system prompt word controls the language model to generate a query based on the scope of the summary vector and the situation reflected by the query reference vector content; the query is a question that the patient is concerned about, the answer scope of the question is the scope of the summary vector, and the clinical data in the generated question comes from the query reference vector content;

[0010] Performing similarity analysis on the content and query of the urology clinical analysis data vector library, and selecting a set number of urology clinical analysis data with the largest similarity scores from the urology clinical analysis data vector library;

[0011] The second system prompt word controls the language model to generate a response result corresponding to the query based on the query and the matched urology clinical analysis data.

[0012] Furthermore, the clinical analysis data related to urology are collected, the tables and texts in the clinical analysis data related to urology are vectorized, and the construction of the urology clinical analysis data vector library includes: under the guidance of medical experts, the medical test parameters in the clinical analysis data are graded according to the health indicators of each medical test parameter; tables and titles are extracted from the clinical analysis data, and the association between the tables and titles is established, and after merging the tables and titles, the tables are stored in the dataframe format, and the table index is configured; texts and page numbers are extracted from the clinical analysis data, and the texts are segmented according to the preset segment length to obtain clinical analysis data segments, and the page numbers involved in the text in each clinical analysis data segment are recorded;

[0013] Building metadata of the clinical analysis data slices and the processed table based on the table index and the clinical analysis data slice associated page number list of the clinical analysis data slices, so as to locate the table and the clinical analysis data slices according to the metadata;

[0014] The tables and clinical analysis data are segmented and encoded through the embedding model to realize the vectorization of clinical analysis data.

[0015] Furthermore, tables and titles are extracted from the clinical analysis data, and the association between the tables and titles is established. After merging the tables and titles, the tables are stored in dataframe format, and the table indexes are configured including:

[0016] Traverse each page of clinical analysis data, and for each traversed page, iterate and perform the following process:

[0017] Obtain the elements in each page of clinical analysis data. If the element is a text container, extract the text from the text container; search chapter codes, abstract codes, and table titles by regular expressions to preliminarily screen texts that may contain titles, and store the matched text, the page number where the text is located, and the location information of the element where the text is located in a possible title list;

[0018] Traverse the text in the possible title list, and further remove the chapter code, summary code, and table code before and after the title through regular expressions to obtain the title to replace the text in the possible title list;

[0019] Get the table in each page of clinical data, get the top coordinates of the table and the page number where the table is located;

[0020] For each table, record the page number and the distance between all titles and the table in the previous page. If they are on the same page, the distance is the position difference between the position of the element where the title is located and the coordinates of the top of the table. If they are on the previous page, the distance is the sum of the position difference between the position of the element where the title is located and the coordinates of the top of the table and the page height.

[0021] Pair and associate the closest titles with the tables to obtain all paired titles and tables;

[0022] Based on the pairing results of the title and table, tables with the same title are merged and the title is added to the table. The data is stored in the dataframe format and the table index is configured.

[0023] Furthermore, the text and page numbers are extracted from the clinical analysis data, and the text is segmented according to the preset segment length to obtain clinical analysis data segments, and the page numbers involved in the text in each clinical analysis data segment are recorded. The process includes: traversing each page of the clinical analysis data, and for each traversed page, iteratively executing the following process:

[0024] Get the elements in each page of clinical analysis data. If the element is a text container, extract the text from the text container and split it into paragraphs.

[0025] For each extracted text paragraph, remove the extra symbols and add it to the temporary text string variable, and save the page number of the text paragraph in the temporary page number list;

[0026] In the temporary text string variable, if the accumulated length of the text paragraphs reaches the segment length and is semantically complete, the content in the text string variable is added to a clinical analysis data segment;

[0027] According to the temporary page number list, the page numbers of all text paragraphs in the text string variable are added to the clinical data segment associated page number list corresponding to the clinical analysis data segment, and then the temporary text string variable and the temporary page number list are updated;

[0028] If the characters of the last remaining text paragraph of the clinical analysis data still cannot reach the segment length after being added to the text string variable, the remaining text paragraphs in the text string variable are added to a clinical data segment; based on the temporary page number list, the page numbers of all text paragraphs in the text string variable are added to the clinical analysis data segment associated page number list corresponding to the clinical data segment.

[0029] Furthermore, the image features and text features of the MRI and the MRI analysis report are extracted respectively by an image feature encoder and a text feature encoder; the image feature encoder and the text feature encoder are trained with the goal of maximizing the similarity between the image features and the text features between the positive pairs and minimizing the similarity between the image features and the text features between the negative pairs.

[0030] The loss function in the contrastive learning process is:

[0031] ;

[0032] Where B is the number of MRIs and MRI analysis reports in the contrastive learning training set, To find the cosine similarity function between variables a and b, is the exponential function of the natural logarithm, and is the MRI features extracted by the image feature encoder, and The text features extracted by the text feature encoder, and For the positive, and For negative pair, is the learnable temperature coefficient.

[0033] Furthermore, a text feature encoder is constructed based on the pre-trained Bio_ClinicalBERT model; a fully connected layer is added to the Bio_ClinicalBERT model to form a text encoder, and compared during learning and training;

[0034] Build an image feature encoder based on the ModifiedResNet model.

[0035] Furthermore, the MRI analysis reports in the data set are enhanced through the existing language model, and the medical-related parameters in the MRI analysis reports are treated as immutable data. Through the existing language model, different forms of textual expressions of the MRI analysis reports are expanded without changing the medical-related parameters in the MRI analysis reports and the semantics of the reports.

[0036] Furthermore, the urology MRI images are enhanced by random flipping, random affine transformation, random addition of Gaussian noise, and random contrast perturbation.

[0037] In a second aspect, the present invention provides a urology question and answer device based on clinical data and magnetic resonance imaging analysis, comprising: at least one processing unit, the processing unit and a storage unit are interconnected by a bus unit, the storage unit stores a computer program, and when the computer program is executed by the processing unit, the urology question and answer method based on clinical data and magnetic resonance imaging analysis is implemented.

[0038] In a third aspect, the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the urology question-and-answer method based on clinical data and magnetic resonance imaging analysis.

[0039] The above technical solution provided by the embodiment of the present invention has the following advantages compared with the prior art:

[0040] The present application vectorizes clinical analysis data related to urology to construct a urology clinical analysis data vector library; vectorizes the summary of urology clinical analysis data to obtain a summary vector, which summarizes the scope of the urology clinical analysis data vector library with less data, and the language model generates a query reference vector with the summary vector rather than the urology clinical analysis data vector library as a prompt, and the amount of data processing is small; the MRI provided to the patient is matched with the MRI analysis report through comparative learning, and the patient's condition reflected by the MRI is textualized; the patient's condition reflected by the MRI is textualized through comparative learning, and the textualized content can be understood by the language model after being quantized; the textualized content is combined with the patient's clinical test data and vectorized to obtain a query reference vector; the first system prompts The word control language model generates a query based on the summary vector and the query reference vector; the query is a question that the patient is concerned about, the answer range of the question is the range of the summary vector, and the clinical data in the generated question comes from the query reference vector content; in this way, the content related to the urology clinical analysis data vector library and the patient's clinical data is fed back to the patient in the form of a query in a faster way, and the urology clinical analysis data with the largest similarity score to the query is screened from the urology clinical analysis data vector library; the patient can choose the content he is concerned about as the target query, and the second system prompts the word control language model to generate the corresponding query response result based on the query and the matched urology clinical analysis data, so that the patient can obtain the analysis intelligence that meets his own situation through this application without the participation of the doctor. This application realizes question and answer through enhanced retrieval using a general language model. This application supports providing corresponding analysis content and medical consultation through clinical data such as patient images and text. It is also conducive to minimizing the information gap between doctors and patients as much as possible, assisting patients to independently master certain medical knowledge, and avoiding conflicts between doctors and patients due to information differences. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0042] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0043] Figure 1 A flowchart of a urology question-answering method based on clinical data and magnetic resonance imaging analysis provided by an embodiment of the present invention;

[0044] Figure 2 A flowchart of vectorizing tables and texts in clinical analysis data provided by an embodiment of the present invention to obtain vector representations that can be understood by a language model and to construct a vector library of urology clinical analysis data;

[0045] Figure 3 A flowchart of extracting tables and titles from clinical analysis data, establishing associations between tables and titles, merging tables and titles, storing tables in dataframe format, and configuring table indexes provided by an embodiment of the present invention;

[0046] Figure 4 A flowchart of extracting text and page numbers from clinical analysis data provided by an embodiment of the present invention, and slicing the text according to a preset slicing length to obtain clinical analysis data slicing, and recording the page numbers involved in the text in each clinical analysis data slicing;

[0047] Figure 5 A comparative learning architecture diagram provided for an embodiment of the present invention;

[0048] Figure 6 A schematic diagram of a urology question-and-answer device based on clinical data and magnetic resonance imaging analysis provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0051] Example 1

[0052] like Figure 1 As shown, the technology of the present invention implements a urology question-answering method based on clinical data and nuclear magnetic resonance imaging analysis, and the present invention includes the following steps:

[0053] Step 1: Collect clinical analysis data related to urology, vectorize the tables and text in the clinical analysis data, obtain vector representations that can be understood by the language model, and build a vector library of urology clinical analysis data. The clinical analysis data collects the analysis content of the doctor's condition at each stage based on a large number of long-term clinical data of patients: including the condition at each stage and precautions at each stage.

[0054] In the specific implementation process, Figure 2 As shown, step 1 includes:

[0055] Under the guidance of medical experts, the medical test parameters of clinical data in the clinical analysis data are graded according to the health indicators of each medical test parameter. For example, the values ​​of creatinine content in the blood are divided into the following ranges: normal creatinine content in the blood, low creatinine content in the blood, extremely low creatinine content in the blood, high creatinine content in the blood, and extremely high creatinine content in the blood. Grading converts the numerical parameters in the clinical data into text that is easy for the language model to understand, avoiding the weakness of the language model's poor ability to understand numbers.

[0056] Extract tables and titles from clinical analysis data, establish associations between tables and titles, merge tables and titles, store tables in dataframe format, and configure table indexes. Figure 3 As shown, the process includes:

[0057] Traverse each page of clinical analysis data, and for each traversed page, iterate and perform the following process:

[0058] Get the elements in each page of clinical analysis data. If the element is a text container, extract the text from the text container. Use regular expressions to search chapter codes, abstract codes, and table titles to preliminarily screen text that may contain titles, and store the matched text, the page number where the text is located, and the location information of the element where the text is located in the possible title list. Traverse the text in the possible title list, and further use regular expressions to remove the chapter codes, abstract codes, and table codes before and after the title to obtain the title to replace the text in the possible title list.

[0059] Get the table in each page of clinical analysis data, and get the top coordinates of the table and the page number where the table is located.

[0060] For each table, record the page number and the distance between all titles and the table in the previous page. When on the same page, the distance is the difference between the position of the element where the title is located and the coordinates of the top of the table. When on the previous page, the distance is the sum of the difference between the position of the element where the title is located and the coordinates of the top of the table and the page height.

[0061] Pair and associate the closest titles with the tables to obtain all paired titles and tables.

[0062] Based on the pairing results of the title and table, tables with the same title are merged and the title is added to the table. The data is stored in the dataframe format and the table index is configured.

[0063] Extract text and page numbers from clinical analysis data, and segment the text according to the preset segment length to obtain clinical analysis data segments, and record the page numbers involved in the text in each clinical analysis data segment. Figure 4 As shown, the process includes:

[0064] Traverse each page of clinical analysis data, and for each traversed page, iterate and perform the following process:

[0065] Get the elements in each page of clinical analysis data. If the element is a text container, extract the text from the text container and split it into paragraphs. For each extracted text paragraph, remove the redundant symbols and add it to the temporary text string variable, and save the page number of the text paragraph in the temporary page number list. In the temporary text string variable, if the cumulative length of the text paragraph reaches the segment length and the semantics are complete, add the content in the text string variable to a clinical analysis data segment. According to the temporary page number list, add the page numbers of all text paragraphs in the text string variable to the clinical analysis data segment associated page number list corresponding to the clinical analysis data segment. Then update the temporary text string variable and the temporary page number list.

[0066] If the characters of the last remaining text paragraph of the clinical analysis data still cannot reach the segment length after being added to the text string variable, the remaining text paragraphs in the text string variable are added to a clinical analysis data segment; according to the temporary page number list, the page numbers of all text paragraphs in the text string variable are added to the clinical analysis data segment associated page number list corresponding to the clinical analysis data segment.

[0067] Metadata of the clinical data slices and the processed table are constructed based on the table index and the clinical analysis data slice associated page number list of the clinical analysis data slices, so as to locate the table and the clinical analysis data slice according to the metadata.

[0068] The table and clinical analysis data slices are encoded through an embedding model to realize the vectorization of clinical analysis data. An exemplary embedding model uses the mxbai-embed-large-v1 embedding model.

[0069] This application is designed for situations where clinical analysis data contains a large amount of tables and text content, and provides an effective table and text vectorization method.

[0070] Step 2: Summarize the urology clinical analysis data in advance through the language model, vectorize the summary of the urology-related clinical analysis data, and obtain a summary vector that the language model can understand. In this application, for a large amount of clinical analysis data, the main analysis topics or key points can be summarized in the form of summaries. In this way, in the process of generating queries, the scope is determined by the summary vector rather than the entire urology clinical analysis data vector library, which greatly reduces the amount of data to be processed and speeds up the processing. The embedding model used for summary vectorization is consistent with the embedding model used for clinical analysis data. An exemplary embedding model uses the mxbai-embed-large-v1 embedding model.

[0071] Step 3: During the application process, the MRI provided to the patient is matched with the corresponding MRI analysis report through comparative learning, and the patient's condition reflected by the MRI is textualized.

[0072] In the specific implementation process, the present application constructs a data set including positive and negative pairs of nuclear magnetic resonance imaging and nuclear magnetic resonance imaging analysis reports. The nuclear magnetic resonance imaging and nuclear magnetic resonance imaging analysis reports in the positive pairs match, and the nuclear magnetic resonance imaging and nuclear magnetic resonance imaging analysis reports in the negative pairs do not match.

[0073] The MRI analysis reports in the data set are enhanced through existing language models. In the specific implementation process, the medical-related parameters in the MRI analysis report are taken as immutable data, and different forms of text expressions of the MRI analysis report are expanded through existing language models, such as the ChatGPT model, without changing the medical-related parameters in the MRI analysis report and the semantics of the report.

[0074] The urology magnetic resonance images are enhanced by random flipping, random affine transformation, random addition of Gaussian noise, and random contrast perturbation.

[0075] Establishing the link between MRIs and MRI analysis reports via a contrastive learning architecture.

[0076] like Figure 5 As shown, the contrastive learning architecture contains an image feature encoder and a text feature encoder.

[0077] A text feature encoder is constructed based on the pre-trained Bio_ClinicalBERT model; the Bio_ClinicalBERT model is a model applied to the biological clinical neighborhood based on the BERT language model, and has a strong ability to understand the content of the biological clinical neighborhood. The Bio_ClinicalBERT model contains 12 Transformer layers. A fully connected layer is added after the Bio_ClinicalBERT model to form a text encoder. The fully connected layer after comparative learning training maps the features output by the Bio_ClinicalBERT model into text features to be compared.

[0078] Build an image feature encoder based on the ModifiedResNet model.

[0079] The image features and text features of the magnetic resonance imaging and magnetic resonance imaging analysis report are extracted by the image feature encoder and the text feature encoder respectively; the image feature encoder and the text feature encoder are trained with the goal of maximizing the similarity of the image features and text features between the positive pairs and minimizing the similarity of the image features and text features between the negative pairs. During the comparative learning process, the parameters of the Bio_ClinicalBERT model are frozen, and the parameters of the fully connected layer and the ModifiedResNet model are adjusted. The image encoder adopts the Transformer Unet model pre-trained for urology. TransformerUnet model is a neural network based on Transformer and UNet for medical image analysis.

[0080] The loss function in the contrastive learning process is:

[0081] ;

[0082] Where B is the number of MRIs and MRI analysis reports in the contrastive learning training set, To find the cosine similarity function between variables a and b, is the exponential function of the natural logarithm, and is the MRI features extracted by the image feature encoder, and The text features extracted by the text feature encoder, and For the positive, and For negative pair, is the learnable temperature coefficient.

[0083] Step 4: After the patient condition reflected by the MRI is textualized, the textualized content is combined with the patient's clinical test data to obtain the patient's clinical data, and the patient's clinical data is vectorized to obtain a query reference vector that the language model can understand. The embedding model used for vectorization of patient clinical data is consistent with the embedding model used for clinical analysis data.

[0084] Step 5, the first system prompt word uses the query reference vector content as the first prompt and the summary vector as the second prompt; the first system prompt word controls the language model to generate a query based on the scope of the second prompt and the situation reflected by the first prompt. The query is a question that the patient is concerned about, the answer scope of the question is the scope of the second prompt, and the clinical data in the generated question comes from the first prompt. For example, the clinical data is a high creatinine content in the blood, and within the scope of the summary vector, there are dialysis time recommendations and dietary precautions for dialysis patients with high blood creatinine content... According to the summary vector scope and clinical data, the query generated by the language model is "How long does it take for dialysis patients to undergo dialysis when their blood creatinine content is high?", "What should be paid attention to in the diet when the blood creatinine content of dialysis patients is high?",... Here, the language ability of the language model is used to combine the key content within the scope of the second prompt with the clinical data of the first prompt to obtain a query.

[0085] Step 6, performing similarity analysis on the content of the urology clinical analysis data vector library and the query, and selecting the urology clinical analysis data with the largest similarity score from the urology clinical analysis data vector library.

[0086] Step 7: The second system prompt word uses the query and the urology clinical analysis data as the third prompt, and the second system prompt word controls the language model to generate a corresponding query response result based on the third prompt. The response result includes the analysis of the corresponding patient clinical data.

[0087] Example 2

[0088] See also Figure 6 As shown, an embodiment of the present invention provides a urology question-and-answer device based on clinical data and nuclear magnetic resonance imaging analysis, including: at least one processing unit, the processing unit and the storage unit are interconnected through a bus unit, and the storage unit is a computer-readable storage medium that can be used to store software programs, computer executable programs, and modules, such as the software programs, computer executable programs, and modules corresponding to a urology question-and-answer method based on clinical data and nuclear magnetic resonance imaging analysis in an embodiment of the present invention. The processing unit implements the above-mentioned urology question-and-answer method based on clinical data and nuclear magnetic resonance imaging analysis by running the software programs, computer executable programs, and modules stored in the storage unit, including:

[0089] Collect clinical analysis data related to urology, vectorize the tables and texts, and build a vector library of clinical analysis data for urology;

[0090] The urology clinical analysis data is summarized through the language model, and the summary is vectorized to obtain a summary vector that the language model can understand.

[0091] In the application process, the MRI provided for the patient is matched with the corresponding MRI analysis report through comparative learning, and the patient's condition reflected by the MRI is textualized;

[0092] After the patient condition reflected by the MRI is textualized, the textualized content is combined with the patient's clinical test data to obtain the patient's clinical data, and the combined patient clinical data is vectorized to obtain a query reference vector that can be understood by the language model;

[0093] The first system prompt word controls the language model to generate a query based on the scope of the summary vector and the situation reflected by the query reference vector content; the query is a question that the patient is concerned about, the answer scope of the question is the scope of the summary vector, and the clinical data in the generated question comes from the query reference vector content;

[0094] Performing similarity analysis on the content and query of the urology clinical analysis data vector library, and selecting a set number of urology clinical analysis data with the largest similarity scores from the urology clinical analysis data vector library;

[0095] The second system prompt word controls the language model to generate a response result corresponding to the query based on the query and the matched urology clinical analysis data.

[0096] Of course, the computer program stored in the storage unit of the urology question and answer device based on clinical data and magnetic resonance imaging analysis provided by an embodiment of the present invention is not limited to the method operations described above, and can also execute related operations in the urology question and answer method based on clinical data and magnetic resonance imaging analysis provided by any embodiment of the present invention.

[0097] Example 3

[0098] An embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program. When the computer program is executed, the urology question-answering method based on clinical data and magnetic resonance imaging analysis is implemented, including:

[0099] Collect clinical analysis data related to urology, vectorize the tables and texts, and build a vector library of clinical analysis data for urology;

[0100] The urology clinical analysis data is summarized through the language model, and the summary is vectorized to obtain a summary vector that the language model can understand.

[0101] In the application process, the MRI provided for the patient is matched with the corresponding MRI analysis report through comparative learning, and the patient's condition reflected by the MRI is textualized;

[0102] After the patient condition reflected by the MRI is textualized, the textualized content is combined with the patient's clinical test data to obtain the patient's clinical data, and the combined patient clinical data is vectorized to obtain a query reference vector that can be understood by the language model;

[0103] The first system prompt word controls the language model to generate a query based on the scope of the summary vector and the situation reflected by the query reference vector content; the query is a question that the patient is concerned about, the answer scope of the question is the scope of the summary vector, and the clinical data in the generated question comes from the query reference vector content;

[0104] Performing similarity analysis on the content and query of the urology clinical analysis data vector library, and selecting a set number of urology clinical analysis data with the largest similarity scores from the urology clinical analysis data vector library;

[0105] The second system prompt word controls the language model to generate a response result corresponding to the query based on the query and the matched urology clinical analysis data.

[0106] A computer-readable storage medium provided in an embodiment of the present invention stores a computer program which is not limited to the method operations described above, but can also execute related operations in a urology question-and-answer method based on clinical data and magnetic resonance imaging analysis provided in any embodiment of the present invention.

[0107] In the embodiments provided by the present invention, it should be understood that the disclosed structures and methods can be implemented in other ways. For example, the structural embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, structures or units, which can be electrical, mechanical or other forms.

[0108] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0109] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0110] The foregoing is merely a specific embodiment of the present invention, which enables those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A urology question-answering method based on clinical data and magnetic resonance imaging analysis, characterized in that: include: Collect clinical analysis data related to urology, vectorize the tables and texts, and build a vector library of clinical analysis data for urology; The urology clinical analysis data is summarized through the language model. For the clinical analysis data, the main analysis topics or key points are summarized through the summary, and the summary is vectorized to obtain the summary vector that the language model can understand. Pre-construct a data set containing positive and negative pairs of MRIs and MRI analysis reports, where the MRIs in the positive pairs match the MRI analysis reports, and the MRIs in the negative pairs do not match the MRI analysis reports; establish a connection between the MRIs and the MRI analysis reports through a contrastive learning architecture; in the application process, match the MRIs provided to the patients with the corresponding MRI analysis reports through contrastive learning, and textualize the patient conditions reflected by the MRIs; After the patient condition reflected by the MRI is textualized, the textualized content is combined with the patient's clinical test data to obtain the patient's clinical data, and the combined patient clinical data is vectorized to obtain a query reference vector that can be understood by the language model; The first system prompt word uses the query reference vector content as the first prompt and the summary vector as the second prompt; The first system prompt word controls the language model to generate a query based on the scope of the second prompt and the situation reflected by the first prompt; The query is a question that the patient is concerned about, the scope of the answer to the question is the scope of the second prompt, and the clinical data in the generated question comes from the first prompt; Performing similarity analysis on the content and query of the urology clinical analysis data vector library, and selecting a set number of urology clinical analysis data with the largest similarity scores from the urology clinical analysis data vector library; The second system prompt word uses the queried and matched urology clinical analysis data as the third prompt, and the second system prompt word controls the language model to generate a response result corresponding to the query based on the third prompt, and the response result includes an analysis of the corresponding patient clinical data.

2. The urology question-answering method based on clinical data and magnetic resonance imaging analysis according to claim 1, characterized in that: Collect clinical analysis data related to urology, vectorize tables and texts in clinical analysis data related to urology, and build a vector library of clinical analysis data for urology, including: Under the guidance of medical experts, classify the medical test parameters in the clinical analysis data according to the health indicators of each medical test parameter; extract tables and titles from the clinical analysis data, and establish the association between the tables and titles. After merging the tables and titles, store the tables in dataframe format and configure the table index; extract text and page numbers from the clinical analysis data, and segment the text according to the preset segment length to obtain clinical analysis data segments, and record the page numbers involved in the text in each clinical analysis data segment; Building metadata of the clinical analysis data slices and the processed table based on the table index and the clinical analysis data slice associated page number list of the clinical analysis data slices, so as to locate the table and the clinical analysis data slices according to the metadata; The tables and clinical analysis data are segmented and encoded through the embedding model to realize the vectorization of clinical analysis data.

3. The urology question-answering method based on clinical data and magnetic resonance imaging analysis according to claim 2, characterized in that: Extract tables and titles from clinical analysis data, establish associations between tables and titles, merge tables and titles, store tables in dataframe format, and configure table indexes including: Traverse each page of clinical analysis data, and for each traversed page, iterate and perform the following process: Obtain the elements in each page of clinical analysis data. If the element is a text container, extract the text from the text container; search chapter codes, abstract codes, and table titles by regular expressions to preliminarily screen texts that may contain titles, and store the matched text, the page number where the text is located, and the location information of the element where the text is located in a possible title list; Traverse the text in the possible title list, and further remove the chapter code, summary code, and table code before and after the title through regular expressions to obtain the title to replace the text in the possible title list; Get the table in each page of clinical data, get the top coordinates of the table and the page number where the table is located; For each table, record the page number and the distance between all titles and the table in the previous page. If they are on the same page, the distance is the position difference between the position of the element where the title is located and the coordinates of the top of the table. If they are on the previous page, the distance is the sum of the position difference between the position of the element where the title is located and the coordinates of the top of the table and the page height. Pair and associate the closest titles with the tables to obtain all paired titles and tables; Based on the pairing results of the title and table, tables with the same title are merged and the title is added to the table. The data is stored in the dataframe format and the table index is configured.

4. The urology question-answering method based on clinical data and magnetic resonance imaging analysis according to claim 2, characterized in that: Extract text and page numbers from clinical analysis data, and segment the text according to a preset segment length to obtain clinical analysis data segments, and record the page numbers involved in the text in each clinical analysis data segment. The process includes: traversing each page of the clinical analysis data, and for each traversed page, iteratively executing the following process: Get the elements in each page of clinical analysis data. If the element is a text container, extract the text from the text container and split it into paragraphs. For each extracted text paragraph, remove the extra symbols and add it to the temporary text string variable, and save the page number of the text paragraph in the temporary page number list; In the temporary text string variable, if the accumulated length of the text paragraphs reaches the segment length and is semantically complete, the content in the text string variable is added to a clinical analysis data segment; According to the temporary page number list, the page numbers of all text paragraphs in the text string variable are added to the clinical data segment associated page number list corresponding to the clinical analysis data segment, and then the temporary text string variable and the temporary page number list are updated; If the characters of the last remaining text paragraph of the clinical analysis data still cannot reach the segment length after being added to the text string variable, the remaining text paragraphs in the text string variable are added to a clinical data segment; based on the temporary page number list, the page numbers of all text paragraphs in the text string variable are added to the clinical analysis data segment associated page number list corresponding to the clinical data segment.

5. The urology question-answering method based on clinical data and magnetic resonance imaging analysis according to claim 1, characterized in that: The image features and text features of the MRI and MRI analysis report are extracted by the image feature encoder and the text feature encoder respectively; the image feature encoder and the text feature encoder are trained with the goal of maximizing the similarity between the image features and the text features between the positive pairs and minimizing the similarity between the image features and the text features between the negative pairs. The loss function in the contrastive learning process is: ； Where B is the number of MRIs and MRI analysis reports in the contrastive learning training set, To find the cosine similarity function between variables a and b, is the exponential function of the natural logarithm, and is the MRI features extracted by the image feature encoder, and The text features extracted by the text feature encoder, and For the positive, and For negative pairs, is the learnable temperature coefficient.

6. The urology question-answering method based on clinical data and nuclear magnetic resonance imaging analysis according to claim 5, characterized in that: Build a text feature encoder based on the pre-trained Bio_ClinicalBERT model; Add a fully connected layer to the Bio_ClinicalBERT model to form a text encoder, and compare it during learning and training; Build an image feature encoder based on the ModifiedResNet model.

7. The urology question-answering method based on clinical data and nuclear magnetic resonance imaging analysis according to claim 5, characterized in that: The MRI analysis reports in the dataset are enhanced through the existing language model, and the medical-related parameters in the MRI analysis reports are taken as immutable data. The existing language model is used to expand the MRI analysis reports into different forms of textual expressions without changing the medical-related parameters and semantics in the MRI analysis reports.

8. The urology question-answering method based on clinical data and nuclear magnetic resonance imaging analysis according to claim 5, characterized in that: The urology magnetic resonance images are enhanced by random flipping, random affine transformation, random addition of Gaussian noise, and random contrast perturbation.

9. A urology question-answering device based on clinical data and magnetic resonance imaging analysis, characterized in that: include: At least one processing unit, the processing unit and the storage unit are interconnected by a bus unit, the storage unit stores a computer program, and when the computer program is executed by the processing unit, the urology question-and-answer method based on clinical data and magnetic resonance imaging analysis as described in any one of claims 1-8 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the urology question-answering method based on clinical data and magnetic resonance imaging analysis as described in any one of claims 1 to 8 is implemented.

Images