Diagnostic report generation method and apparatus based on search enhancement, and electronic device

Abstract

This disclosure provides a retrieval-enhanced diagnostic report generation method, apparatus, and electronic device, comprising: generating a preliminary diagnostic report for medical images of a case to be analyzed using a first preset large model; calculating the similarity between the preliminary diagnostic report and pre-stored diagnostic reports in a database to obtain similar cases; using the images of similar cases and their actual diagnostic reports as reference information, and inputting the medical images of the case to be analyzed and the reference information into a second preset large model to generate a final diagnostic report. This achieves a dual-driven diagnosis combining visual and semantic elements, accurately capturing pathological semantic relationships, avoiding the neglect of pathological diagnostic logic due to relying solely on visual features, and improving generalization ability.

Description

Technical Field

[0001] This disclosure relates to the fields of medical image analysis and artificial intelligence-assisted diagnosis, and in particular to a method, apparatus and electronic device for generating diagnostic reports based on retrieval enhancement. Background Technology

[0002] Radiological diagnosis is an important component of medical imaging. Physicians typically rely on the systematic analysis of medical images such as X-rays, CT scans, and MRIs to detect early lesions, determine the type and severity of diseases, and formulate clinical treatment plans accordingly. In this process, radiologists need to extract numerous subtle features from the images, develop a structured diagnostic approach, and ultimately write detailed diagnostic reports. These reports provide crucial guidance for subsequent diagnosis and treatment by clinicians.

[0003] However, there are several problems with manually writing diagnostic reports: First, the diagnostic process is often time-consuming, especially when the number of cases is increasing rapidly, which significantly increases the workload of doctors; second, the quality of diagnosis depends on the doctor's personal experience, and there may be differences between reports from different doctors, resulting in insufficient consistency; third, the imaging data itself is complex, and if it is relied on manual analysis of each case, it is very easy to cause omissions or errors in information.

[0004] In recent years, with the development of deep learning and natural language processing technologies, automated diagnostic report generation has become a research hotspot. The basic idea is to extract features from medical images using deep neural network models and generate diagnostic reports in natural language based on a trained language model. These methods can alleviate the burden on doctors to some extent and improve the standardization of diagnoses. However, existing automated generation methods still face two major problems: firstly, the models tend to overlook the implicit pathological similarities between cases, resulting in a lack of clinical logic in the generated results; secondly, the generated content often contains inaccuracies or "illusions," making the generated reports difficult to use directly in clinical practice.

[0005] To improve the reliability of automatically generated reports, Retrieval-Augmented Generation (RAG) technology has been introduced into medical imaging diagnosis. The core idea of ​​this technology is to retrieve the most relevant information about the current case from a case database before the generation task, and then input this information as additional context into the model. By drawing on diagnostic information from previous cases, the model can better interpret and reason about new cases, thereby generating diagnostic reports that are more consistent with clinical logic. However, a key challenge in applying RAG to medical imaging diagnosis is how to accurately measure the similarity between cases.

[0006] Existing similarity calculation methods can be broadly classified into two categories: (1) Similarity calculation methods based on image features. These methods typically extract image features through convolutional neural networks or visual Transformer models and use these feature vectors to calculate the distance between cases. Although they have a certain ability to distinguish appearance and structure, since many diseases appear similar at the imaging level, it is difficult to accurately distinguish deep pathological differences by relying solely on visual features.

[0007] (2) Retrieval methods based on structured fields. These methods retrieve cases by extracting key fields (such as lesion location and lesion type) from diagnostic reports and comparing them. This method is effective for some specific tasks (such as tumor number statistics), but it is often difficult to cover all information when dealing with free text descriptions or complex pathological conditions, and it relies on manually defined fields, which is not flexible enough.

[0008] Therefore, existing methods have significant shortcomings in the retrieval enhancement generation of medical image diagnosis: they either rely too much on image visual features and fail to capture pathological semantics, or they rely on manually defined fields, resulting in poor generalization of the method. Summary of the Invention

[0009] This disclosure provides a retrieval-enhanced diagnostic report generation method, apparatus, and electronic device to address the problems of existing methods either relying too heavily on image visual features and failing to capture pathological semantics, or relying on manually defined fields, resulting in poor generalization.

[0010] In view of the above problems, in a first aspect, the retrieval-enhanced diagnostic report generation method provided in this disclosure is characterized by comprising: A preliminary diagnostic report is generated from the medical images of the case to be analyzed using the first pre-set large model; Calculate the similarity between the preliminary diagnostic report and pre-stored diagnostic reports in the database to obtain similar cases; The medical images and actual diagnostic reports of the similar cases are used as reference information, and the medical images of the case to be analyzed and the reference information are input into a second preset large model to generate a final diagnostic report.

[0011] Secondly, a diagnostic report generation device based on retrieval enhancement is provided, comprising: The preliminary diagnosis report generation module is used to generate a preliminary diagnosis report for the medical images of the case to be analyzed using a first preset large model; A similar case calculation module is used to calculate the similarity between the preliminary diagnostic report and the diagnostic reports pre-stored in the database to obtain similar cases; The final diagnostic report generation module is used to take the medical images and real diagnostic reports of the similar cases as reference information, and input the medical images of the case to be analyzed and the reference information into a second preset large model to generate a final diagnostic report.

[0012] Thirdly, an electronic device is provided, comprising: a processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor communicates with the memory via the bus, and when the machine-readable instructions are executed by the processor, perform the steps of the retrieval-enhanced diagnostic report generation method as described in the first aspect.

[0013] Fourthly, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, performs the steps of the retrieval-enhanced diagnostic report generation method as described in the first aspect.

[0014] The beneficial effects of the embodiments disclosed herein include: This disclosure provides a method, apparatus, and electronic device for generating diagnostic reports based on enhanced retrieval, comprising: generating a preliminary diagnostic report for medical images of a case to be analyzed using a first preset large model; calculating the similarity between the preliminary diagnostic report and pre-stored diagnostic reports in a database to obtain similar cases; using the images of similar cases and their actual diagnostic reports as reference information, and inputting the medical images of the case to be analyzed and the reference information into a second preset large model to generate a final diagnostic report. Compared with related technologies that rely too heavily on the visual features of images, this method introduces a textual semantic dimension, transforming the visual features of images into structured case semantics. Subsequent similarity matching is based on the diagnostic report, essentially a semantic-level association retrieval rather than simply relying on visual image matching. This achieves a dual-driven diagnosis combining visual and semantic elements, accurately capturing pathological semantic associations and avoiding ignoring pathological diagnostic logic by only looking at visual features. Compared with related technologies that rely too heavily on manually defined fields, the generation of diagnostic reports and similarity matching are both based on the semantic understanding capabilities of the large model, eliminating the need for manually defined fields. This allows it to adapt to different diseases and different diagnostic styles, improving its generalization ability. Attached Figure Description

[0015] Figure 1 One of the flowcharts for a retrieval-enhanced diagnostic report generation method provided in this disclosure embodiment; Figure 2 A second flowchart illustrating a retrieval-enhanced diagnostic report generation method provided in this embodiment of the present disclosure; Figure 3 This is a schematic diagram of a retrieval-enhanced diagnostic report generation device provided in an embodiment of this disclosure. Detailed Implementation

[0016] This disclosure provides a method, apparatus, and electronic device for generating diagnostic reports based on search enhancement. Preferred embodiments of this disclosure are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustrative and explanatory purposes only and are not intended to limit the scope of this disclosure. Furthermore, the embodiments and features described herein can be combined with each other unless otherwise specified.

[0017] This disclosure provides a method for generating diagnostic reports based on retrieval enhancement, such as... Figure 1 As shown, it includes: S101. Use the first preset large model to generate a preliminary diagnostic report for the medical images of the case to be analyzed; S102. Calculate the similarity between the preliminary diagnostic report and the pre-stored diagnostic reports in the database to obtain similar cases; S103. Use medical images and real diagnostic reports of similar cases as reference information, and input the medical images and reference information of the case to be analyzed into the second preset large model to generate the final diagnostic report.

[0018] In this embodiment, a preliminary diagnostic report is generated using a first preset large model, transforming the visual features of the images (such as lesion morphology and location) into structured pathological semantics. Subsequent similarity matching is based on the generated preliminary diagnostic report text, essentially a semantic-level association retrieval rather than simply relying on visual image matching. By filtering similar cases through the generated preliminary diagnostic report similarity, it ensures that the retrieved cases have reference value at the pathological diagnostic level, rather than cases that are only visually similar but have completely different pathological meanings (such as the visual similarity between inflammatory pseudotumors and malignant nodules). It accurately captures pathological semantic associations (e.g., the semantic association between "ground-glass nodules" and "early lung adenocarcinoma"), avoiding the neglect of pathological diagnostic logic due to relying solely on visual features (e.g., lesions with similar visual morphology may correspond to different pathological types). This provides accurate semantic-level reference for the final diagnosis. Finally, the images of the case to be analyzed, images of similar cases, and real diagnostic reports are input into the second preset large model to achieve dual verification of visual features and semantic reference supplementation. This avoids the one-sidedness of relying solely on visual images and corrects semantic deviations in the preliminary diagnosis through semantic reference of real diagnostic reports, so that the final diagnosis takes into account both visual authenticity and pathological semantic accuracy.

[0019] Furthermore, the diagnostic report is a structured or semi-structured text record made by clinicians based on medical images (such as CT and MRI), laboratory tests, and other information regarding the patient's condition. In this embodiment of the disclosure, the actual diagnostic report is a text record manually made by clinicians based on medical images using their medical knowledge; that is, a diagnostic text.

[0020] In addition, medical images can include X-ray images, CT images, or MRI images, etc., and there are no restrictions here.

[0021] In yet another embodiment provided in this disclosure, the method further includes the following steps: Step 1: Collect medical case data covering the target clinical scenario; medical case data includes: medical imaging data and diagnostic reports written by clinicians based on the medical imaging data; Step 2: Input the medical case data and corresponding prompts into the first preset large model to generate a diagnostic report for the corresponding medical case data; Step 3: Store the medical case data and the generated diagnostic reports in the database.

[0022] In this embodiment, medical case data for the target clinical scenario can be collected in advance as a pre-executed step, such as chest CT scans, brain MRIs, and mammograms, to ensure the data is targeted. This medical case data includes both medical imaging data and diagnostic reports written by clinicians based on the medical imaging data. The diagnostic reports may include medical history, imaging findings, diagnostic conclusions, and recommendations. Furthermore, the collected data needs to reach a preset scale to ensure the coverage and diversity of the database.

[0023] In step two, before inputting medical case data into the first preset large model, preprocessing is required. For example, the medical image data is standardized in terms of image resolution, window width and window level (e.g., lung window and mediastinal window in chest CT), and converted into a format that the model can input (e.g., PNG, JPG); the diagnostic report is cleaned to remove irrelevant content (e.g., doctor's signature, date, etc.) and the terminology is standardized (e.g., using a unified disease naming standard).

[0024] Furthermore, proper design of prompts can ensure that the generated report aligns with the structure of the actual report. For example: Task: As a specialist physician in a specific field, such as chest radiology, analyze the provided medical images and generate a complete diagnostic report.

[0025] Report structure: Must include the following sections: 1. Imaging features: Describe in detail the abnormal or normal structures observed in the images (location, size, shape, density / signal, etc.). 2. Diagnostic conclusion: Based on imaging findings, provide a clear diagnosis or differential diagnosis; 3. Recommendations: Provide recommendations for further examination or treatment.

[0026] Requirements: Professional language skills, clear logic, consistent with the style of reports written by clinicians, and avoid vague expressions.

[0027] Finally, the medical case data and generated diagnostic reports are stored in the database. During implementation, a unique ID can be assigned to each case, establishing a one-to-one correspondence between case ID, images, actual diagnostic reports, and reports generated by the large model.

[0028] Further, in step S103 above, medical images of similar cases and their actual diagnostic reports can be used as reference information. The medical images of the case to be analyzed and the reference information are input into the second preset large model to generate the final diagnostic report. Here, the first and second preset large models can be the same or different large language models, such as Google's gemma3-27b model. The following information can be input into the second preset large model: medical images of the case to be analyzed, medical images of similar cases, actual diagnostic reports of similar cases, and prompt words. The prompt word design here can also guide the model to refer to precedents for diagnosis, for example: Task: As a professional physician specializing in the [target area], generate a final diagnostic report for the case to be analyzed based on the following information.

[0029] Input information: 1. Case images to be analyzed: [Image Input] 2. Reference case information: [Reference Case 1] Imaging: [Image 1], Actual Diagnostic Report: [Report 1] [Reference Case 2] Imaging: [Image 2], Actual Diagnostic Report: [Report 2] ... [Reference Case K] Imaging: [Imaging K], Actual Diagnostic Report: [Report K] Require: 1. Carefully compare the imaging findings and diagnostic conclusions of the case to be analyzed with those of the reference case, and refer to clinical precedents of similar cases; 2. If the case to be analyzed is highly similar to a reference case, its diagnostic approach can be used as a reference; if there are differences, they must be clearly pointed out and differential diagnosis should be performed. 3. The final report should include three parts: imaging findings, diagnostic conclusions, and recommendations. The language should be professional and the logic rigorous. Diagnostic accuracy should take precedence over conciseness.

[0030] The integrated input information and prompts are then fed into the multimodal large language model to obtain the final diagnostic report for the case to be analyzed. This provides rich reference data for the large model's inference based on similar case images and real diagnostic reports, essentially upgrading individual diagnosis to a more precise diagnosis supported by group experience, thus improving the accuracy of the large model's diagnosis.

[0031] In another embodiment provided in this disclosure, the preliminary diagnostic report may be a structured report; The method further includes the following steps: Step 1: Store the preliminary diagnostic report in a structured format beforehand, where the fields in the structured format correspond to the fields in the structured report; The above step "S102, calculate the similarity between the preliminary diagnostic report and the pre-stored diagnostic reports in the database to obtain similar cases" can be implemented as follows: Step 2: For each field in the structured format, match the value of the field in the preliminary diagnostic report with the value of the field in the stored diagnostic report according to the matching rules corresponding to the type of the field, and determine the score of the field based on the matching results; Step 3: Determine the overall similarity between the preliminary diagnostic report and the stored diagnostic report based on the weight and score of each field; Step 4: Sort the obtained overall similarity scores from high to low, and determine the diagnostic reports corresponding to the first preset number of overall similarity scores as similar cases.

[0032] In this embodiment, the diagnostic report may include structured and unstructured reports. A structured report can be a diagnostic report generated according to a fixed template, preset fields, and standardized format. Its text content is broken down into independent modules or fields that can be directly extracted and parsed by machines, allowing core information to be obtained without additional semantic analysis. For example, fixed fields include: examination type, examination site, lesion type, lesion size, etc. An unstructured report can be a natural language text report freely written by a doctor, without a fixed template or preset fields. Core information is scattered throughout continuous text, which cannot be directly extracted by a machine (semantic parsing can be performed using NLP or large models).

[0033] For cases where the diagnostic report is a structured report, the preliminary diagnostic report and the aforementioned diagnostic report generated from the collected medical case data can be pre-converted into structured data according to a structured format and stored. During implementation, when generating the preliminary diagnostic report and the diagnostic report from the collected medical case data, prompts can be used to instruct the first preset large model to generate the report in a structured format. The fields included in the structured format can correspond to the fields in the structured report, such as: number of lesions, lesion location, lesion type, diagnostic conclusion, etc. During storage, the values ​​of the corresponding fields in the diagnostic report can be stored accordingly.

[0034] Furthermore, after storing the preliminary diagnostic report and the diagnostic report generated from the collected medical case data in a structured format, the values ​​of each field in the structured format can be matched with the values ​​of the same field in the preliminary diagnostic report and the stored diagnostic report according to the matching rules corresponding to the field's type. That is, each field can correspond to a field type. For example, the field for the number of lesions can be of type numeric, while the field for the location of lesions can be of type enumeration. Therefore, when matching corresponding fields, different types of fields can use corresponding matching methods. For example, for numeric fields, numerical comparisons can be performed; a perfect match scores the field 1, a difference of 1 scores it 0.8, and so on. For enumeration fields, it can be checked whether the enumerated values ​​match perfectly; a perfect match scores the field 1, and a mismatch scores it 0. This way, the scores corresponding to each field can be obtained.

[0035] Furthermore, corresponding weights can be assigned to each field to reflect its medical importance. These weights can be set in conjunction with the experience of clinical experts. However, it's important to note that the sum of all field weights can be 1, and fields with higher medical priority have greater weights. Weighted summation can then be used to obtain the overall similarity between the preliminary diagnostic report and the stored diagnostic reports.

[0036] Furthermore, the overall similarity scores can be sorted from highest to lowest, and the diagnostic reports corresponding to the top K overall similarity scores can be identified as similar cases. Additionally, if multiple cases have the same similarity, cases with higher scores in core fields (e.g., diagnostic conclusion, lesion location) can be prioritized to further enhance their reference value.

[0037] In another embodiment provided in this disclosure, the preliminary diagnostic report may be an unstructured report; The method further includes the following steps: Step 1: Convert the preliminary diagnostic report into a first feature vector; The above step "S102, calculate the similarity between the preliminary diagnostic report and the pre-stored diagnostic reports in the database to obtain similar cases" can be implemented as follows: Step 2: Calculate the cosine similarity between the first feature vector and the second feature vector corresponding to the pre-stored diagnostic report; Step 3: Sort the obtained cosine similarity scores from high to low, and determine the diagnostic reports corresponding to the first preset number of cosine similarity scores as similar cases.

[0038] In this embodiment, the preliminary diagnostic report and the aforementioned diagnostic report generated from the collected medical case data can be pre-converted into feature vectors. During implementation, an appropriate encoder (e.g., a pre-trained text encoder in the medical field, such as PubMedBERT) can be selected to encode the preliminary diagnostic report and the aforementioned diagnostic report generated from the collected medical case data, obtaining the corresponding first feature vector and second feature vector. It is important to note that the dimensions of the first feature vector and the second feature vector must be consistent to facilitate subsequent similarity calculations.

[0039] Furthermore, cosine similarity can be used to calculate the cosine similarity between the first feature vector and each of the second feature vectors. Finally, the obtained cosine similarities are sorted in descending order of similarity, and the diagnostic reports corresponding to the top preset number (K) cosine similarities are identified as similar cases.

[0040] In yet another embodiment provided in this disclosure, the method further includes the following steps performed in advance: Step 1: Use a pre-set neural network to extract visual features from the medical images of the case to be analyzed, and obtain the visual features; Step 2: Extract semantic features from the preliminary diagnostic report using a preset encoder to obtain semantic features; Step 3: Fuse visual features and semantic features to obtain joint features; The above step "S102, calculate the similarity between the preliminary diagnostic report and the pre-stored diagnostic reports in the database to obtain similar cases" can be implemented as follows: Step 4: Calculate the similarity between the joint features of the case to be analyzed and the joint features corresponding to the pre-stored diagnostic reports; Step 5: Sort the obtained similarity scores from high to low, and determine the diagnostic reports corresponding to the first preset number of similarity scores as similar cases.

[0041] In this embodiment, the visual features corresponding to the medical images and the semantic features corresponding to the diagnostic reports can be fused to obtain joint features. Based on these joint features, the similarity between the case to be analyzed and the aforementioned uncollected medical case data is determined, ultimately identifying similar cases. In practice, visual and semantic features can be pre-extracted for the medical images and generated diagnostic reports corresponding to the case to be analyzed and the aforementioned uncollected medical case data, respectively. Visual feature extraction can employ a pre-defined neural network (e.g., medical-specific CNNs (such as CheXNet, ResNet50), visual Transformers (such as MedViT, Swin-UNETR)); semantic feature extraction can employ a pre-defined encoder (e.g., medical LLM / encoder). The visual and semantic features of the corresponding case are then multimodally fused to obtain the joint features corresponding to that case.

[0042] Furthermore, similar cases to the case to be analyzed are identified from pre-stored diagnostic reports based on joint similarity. In practice, cosine similarity can be used to calculate the similarity between the case to be analyzed and cases in the database, which will not be elaborated upon here.

[0043] In another embodiment provided in this disclosure, the above-mentioned "step 3, fusing visual features and semantic features to obtain joint features" can be implemented as follows: Step 1: Unify the dimensions of visual and semantic features; then concatenate the unified visual and semantic features to obtain joint features; In this step, visual and semantic features can be directly concatenated to obtain joint features. It's important to note that the visual and semantic features should be unified in dimension before concatenation. For example, if the visual feature V_img is 1024-dimensional and the semantic feature V_text is 768-dimensional, a linear layer can be used to map V_text to 1024 dimensions (or vice versa), resulting in V_img' and V_text' of the same dimension. Furthermore, L2 normalization can be performed on V_img' and V_text' respectively (making the vector magnitude 1) to avoid the differences in single-modal feature ranges dominating the fusion result. With both V_img' and V_text' being 1024-dimensional, the directly concatenated joint feature V_fusion can have a dimension of 2048.

[0044] In another embodiment provided in this disclosure, the above-mentioned "step 3, fusing visual features and semantic features to obtain joint features" can be implemented as follows: Step 1: Decompose the visual features into local features, and bind a region semantic label to each local feature through image coordinate mapping; Step 2: For each pathological element, extract the corresponding feature vector from the semantic features to obtain the text element features, and add a label to each text element feature; Step 3: Input the local features and text element features into the preset attention model, and determine the joint features through the attention mechanism.

[0045] In this embodiment of the disclosure, an attention mechanism can be used to enable the attention model to automatically learn the correlation between medical images and diagnostic reports (e.g., ground-glass nodules in images and lung nodules in corresponding text). In steps one and two, before proceeding to attention calculation, the extracted visual and semantic features can be structured to ensure that the two are dimensionally compatible and semantically related, laying the foundation for subsequent attention weight calculation.

[0046] In implementation, visual features can be divided into multiple fixed-size patches (e.g., 16×16 pixels), each patch corresponding to a feature vector with dimensions [N, D] (N represents the number of patches, e.g., a 224×224 visual vector is split into 196 patches). Each patch can correspond to a specific region in the image (e.g., the upper lobe of the left lung, the lower lobe of the right lung, etc.). For each patch feature, the image coordinates corresponding to the region, and the semantic label of the region corresponding to those coordinates, are bound to a corresponding semantic label. Alternatively, only the features of the patch containing the lesion area can be retained, while background patch features are removed to improve processing efficiency and reduce unnecessary computation.

[0047] Furthermore, for semantic features, core pathological elements related to the images can be extracted to avoid interference from redundant information in the text (such as doctor's signature and examination date). In implementation, regular expression matching or named entity recognition (NER) models (such as BioBERT-NER) can be used to extract core pathological elements (e.g., lesion name, lesion location, lesion severity, lesion morphology, etc.) from the diagnostic report. For each extracted pathological element, its corresponding textual element features in the text sequence are extracted, and labels are added to the textual element features corresponding to each element (e.g., lesion location - right upper lobe, lesion type - ground-glass nodule).

[0048] Finally, the local features and text element features are input into a pre-defined attention model, and the joint features are determined through the attention mechanism. It is necessary to ensure that the dimensions and value ranges of the local features and text element features input into the attention model are consistent to avoid deviations in attention calculation due to dimensional differences.

[0049] In yet another embodiment provided in this disclosure, the preset attention model in the above embodiments can be trained in the following manner: Method 1: Step 1: Obtain training samples; training samples include medical image samples and corresponding real diagnostic report samples; Step 2: Extract local features from medical image samples and construct positive sample pairs with the corresponding pathological element text features in the corresponding real diagnostic report samples, and construct negative sample pairs with the text features of non-corresponding pathological elements. Step 3: Input positive and negative sample pairs into the preset attention model for training. Through the attention learning mechanism, learn the alignment relationship between the local features of the medical image samples and the text features of pathological elements.

[0050] Method 2: Step 1: Obtain training samples; training samples include medical image samples and corresponding real diagnostic report samples; Step 2: Extract local features from medical image samples and construct positive sample pairs with the corresponding pathological element text features in the corresponding real diagnostic report samples, and construct negative sample pairs with the text features of non-corresponding pathological elements. Step 3: Input the positive sample pairs and negative sample pairs into the preset teacher model, train the preset teacher model, and obtain the trained teacher model; Step 4: Use the preset attention model as the student model, and input all or part of the positive sample pairs and negative sample pairs into the preset attention model and the teacher model that has been trained. Step 5: Train the preset attention model using the semantics of the corresponding positive and negative sample pairs and the output of the trained teacher model.

[0051] In Method 1, a pre-defined attention model can be trained directly. This attention model can be one with a built-in attention mechanism. Existing model architectures such as Transformer, BERT, and ViT can be used. When acquiring training samples, medical pathology data covering the target clinical scenario can be collected. Each medical pathology report should include at least multiple medical images and real diagnostic reports written by clinicians. These real diagnostic reports can be parsed into a set of pathological elements, such as lesion location, morphological features, density / signal features, enhancement patterns, distribution range, and diagnostic conclusions. During implementation, a rule engine or natural language processing model can be used to automatically or semi-automatically map pathological elements to image regions, forming semantic labels for those regions.

[0052] In step 2, for the same pathology, local features in medical images and their corresponding pathological element text features can be constructed as positive sample pairs; for non-corresponding elements in different cases or in the same case, local image features and unrelated pathological element text features can be constructed as negative sample pairs. Optionally, a hard negative sample strategy can be adopted, using pathological elements with similar semantics but inconsistent diagnostic conclusions as negative samples to enhance the model's discriminative ability.

[0053] Furthermore, the pre-defined attention model can be trained under supervised or semi-supervised conditions. Based on the aforementioned positive and negative sample pairs, the pre-defined attention model learns the alignment relationship between local image features and textual features of pathological elements through an attention mechanism to obtain joint features. In implementation, local image features, textual features of pathological elements, and their labels can be input into the pre-defined attention model. The model calculates the weight distribution between image features and textual features through a cross-modal attention mechanism to achieve explicit semantic alignment and outputs a joint feature representation. The joint feature represents the correspondence between image evidence and diagnostic semantics.

[0054] Based on joint features, multi-objective joint optimization can be performed when determining the loss. At least one of the following loss functions can be used for parameter updates: an alignment loss to constrain the similarity between local image features and corresponding pathological text feature spaces to be higher than that of non-corresponding features; a matching loss to determine whether local image features match text elements; and a generation loss to constrain the consistency of the diagnostic report generated from the joint features. This involves inputting the joint features into the report generation module, aiming at a realistic diagnostic report, and optimizing the consistency of the generated results. In implementation, the corresponding weight coefficients for different losses can be determined according to actual needs to determine the final loss.

[0055] In Method 2, a multimodal or language model with a larger parameter scale can be introduced as the teacher model to provide soft labels to the preset attention model, including attention distribution, feature alignment relationship or generation result probability distribution. The student model is trained by constraining the distillation loss, thereby improving the model's generalization ability.

[0056] In practice, the above positive and negative sample pairs can be input into the preset teacher model to train the preset teacher model and obtain the trained teacher model; the preset attention model can be used as the student model, and the student model can be trained based on the preset teacher model. When calculating the loss, in addition to at least one of the above alignment loss, matching loss and generation loss, distillation loss can also be included, that is, the soft label of the student imitating the teacher.

[0057] In another alternative implementation, the trained model is fine-tuned or optimized by instructions to enable the model to stably output diagnostic reports that conform to clinical logic and medical standards under given prompts, reference cases, and format constraints.

[0058] This disclosure also provides the following embodiments: Example 1 like Figure 2 As shown, a retrieval-enhanced diagnostic report generation method includes the following steps: S1: Construct an image-report database. Collect a medical image dataset covering the target clinical scenario, where each case includes at least one medical image and a real diagnostic report written by a physician. Then, use the same large language model to generate a large model diagnostic report for each image. The generated diagnostic reports correspond one-to-one with the real reports and images, forming an image-report database.

[0059] S2: Generate a preliminary diagnostic report. For each case to be analyzed, its medical images are input into the large language model to generate a preliminary diagnostic report. Since the report is generated in the same way as those generated in the database, semantic space comparability is ensured.

[0060] S3: Calculate report similarity. Use a text encoder to extract features from the preliminary diagnostic report and the generated reports in the database to obtain vectorized representations; then apply the cosine similarity formula:

[0061] VA and VB are the feature vectors of the report to be analyzed and the database report, respectively. The higher the similarity value, the closer the pathological semantics of the two reports are.

[0062] S4: Screening similar cases. Sort the cases according to their similarity scores, select the top K cases with the highest similarity, extract their corresponding images and actual diagnostic reports, and form a candidate reference case set.

[0063] S5: Generate the final diagnostic report. Input the images of the case to be analyzed, the images of the selected reference cases, and the actual diagnostic report into the large language model. The model is prompted to refer to these clinical precedents when generating the final diagnostic report, thereby obtaining a more accurate final diagnostic report.

[0064] In this embodiment, the similarity between the preliminary diagnostic report and the database-generated report relies solely on semantic space calculations, which effectively avoids situations where medical images appear similar but have significant pathological differences, thus improving the reliability of the enhanced retrieval generation.

[0065] Example 2 In another embodiment, both the preliminary diagnostic report and the database report are structured reports. The method calculates similarity in step S3 using a field-by-field comparison approach: (1) The diagnostic report is split into multiple diagnostic element fields, such as "number of lesions", "location of lesions", and "type of lesions"; (2) Calculate the matching degree for each field separately, and record the matching result as the score si; (3) Set the weight wi for each field, and obtain the overall similarity by weighted summation:

[0066] The weighting can be adjusted according to the actual clinical situation. For example, the weight of lesion location and lesion type is higher, while the weight of lesion number is relatively lower.

[0067] By using this field-by-field calculation method, the similarity results can more accurately reflect the pathological similarity of cases, which is especially suitable for tasks such as brain MRI and chest CT.

[0068] Example 3 In another embodiment, similarity calculation simultaneously considers the fusion of image features and text features. Specifically, this includes: (1) Extract visual features of medical images using convolutional neural networks or visual Transformer models; (2) Extract semantic vector features from the diagnostic report generated by the large language model; (3) The two types of features are concatenated or fused through a multimodal attention mechanism to form a joint representation; (4) Similarity is calculated based on joint representation to obtain matching results that are more consistent with clinical pathology.

[0069] In this embodiment, by combining image and text information, the robustness of case retrieval can be further improved, and the misjudgment that may occur under a single modality can be reduced.

[0070] The present invention provides a method, apparatus, and electronic device for generating diagnostic reports based on retrieval enhancement, which has the following beneficial effects: 1. Implement similarity calculation at the pathological semantic level. This invention utilizes a large language model to generate preliminary diagnostic reports corresponding to images, projecting medical images into a deep semantic space, thereby enabling the measurement of differences between cases at the pathological semantic level. Compared to methods that rely solely on the appearance features of images, this invention can more accurately distinguish key pathological information such as lesion type and location, avoiding the misclassification of cases with highly similar visual features but significant pathological differences as similar.

[0071] 2. Improve the accuracy and reliability of search enhancement generation Before generating the final diagnostic report, this invention provides the model with additional clinical reference information through similar case retrieval, making the generated results more consistent with clinical logic and reducing the occurrence of "illusionary" content. Compared with traditional end-to-end generation methods, this invention can significantly improve the accuracy and clinical usability of the generated report.

[0072] 3. Reduce reliance on large-scale image pre-training Many existing medical image analysis methods rely on large-scale labeled datasets for pre-training, which is costly to build and difficult to transfer directly across hospitals and task scenarios. This invention uses a language model to generate semantic representations for similarity calculation, achieving stable retrieval and generation results without the need for additional large-scale image pre-training, thus reducing the development and deployment costs of the system.

[0073] 4. Possesses good scalability and versatility. The method of this invention is applicable to both unstructured and structured reporting scenarios, and can be extended to multimodal fusion methods to jointly model image and text features. Furthermore, this method can be widely applied to diagnostic tasks involving different image modalities (such as X-ray, CT, and MRI) and different disease types, demonstrating good versatility and potential for wider application.

[0074] 5. Improve the efficiency and consistency of clinical diagnosis. This invention, through automated retrieval and generation, can effectively reduce the workload of radiologists and ensure the consistency of diagnostic reports for different cases, thereby improving diagnostic efficiency and quality, reducing the potential risks caused by human error, and has significant clinical application prospects.

[0075] Based on the same disclosed concept, this disclosure also provides a retrieval-enhanced diagnostic report generation device and electronic device. Since the principle of solving the problem by these devices and electronic devices is similar to the aforementioned retrieval-enhanced diagnostic report generation method, the implementation of these devices and electronic devices can refer to the implementation of the aforementioned method, and the repeated parts will not be described again.

[0076] This disclosure provides a diagnostic report generation apparatus based on retrieval enhancement, such as... Figure 3 As shown, it includes: The preliminary diagnosis report generation module 301 is used to generate a preliminary diagnosis report for the medical images of the case to be analyzed using a first preset large model; The similar case calculation module 302 is used to calculate the similarity between the preliminary diagnosis report and the diagnosis reports pre-stored in the database to obtain similar cases; The final diagnosis report generation module 303 is used to take the medical images of the similar cases and their actual diagnosis reports as reference information, and input the medical images of the case to be analyzed and the reference information into a second preset large model to generate a final diagnosis report.

[0077] This disclosure provides an electronic device, including a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the computer device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, they perform the steps of the retrieval-enhanced diagnostic report generation method as described in any of the above embodiments.

[0078] This disclosure provides a computer-readable storage medium storing a computer program that, when executed by a processor, performs the steps of the retrieval-enhanced diagnostic report generation method as described in any of the above embodiments.

[0079] Through the above description of the embodiments, those skilled in the art can clearly understand that the embodiments of this disclosure can be implemented in hardware or by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions of the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, mobile hard drive, etc.) and includes several instructions to cause a computer device (such as a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of this disclosure.

[0080] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of a preferred embodiment, and the modules or processes in the drawings are not necessarily essential for implementing this disclosure.

[0081] Those skilled in the art will understand that the modules in the apparatus of the embodiments can be distributed in the apparatus of the embodiments as described in the embodiments, or they can be located in one or more devices different from this embodiment with corresponding changes. The modules of the above embodiments can be combined into one module, or they can be further divided into multiple sub-modules.

[0082] The sequence numbers of the embodiments disclosed above are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0083] Obviously, those skilled in the art can make various modifications and variations to this disclosure without departing from its spirit and scope. Therefore, if such modifications and variations fall within the scope of the claims of this disclosure and their equivalents, this disclosure is also intended to include such modifications and variations.

Claims

1. A method for generating diagnostic reports based on retrieval enhancement, characterized in that, include: A preliminary diagnostic report is generated from the medical images of the case to be analyzed using the first pre-set large model; Calculate the similarity between the preliminary diagnostic report and pre-stored diagnostic reports in the database to obtain similar cases; The medical images and actual diagnostic reports of the similar cases are used as reference information, and the medical images of the case to be analyzed and the reference information are input into a second preset large model to generate a final diagnostic report.

2. The method as described in claim 1, characterized in that, Also includes: Collect medical case data covering the target clinical scenarios; The medical case data includes: medical imaging data and diagnostic reports written by clinicians based on the medical imaging data; Input the medical case data and corresponding prompt words into the first preset large model to generate a diagnostic report for the corresponding medical case data; The medical case data and the generated diagnostic reports are stored in the database.

3. The method as described in claim 1, characterized in that, The preliminary diagnostic report is a structured report; The method further includes: The preliminary diagnostic report is stored in advance in a structured format, wherein the fields contained in the structured format correspond to the fields of the structured report; Calculate the similarity between the preliminary diagnostic report and pre-stored diagnostic reports in the database to obtain similar cases, including: For each field in the structured format, the value of that field in the preliminary diagnostic report and the value of that field in the stored diagnostic report are matched according to the matching rules corresponding to the type to which that field belongs, and the score of that field is determined based on the matching results; Based on the weight and score of each field, the overall similarity between the preliminary diagnostic report and the stored diagnostic report is determined. The overall similarity scores are sorted from highest to lowest, and the diagnostic reports corresponding to the first preset number of overall similarity scores are identified as similar cases.

4. The method as described in claim 1, characterized in that, The preliminary diagnostic report is an unstructured report; The method further includes: The preliminary diagnostic report is first converted into a first feature vector; Calculate the similarity between the preliminary diagnostic report and pre-stored diagnostic reports in the database to obtain similar cases, including: Calculate the cosine similarity between the first feature vector and the second feature vector corresponding to the pre-stored diagnostic report; The obtained cosine similarities are sorted in descending order of similarity, and the diagnostic reports corresponding to the first preset number of cosine similarities are identified as similar cases.

5. The method as described in claim 1, characterized in that, It also includes the following steps that are performed in advance: Visual features are extracted from the medical images of the case to be analyzed using a pre-defined neural network. The semantic features are extracted from the preliminary diagnostic report using a preset encoder. The visual features and the semantic features are fused to obtain joint features; Calculate the similarity between the preliminary diagnostic report and pre-stored diagnostic reports in the database to obtain similar cases, including: Calculate the similarity between the joint features of the case to be analyzed and the joint features corresponding to the pre-stored diagnostic reports; The obtained similarity scores are sorted from high to low, and the diagnostic reports corresponding to the first preset number of similarity scores are identified as similar cases.

6. The method as described in claim 5, characterized in that, The visual features and the semantic features are fused to obtain joint features, including: The visual features and semantic features are processed to a unified dimension; and the unified visual features and semantic features are concatenated to obtain joint features; and / or The visual features are broken down into local features, and each local feature is bound with a region semantic label by image coordinate mapping. For each pathological element, the corresponding feature vector in the semantic features is extracted to obtain the text element features, and a label is added to each text element feature; The local features and the text element features are input into a preset attention model, and the joint features are determined through the attention mechanism.

7. The method as described in claim 6, characterized in that, The preset attention model is trained using the following method: Obtain training samples; the training samples include medical image samples and corresponding real diagnostic report samples; Local features are extracted from the medical image samples, and positive sample pairs are constructed with the corresponding pathological element text features in the corresponding real diagnostic report samples, and negative sample pairs are constructed with the non-corresponding pathological element text features. The positive sample pairs and the negative sample pairs are input into a preset attention model for training. The alignment relationship between the local features of the medical image samples and the text features of pathological elements is learned through the attention learning mechanism. or The positive sample pairs and the negative sample pairs are input into a preset teacher model, and the preset teacher model is trained to obtain a trained teacher model. The preset attention model is used as a student model, and all or part of the positive sample pairs and the negative sample pairs are input into the preset attention model and the trained teacher model. The preset attention model is trained by using the semantics of the corresponding positive sample pairs and negative sample pairs and the output of the trained teacher model.

8. A diagnostic report generation device based on retrieval enhancement, characterized in that, include: The preliminary diagnosis report generation module is used to generate a preliminary diagnosis report for the medical images of the case to be analyzed using a first preset large model; A similar case calculation module is used to calculate the similarity between the preliminary diagnostic report and the diagnostic reports pre-stored in the database to obtain similar cases; The final diagnostic report generation module is used to take the medical images and real diagnostic reports of the similar cases as reference information, and input the medical images of the case to be analyzed and the reference information into a second preset large model to generate a final diagnostic report.

9. An electronic device, characterized in that, include: The computer device includes a processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor, and the processor communicates with the memory via the bus when the computer device is running, and the machine-readable instructions, when executed by the processor, perform the steps of the retrieval-enhanced diagnostic report generation method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the retrieval-enhanced diagnostic report generation method as described in any one of claims 1 to 7.

Images