Label-assisted report generation method and device

Abstract

The invention relates to a method and a device for generating a report under the assistance of a label, and the method comprises the following steps: 1) extracting a structured label set from a text report of a sample based on a large language model, wherein the structured label set comprises a multi-classification group consisting of dichotomous labels and mutual exclusion options; 2) aggregating the labels in batches, after a threshold value is reached, merging and de-weighting, performing specification and mutual exclusion group merging on synonymous, near-synonymous and redundant labels, and converging into a unified label library; 3) based on the text report and the tag library, outputting a tag subset of each sample through a large language model; 4) multi-modal multi-label classification model training: extracting each visual modal feature, performing weighted aggregation and splicing, and outputting each label group logits through a classification head to perform weighted group loss optimization; (5) carrying out joint training on the multi-modal large language model by using samples of'only images-reports' and'images + labels-reports', and (6) carrying out label prediction and screening on the images by using the classification model, and inputting'images + prediction labels' into the multi-modal large language model to obtain a final report.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of multi-modal artificial intelligence and natural language generation, and in particular to a method and device for generating professional text reports using multi-label classification results to assist a multi-modal large language model. BACKGROUND

[0002] Multi-modal large language models (MLLMs) can jointly model visual modalities such as images and text modalities, and can be used to generate structured or natural language reports from images. However, directly supervising and fine-tuning MLLMs in an end-to-end manner to generate long text often results in problems such as insufficient grasp of key information, unstable facts, redundant expressions, or missing key points. The root cause is that direct mapping from images to text cannot simultaneously cover global key concepts and fine-grained details, and the saliency signals in the training data may not be consistent with the generation target. To improve the accuracy and coverage of the generated text, in practice, a set of discriminative "semantic labels" is often first extracted, and then these labels are used as high-level semantic constraints to guide the multi-modal large language model to complete more robust report generation. Existing solutions often rely on manually defined or static label systems, which are difficult to transfer across scenarios and cannot be continuously expanded and de-duplicated on a large amount of data. The present application proposes an integrated technical route of "automatic label construction - multi-label classification - label enhanced generation", which balances the automation and scalability of the label system, the generalizability of the classification model, and the controllability and factuality of the multi-modal large language model. SUMMARY

[0003] The present application overcomes the above-mentioned shortcomings of the prior art and provides a method and device for label-assisted report generation.

[0004] The first aspect of the present application relates to a method for label-assisted report generation, comprising: (1) Label set construction: batch processing of samples containing text reports, extracting discriminative concept-type labels from the reports based on a large language model through instructions, and outputting a structured label set composed of a mixture of "binary classification labels" and "multi-classification groups consisting of mutually exclusive options"; (2) Label iterative merging and de-duplication: aggregating the label sets obtained from different batches in order, and once the total number of labels in the temporary group reaches a preset threshold k, triggering a merge and de-duplicate operation, reducing synonymous, near-synonymous, and redundant labels through a large language model, and merging mutually exclusive concepts into the same multi-classification group until convergence to a unified label library; (3) Sample label annotation: given a text report and the label library for a single sample, output the selected index of the binary classification label and the unique selection index of each mutually exclusive group through a large language model to form a label subset for the sample; (4) Multi-modal multi-label classification model training: For each visual modality, extract features, and for multiple images under the same modality, perform weighted average according to the effective mask to obtain the aggregated features of the modality. The features of each modality are spliced and input into the classification head. The corresponding logits are output according to the label group, and the weighted sum of the group losses is optimized. (5) Multi-modal large language model training: Joint training of the multi-modal large language model with two types of samples "image only -> report" and "image + label -> report" to generate more accurate text reports under the label prior constraint. (6) Inference: First, the classification model predicts the relevant labels and filters the input images, and then inputs "image + predicted label" into the multi-modal large language model as a condition to obtain the final report.

[0005] Among them, the label extraction in step (1) uses structured output constraints to limit the output to only JSON arrays containing string labels and string arrays (representing mutually exclusive groups); The extraction rules at least include: avoiding overly general descriptions, prioritizing concepts with distinguishing features, merging mutually exclusive concepts into the same multi-classification group, and only retaining positive expressions.

[0006] Among them, the iterative merging process in step (2) aggregates a number of label groups in order until it reaches a threshold k, and then performs merging and deduplication, which includes deleting completely duplicated items, normalizing synonymous items, merging mutually exclusive concepts into multi-classification groups, and maintaining only the output of structured label sets.

[0007] Among them, the multi-modal multi-label classification model in step (4) includes: Single or multiple visual encoders (depending on the number of image modalities), corresponding to input images of different modalities; Masked mean aggregation module for weighted averaging of features of multiple images under the same modality according to the effective mask to obtain the aggregated features of the modality; Feature fusion and classification head, which splices and maps the aggregated features of each modality to the total output dimension, and the total output is divided into multiple group logits according to the label group.

[0008] Among them, the classification loss includes: cross-entropy loss for mutually exclusive groups, equivalent two-class cross-entropy or binary cross-entropy for binary classification labels, and the total loss is the weighted sum of each group loss; The classification head contains nonlinear activation and random inactivation to improve generalization ability.

[0009] Among them, the label filtering in step (6) includes: filtering binary classification labels according to probability threshold or Top-K, and selecting the maximum probability item for mutually exclusive groups; The label is encoded into a prompt prefix and an image input into a multi-modal large language model.

[0010] The second aspect of the present application relates to a device for label-assisted report generation, comprising a memory for storing a program and a processor configured to execute the program to implement a method for label-assisted report generation of the present application.

[0011] The third aspect of the present application relates to an electronic device comprising at least one processor, a memory and a communication interface, the memory storing a computer program, the processor being configured to implement a method for label-assisted report generation of the present application when executing the program.

[0012] The fourth aspect of the present application relates to a computer readable medium having stored thereon a computer program for implementing a method for label-assisted report generation of the present application when executed by a processor.

[0013] The working principle of the present application is: using the structured knowledge extraction and conditional generation capability of large language model, the end-to-end "image → long text" complex mapping is decoupled into "image → structured key concept" and "key concept + image → long text" two more easily optimized sub-tasks. When generating a long report directly from an image, a multi-modal large language model needs to complete three tasks of visual understanding, key point extraction and language organization at the same time, and the direct mapping in the high-dimensional feature space is easy to cause random omission of key information; and the present application introduces an explicit "diagnostic label" as a semantic bottleneck layer, uses the understanding ability of large language model for medical text to automatically construct a label system and label data, and then uses an independent classification model to forcibly extract visual key features, and finally lets the multi-modal large language model generate under the constraint of the label - this paradigm of transforming implicit end-to-end learning into "explicit structured guided generation" essentially reduces the solution space complexity and introduces intermediate supervision signals, so that the attention mechanism of the model can more stably focus on the key visual-semantic correspondence related to diagnosis, thereby improving the factuality and completeness of the generated text.

[0014] The innovation of the present application is: 1. Automation and scalability of the label system: without manual definition of labels, the LLM automatically mines and normalizes from data, which can be transferred across scenarios.

[0015] 2. Two-stage decoupling architecture of "classification first and then generation": the key concept extraction is decoupled into an independent classification task, and the classification result is used as an explicit intermediate prior constraint to guide the MLLM generation. This architecture not only reduces the burden of end-to-end generation, but also avoids key information omission through structured labels, significantly improving the factuality and key point coverage of the generated report.

[0016] The beneficial effects of the present application are: improving the factuality and key point coverage of report generation, reducing the probability of missing key concepts; without relying on specific data sets, automatically obtaining a transferable label system with strong universality; multi-modal feature fusion and mask mean aggregation strategy improve the robustness of variable number of input images; the "classification first and then generation" process in the reasoning stage improves the controllability and consistency of generation. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 is a training process schematic diagram of the method of the present application, including automatic construction of label set, label merging and deduplication, sample label annotation, multi-modal multi-label classification model training, and multi-modal large language model training; Figure 2 is a reasoning process schematic diagram of the method of the present application, including label prediction and screening, and label enhancement generation. The data flow between the modules is indicated by arrows, and the specific steps are described in detail in the specific embodiments section. DETAILED DESCRIPTION

[0018] In order for those skilled in the art to better understand the present application, the technical solutions are described in detail below in conjunction with the embodiments. It should be understood that these embodiments are intended to illustrate but not limit the protection scope of the present application.

[0019] Embodiment 1

[0020] The method of the present application is applied to a method of generating a radiology diagnosis report from a chest X-ray film. This embodiment is aimed at the computer-aided screening scene of pulmonary tuberculosis, and a system for automatically generating a structured diagnosis report from a chest X-ray film is constructed. The system is based on two public chest X-ray datasets for computer-aided screening of pulmonary diseases, which include 561 training data, 80 validation data and 160 test data, each containing a chest X-ray film and a corresponding diagnosis report. The specific steps include the following: I. Automatic construction of medical diagnosis key concepts (labels); 1. Label extraction for pulmonary tuberculosis diagnosis report: group the 561 training set reports by batch (e.g. 50 per batch), and construct a special extraction instruction for each batch of reports, requiring the large language model to extract key concepts related to pulmonary tuberculosis diagnosis, including: Lesion feature class labels: such as "pulmonary infiltrative lesions", "nodule shadows", "cavity formation", "calcification foci", etc.; Lesion location class labels: such as "upper lung field lesions", "lower lung field lesions", "bilateral lesions", etc. mutually exclusive groups; Severity-based labels: mutually exclusive groups such as "mild," "moderate," and "severe"; Diagnostic conclusion labels: such as mutually exclusive groups like "positive", "negative", and "suspected".

[0021] 2. Iterative Merging and Standardization of Medical Terminology: Tags extracted from multiple batches are merged and integrated. Whenever the cumulative number of tags reaches approximately 100, a large language model is invoked to perform deduplication and synonym merging (e.g., "lung shadow" and "lung lesion"), and mutually exclusive diagnostic concepts are grouped into the same multi-category group. After multiple iterations, a unified tag library is ultimately formed, covering the main clinical concepts for pulmonary tuberculosis diagnosis.

[0022] 3. Medical validation of the tag library: The constructed tag collection is saved in JSONL format, which supports subsequent manual review and verification by medical experts to ensure that the tags comply with clinical diagnostic standards.

[0023] II. Automatic sample labeling based on a label library; 1. Label matching based on diagnostic reports: For each report in the training set, a pre-built label library is input, and a large language model analyzes the report content and outputs the results. binary_labels: The binary label index of this case (such as "nodular shadow", "calcification", etc.); multi_class_selections: Select each mutually exclusive group (e.g., select "upper lung field lesion" for lesion location and "positive" for diagnosis conclusion).

[0024] 2. Consistency verification of annotation results: The output is parsed and verified, including index range checks and uniqueness verification of mutually exclusive groups. If the output format of the large language model is abnormal, automatic repair or manual review is performed to ensure that each data point obtains a complete and consistent label set.

[0025] 3. Label Frequency Statistics and Long-Tail Processing: The frequency of each label in the 561 training data examples was statistically analyzed. Low-frequency labels with too few occurrences (e.g., <15 times) were removed to avoid overfitting, and the distribution of each label was recorded for subsequent class imbalance optimization of the classification model.

[0026] III. A multimodal, multi-label classification model for the diagnosis of pulmonary tuberculosis; 1. Model structure for chest X-ray: Visual encoder selection: DINOv3-ConvNeXt-Base was used as the pre-trained visual encoder, as this model performs excellently in medical image feature extraction. Forward encoding was performed on each chest X-ray (scaled to 224×224 pixels) to obtain high-dimensional visual features; Feature pooling and classification head: After global average pooling of visual features, they are sent to a multi-layer perceptron containing Dropout (to prevent overfitting) to map to the total output dimension. The output dimension is equal to the sum of the number of binary labels and the number of mutually exclusive group categories (about 80 dimensions in this case); Group-wise split of output logits: According to the group configuration of the label library, the total output is divided into multiple subsets corresponding to different label groups, which facilitates the calculation of loss.

[0027] 2. Training strategy for long-tailed distribution: Mixed precision training (FP16) is used for full-parameter fine-tuning on a single NVIDIA RTX A6000 GPU; Training hyperparameters: learning rate 2x1 , batch size 16, training for 30 epochs; Loss function: Cross-entropy loss is used for mutually exclusive groups (such as diagnostic conclusions), and binary cross-entropy with class weighting is used for binary labels. The weight of low-frequency labels is increased to alleviate the long-tailed problem; Model selection: Monitor the micro-F1 score on the validation set and select the checkpoint with the best performance as the final classification model.

[0028] Four, label-enhanced pulmonary tuberculosis diagnosis report generation model; 1. Selection and training data construction of multi-modal large language model: Model selection: Qwen2.5-VL-7B is used as the basic multi-modal large language model, which has strong visual-language joint modeling capability; Training data format: For each data in the training set, two types of input are constructed: a) Pure image input: only use chest X-ray images as conditions, and require the model to generate a diagnosis report; b) Image + label input: input the chest X-ray and the corresponding label set (encoded as natural language prompts according to the template, such as "diagnostic features: pulmonary infiltrative lesions, upper lung field lesions, positive") together, and require the model to generate a report.

[0029] 2. LoRA low-rank fine-tuning strategy: The language decoder, visual encoder, and visual projection layer of Qwen2.5-VL are fine-tuned using LoRA (Low-Rank Adaptation), which efficiently adapts to the medical diagnosis scenario while maintaining the capabilities of the base model; Training hyperparameters: learning rate 2x1 , weight decay 0.1, batch size 1 per device, gradient accumulation step 4, and training for 1 epoch; Training environment: Distributed training with BF16 mixed precision on 4 NVIDIA RTX A6000 GPUs Training objective: Adopting autoregressive language modeling loss to make the model learn to generate more accurate and complete diagnosis reports under the constraint of label priori.

[0030] Five, the reasoning generation process of the tuberculosis diagnosis report; 1. Label prediction stage: For each chest X-ray in the test set, first input the trained multi-label classification model to get the predicted confidence of each label. For binary classification labels, a threshold (such as 0.5) is used for screening; for mutually exclusive groups, the class with the highest probability is selected.

[0031] 2. Label-enhanced report generation: input the chest X-ray image and the predicted label set (encoded as natural language prompts) into the fine-tuned Qwen2.5-VL model, and generate a complete diagnosis report from the model autoregression.

[0032] 3. Effect verification: On the 160 test sets, compared with directly fine-tuning the multi-modal large language model, the clinical effectiveness index (RadFact F1 score) is improved from 0.0286 to 0.2831, and the natural language generation index (BLEU-4) is improved from 0.0265 to 0.2129, significantly reducing the omission rate of key diagnostic information and improving the factuality and accuracy of the report.

[0033] Embodiment 2

[0034] The embodiment provides a label-assisted report generation device: including a memory and a processor. The memory is used to store programs and model parameters; the processor is configured to call the programs to execute the steps of the method of the present application, including label extraction and merging, sample label annotation, multi-modal multi-label classification training and reasoning, and label-enhanced report generation.

[0035] Embodiment 3

[0036] The embodiment provides an electronic device: including at least one processor, memory and communication interface; the memory stores programs executable on the processor, and the programs make the electronic device realize the method of label-assisted report generation according to embodiment 1 when executed.

[0037] Embodiment 4

[0038] Computer readable medium: for storing computer instructions, when the instructions are executed by the processor, the processor executes the method of label-assisted report generation according to embodiment 1.

[0039] The above embodiments can be implemented on different hardware and software platforms, and the models, loss functions, hyperparameters, prompt templates, threshold strategies, network structures, etc. involved can be replaced or equivalently deformed according to specific applications without departing from the spirit and scope of the present application.

Claims

1. A method for generating reports with the assistance of tags, comprising the following steps: (1) Construction of tag set: Batch processing of samples containing text reports, extracting distinguishable conceptual tags from the reports based on the large language model, and outputting a structured tag set composed of a mixture of "binary tags" and "multi-class groups composed of mutually exclusive options"; (2) Iterative merging and deduplication of tags: The tag sets obtained from different batches are aggregated in sequence until the total number of tags in the temporary group reaches the preset threshold k. Then, a merging and deduplication is triggered. Synonyms, near-synonyms and redundant tags are reduced by the large language model, and mutually exclusive concepts are merged into the same multi-class group until they converge into a unified tag library. (3) Sample labeling: Given a text report of a single sample and the label library, the selection index of the binary classification label and the unique selection index of each mutually exclusive group are output by the large language model to form a subset of the labels of the sample; (4) Multimodal multi-label classification model training: extract features for each visual modality, perform weighted average of multiple images under the same modality according to the effective mask to obtain the aggregated features of the modality, concatenate the features of each modality and input them into the classification head, output the corresponding logits according to the label group, and optimize by weighted sum of group loss; (5) Multimodal large language model training: The multimodal large language model is jointly trained with two types of samples, "image only → report" and "image + label → report", so that it can generate more accurate text reports under the label prior constraint; (6) Reasoning: First, the classification model predicts and filters relevant labels for the input image, and then the "image + predicted label" is used as a condition to input into the multimodal large language model to obtain the final report.

2. A method of tag-assisted report generation as claimed in claim 1, wherein, The label extraction in step (1) uses structured output constraints, limiting the output to only JSON arrays containing string labels and string arrays (representing mutually exclusive groups); the extraction rules include at least: avoiding overly general descriptions, prioritizing the selection of distinctive concepts, merging mutually exclusive concepts into the same multi-class group and retaining only positive descriptions.

3. A method of tag-assisted report generation as claimed in claim 1, wherein, The iterative merging process in step (2) aggregates several label groups in sequence until a threshold k is reached, and then merges and removes duplicates. The merging and deduplication includes: deleting completely duplicate items, normalizing synonyms, merging mutually exclusive concepts into multi-class groups, and maintaining only the output of structured label sets.

4. A method of tag-assisted report generation as claimed in claim 1, wherein, The multimodal, multi-label classification model in step (4) includes: One or more visual encoders, each corresponding to an input image of a different modality; The mask mean aggregation module is used to perform a weighted average of the features of multiple images under the same modality according to the effective mask to obtain the aggregated features of that modality; The feature fusion and classification head concatenates the aggregated features of each modality and maps them to the total output dimension. The total output is divided into multiple group logits according to the label group.

5. A method for tag-assisted reporting of a generation of a report as recited in claim 1, wherein, The classification loss in step (4) includes: cross-entropy loss for mutually exclusive groups, equivalent two-class cross-entropy or sigmoid binary cross-entropy for binary labels, and the total loss is the weighted sum of the losses of each group; the classification head includes non-linear activation and random deactivation to improve generalization ability.

6. The method for generating a report with tag assistance as described in claim 1, characterized in that, The label selection in step (6) includes: selecting binary classification labels according to probability thresholds or Top-K, and selecting the highest probability item for mutually exclusive groups; the labels are encoded as prompt word prefixes by a fixed template and input together with the image into the multimodal large language model.

7. A device for generating reports with the assistance of tags, characterized in that, It includes a memory and a processor, the memory being used to store a program, and the processor being configured to execute the program to implement a method for generating a tag-assisted report according to any one of claims 1-6.

8. A computer-readable medium, characterized in that, It stores a computer program that, when executed by a processor, is used to implement a method for generating a report with label assistance according to any one of claims 1-6.

Images