Structured medical report generation method based on multi-stage decoding and hyper-lexical elements
Through the multi-stage 3D visual feature extraction and implicit super-word generation method, the problem of structured generation of three-dimensional medical image reports was solved, high-quality, coherent and clinically relevant structured reports were achieved, and the accuracy and clarity of the reports were improved.
Patent Information
- Application Number
- CN202510782980.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-12
- Publication Date
- 2025-09-26
AI Technical Summary
Existing technologies have difficulty generating high-quality, structured three-dimensional medical image reports, especially in terms of inconsistent terminology and structural differences, and evaluation methods lack clinical information, resulting in insufficient confidence in artificial intelligence technology among doctors and patients.
A multi-stage decoding process with multi-stage 3D visual feature extractors and implicit super-word generation is adopted. By training a dedicated 3D visual feature extractor for each stage, combined with a transformer encoder and a super-word decoder, high-quality structured reports are generated.
It improves the accuracy and consistency of structured reports, achieves terminology standardization, enhances the clarity of reports, facilitates clinical data mining and analysis, and improves diagnosis and treatment efficiency and economic benefits.
Smart Images
Figure CN120708791A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing, and in particular to a structured medical report generation method based on multi-stage decoding and superwords. Background Art
[0002] Automatic medical image report generation, which automatically generates diagnostic reports by analyzing medical images, has become a key research direction for improving clinical efficiency. This technology not only optimizes workflows and reduces physician burden, but also improves diagnostic accuracy. Early research focused on two-dimensional images (such as chest X-rays), but has now expanded to three-dimensional images (such as CT scans) that contain richer spatial information. Existing technologies often generate unstructured free-text reports based on datasets such as MIMIC-CXR, but these technologies suffer from limitations such as inconsistent terminology and structural differences. Summary of the Invention
[0003] The purpose of the present invention is to provide a structured medical report generation method based on multi-stage decoding and super-words.
[0004] The technical solution for achieving the purpose of the present invention is: a method for generating a structured medical report based on multi-stage decoding and super-words, comprising:
[0005] S1, a 3D visual feature extractor specifically trained for multi-phase CT images; using the 3D visual feature extractor to extract features, obtain 4-phase image features; the extracted 4-phase image feature maps are connected along the channel dimension to form a fused feature representation, thereby obtaining a fused feature;
[0006] In step S2, the fused features are fed into the transformer encoder to obtain the encoded implicit state. Based on the implicit state, block decoding is used to generate super-grams. The generated super-grams are fed into the super-gram decoder, and the decoder is forced to rely solely on super-grams to reconstruct the report fragment text, thereby prompting the fragment decoder to learn to generate high-quality super-grams containing report text information. Step S2 is repeated to obtain super-grams for all fragments.
[0007] S3, the fusion decoder receives the encoded image feature implicit state and the connected super-words of all segments to generate the final structured report.
[0008] Furthermore, in step S1, given a set of CT scan images of four periods, {I p} p∈{C,A,P,V} , where each Represents the scan image during the p period, C represents the plain scan period, A represents the arterial phase, P represents the portal venous phase, and V represents the delayed phase. Four independent 3D visual feature extractors are trained: and These extractors extract four aspects of image features: F C 、F A 、F P and F V .
[0009] Connect the four-dimensional image feature maps along the channel dimension to form a fusion feature:
[0010] F cat =concat(F C ,F A ,F P ,F V ).
[0011] Furthermore, in step S2, the fusion feature F cat Input transformer encoder To get the implicit state of the encoding
[0012]
[0013] For each segment B of the structured report k , where k = 1, 2, ..., K, K is the segment in the structured report, the total number is 6, train a block decoder The decoder generates text for the segment And train it to generate a single additional token at the end of the sequence, designating it as a super-word Superwords are not predefined tags, they are learned during training. This stage is represented as
[0014]
[0015] For each segment B k , the generated superword Then used as a separate superword decoder Φ meta The only input of this super-word decoder is ; this super-word decoder is trained to reconstruct the text of the fragment without access to the original visual features or the encoded implicit state:
[0016]
[0017] Furthermore, step S3 is specifically as follows:
[0018] Using the fusion decoder Φ fusion , the encoded image feature implicit state h and the concatenated super-word of all segments As input, predict the final structured report
[0019]
[0020] Joint training of block decoder and super word decoder: The goal of joint training is to minimize the comprehensive loss For each fragment
[0021]
[0022] in is the cross entropy loss of the block decoder; T k , and Report segment B k The length of the true value text, the tth token and the previous token, h is the implicit state of the image feature, is a block decoder;
[0023] is the cross entropy loss of super-word decoder reconstruction, It is a super-word. is the super-word decoder; α is used to balance the two loss terms;
[0024] Fusion decoder training: The fusion decoder receives the encoded image feature latent state h and the concatenated super-words of all segments: Generate final structured report The training goal is to minimize the cross entropy loss of the fusion decoder parameters
[0025]
[0026] Where T and is the length of the complete ground truth structured report and the Tth token; the optimization parameter θ fusion To minimize the fusion decoder loss
[0027] In a second aspect, the present invention provides a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method described in the first aspect when executing the program.
[0028] In a third aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in the first aspect.
[0029] In a fourth aspect, the present invention provides a computer program product, comprising a computer program, which implements the steps of the method described in the first aspect when executed by a processor.
[0030] Compared with the prior art, the present invention has the following beneficial effects:
[0031] (1) The present invention proposes a multi-stage 3D visual feature extractor. By using a dedicated 3D visual feature extractor for each stage, the model learns the features of a specific stage, effectively capturing stage-specific information and inter-stage information, and providing critical temporal information of tissue characteristics and lesion behavior for subsequent operations.
[0032] (2) The present invention proposes a multi-stage decoding process based on implicit super-word generation. By jointly training the block decoder and super-word decoder of report segments, the model is allowed to implicitly learn a concise representation (super-word) for each segment; these super-words are combined with visual features to improve the accuracy of the model's prediction of structured medical reports.
[0033] (3) The present invention proposes to use multiple evaluation indicators to perform high-quality assessment of the quality of generated reports, and evaluates them from multiple aspects such as professional knowledge and sentence word order matching, accurately and effectively proving that the present invention can generate accurate, coherent and clinically relevant structured reports. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] Figure 1 This is a structural diagram of the CT2SMR of the present invention.
[0035] Figure 2 This is a flowchart of CT2SMR generating structured medical reports. DETAILED DESCRIPTION
[0036] Compared with unstructured free-text reports, structured reports using predefined templates have significant advantages: they can not only standardize terminology and improve report clarity, but also facilitate clinical data mining and analysis, and at the same time demonstrate important value in terms of economic benefits and diagnosis and treatment efficiency.
[0037] This invention studies the generation of structured medical reports from multi-stage CT scans, which is a relatively underexplored area in medical image analysis. The reasons for this underexploration are: 1) The scarcity of high-quality three-dimensional image datasets based on structured reports. Existing public datasets are all single-stage image and unstructured report datasets, which cannot capture the dynamic evolution of lesion characteristics over time; 2) The challenge in automatically generating structured reports is to generate coherent and accurate text in each segment and ensure the overall consistency and coherence of each segment to present a holistic view of the patient's condition; 3) The current evaluation of medical report generation is mostly based on evaluation indicators generated by natural language. These evaluation methods mainly evaluate from aspects such as word order and sentence structure similarity, and lack accurate evaluation of clinical information, resulting in doctors and patients lacking confidence in artificial intelligence technology in clinical applications.
[0038] The present invention improves the three-dimensional image report generation model CT2Rep, uses multi-phase CT image feature-specific extraction and multi-stage decoding architecture, and constructs a CT image structured medical report generation model CT2SMR.
[0039] The present invention adopts a multi-stage decoding architecture that uses super-words to improve intra-segment consistency and inter-segment consistency. The model consists of two main components: (1) a multi-stage 3D visual feature extractor (2) a two-stage decoding process with implicit super-word generation, and a final fusion decoder.
[0040] The first component of the present invention is to train a dedicated 3D visual feature extractor for multi-phase CT images.
[0041] The technical solution to implement this component is as follows: Multi-phase CT scans provide critical temporal information about tissue characteristics and lesion behavior. To effectively capture this information, we use a dedicated 3D visual feature extractor for each phase (plain phase, arterial phase, portal venous phase, and delayed phase). This approach allows the model to learn phase-specific features, which are essential for accurately describing findings in structured reports. Given a set of CT scan images from four phases, {I p} p∈{C,A,P,V} , where each Representing the p-period scan image (C: plain phase, A: arterial phase, P: portal venous phase, V: delayed phase), we trained four independent 3D visual feature extractors: and These extractors extract specific image features: C , F A , F P and F V To exploit the complementary information between stages, we concatenate these feature maps along the channel dimension to form a fused feature representation
[0042] F cat =concat(F C ,F A ,F P ,F V )
[0043] This fused feature representation F cat Both stage-specific and inter-stage information are captured, providing rich 3D CT image feature representation for subsequent report generation.
[0044] The second component of the present invention is the multi-stage decoding process for implicit super-gram generation, which is the core of CT2SMR. Based on the encoder-decoder converter architecture, super-grams and multi-stage decoding methods are introduced to improve intra-segment consistency and cross-segment consistency.
[0045] The technical solution to realize this component is:
[0046] Phase 1: Specific block decoding and superword generation. The focus of the first phase is to generate the individual fragments of the structured report. We use the fused visual feature F cat and transformer encoder To get the implicit state of the encoding
[0047]
[0048] in We use an attention mechanism to capture dependencies between features, just like in standard transformers. We also combine relational memory (RM) and memory-driven conditional layer normalization (MCLN) to enhance context awareness.
[0049] For each segment B of the structured report k (where k = 1, 2, ..., K), K is the total number of segments in the structured report (6), we train a block decoder The decoder generates text for the segment and trains it to generate a single additional token at the end of the sequence, which we designate as a super-word The super-word is not a predefined token; it is learned during training. This stage can be expressed as
[0050]
[0051] Stage 2: Superword decoding. For each segment B k , the generated superword Then used as a separate super-word decoder Φ meta The super-word decoder is trained to reconstruct the fragment text without access to the original visual features or the encoded implicit state.
[0052]
[0053] This is a crucial step. By forcing the superword decoder to reconstruct the fragment text only from superwords, we implicitly train the fragment decoder to generate a superword that encapsulates all the necessary information about the visual features of the fragment. This promotes the generation of highly informative and representative superwords. We train a separate chunk decoder and superword decoder for each of the K fragments in the structured report.
[0054] Stage 3: Fusion decoding generates a complete report. After training the block decoder and super word decoder for each segment, we freeze their weights. To generate the final structured report, we use the fusion decoder Φ fusion The decoder takes as input the latent state h encoding the image features and the concatenated super-words of all segments
[0055]
[0056] Joint training of block decoder and super word decoder: The goal of joint training is to minimize the comprehensive loss For each fragment
[0057]
[0058] in
[0059] is the cross entropy loss of the block decoder. T k , and Report segment B k The true value of the length of the text, the tth token and the previous token,
[0060] h is the implicit state of the image feature, It is a block decoder.
[0061] is the cross entropy loss of super-word decoder reconstruction, It is a super-word. It is a super word decoder.
[0062] — α balances the two loss terms (set to 1 by default).
[0063] Fusion decoder training: After stage 1 and stage 2, all block decoders and super word decoders ( and ) are frozen. The fusion decoder receives the encoded image feature implicit state h, and the concatenated super-words of all segments: It generates a final structured report The training goal is to minimize the cross entropy loss of the fusion decoder parameters
[0064]
[0065] Where T and is the length of the complete ground truth structured report and the Tth token. The optimization parameter θ fusion To minimize the fusion decoder loss
[0066] For the final generated structured medical report, we use three categories of metrics to evaluate the CT2SMR performance: radiology-specific evaluation metrics and natural language generation (NLG) evaluation metrics, as well as LLM-based evaluation metrics.
[0067] Radiology Report Generation (RRG) metrics: Use Rate-Score, a recently developed method designed to assess the quality of generated medical reports.
[0068] Natural Language Generation (NLG) metrics: Standard NLG metrics BLEU, METEOR, and ROUGE-L are used, which evaluate from the aspects of word matching, synonym and word order similarity, and word sequence matching, respectively.
[0069] Large Language Model (LLM) Scoring: Using the Qwen-Max-Latest version of the Qwen large language model API (Qwen 2.5 series, with hyperparameters: temperature = 0.7, top-k = 0.8, top-p = 20), two rounds of independent prompt evaluation were conducted, as follows: (1) A 7-level percentage score was set (90-100 is the highest, 40 and below is the lowest), and the scores were divided according to the accuracy of core information and secondary information; (2) Sentence-by-sentence semantic analysis, including core information priority verification, synonym rationality, and data deviation penalty; (3) Differentiated deductions, such as 10-20 points for core information errors (such as incorrect lesion location) and 2-5 points for secondary information errors (such as descriptive deviation); (4) Scores and reasons were provided after each evaluation, and the second evaluation would refer to the scores and reasons of the first evaluation.
[0070] The present invention will be described in detail below with reference to the accompanying drawings and embodiments.
[0071] Example
[0072] Figure 1 This is a structural diagram of the present invention's generation of structured medical reports CT2SMR based on multi-stage decoding and superwords. Figure 2 This is a flowchart of CT2SMR generating structured medical reports.
[0073] like Figure 1 、 Figure 2 As shown in the figure, the specific implementation process of CT2SMR is as follows:
[0074] ① Train a dedicated 3D visual feature extractor for multi-phase CT images: and
[0075] ②Use and Extract the corresponding image features F C, F A , F P and F V .
[0076] ③ Feature fusion: The extracted F C , F A , F P and F V , connect these feature maps along the channel dimension to form a fused feature representation and obtain the fused feature F cat .
[0077] ④ Fusion feature F cat Input transformer encoder Get the encoded implicit state h.
[0078] ⑤ Based on the implicit state h, use block decoding to generate super-words.
[0079] ⑥ Input the generated super-word into the super-word decoder, and by forcing the decoder to reconstruct the report fragment text only based on the super-word, the fragment decoder is prompted to learn to generate high-quality super-words containing report text information.
[0080] ⑦ Repeat steps ④⑤⑥ to obtain superwords of all segments;
[0081] ⑧The fusion decoder receives the encoded image feature implicit state h and the connected super-words of all segments Generate final structured report
[0082] The above embodiments merely illustrate several specific implementation methods of the present application. Although described in considerable detail, they should not be construed as limiting the scope of the present invention. It should be emphasized that a person skilled in the art may make various modifications and improvements without departing from the basic concept of the present application, and such modifications and improvements should be included within the scope of protection of the present application. Therefore, the scope of protection of this patent shall be subject to the appended claims.
Claims
1. A method for generating structured medical reports based on multi-stage decoding and superwords, characterized in that: include: S1 is a 3D visual feature extractor trained specifically on CT images in multiple phases; Use a 3D visual feature extractor to extract features and obtain four-dimensional image features; connect the extracted four-dimensional image feature maps along the channel dimension to form a fused feature representation to obtain a fused feature; In step S2, the fused features are fed into the transformer encoder to obtain the encoded implicit state. Based on the implicit state, block decoding is used to generate super-grams. The generated super-grams are fed into the super-gram decoder, and the decoder is forced to rely solely on super-grams to reconstruct the report fragment text, thereby prompting the fragment decoder to learn to generate high-quality super-grams containing report text information. Step S2 is repeated to obtain super-grams for all fragments. S3, the fusion decoder receives the encoded image feature implicit state and the connected super-words of all segments to generate the final structured report.
2. The method for generating a structured medical report based on multi-stage decoding and superwords according to claim 1, characterized in that: In step S1, given a set of CT scan images from four periods, {I p } p∈{C,A,P,V} , where each Represents the scan image during the p period, C represents the plain scan period, A represents the arterial phase, P represents the portal venous phase, and V represents the delayed phase. Four independent 3D visual feature extractors are trained: and These extractors extract four aspects of image features: F C 、F A 、F P and F V .
3. The method for generating a structured medical report based on multi-stage decoding and superwords according to claim 2, characterized in that: Connect the four-dimensional image feature maps along the channel dimension to form a fusion feature: F cat =concat(F C ,F A ,F P ,F V )。 4. The method for generating a structured medical report based on multi-stage decoding and superwords according to claim 3, characterized in that: In step S2, the fusion feature F cat Input transformer encoder To get the implicit state of the encoding For each segment B of the structured report k , where k = 1, 2, ..., K, K is the segment in the structured report, the total number is 6, train a block decoder The decoder generates text for the segment And train it to generate a single additional token at the end of the sequence, designating it as a super-word Superwords are not predefined tags, they are learned during training. This stage is represented as For each segment B k , the generated superword Then used as a separate super-word decoder Φ meta The only input of This super-word decoder is trained to reconstruct the text of a fragment without access to the original visual features or the encoded implicit state:
5. The method for generating a structured medical report based on multi-stage decoding and superwords according to claim 4, characterized in that: Step S3 is specifically as follows: Using the fusion decoder Φ fusion , the encoded image feature implicit state h and the concatenated super-word of all segments As input, predict the final structured report Joint training of block decoder and super word decoder: The goal of joint training is to minimize the comprehensive loss For each fragment in is the cross entropy loss of the block decoder; T k , and Report segment B k The length of the true value text, the tth token and the previous token, h is the implicit state of the image feature, is a block decoder; is the cross entropy loss of super-word decoder reconstruction, It is a super-word. is the super-word decoder; α is used to balance the two loss terms; Fusion decoder training: The fusion decoder receives the encoded image feature latent state h and the concatenated super-words of all segments: Generate final structured report The training goal is to minimize the cross entropy loss of the fusion decoder parameters Where T and is the length of the complete ground truth structured report and the Tth token; the optimization parameter θ fusion To minimize the fusion decoder loss 6. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the method according to any one of claims 1 to 5 are implemented.
7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.
8. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.