Medical image automatic reporting system and method

Abstract

The invention relates to an automatic medical image reporting system and method, and aims to output a natural, accurate and professional medical image report. The method comprises the following steps: inputting an X-ray image into a preset feature extraction model for feature extraction, and outputting to obtain a preset number of region visual features; inputting the preset number of area visual features into a multi-layer perceptron model to carry out abnormal area binary classification, and outputting to obtain abnormal area visual features; inputting the abnormal region visual features into a language model, and generating feature statements corresponding to the abnormal region visual features; and merging the feature statements to obtain a medical image report.

Description

Technical Field

[0001] This application relates to the field of medical image processing technology, and in particular to an automatic medical image reporting system and method. Background Technology

[0002] While current research on automatically generating medical image reports has achieved some progress, it still suffers from several technical shortcomings. Traditional methods typically generate single-segment text based on the overall features of the image, failing to reflect the semantic boundaries between anatomical structures in medical images, resulting in a lack of logical coherence and regional correspondence in the generated content. Furthermore, existing, better systems generally output reports in English, making it difficult to directly serve the Chinese medical environment and severely limiting their promotion and application in the domestic medical system. This is especially true in non-English-speaking countries like China, where medical staff have a strong demand for Chinese reports. Although some attempts have been made to use machine translation tools for language conversion, issues such as inaccurate terminology, disordered word order, and grammatical inconsistencies result in Chinese texts that are difficult to meet clinical requirements.

[0003] A search of existing technical literature revealed a patent application (CN118230889A) entitled "Method and System for Automatic Generation of Medical Image Reports Based on Deep Learning." This patent collects a large amount of first medical image data and corresponding report data, constructs a training dataset, and uses the training dataset to train a pre-constructed generative model. It then determines the associated feature data of the received second medical image data, determines a first segmentation resolution based on the associated feature data, and segments the second medical image data into several third medical image data according to the first segmentation resolution. Finally, it uses convolutional neural network technology to extract first medical image features from each third medical image data, inputs these first medical image features into the generative model, and the generative model converts the first medical image features into a natural language report and outputs it. However, this patent suffers from problems such as unclear content structure, poor interpretability, and insufficient professionalism in the reports. Summary of the Invention

[0004] Therefore, it is necessary to provide an automated medical image reporting system and method to address the aforementioned technical problems and achieve the output of natural, accurate, and professional medical image reports.

[0005] To solve the above-mentioned technical problems, the present invention provides the following technical solution: On one hand, the present invention provides an automatic reporting method for medical images, the method comprising: S1: Input the X-ray image into the preset feature extraction model to extract features, and output the visual features of the preset regions; S2: Input the preset visual features of each region into the multilayer perceptron model to perform binary classification of abnormal regions, and output the visual features of the abnormal regions. S3: Input the visual features of the abnormal region into the language model to generate feature sentences corresponding to the visual features of the abnormal region; S4: Merge the feature statements to obtain a medical image report.

[0006] Preferably, before step S1, the X-ray image is further subjected to normalization preprocessing.

[0007] Preferably, the preset feature extraction model includes: a feature extraction backbone layer, a multi-scale sampling adapter layer, a bidirectional learnable fusion feature pyramid network layer, and a mask R-CNN component layer; The feature extraction backbone layer includes a window attention block, a first global attention block, a second global attention block, a third global attention block, and a fourth global attention block. The normalized preprocessed X-ray image is processed sequentially through the window attention block and the first global attention block to output a first feature map, sequentially through the window attention block, the first global attention block, and the second global attention block to output a second feature map, sequentially through the window attention block, the first global attention block, the second global attention block, and the third global attention block to output a third feature map, and sequentially through the window attention block, the first global attention block, the second global attention block, the third global attention block, and the fourth global attention block to output a fourth feature map. The first feature map, the second feature map, the third feature map, and the fourth feature map are respectively processed by the multi-scale sampling adapter layer to output an adjusted first feature map, an adjusted second feature map, an adjusted third feature map, and an adjusted fourth feature map; The adjusted first feature map, the adjusted second feature map, the adjusted third feature map, and the adjusted fourth feature map are jointly processed by the bidirectional learnable fusion feature pyramid network layer to output a bottom-up first feature map, a bottom-up second feature map, a bottom-up third feature map, a bottom-up fourth feature map, and a bottom-up fifth feature map; The normalized preprocessed X-ray image, the bottom-up first feature map, the bottom-up second feature map, the bottom-up third feature map, the bottom-up fourth feature map, and the bottom-up fifth feature map are all passed through the mask R-CNN component layer to output a preset number of visual features for each region.

[0008] Preferably, the bidirectional learnable fusion feature pyramid network layer includes: The adjusted fourth feature map is processed by the first convolution to obtain a top-down fourth feature map. The top-down fourth feature map is then upsampled and fused with the adjusted third feature map through a gated fusion mechanism to obtain a top-down third feature map. The top-down third feature map is then upsampled and fused with the adjusted second feature map through a gated fusion mechanism to obtain a top-down second feature map. The top-down second feature map is then upsampled and fused with the adjusted first feature map through a gated fusion mechanism to obtain a top-down first feature map. The first feature map from top to bottom is obtained by a second convolution. The first feature map from bottom to top is then downsampled and fused with the second feature map from top to bottom through a gating fusion mechanism to obtain the second feature map from bottom to top. The second feature map from bottom to top is then downsampled and fused with the third feature map from top to bottom through a gating fusion mechanism to obtain the third feature map from bottom to top. The third feature map from bottom to top is then downsampled and fused with the fourth feature map from top to bottom through a gating fusion mechanism to obtain the fourth feature map from bottom to top. The fourth feature map from bottom to top is then downsampled and obtained as the fifth feature map from bottom to top.

[0009] Preferably, the multilayer perceptron model includes: The preset visual features of each region are sequentially passed through a first linear layer, a first ReLU activation layer, a second linear layer, a second ReLU activation layer, a third linear layer, and a third ReLU activation layer to output the visual features of the abnormal region.

[0010] Preferably, the language model adopts the GPT-2 model.

[0011] Preferably, before step S3, the method further includes: The GPT-2 model is trained and its parameters are adjusted to obtain a fine-tuned GPT-2 model.

[0012] Preferably, the medical imaging report is in English, and after step S4, the following steps are also included: The medical imaging report and medical terms from the medical terminology knowledge base are input into the grammar optimization model for translation, and a Chinese version of the medical imaging report is output.

[0013] Preferably, the syntax optimization model includes any one of T5-Base, mT5-Small, BART, PEGASUS, and a prompt-based Chinese GPT decoder structure.

[0014] On the other hand, the present invention provides an automatic medical image reporting system, the system comprising: a feature extraction module, an abnormal region identification module, a sentence generation module, and a report generation module; The feature extraction module is used to input the X-ray image into a preset feature extraction model for feature extraction and output the visual features of a preset region. The abnormal region identification module is used to input the preset visual features of the regions into the multilayer perceptron model to perform binary classification of the abnormal regions and output the visual features of the abnormal regions. The statement generation module is used to input the visual features of the abnormal region into the language model and generate feature statements corresponding to the visual features of the abnormal region. The report generation module is used to merge the feature statements to obtain a medical image report.

[0015] Compared with the prior art, the beneficial effects of the present invention are: This invention provides an automatic medical image reporting system and method. Based on the existing Faster R-CNN_ResNet50_FPN_V2 model, an improved feature extraction model is used for feature extraction, enhancing the model's target detection capabilities. Furthermore, the model innovatively employs a bidirectional learnable fusion feature pyramid network to replace the traditional FPN structure, fusing multi-scale features bidirectionally (top-down and bottom-up). A gating fusion mechanism is introduced during the fusion process to enhance feature representation, better capturing contextual information. This ensures that each diagnostic statement establishes a one-to-one correspondence with a specific anatomical region in the image, significantly improving the structure and interpretability of the report. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of a medical image automatic reporting method in one embodiment; Figure 2 This is a schematic diagram of the Faster R-CNN_ResNet50_FPN_V2 model structure, which improves the automatic medical image reporting method in one embodiment. Figure 3 This is a schematic diagram of a multilayer perceptron model structure for an automated medical image reporting method in one embodiment. Figure 4 This is a schematic diagram of a first X-ray film in one embodiment of the automatic medical image reporting method; Figure 5 This is a schematic diagram of a second X-ray film in one embodiment of the automatic medical image reporting method; Figure 6 This is a schematic diagram of the architecture of an automated medical image reporting system in one embodiment; Figure 7 This is a schematic diagram of the complete process of an automated medical image reporting method in one embodiment. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0018] Example 1 like Figure 1 As shown in the figure, this embodiment proposes an automatic medical image reporting method, the method comprising: S1: Input the X-ray image into the preset feature extraction model to extract features, and output the visual features of the preset regions; The specific implementation of this step is as follows: the preset feature extraction model is an improved Faster RCNN_ResNet50_FPN_V2 model. On the top-down path of the original Feature Pyramid Network (FPN), a deep neural network with bidirectional learnable fusion, content-aware resampling, and lightweight attention / alignment mechanisms is introduced to perform target detection on X-ray images, extracting 11 anatomical regions and obtaining their features. The preset feature extraction model structure is as follows: Figure 2 As shown, the overall process can be understood in the order of input, feature extraction, multi-scale alignment, bidirectional fusion FPN, and R-CNN detection, corresponding to the traditional structure. The traditional "ResNet-50 backbone + FPN + R-CNN" of Faster R-CNN_ResNet50_FPN_v2 is replaced here with a backbone feature extraction structure composed of a hierarchical Transformer encoder and a multi-scale sampling adapter as shown in the figure. Then, the bidirectional learnable fusion FPN completes feature fusion, and finally, the R-CNN component completes detection and feature extraction. Therefore, the essential structure can still be classified as "backbone network + FPN + R-CNN", with innovative designs implemented within the backbone network and FPN.

[0019] S2: Input the preset visual features of each region into the multilayer perceptron model to perform binary classification of abnormal regions, and output the visual features of the abnormal regions. S3: Input the visual features of the abnormal region into the language model to generate feature sentences corresponding to the visual features of the abnormal region; S4: Merge the feature statements to obtain a medical image report.

[0020] Example 2 This embodiment further provides a supplementary explanation of the automatic medical image reporting method proposed in the above embodiments.

[0021] Before step S1, the X-ray image is further subjected to normalization preprocessing.

[0022] The specific implementation of this step is as follows: the input X-ray image size is [3, H, W] (assuming it is converted to 3 channels), where H and W are the height and width of the image. The image is preprocessed (such as normalized) and then sent to the feature extraction backbone layer.

[0023] The preset feature extraction model includes: a feature extraction backbone layer, a multi-scale sampling adapter layer, a bidirectional learnable fusion feature pyramid network layer, and a mask R-CNN component layer; The feature extraction backbone layer includes a window attention block, a first global attention block, a second global attention block, a third global attention block, and a fourth global attention block. The normalized preprocessed X-ray image is processed sequentially through the window attention block and the first global attention block to output a first feature map, sequentially through the window attention block, the first global attention block, and the second global attention block to output a second feature map, sequentially through the window attention block, the first global attention block, the second global attention block, and the third global attention block to output a third feature map, and sequentially through the window attention block, the first global attention block, the second global attention block, the third global attention block, and the fourth global attention block to output a fourth feature map. The first feature map, the second feature map, the third feature map, and the fourth feature map are respectively processed by the multi-scale sampling adapter layer to output an adjusted first feature map, an adjusted second feature map, an adjusted third feature map, and an adjusted fourth feature map; The adjusted first feature map, the adjusted second feature map, the adjusted third feature map, and the adjusted fourth feature map are jointly processed by the bidirectional learnable fusion feature pyramid network layer to output a bottom-up first feature map, a bottom-up second feature map, a bottom-up third feature map, a bottom-up fourth feature map, and a bottom-up fifth feature map; The normalized preprocessed X-ray image, the bottom-up first feature map, the bottom-up second feature map, the bottom-up third feature map, the bottom-up fourth feature map, and the bottom-up fifth feature map are all passed through the mask R-CNN component layer to output a preset number of visual features for each region.

[0024] The specific implementation of this step is as follows, such as Figure 2As shown, the feature extraction backbone layer consists of multiple Transformer blocks, including window attention blocks and global attention blocks, used to extract multi-scale features. The figure shows the "window block" repeated four times, followed by "block 1" (window attention block), "block d / 4" (first global attention block), "block 2d / 4" (second global attention block), "block 3d / 4" (third global attention block), and "block d" (fourth global attention block). Here, d represents the total depth of the backbone (total number of blocks). The attention mechanism gradually transitions from window attention to global attention to balance computational efficiency and feature capture capability. After passing through the feature extraction backbone layer, the normalized preprocessed X-ray image of size [3, H, W] outputs four feature maps at different depths (similar to stage outputs in CNNs), denoted as C1, C2, C3, and C4 (first feature map, second feature map, third feature map, and fourth feature map), where: C1 comes from a shallower layer (such as after block 1) and has a higher resolution (e.g., [H / 4, W / 4, C]). C2 comes from intermediate layers (such as after block d / 4) and has a medium resolution. C3 comes from deeper layers (such as after block 2d / 4) and has lower resolution; C4 comes from the deepest layer (such as after block d) and has the lowest resolution (e.g., [H / 32, W / 32, C]).

[0025] The multi-scale sampling adapter layer is a scale alignment module that adjusts the size of the feature maps output by the feature extraction backbone layer through upsampling, downsampling, and identity mapping. This ensures that features at different scales are aligned, preparing for the subsequent fusion of the bidirectional learnable fusion feature pyramid network layer. The figure shows the operations of "upsampling 4x", "upsampling 2x", "downsampling 2x", and "identity activation". After C1, C2, C3, and C4 are input to the multi-scale sampling adapter layer, C1 is downsampled by 2x to reduce the resolution, C2 is activated by identity (keeping it as is), C3 is upsampled by 2x to increase the resolution, and C4 is upsampled by 4x to significantly increase the resolution. The final output is the adjusted multi-scale feature maps, denoted as A1, A2, A3, and A4 (adjusting the first, second, third, and fourth feature maps). All feature maps are adjusted to the same number of channels but with different resolutions, forming a preliminary pyramid structure.

[0026] The Bidirectional Learnable Fusion Feature Pyramid Network layer is an improved feature pyramid network that bidirectionally (top-down and bottom-up) fuses multi-scale features and introduces gating mechanisms and alignment operations to enhance feature representation. It replaces the traditional FPN and can better capture contextual information. After A1, A2, A3, and A4 are input to the Bidirectional Learnable Fusion Feature Pyramid Network layer, it first fuses high-resolution features (A1) to low-resolution features (A4) (top-down path), using 1x1 convolutions and upsampling to pass semantic information, and then fuses low-resolution features (A4) to high-resolution features (A4). The resolution features (A1) are fused (bottom-up path) using downsampling and convolution to pass in detail information. Each fusion step uses learnable gating weights to adjust feature contributions and ensure feature alignment. The final outputs are P1, P2, P3, P4, and P5 (bottom-up first feature map, bottom-up second feature map, bottom-up third feature map, bottom-up fourth feature map, and bottom-up fifth feature map). Each level has the same number of channels, but the resolution decreases (P1 has the highest resolution, and P5 has the lowest). These feature maps are used for subsequent region proposal and feature extraction.

[0027] The Mask R-CNN component layer includes a Region Proposal Network (RPN), ROI pooling (or ROI alignment), a classification head, a bounding box regression head, and a masking head. It is responsible for generating region proposals and extracting visual features for each region. The feature pyramids P1 to P5 output by the bidirectional FPN, along with the normalized preprocessed X-ray image, are input to the Mask R-CNN component layer. The RPN then generates candidate region proposals (ROIs) by sliding a window across the feature pyramid. The RPN uses an anchoring mechanism to output multiple proposals and their scores. Based on the proposals generated by the RPN, fixed-size features (e.g., 7x7) are extracted from the feature pyramid through ROI alignment. The model is configured to output a fixed number of ROIs, here 11. These 11 extracted features are fed into a fully connected layer for classification (object category) and bounding box regression. In parallel, these 11 features are also used for mask prediction, generating a segmentation mask for each object. The final output consists of 11 visual features for each region, each a feature vector corresponding to one of the 11 ROIs. These features are used in downstream tasks, and the detection results (bounding boxes, class labels, and segmentation masks) are also output.

[0028] The bidirectional learnable fusion feature pyramid network layer includes: The adjusted fourth feature map is processed by the first convolution to obtain a top-down fourth feature map. The top-down fourth feature map is then upsampled and fused with the adjusted third feature map through a gated fusion mechanism to obtain a top-down third feature map. The top-down third feature map is then upsampled and fused with the adjusted second feature map through a gated fusion mechanism to obtain a top-down second feature map. The top-down second feature map is then upsampled and fused with the adjusted first feature map through a gated fusion mechanism to obtain a top-down first feature map. The first feature map from top to bottom is obtained by a second convolution. The first feature map from bottom to top is then downsampled and fused with the second feature map from top to bottom through a gating fusion mechanism to obtain the second feature map from bottom to top. The second feature map from bottom to top is then downsampled and fused with the third feature map from top to bottom through a gating fusion mechanism to obtain the third feature map from bottom to top. The third feature map from bottom to top is then downsampled and fused with the fourth feature map from top to bottom through a gating fusion mechanism to obtain the fourth feature map from bottom to top. The fourth feature map from bottom to top is then downsampled and obtained as the fifth feature map from bottom to top.

[0029] The specific implementation of this step is as follows: In the bidirectional learnable fusion feature pyramid network, the transformation process of the A1-A4 feature maps forms a precise bidirectional information flow system. The A1-A4 feature maps output by the multi-scale sampling adapter layer first enter the top-down path. Starting from the deepest layer A4, after unifying the number of channels through a 1x1 convolution (first convolution), the initial P4_1 (top-down fourth feature map) is obtained. Then, it is upsampled by 2 times (first upsampling) and fused with the previous layer feature A3 to generate P3_1 (top-down third feature map). P3_1 is upsampled by 2 times again (second upsampling) and fused with A2 to obtain P2_1 (top-down second feature map). Finally, P2_1 is upsampled by 2 times again (third upsampling) and fused with A1 to generate P1_1 (top-down first feature map). This completes the top-down semantic information propagation path. Next, the network enters a bottom-up path, starting with the highest resolution feature P1_1 obtained earlier and performing reverse information transfer. P1_1 undergoes a 1x1 convolution (second convolution) to obtain P1 (the first bottom-up feature map). Then, after a 2x downsampling (first downsampling), it is fused with P2_1 to generate P2 (the second bottom-up feature map). P2 is then downsampled again (second downsampling) and fused with P3_1 to obtain P3 (the third bottom-up feature map). P3 is further downsampled again (third downsampling) and fused with P4_1 to generate P4 (the fourth bottom-up feature map). Finally, an additional 2x downsampling operation is performed on P4 (fourth downsampling) to obtain P5 (the fifth bottom-up feature map). This bottom-up path effectively transfers high-resolution detail information to deeper features. After completing the bidirectional fusion, channel attention and dilated convolution can be applied to the fusion result to ensure that the pyramid features retain fine-grained attention to small targets while enhancing the expression of long-range semantics, while maintaining low computational cost.

[0030] Throughout the transformation process, the gating fusion mechanism plays a crucial role. Each fusion node adaptively adjusts the importance of features from different sources through learnable gating weights, while using alignment operations to ensure precise spatial correspondence of multi-scale features. This design enables the network to intelligently balance the contribution of features from each layer, avoiding information conflicts that may arise from simple feature addition. The resulting P1-P5 feature pyramid constitutes a hierarchical multi-scale representation system. P1 retains the richest spatial detail information, suitable for detecting small targets; P2 and P3 achieve a good balance between spatial detail and semantic information; P4 and P5 provide a larger receptive field, specifically for handling medium and large target detection. This bidirectional fusion mechanism ensures that the features at each level contain both deep semantic information and shallow detail features, providing strong multi-scale feature support for subsequent detection tasks.

[0031] Compared to the standard FPN which only fuses features from top to bottom, the bidirectional learnable feature pyramid network has four key improvements: Two-way information flow: The fusion process is no longer unidirectional, but a two-way back-and-forth "downward first, then upward," allowing shallow textures and deep semantics to fully exchange information in two rounds of propagation. The aligned multi-scale features are sequentially fused downward (enhancing semantics) and upward (restoring details), ultimately resulting in a set of semantically enhanced and structurally detailed pyramid features; Learnable normalized weighting: At each fusion node, inputs from different scales are first passed through non-negative weights and normalized summation is performed. This is equivalent to setting a learnable gating, which enables the network to automatically allocate the contribution ratio of shallow and deep layers according to the input content. The final output fusion feature is more stable than that of the standard FPN. Content-aware upsampling and alignment: Upsampling uses a content-aware reconstruction method, while downsampling uses separable convolution to reduce misalignment and scaling artifacts during cross-layer fusion, making the fused features more accurate in structural regions such as edges and skeletal contours. Lightweight attention enhancement: After completing the bidirectional fusion, channel attention and dilated convolution can be optionally applied to the fusion result, so that the pyramid features can retain fine-grained attention to small targets and enhance the expression of long-range semantics, while maintaining low computational cost.

[0032] The multilayer perceptron model includes: The preset visual features of each region are sequentially passed through a first linear layer, a first ReLU activation layer, a second linear layer, a second ReLU activation layer, a third linear layer, and a third ReLU activation layer to output the visual features of the abnormal region.

[0033] The specific implementation of this step is as follows, such as Figure 3 As shown, 11 visual features (shape [11, d_v]) of different regions are input into a binary classifier (multilayer perceptron model) or a more complex structure (such as a self-attention module) to determine whether each region is abnormal. Each region's visual feature is input into the network for binary classification, and a binary label (abnormal or normal) is output. Only the features of regions classified as abnormal are retained. For example, if K of the 11 regions are judged as abnormal, then 5 abnormal region visual features with shape [K, d_v] are output. These feature vectors will be fed into the language model.

[0034] The language model uses the GPT-2 model. Before step S3, the GPT-2 model is trained and its parameters are adjusted to obtain a fine-tuned GPT-2 model.

[0035] The specific implementation of this step is as follows: the input to GPT-2 is usually a starting token (such as <|startoftext|>) or a sequence of preceding tokens, used to initialize sentence generation. For each anomalous region, a pseudo-self-attention mechanism is introduced based on the fine-tuned GPT-2 model to inject visual features. The specific association and injection process is as follows: In the standard GPT-2 self-attention mechanism, each text token calculates attention through a Query(Q), Key(K), and Value(V) matrix, focusing on the preceding token sequence. In the pseudo-self-attention mechanism, we add a visual information pathway, enabling the text token to simultaneously focus on the visual features of the current region during generation. For each anomalous region, its anomalous region visual feature vector... (Dimension dv) First, it is projected onto a dimensional d model with the same dimensions as the text token through two independent linear layers (the hidden dimension of GPT-2 used in this embodiment is 768) to generate a visual key. and visual value :

[0036]

[0037] in and It is a learnable weight matrix, output and The shape of each element is [1, d_model]. In each self-attention layer of GPT-2, for a text token sequence T=[t1, t2, ..., tn] (each ti dimension is d_model), the text query is calculated. =TWq、Text Key =TWk、Text Value =TW. Then the visual key and visual value By concatenating the text key and value, we get =[ , ], =[ , The concatenation operation here is performed along the sequence length dimension, therefore The shape is [n+1, d_model]. They have the same shape. The attention output is calculated as follows:

[0038] The attention weights now consider both text tokens and visual features, where the text tokens "see" visual information. For each anomalous region, we use its corresponding visual feature vector. The standalone GPT-2 generation process is as follows: Initialize the text sequence as the starting token; At each step of generating a new token, the pseudo-self-attention mechanism will... As an additional key / value injection, this ensures that the generated sentences are influenced by the visual features of that region; The generation process continues until the end token (e.g., <|endoftext|>) or the maximum length is reached; For each abnormal region, generate a separate English sentence describing the abnormal situation in that region. Finally, output K sentences, each corresponding to an abnormal region.

[0039] The medical imaging report is in English. After step S4, the following steps are also included: The medical imaging report and medical terms from the medical terminology knowledge base are input into the grammar optimization model for translation, and a Chinese version of the medical imaging report is output.

[0040] The syntax optimization model includes any one of T5-Base, mT5-Small, BART, PEGASUS, and a Chinese GPT decoder structure based on prompting learning.

[0041] The specific implementation of this step is as follows: the complete process of generating a medical image report, including a Chinese conversion module, is as follows. Figure 7 As shown, this system integrates region detection, anomaly recognition, diagnostic text generation, and Chinese language conversion. This embodiment uses a first X-ray film and a second X-ray film to generate a medical image report, as shown below. Figure 4 As shown, the reference report corresponding to the first X-ray film is as follows: Lung volumes remain low. There are innumerable bilateral scattered small pulmonary nodules, which are better demonstrated on recent CT. Mild pulmonary vascular congestion is stable. The cardiomediastinal silhouette and hilar contours are unchanged. Small pleural effusion in the right middle fissure is new. There is no new focal opacity suggesting pneumonia. There is no pneumothorax. The English version of the medical image report generated by the method in this embodiment is as follows: The lung volumes are low. There is no pneumothorax. Bibasilaratelectasis unchanged. No pulmonary edema. No free air below the righthemidiaphragm is seen. There is no pneumothorax or pleural effusion. There ismild bibasilar atelectasis. The Chinese version of the medical image report translated using the method in this embodiment is as follows: Low lung capacity. No pneumothorax. No changes in bilateral base atelectasis. No pulmonary edema. No free air seen below the diaphragm on the right side. No pneumothorax or pleural effusion. Mild bilateral base atelectasis. No acute bone abnormalities. Mediastinal contour within normal range. NG tube tip in the stomach.

[0042] The intermediate process of generating the report provides the generation statements for 11 regions, including the right lung and right lower lung: Right lung: The lung volumes are low. There is no pleural effusion or pneumothorax. In the right lower lung zone, bibasilar atelectasis remains unchanged. Right hilar structures: No pulmonary edema. Right hemidiaphragm: No free air is visible below the right hemidiaphragm. Left lung: There is no pneumothorax or pleural effusion. Left lower lung zone: Mild bibasilar atelectasis. Spine: No acute osseous abnormalities. Mediastinum: The cardiomediastinal silhouette is within normal limits. Upper mediastinum: The cardiomediastinal silhouette is within normal limits. Cardiac silhouette: The cardiomediastinal silhouette is normal. Abdomen: The tip of the NG tube is in the stomach. During the intermediate process of generating the report, the generated statements in some areas are fine-tuned to create a report that is more realistic: There is no pneumothorax or pleural effusion. (This means there is no pneumothorax or pleural effusion.) The cardiomediastinal silhouette is within normal limits. (This means the cardiomediastinal silhouette is normal.) like Figure 5 As shown, the reference report corresponding to the second X-ray film is as follows: Heart size is at the upper limits of normal. Mediastinal hilar contours are normal. Lungs are clear except for linear bibasilar atelectasis and / or scarring. Skeletal structures have been more fully assessed by a recent skeletal survey of 1 day earlier. The English version of the medical image report generated by the method in this embodiment is as follows: There is no pleural effusion or pneumothorax. No acutecardiopulmonaryprocess. The mediastinal and hilar contours are normal. Nofree air below the righthemidiaphragm is seen. Bibasilar atelectasis isunchanged. The cardiomediastinal silhouetteis within normal limits. Heartsize is normal. The Chinese version of the medical image report translated using the method in this embodiment is as follows: No pleural effusion or pneumothorax. No acute cardiopulmonary disease. Normal mediastinal and hilar contours. No free air seen below the diaphragm on the right side. No changes in atelectasis at the lung bases. Mediastinal contours within the normal range. Normal heart size.

[0043] The intermediate process of generating the report provides the generation statements for 10 regions, including the right lung and right lower lung: Right lung: There is no pleural effusion or pneumothorax. No acute cardiopulmonary process. Bibasilar atelectasis is noted in the right lower lung zone. Right hilar structures: The mediastinal and hilar contours are normal. Right hemidiaphragm: No free air is seen below the right hemidiaphragm. Left lung: There is no pleural effusion or pneumothorax. In the left lower lung zone, bibasilar atelectasis remains unchanged. Left hilar structures: The mediastinal and hilar contours are normal. Mediastinum: The cardiomediastinal silhouette is normal. Upper mediastinum: The cardiomediastinal silhouette is within normal limits. Cardiac silhouette: Heart size is normal. During the intermediate process of generating the report, the generated statements in some areas are fine-tuned to create a report that is more realistic: Bibasilar atelectasis is unchanged. == Bibasilar atelectasis is noted. (No change in basal atelectasis. == Bibasilar atelectasis is noted.) The cardiomediastinal silhouette is within normal limits. (This means the cardiomediastinal silhouette is normal.) Example 3 like Figure 6 As shown, this embodiment proposes an automatic medical image reporting system, including: a feature extraction module, an abnormal region identification module, a sentence generation module, and a report generation module; The feature extraction module is used to input the X-ray image into a preset feature extraction model for feature extraction and output the visual features of a preset region. The abnormal region identification module is used to input the preset visual features of the regions into the multilayer perceptron model to perform binary classification of the abnormal regions and output the visual features of the abnormal regions. The statement generation module is used to input the visual features of the abnormal region into the language model and generate feature statements corresponding to the visual features of the abnormal region. The report generation module is used to merge the feature statements to obtain a medical image report.

Claims

1. A method of automatic reporting of medical images, characterized in that, The method comprises the following steps: S1: inputting an X-ray image into a preset feature extraction model for feature extraction, and outputting a preset number of regional visual features; S2: inputting the preset number of regional visual features into a multi-layer perception model for binary classification of abnormal regions, and outputting abnormal region visual features; S3: inputting the abnormal region visual features into a language model to generate feature sentences corresponding to the abnormal region visual features; S4: merging the feature sentences to obtain a medical image report.

2. The medical image automatic reporting method according to claim 1, characterized in that, Before step S1, the X-ray image is subjected to normalization preprocessing.

3. The medical image automatic reporting method according to claim 2, characterized in that, The preset feature extraction model comprises a feature extraction backbone layer, a multi-scale sampling adapter layer, a bidirectional learnable fusion feature pyramid network layer, and a mask R-CNN component layer. The feature extraction backbone layer comprises a window attention block, a first global attention block, a second global attention block, a third global attention block, and a fourth global attention block. The normalized X-ray image is sequentially input into the window attention block and the first global attention block to output a first feature map, sequentially input into the window attention block, the first global attention block, and the second global attention block to output a second feature map, sequentially input into the window attention block, the first global attention block, the second global attention block, and the third global attention block to output a third feature map, and sequentially input into the window attention block, the first global attention block, the second global attention block, the third global attention block, and the fourth global attention block to output a fourth feature map. The first feature map, the second feature map, the third feature map, and the fourth feature map are respectively input into the multi-scale sampling adapter layer to output an adjusted first feature map, an adjusted second feature map, an adjusted third feature map, and an adjusted fourth feature map. The adjusted first feature map, the adjusted second feature map, the adjusted third feature map, and the adjusted fourth feature map are collectively input into the bidirectional learnable fusion feature pyramid network layer to output a bottom-up first feature map, a bottom-up second feature map, a bottom-up third feature map, a bottom-up fourth feature map, and a bottom-up fifth feature map. The normalized X-ray image, the bottom-up first feature map, the bottom-up second feature map, the bottom-up third feature map, the bottom-up fourth feature map, and the bottom-up fifth feature map are collectively input into the mask R-CNN component layer to output a preset number of regional visual features.

4. The medical image automatic reporting method according to claim 3, characterized in that, The bidirectional learnable fusion feature pyramid network layer comprises: The adjusted fourth feature map is input into a first convolution to obtain a top-down fourth feature map, the top-down fourth feature map is input into a first upsampling to fuse with the adjusted third feature map through a gating fusion mechanism to obtain a top-down third feature map, the top-down third feature map is input into a second upsampling to fuse with the adjusted second feature map through a gating fusion mechanism to obtain a top-down second feature map, and the top-down second feature map is input into a third upsampling to fuse with the adjusted first feature map through a gating fusion mechanism to obtain a top-down first feature map. The top-down first feature map is obtained after the second convolution, and the bottom-up first feature map is obtained after the first down-sampling.

5. The medical image automatic reporting method of claim 1, wherein, The multi-layer perception model comprises: The abnormal region visual features are output after the preset region visual features sequentially pass through a first linear layer, a first ReLU activation layer, a second linear layer, a second ReLU activation layer, a third linear layer, and a third ReLU activation layer.

6. The medical image automatic reporting method of claim 1, wherein, The language model adopts a GPT-2 model.

7. The medical image automatic reporting method according to claim 6, wherein, Before step S3, the method further comprises: The GPT-2 model is trained and adjusted to obtain a fine-tuned GPT-2 model.

8. The medical image automatic reporting method according to claim 7, characterized in that, The medical image report is an English version, and after step S4, the method further comprises: The medical image report and medical terms in the medical term knowledge base are input into a grammar optimization model for translation, and a Chinese version of the medical image report is output.

9. The medical image automatic reporting method according to claim 8, wherein, The grammar optimization model comprises any one of T5-Base, mT5-Small, BART, PEGASUS, and a Chinese GPT decoder structure based on prompt learning.

10. An automated medical image reporting system, characterized by Comprise: a feature extraction module, an abnormal region identification module, a sentence generation module, and a report generation module; The feature extraction module is configured to input an X-ray image into a preset feature extraction model for feature extraction, and output a preset region visual feature; The abnormal region identification module is configured to input the preset region visual feature into a multi-layer perception model for abnormal region binary classification, and output an abnormal region visual feature; The sentence generation module is configured to input the abnormal region visual feature into a language model to generate a feature sentence corresponding to the abnormal region visual feature; The report generation module is configured to merge the feature sentence to obtain a medical image report.

Images