A medical image report generation method and system based on multi-feature enhancement

Abstract

The application discloses a kind of based on multi-feature enhancement medical image report generation method and system, method includes: using superpixel segmentation technology is segmented to medical image, obtain region of interest;Using pre-trained DenseNet is extracted to local feature to region of interest, and local feature is as graph node to construct relation graph;The relation graph is input into graph network GAT to learn graph feature, and the graph feature is input into encoder;The feature obtained by each layer of the encoder is output to the corresponding decoder layer in the form of U-type connection, the decoder predicts the next word based on the image feature obtained in the encoder and the generated text sequence feature, until all words are generated, finally obtain complete image report.Through the technical scheme of the application, the recognition of local lesions is enhanced, the interaction between text features and image features is strengthened, the semantic details and feature relationships of medical images are strengthened, and the accuracy of the generated report is improved.

Description

Technical Field

[0001] This invention relates to the field of pattern recognition technology, and in particular to a method and system for generating medical image reports based on multi-feature enhancement. Background Technology

[0002] Medical imaging technology is the use of various imaging methods to acquire information about the internal structure and function of the human body, encompassing CT (computed tomography), MRI (magnetic resonance imaging), X-rays, ultrasound, and many others. These images provide doctors with a direct view, enabling them to accurately locate lesions, assess their size and shape, and analyze their relationship with surrounding tissues. This allows for a deeper understanding of the patient's internal structure and the nature of the disease, providing a basis for diagnosis. In clinical practice, experienced radiologists typically write preliminary analysis reports based on the imaging data. These reports not only include a preliminary diagnosis of the lesion location in the images but also provide important references for the attending physician, assisting in the development of subsequent treatment plans.

[0003] In automatic chest image report generation algorithms, report generation primarily relies on relevant information extracted and understood from the image. However, medical images differ from natural images; they typically have higher resolution and smaller regions of interest (such as lesions). These lesion regions occupy a very small proportion of the overall image, and therefore may be overlooked during feature extraction due to their small size, leading to the loss of crucial information. Furthermore, the tissue structures in medical images are often quite similar, resulting in low contrast between lesions and the background, further increasing the difficulty of lesion identification. In addition, variations in shooting angle and time can cause inconsistencies in the brightness of medical images; these factors combined can lead to image quality degradation, thus interfering with accurate lesion identification.

[0004] In actual diagnostic processes, radiological reports generated from medical images are crucial for doctors to diagnose diseases. The accuracy of these reports often depends on small, easily overlooked local lesions in the images. Therefore, effectively extracting the features of these subtle lesions is essential for improving diagnostic accuracy. Currently, many medical image feature extraction methods rely on traditional deep neural network structures, such as ResNet and DenseNet. However, these networks have limitations in capturing long-range dependencies. Although the ViT (Vision Transformer) model has made significant progress in long-range dependency modeling through its self-attention mechanism, its performance is often inferior to convolutional neural network (CNN)-based methods when dealing with smaller medical datasets. Furthermore, a major problem with ViT is its simple division of images into multiple patches, which ignores the spatial relationships between tissues in medical images—relationships crucial for accurately identifying the nature and location of lesions.

[0005] Existing medical image analysis methods often neglect the extraction of subtle lesion information in images, especially the spatial relationships and functional connections between local lesions. For example, although some studies have used Faster R-CNN models to obtain information about local abnormal regions, these methods fail to fully consider the potential layout anomalies or functional connections between different lesions. While Faster R-CNN can effectively detect local anomalies, it is an independent object detection framework and does not effectively model the contextual relationships between lesions. Therefore, although it can identify the location of lesions, it does not consider the mutual influence and connections between lesions, greatly limiting the comprehensive understanding of disease development. Furthermore, the training process of Faster R-CNN itself has high data requirements and requires additional manual annotation and training, which undoubtedly significantly increases the workload and time cost of system development.

[0006] Another pressing issue is the fusion of image and text features. Existing methods typically process image and text features separately, failing to fully explore the potential connections between them. In medical image analysis, image information is closely related to relevant medical reports or diagnostic texts. How to effectively fuse these two types of information to improve lesion detection and classification remains an important research direction. Summary of the Invention

[0007] To address the aforementioned issues, this invention provides a medical image report generation method and system based on multi-feature enhancement. It utilizes superpixel segmentation technology to divide medical images into small blocks with similar texture and color features, effectively separating regions of interest. This allows for better extraction of key information from medical images and enhances the identification of local lesions. Furthermore, it employs a graph network (GAT) to learn the relationships between different regions, thereby learning the relationships between the tissue structures of the medical image. The regional features learned by the graph network are then fused with the global features of the entire image and input into an encoder-decoder network for sentence generation. This strengthens the interaction between text features and image features, enhancing the semantic details and feature relationships of the medical image and improving the accuracy of the generated report.

[0008] To achieve the above objectives, the present invention provides a method for generating medical image reports based on multi-feature enhancement, comprising:

[0009] Superpixel segmentation technology is used to segment medical images to obtain regions of interest;

[0010] The pre-trained DenseNet is used to extract local features from the region of interest, and the extracted local features are used as graph nodes to construct a relationship graph.

[0011] The relationship graph is input into the graph network GAT to learn graph features, and the graph features are then input into the encoder;

[0012] The features obtained from each layer of the encoder are output to the corresponding decoder layer in a U-shaped connection. The decoder predicts the next word based on the image features obtained from the encoder and the features of the generated text sequence, until all words are generated, and finally a complete image report is obtained.

[0013] In the above technical solution, preferably, the step of segmenting the medical image using superpixel segmentation technology to obtain the region of interest includes the following specific process:

[0014] Input a medical image, and segment the medical image using superpixel segmentation (SLIC) technology according to the texture and color features of the medical image;

[0015] The medical image is divided into a preset number of local regions to separate the corresponding number of regions of interest.

[0016] In the above technical solution, preferably, the step of using a pre-trained DenseNet to extract local features from the region of interest and using the extracted local features as graph nodes to construct a relationship graph includes the following specific steps:

[0017] A pre-trained DenseNet is used to extract local features from a predetermined number of regions of interest;

[0018] Each extracted local feature is treated as a graph node, and the mean color difference between adjacent regions of interest is calculated as the weight of the edge between each pair of adjacent regions to construct a relationship graph.

[0019] In the above technical solution, preferably, the relationship graph is input into a graph network (GAT) to learn graph features, and the graph features are input into an encoder. The specific process includes:

[0020] The relationship graph is input into the graph network GAT to learn the relationship between the local regions corresponding to each graph node, thereby obtaining the graph features;

[0021] The graph features are input into the encoder in a U-transformer-based codec.

[0022] In the above technical solution, preferably, the features obtained from each layer of the encoder are output to the corresponding decoder layer in a U-shaped connection. The decoder predicts the next word based on the image features obtained from the encoder and the features of the generated text sequence, until all words are generated, and finally a complete image report is obtained. The specific process includes:

[0023] A U-shaped connection is constructed between the encoder and decoder in the transformer codec network to form a U-transformer codec.

[0024] The image features obtained from the first layer of the encoder in the U-transformer encoder-decoder are output to the last layer of the decoder, and the outputs are connected sequentially until the image features obtained from the last layer of the encoder are output to the first layer of the decoder.

[0025] The decoder predicts the next word based on the image features obtained from the encoder and the text features of the generated sequence, and so on, until a complete image report is generated.

[0026] This invention also proposes a medical image report generation system based on multi-feature enhancement, which applies the medical image report generation method based on multi-feature enhancement disclosed in any of the above technical solutions, including:

[0027] The image segmentation module is used to segment medical images using superpixel segmentation technology to obtain regions of interest;

[0028] The feature extraction module is used to extract local features from the region of interest using a pre-trained DenseNet, and to use the extracted local features as graph nodes to construct a relationship graph.

[0029] The feature learning module is used to input the relationship graph into the graph network GAT to learn graph features, and input the graph features into the encoder;

[0030] The report generation module is used to output the features obtained from each layer of the encoder to the corresponding decoder layer in a U-shaped connection. The decoder predicts the next word based on the image features obtained from the encoder and the features of the generated text sequence, until all words are generated, and finally a complete image report is obtained.

[0031] In the above technical solution, preferably, the image segmentation module is specifically used for:

[0032] Input a medical image, and segment the medical image using superpixel segmentation (SLIC) technology according to the texture and color features of the medical image;

[0033] The medical image is divided into a preset number of local regions to separate the corresponding number of regions of interest.

[0034] In the above technical solution, preferably, the feature extraction module is specifically used for:

[0035] A pre-trained DenseNet is used to extract local features from a predetermined number of regions of interest;

[0036] Each extracted local feature is treated as a graph node, and the mean color difference between adjacent regions of interest is calculated as the weight of the edge between each pair of adjacent regions to construct a relationship graph.

[0037] In the above technical solution, preferably, the feature learning module is specifically used for:

[0038] The relationship graph is input into the graph network GAT to learn the relationship between the local regions corresponding to each graph node, thereby obtaining the graph features;

[0039] The graph features are input into the encoder in a U-transformer-based codec.

[0040] In the above technical solution, preferably, the report generation module is specifically used for:

[0041] A U-shaped connection is constructed between the encoder and decoder in the transformer codec network to form a U-transformer codec;

[0042] The image features obtained from the first layer of the encoder in the U-transformer encoder-decoder are output to the last layer of the decoder, and the outputs are connected sequentially until the image features obtained from the last layer of the encoder are output to the first layer of the decoder.

[0043] The decoder predicts the next word based on the image features obtained from the encoder and the text features of the generated sequence, and so on, until a complete image report is generated.

[0044] Compared with existing technologies, the beneficial effects of this invention are as follows: By using superpixel segmentation technology to segment medical images into small blocks with similar texture and color features, the region of interest can be effectively separated, allowing for better extraction of key information from medical images and enhancing the identification of local lesions. Furthermore, the graph network GAT is used to learn the relationships between different regions, thereby learning the relationships between the tissue structures of medical images. The regional features learned by the graph network are fused with the global features of the entire image and input into a U-shaped encoder-decoder network for sentence generation tasks, strengthening the interaction between text features and image features, enhancing the semantic details and feature relationships of medical images, and improving the accuracy of generated reports. Attached Figure Description

[0045] Figure 1 This is a schematic diagram illustrating the principle of a medical image report generation method based on multi-feature enhancement disclosed in one embodiment of the present invention;

[0046] Figure 2 This is a schematic diagram illustrating the principle of a superpixel graph network learning module disclosed in one embodiment of the present invention;

[0047] Figure 3 This is a schematic diagram illustrating the principle of the U-transformer codec disclosed in one embodiment of the present invention. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0049] The present invention will now be described in further detail with reference to the accompanying drawings:

[0050] like Figure 1 As shown, a medical image report generation method based on multi-feature enhancement according to the present invention includes:

[0051] Superpixel segmentation technology is used to segment medical images to obtain regions of interest;

[0052] A pre-trained DenseNet is used to extract local features from the region of interest, and the extracted local features are used as graph nodes to construct a relationship graph.

[0053] The relational graph is input into the graph network GAT to learn graph features, and the graph features are then input into the encoder and decoder.

[0054] The features obtained from each layer of the encoder are output to the corresponding decoder layer in a U-shaped connection. The decoder predicts the next word based on the image features obtained from the encoder and the features of the generated text sequence, until all words are generated, and finally a complete image report is obtained.

[0055] In this implementation, superpixel segmentation technology is used to segment medical images into small blocks with similar texture and color features, effectively separating regions of interest. This allows for better extraction of key information from medical images and enhances the identification of local lesions. A graph network (GAT) is used to learn the relationships between different regions, thereby learning the relationships between the tissue structures of the medical image. The regional features learned by the graph network are fused with the global features of the entire image and input into a U-shaped encoder-decoder network for sentence generation. The encoder-decoder uses U-shaped connections to strengthen the interaction between text features and image features, thereby enhancing the semantic details and feature relationships of the medical image and improving the accuracy of the generated report.

[0056] Specifically, the superpixel image network learning module separates important regions from the entire image to obtain key regions in medical images, and learns the relationships between these regions to enhance the acquisition of important semantic information in the image. Finally, the global and local features are merged and input into a U-shaped encoder-decoder for sentence generation. The encoder-decoder uses a U-shaped connection to enable the low-dimensional and high-dimensional features of the image to interact with the text simultaneously, generating a medical image report.

[0057] In the above embodiments, preferably, superpixel segmentation technology is used to segment the medical image to obtain the region of interest. The specific process includes:

[0058] Input medical images and segment them using SuperPixel Segmentation Technology (SLIC) based on the texture and color features of the medical images.

[0059] The medical image is divided into a predetermined number of local regions to separate the corresponding number of regions of interest.

[0060] During implementation, preliminary experiments determined that the optimal number of segments for medical images using superpixel segmentation technology is 60 local regions. Therefore, the medical images were segmented into 60 local regions based on features such as texture and color.

[0061] like Figure 2As shown, in the above embodiment, preferably, a pre-trained DenseNet is used to extract local features from the region of interest, and the extracted local features are used as graph nodes to construct a relationship graph. The specific process includes:

[0062] A pre-trained DenseNet is used to extract local features from a predetermined number of regions of interest;

[0063] Each extracted local feature is treated as a graph node, and the mean color difference between adjacent regions of interest is calculated as the weight of the edge between each pair of adjacent regions to construct a relationship graph.

[0064] Specifically, a pre-trained DenseNet is used to extract 60 local features from these 60 regions. Each local feature is treated as a graph node. The relationship graph is constructed by calculating the mean color difference between adjacent regions in the image and using it as the weight of the edge between adjacent regions.

[0065] In the above embodiment, preferably, the relationship graph is input into the graph network GAT to learn graph features, and the graph features are input into the encoder. The specific process includes:

[0066] The relationship graph is input into the graph network GAT to learn the relationships between the local regions corresponding to each graph node, thus obtaining graph features;

[0067] The graph features are input into the encoder in the U-transformer-based codec.

[0068] like Figure 3 As shown, in the above embodiment, preferably, the features obtained from each layer of the encoder are output to the corresponding decoder layer in a U-shaped connection. The decoder predicts the next word based on the image features obtained from the encoder and the features of the generated text sequence, until all words are generated, and finally a complete image report is obtained. The specific process includes:

[0069] The image features obtained from the first layer of the encoder in the U-transformer codec are output to the last layer of the decoder, and the outputs are connected sequentially until the image features obtained from the last layer of the encoder are output to the first layer of the decoder.

[0070] The decoder predicts the next word based on the image features obtained from the encoder and the text features of the generated sequence, and so on, until a complete image report is generated.

[0071] In implementation, the transformer is used as the encoder-decoder because it can handle long sequence dependencies well. However, in a standard transformer decoder, each layer's input comes from the last layer's output of the encoder; that is, only the last layer's output of the encoder is used. To simultaneously utilize both low-dimensional and high-dimensional features of the image, this invention constructs a U-shaped connection connecting the layers of the encoder and decoder to enhance the interaction between different modalities. The decoder aggregates the visual features of all layers through the U-shaped connection, using the output of each layer of the encoder in the corresponding position of the decoder. This allows text features to simultaneously focus on image features with different information, thereby learning richer semantic information and enhancing report generation.

[0072] Specifically, the image is encoded to obtain image features, which are then fed into the decoder along with the text features extracted from the already generated text sequence to generate the next word. The decoder predicts the probability of all words and selects the word with the highest probability as the next word. This process is repeated until all words are predicted, generating the final image report.

[0073] The medical image report generation method based on multi-feature enhancement disclosed in the above implementation method uses a cross-entropy loss function to measure the difference between the model-generated description and the actual description. The loss function is as follows:

[0074] ;

[0075] in, This represents the word predicted at time t. This represents the text from the beginning to time t-1.

[0076] The medical image report generation method based on multi-feature enhancement disclosed in the above embodiments has achieved good results in preliminary experiments. It enhances the interaction between text features and image features, strengthens the semantic details and feature relationships of medical images, and improves the accuracy of the generated reports. It has great application value and social significance in the field of automatic generation of imaging reports.

[0077] This invention also proposes a medical image report generation system based on multi-feature enhancement, which applies the medical image report generation method based on multi-feature enhancement disclosed in any of the above embodiments, including:

[0078] The image segmentation module is used to segment medical images using superpixel segmentation technology to obtain regions of interest;

[0079] The feature extraction module is used to extract local features from the region of interest using a pre-trained DenseNet, and to use the extracted local features as graph nodes to construct a relationship graph.

[0080] The feature learning module is used to input the relationship graph into the graph network GAT to learn graph features, and then input the graph features into the encoder;

[0081] The report generation module outputs the features obtained from each layer of the encoder to the corresponding decoder layer in a U-shaped connection. The decoder predicts the next word based on the image features obtained from the encoder and the features of the generated text sequence, until all words are generated, and finally a complete image report is obtained.

[0082] In this implementation, superpixel segmentation technology is used to segment medical images into small blocks with similar texture and color features, effectively separating regions of interest. This allows for better extraction of key information from medical images and enhances the identification of local lesions. A graph network (GAT) is used to learn the relationships between different regions, thereby learning the relationships between the tissue structures of the medical image. The regional features learned by the graph network are fused with the global features of the entire image and input into a U-shaped encoder-decoder network for sentence generation. The encoder-decoder uses U-shaped connections to strengthen the interaction between text features and image features, thereby enhancing the semantic details and feature relationships of the medical image and improving the accuracy of the generated report.

[0083] In the above embodiments, preferably, the image segmentation module is specifically used for:

[0084] Input medical images and segment them using SuperPixel Segmentation Technology (SLIC) based on the texture and color features of the medical images.

[0085] The medical image is divided into a predetermined number of local regions to separate the corresponding number of regions of interest.

[0086] In the above embodiments, preferably, the feature extraction module is specifically used for:

[0087] A pre-trained DenseNet is used to extract local features from a predetermined number of regions of interest;

[0088] Each extracted local feature is treated as a graph node, and the mean color difference between adjacent regions of interest is calculated as the weight of the edge between each pair of adjacent regions to construct a relationship graph.

[0089] In the above embodiments, preferably, the feature learning module is specifically used for:

[0090] The relationship graph is input into the graph network GAT to learn the relationships between the local regions corresponding to each graph node, thus obtaining graph features;

[0091] The graph features are input into a U-transformer-based encoder and decoder.

[0092] In the above embodiments, preferably, the report generation module is specifically used for:

[0093] A U-shaped connection is constructed between the encoders and decoders of each layer in the transformer codec network to form a U-transformer codec.

[0094] The image features obtained from the first layer of the encoder in the U-transformer codec are output to the last layer of the decoder, and the outputs are connected sequentially until the image features obtained from the last layer of the encoder are output to the first layer of the decoder.

[0095] The decoder predicts the next word based on the image features obtained from the encoder and the text features of the generated sequence, and so on, until a complete image report is generated.

[0096] The functions of each module of the medical image report generation system based on multi-feature enhancement disclosed in the above embodiments correspond to the steps of the medical image report generation method based on multi-feature enhancement disclosed in the above embodiments. In the implementation process, the operation is carried out with reference to the above embodiments, and will not be repeated here.

[0097] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for generating medical image reports based on multi-feature enhancement, characterized in that, include: The medical image is segmented using superpixel segmentation (SLIC) technology. According to the texture and color features of the medical image, the medical image is divided into a preset number of local regions to separate the corresponding number of regions of interest. The pre-trained DenseNet is used to extract local features from the region of interest, and the extracted local features are used as graph nodes to construct a relationship graph. The specific process includes: A pre-trained DenseNet is used to extract local features from a preset number of regions of interest; each extracted local feature is used as a graph node, and the mean color difference between adjacent regions of interest is calculated as the weight of the edge between each pair of adjacent regions to construct a relationship graph; The relationship graph is input into the graph network GAT to learn graph features, and the graph features are then input into the encoder; The features obtained from each layer of the encoder are output to the corresponding decoder layer in a U-shaped connection. The decoder predicts the next word based on the image features obtained from the encoder and the features of the generated text sequence, until all words are generated, and finally a complete image report is obtained. The specific process includes: A U-shaped connection is constructed between the encoders and decoders of each layer in the transformer codec network to form a U-transformer codec. The image features obtained from the first layer of the encoder in the U-transformer encoder-decoder are output to the last layer of the decoder, and the outputs are connected sequentially until the image features obtained from the last layer of the encoder are output to the first layer of the decoder. The decoder predicts the next word based on the image features obtained from the encoder and the text features of the generated sequence, and so on, until a complete image report is generated.

2. The medical image report generation method based on multi-feature enhancement according to claim 1, characterized in that, The relationship graph is input into a graph network (GAT) to learn graph features, and the graph features are then input into an encoder. The specific process includes: The relationship graph is input into the graph network GAT to learn the relationship between the local regions corresponding to each graph node, thereby obtaining the graph features; The graph features are input into the encoder in a U-transformer-based codec.

3. A medical image report generation system based on multi-feature enhancement, characterized in that, The medical image report generation method based on multi-feature enhancement as described in claim 1 or 2 includes: The image segmentation module is used to segment medical images using superpixel segmentation technology to obtain regions of interest; The feature extraction module is used to extract local features from the region of interest using a pre-trained DenseNet, and to use the extracted local features as graph nodes to construct a relationship graph. Specifically, it is used to: extract local features from a preset number of regions of interest using a pre-trained DenseNet; treat each extracted local feature as a graph node, and calculate the mean color difference between adjacent regions of interest as the weight of the edge between each pair of adjacent regions to construct a relationship graph; The feature learning module is used to input the relationship graph into the graph network GAT to learn graph features, and input the graph features into the encoder; The report generation module is used to output the features obtained from each layer of the encoder to the corresponding decoder layer in a U-shaped connection. The decoder predicts the next word based on the image features obtained from the encoder and the features of the generated text sequence, until all words are generated, and finally a complete image report is obtained.

4. The medical image report generation system based on multi-feature enhancement according to claim 3, characterized in that, The image segmentation module is specifically used for: Input a medical image, and segment the medical image using superpixel segmentation (SLIC) technology according to the texture and color features of the medical image; The medical image is divided into a preset number of local regions to separate the corresponding number of regions of interest.

5. The medical image report generation system based on multi-feature enhancement according to claim 3, characterized in that, The feature learning module is specifically used for: The relationship graph is input into the graph network GAT to learn the relationship between the local regions corresponding to each graph node, thereby obtaining the graph features; The graph features are input into the encoder in a U-transformer-based codec.

6. The medical image report generation system based on multi-feature enhancement according to claim 5, characterized in that, The report generation module is specifically used for: A U-shaped connection is constructed between the encoders and decoders of each layer in the transformer codec network to form a U-transformer codec. The image features obtained from the first layer of the encoder in the U-transformer encoder-decoder are output to the last layer of the decoder, and the outputs are connected sequentially until the image features obtained from the last layer of the encoder are output to the first layer of the decoder. The decoder predicts the next word based on the image features obtained from the encoder and the text features of the generated sequence, and so on, until a complete image report is generated.

Images