A medical imaging report generation method and system based on dual graph convolutional network

Abstract

The present invention belongs to the technical field of medical image-assisted diagnosis, and specifically relates to a medical image report generation method and system of a dual graph convolutional network. The method comprises: obtaining a medical image picture set and a corresponding medical image report set; wherein the medical image report set is used as a label corresponding to the medical image picture set; for each medical image picture in the medical image picture set, using a dual graph convolutional network to extract global-level features and local-level features from the medical image picture, and performing feature fusion based on the global-level features and the local-level features to obtain the aggregated features of the medical image picture; training a codec model based on all the aggregated features and corresponding labels of the medical image picture set, so that the trained codec model can output the corresponding medical image report based on the input medical image picture.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical image-assisted diagnosis, and in particular relates to a medical image report generation method and system based on a dual graph convolutional network. Background Art

[0002] Medical imaging refers to an inspection method that uses various medical imaging devices to convert the morphology, function and other information of the internal structure and organs of the human body into images. These images can clearly show the conditions of various parts of the human body, such as bones, muscles, organs, etc., so as to help doctors accurately diagnose diseases. Medical imaging is of great significance in modern medicine. It can not only provide doctors with intuitive and accurate diagnostic basis, but also help doctors formulate scientific and reasonable treatment plans. Through medical imaging, doctors can timely detect lesions, assess the severity of the disease, monitor the treatment effect and predict the trend of disease progression, so as to provide patients with more accurate and effective treatment services. However, writing imaging reports is time-consuming and error-prone, especially when radiologists need to analyze many images, which is undoubtedly a huge workload for radiologists. Therefore, there is an urgent need for a method to automatically generate imaging medical reports to assist radiologists in report writing and reduce the burden on doctors.

[0003] At present, most mainstream medical report generation methods are derived from image caption generation methods. Image caption is the task of generating natural language descriptions for images. This method combines computer vision and natural language processing techniques to automatically generate text captions that describe the content of the image. However, when image description methods are directly applied to medical report generation tasks, there are the following problems: on the one hand, the limited ability to understand complex scenes and the insufficient ability to recognize subtle differences may lead to errors in identifying features with high similarity in medical images. On the other hand, although the symptoms described in medical reports have been widely used as labels to guide image encoding modules to use attention mechanisms to focus on salient areas in the image, this method ignores the relationship between symptom features corresponding to different labels, as well as the association between these local-level features and global-level features.

[0004] Therefore, existing medical report generation methods cannot effectively capture abnormal information and are difficult to meet the actual needs of clinical applications. Summary of the invention

[0005] In view of the shortcomings of the prior art described above, the purpose of the present invention is to propose a medical imaging report generation method based on a dual graph convolutional network, which innovatively integrates the grid feature map and the category feature map into the report generation process, explores the relationship between the grid areas on the entire image through the grid feature map, and uses the category feature map to mine the relationship between different disease labels and their corresponding symptom features. Finally, the relationship between the global and local features between the grid feature map and the category feature map is explored to identify more effective medical information and automatically generate medical imaging reports.

[0006] To achieve the above-mentioned purpose and other related purposes, the present invention provides a medical image report generation method of a dual graph convolutional network, comprising: obtaining a medical image picture set and a corresponding medical image report set; wherein the medical image report set serves as a label corresponding to the medical image picture set; for each medical image picture in the medical image picture set, using a dual graph convolutional network to extract global-level features and local-level features from the medical image picture, and performing feature fusion based on the global-level features and the local-level features to obtain aggregated features of the medical image picture; training a codec model based on all the aggregated features and corresponding labels of the medical image picture set, so that the trained codec model can output a corresponding medical image report based on the input medical image picture.

[0007] According to a specific embodiment of the present invention, a dual graph convolutional network is used to extract global-level features and local-level features from the medical image, and feature fusion is performed based on the global-level features and the local-level features to obtain the aggregated features of the medical image, including: based on the medical image, a grid feature map of the medical image is constructed using a first main graph convolutional network; based on the grid feature map, a category feature map of the medical image is constructed using a second main graph convolutional network; global-level features and local-level features are extracted from the grid feature map and the category feature map respectively using an auxiliary graph convolutional network; the global-level features and the local-level features are aggregated through convolutional association operations, and the correlation of global-local relationship weights is calculated using a multi-head attention function to obtain the aggregated features of the medical image; wherein the dual graph convolutional network is composed of the first main graph convolutional network, the second main graph convolutional network, and the auxiliary graph convolutional network.

[0008] According to a specific embodiment of the present invention, based on the medical image, the step of using the first main graph convolutional network to construct a grid feature map of the medical image includes: using the first main graph convolutional network to map and extract the feature map corresponding to the medical image, and reshape the feature map into a feature array, and the elements in the feature array represent a grid; for each element in the feature array, identifying its adjacent relationship with other elements to construct an edge set between grids; constructing a grid feature map of the medical image based on the grids contained in the feature array and the edge sets between the grids; wherein the first main graph convolutional network does not include a pooling layer and a fully connected layer.

[0009] According to a specific embodiment of the present invention, the step of constructing the category feature map of the medical image using the second main graph convolutional network according to the grid feature map includes:

[0010] The grid feature map is processed by using the second main graph convolutional network to extract category features corresponding to the medical image; the corresponding category features are associated according to preset category labels to construct edge sets between category features; and a category feature map of the medical image is constructed based on the category features and the edge sets between the category features; wherein the second main graph convolutional network is pre-trained using category features manually defined by medical experts and category features obtained by analyzing radiology reports in a data-driven manner.

[0011] According to a specific embodiment of the present invention, the formation formula of the auxiliary graph convolutional network is as follows:

[0012] ,

[0013] ,

[0014] ,

[0015] Among them, the pre-built graph , represents a node set, represents the node with Nth dimension D, represents the edge set, m represents the number of edges, and X represents the edge set formed by all nodes. Feature matrix, adjacency matrix A represents the relationship between nodes, and its elements represents the weight of the edge between node i and node j, Indicates a The diagonal matrix of , W represents the learnable weight matrix, represents the activation function, represents batch normalization, represents the graph convolution operation, represents the connection operation of the convolution association operation, Represents the feature vector of the node in layer l.

[0016] According to a specific embodiment of the present invention, the step of aggregating the global-level features and the local-level features using a convolutional correlation operation, and calculating the correlation of the global-local relationship weights using a multi-head attention function to obtain the aggregated features of the medical image includes: calculating the global-local relationship weight e according to the following formula:

[0017] ,

[0018] Performing a dot multiplication on the global-local relationship weight e to obtain a weighted feature representation, and using it as the aggregated feature of the medical image; wherein, represents the multi-head attention function, represents the connection operation of the convolution association operation, represents a node in the category feature graph, represents a node in the grid feature map, and Respectively represent and The associated first learnable weight parameters.

[0019] According to a specific embodiment of the present invention, the weighted feature is calculated according to the following formula: :

[0020] ,

[0021] in, represents channel addition, represents element-wise multiplication, and Respectively represent and The associated second learnable weight parameter.

[0022] According to a specific embodiment of the present invention, the dual graph convolutional network includes two auxiliary graph convolutional networks, corresponding to the first main graph convolutional network and the second main graph convolutional network, respectively.

[0023] According to a specific embodiment of the present invention, the encoder and decoder in the encoding and decoding model are respectively composed of three sub-layers, and each sub-layer includes a multi-head attention layer and a feed-forward layer.

[0024] A medical image report generation system of a dual graph convolutional network comprises: a training data collection module, used to obtain a medical image picture set and a corresponding medical image report set; wherein the medical image report set serves as a label corresponding to the medical image picture set; a data feature extraction module, used to extract global-level features and local-level features from each medical image picture in the medical image picture set using a dual graph convolutional network, and perform feature fusion based on the global-level features and the local-level features to obtain aggregate features of the medical image picture; and a model training module, used to train a codec model based on all aggregate features and corresponding labels of the medical image picture set, so that the trained codec model can output a corresponding medical image report based on the input medical image picture.

[0025] The present invention provides a medical imaging report generation method based on a dual graph convolutional network. The method integrates grid feature maps and category feature maps into the medical imaging report generation process, and fuses different types of features to mine more useful relationship information in medical imaging pictures.

[0026] The present invention can be used to automatically generate diagnostic reports for medical images, such as X-rays, CT scans, MRIs, etc. By performing deep learning and analysis on medical images, it can assist doctors in quickly and accurately diagnosing diseases.

[0027] The present invention can provide quantitative and qualitative analysis based on imaging data in the medical decision-making process, helping doctors to formulate more reasonable treatment plans, especially in complex cases or when multidisciplinary consultation is required, which can speed up the decision-making process.

[0028] The present invention can be used for medical research and teaching, providing standardized image analysis cases and report templates, which is helpful for medical education and training and improving the understanding and skills of medical professionals in image diagnosis. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 A schematic flow chart of a specific embodiment of a medical imaging report generation method using a dual graph convolutional network provided by the present invention;

[0030] Figure 2 A schematic diagram of a model of a specific embodiment of a medical imaging report generation method using a dual graph convolutional network provided by the present invention;

[0031] Figure 3 A schematic diagram of the structure of a specific embodiment of a medical imaging report generation system of a dual graph convolutional network provided by the present invention;

[0032] Figure 4 The present invention provides a structural block diagram of a specific embodiment of an electronic device. DETAILED DESCRIPTION

[0033] In order to facilitate understanding of the present application, the present application will be described more fully below with reference to the relevant drawings. Embodiments of the present application are provided in the drawings. However, the present application can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present application more thorough and comprehensive.

[0034] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application.

[0035] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0036] In the following description, numerous details are discussed to provide a more thorough explanation of the embodiments of the present invention. However, it is obvious to those skilled in the art that the embodiments of the present invention can be implemented without these specific details. In other embodiments, well-known structures and devices are shown in the form of block diagrams rather than in detail to avoid making the embodiments of the present invention difficult to understand.

[0037] First of all, it should be explained that in order to enable people in this technical field to better understand the present application scheme, the technical background of this application is explained accordingly.

[0038] Chest X-rays are an important part of medical diagnosis, providing an effective analysis of a patient's health status, such as potential lung infections, pleural effusions, and other abnormalities. Radiologists analyze these images to diagnose the patient's condition and write a report outlining their conclusions and recommendations. However, handwritten reports are time-consuming and prone to errors, especially when radiologists need to analyze many images or are new to the process.

[0039] Automatic chest X-ray report generation refers to the automatic generation of reports based on the image features captured in chest X-rays. It can greatly reduce the workload of radiologists, assist them to focus on more serious cases, and diagnose and treat patients as soon as possible by reducing the turnaround time of radiologists. In addition, it can also improve the accuracy of diagnosis by combining data-driven insights and consistency in generating reports.

[0040] In recent years, there has been significant development in the field of artificial intelligence and machine learning technology, which has opened up more possibilities for medical research. With the advancement of deep learning technology, an image model can be built to recognize and analyze X-ray images and automatically generate reports. This technology is a boon for patients who need accurate reports to be issued as soon as possible for follow-up treatment.

[0041] Image subtitle generation methods have achieved good results in solving the above problems. Image subtitle generation methods generally use empirical knowledge related to medical reports to manually build a knowledge graph in advance to extract image feature nodes, and finally use a graph convolutional network (GCN) to obtain a graph. However, as a visual text task, image subtitle generation methods still have the following problems in medical report generation tasks:

[0042] The images in chest X-rays are highly similar, and the abnormal areas are usually small. Since there are few abnormal samples of the same type, it is difficult for the feature extraction and encoding modules to capture the corresponding features.

[0043] In addition, in the reports, there are few descriptions of abnormal text, while standard descriptions dominate the entire dataset. Therefore, the formulaic image description method may tend to favor standard descriptions, fail to capture abnormal features, and be difficult to meet the requirements of clinical applications.

[0044] Therefore, this application proposes a new medical report generation framework that can learn the relationship between grid-guided global-level features and mine the aggregation relationship between the graph representation of local-level features and the graph representation of global-level features. Finally, the enhanced graph representation is obtained and encoded and decoded to generate an accurate report.

[0045] Example 1

[0046] See also Figure 1 , a medical imaging report generation method using a dual graph convolutional network as shown in 2, comprising:

[0047] Step S100, obtaining a medical image picture set and a corresponding medical image report set; wherein the medical image report set serves as a label corresponding to the medical image picture set.

[0048] Step S200, for each medical image in the medical image set, extract global-level features and local-level features from the medical image using a dual graph convolutional network, perform feature fusion based on the global-level features and the local-level features, and obtain aggregated features of the medical image.

[0049] Among them, the Dual Graph Convolutional Network (DGCN) is a special graph neural network architecture that uses two standard graph convolutional networks with the same structure to process graph data in parallel. This design aims to more comprehensively capture the feature information and structural information in the graph data through the collaborative work of the two networks, and realize information aggregation and feature extraction of nodes and edges in medical images.

[0050] Specifically, the main graph convolutional network is mainly responsible for extracting basic feature representations from the original graph data. These features may include local attributes of nodes, connection relationships of edges, etc. The main graph convolutional network gradually aggregates the neighborhood information of nodes into the node representation through the stacking of multiple graph convolutional layers to form a richer feature vector. The auxiliary graph convolutional network further mines and supplements the feature information in the graph data based on the main graph convolutional network. The auxiliary graph convolutional network may focus on different feature dimensions or structural patterns, and provide additional information for the final node representation through its unique weight parameters and learning process. This information may help improve the performance of the network on specific tasks, such as node classification, link prediction, etc. It can be seen that the main graph convolutional network can ensure that the network can capture the basic structure and attributes of the graph data. The auxiliary graph convolutional network enhances the DGCN's ability to represent graph data through its unique perspective and learning ability, thereby generating better feature representations and fully mining the feature information in medical images.

[0051] Furthermore, in this embodiment, a dual graph convolution network is formed based on the first main graph convolution network and its corresponding auxiliary graph convolution network, and the second main graph convolution network and its corresponding auxiliary graph convolution network. Specifically, first, a grid feature map of the input medical image is constructed through the first main graph convolution network, and the global-level features of the medical image are extracted from the grid feature map using the corresponding auxiliary graph convolution network. Secondly, a category feature map is constructed through the second main graph convolution network based on the grid feature map, and the local-level features of the category feature map are extracted from the category feature map using the corresponding auxiliary graph convolution network.

[0052] Among them, the first main graph convolution network used to construct the grid feature map removes the pooling layer and the fully connected layer, only uses the last convolution layer to map the output feature map corresponding to the medical image, and reshapes the feature map into a feature array ,in, Refers to the dimension of each grid, and n is the number of grids, that is, the reshaped size of the feature map. Since each element in the feature array represents a grid, the edge set between grids is constructed by identifying the adjacent relationship between each element and other elements. , Representation Grid and Grid The edges between them are connected by adjacent relations. Finally, the grid feature map can be constructed .

[0053] It should be noted here that after removing the pooling layer and the fully connected layer, the graph convolutional network can still output a grid map. The construction of the grid map is based on the spatial structure of the feature map, rather than relying on the output of the pooling layer and the fully connected layer. As long as there is a medical image as input, it can be divided into several grid areas to construct a grid map.

[0054] The second main graph convolutional network used to construct the category feature map can be pre-trained with the category features manually defined by medical experts and the category features obtained by analyzing radiology reports in a data-driven manner, so that the category features of medical images can be extracted from the grid feature map. Each category feature can be regarded as a node, and the binary co-occurrence matrix of the category label is used as the adjacency matrix shared by all batches during training to associate the category labels with different nodes to construct the edge set between the nodes. For example, the category label is fever, and its corresponding category features include elevated body temperature, cough, runny nose, etc. Finally, the category feature map can be constructed , Represents the nodes, that is, the category features.

[0055] It should also be noted that the first main graph convolutional network and the second main graph convolutional network in this embodiment both use DenseNet (Densely Connected Convolutional Networks). The two networks are consistent in structure, but may learn different weight parameters during the training process, so as to capture different aspects of the graph data.

[0056] The construction of the auxiliary graph convolutional network can be specifically described as follows:

[0057] Assume there is a pre-built graph ,in is a node set, represents the node with Nth dimension D, represents the edge set. All nodes X form an dimensional feature matrix, and the relationship between each node is defined as The adjacency matrix A, where each element represents the weight of the edge between node i and node j, and the formula is as follows:

[0058] ,

[0059] The auxiliary graph convolutional network is formed as follows:

[0060] ,

[0061] ,

[0062] ,

[0063] in, Indicates a The diagonal matrix of , W represents the learnable weight matrix, represents the activation function, represents batch normalization, represents the graph convolution operation, represents the connection operation of the convolution association operation, Represents the feature vector of the node in the lth layer, and updates the feature vectors from the lth layer to the (l+1)th layer through convolution operations, that is, the graph representation.

[0064] It can be seen that the grid feature map and the category feature map, as input data of the auxiliary graph convolutional network, provide it with necessary structural information, which enables the auxiliary graph convolutional network to understand and use this structural information to perform subsequent tasks, that is, to use the node and edge information in the grid feature map and the category feature map to aggregate the features of neighboring nodes through convolution operations and update the representation of each node. This process enables the network to learn the complex relationships between nodes so as to facilitate subsequent classification, regression or other tasks.

[0065] In addition, the global-level features and local-level features of an image are closely related. Effectively learning the relationship between them can obtain more discriminative feature representations for generating medical imaging reports. When we obtain accurate global-level and local-level feature representations, the incorrect or missing interactions between the two will also seriously affect the accuracy of report generation. Therefore, in order to generate accurate feature representations of global-local relationships, a FFARM (Feature Fusion And Relationship Mining) network is proposed to aggregate the two types of features and mine the relationship information between them.

[0066] However, the direct addition feature fusion method lacks deep mining of the correlation between the two and lacks precise calculation of the features, so the obtained aggregated features have more redundant data and noise. In this regard, the FFARM network with multi-head attention mechanism as the core is used to learn the relationship between features and enhanced feature representation. Specifically, the global-level features and local-level features are linearly transformed, and the convolutional correlation operation is used to aggregate the two types of features. Then, the multi-head attention function is used to calculate the correlation of the global-local relationship weights to obtain the aggregated features of medical images.

[0067] The specific steps are as follows:

[0068] First, the global-local relationship weight e is calculated according to the following formula:

[0069] ,

[0070] represents the multi-head attention function, represents the connection operation of the convolution association operation, represents a node in the category feature graph, represents a node in the grid feature map, and Respectively represent and The associated first learnable weight parameters.

[0071] Secondly, dot multiplication of the global-local relationship weight e is performed to obtain the weighted feature Represented as the aggregated feature of medical imaging pictures, and the calculation formula is as follows:

[0072] ,

[0073] represents channel addition, represents element-wise multiplication, and Respectively represent and The associated second learnable weight parameter.

[0074] Step S300: training a codec model based on all aggregated features and corresponding labels of the medical image picture set, so that the trained codec model can output a corresponding medical image report based on the input medical image picture.

[0075] It is understandable that the aggregate features of the medical image pictures obtained in the above steps are obtained by computer equipment processing, and accordingly cannot be understood by humans. For this reason, it is necessary to convert it into text representation to obtain the corresponding medical image report. Secondly, in this embodiment, a medical image picture set and a corresponding medical image report set are used as training sets to train the encoding and decoding model so that it can correctly interpret the aggregate features output by the dual graph convolutional network after the training is completed, and generate an accurate medical image report.

[0076] Accordingly, in this embodiment, a standard Transformer is used as the encoding and decoding model, which includes an encoder and a decoder, and the encoder and decoder are respectively composed of three sub-layers, each of which includes a multi-head attention layer and a feed-forward layer. The specific training steps of Transformer are as follows:

[0077] First, the purpose of the medical imaging report generation task is to generate a correct radiology diagnosis report R from a medical image I. The medical imaging reports used as training sets are written or formulated by humans. Computer equipment needs to interpret them, convert them into corresponding labels, and then compare whether the output of the Transformer is consistent with the label, so as to complete the model training. Among them, the medical imaging report consists of L words, denoted as ,in represents the lth word, C is the vocabulary size, and L is the report length. Accordingly, the above is equivalent to fitting a parameterized model , from a probability perspective, its goal is to maximize the conditional log-likelihood of generating a correct report given a medical image, and is defined as:

[0078] ,

[0079] ,

[0080] in, represents the model parameters, N represents the total number of training samples, Represents the corresponding input, represents the observed results, Indicates that given parameters Next, given input The results were observed probability.

[0081] In addition, the model parameters are trained by minimizing the negative log-likelihood using the aggregated features extracted from the medical image set by the dual graph convolutional network and the corresponding labels. Finally, the dual graph convolutional network and the trained Transformer can be used to automatically generate medical imaging reports based on the input medical imaging images.

[0082] For the medical imaging report generation method of the dual graph convolutional network provided above, and its corresponding model, a medical impact report generated by a set of chest X-ray data sets is used to demonstrate the role and effect of the present application.

[0083] In order to evaluate the report generation results, this example calculates the scores of the most commonly used evaluation indicators in chest X-ray report generation research: BLEU-1, BLEU-2, BLEU-3, BLEU-4, ROUGE, METEOR, which are derived from machine translation, text summarization, and image captioning tasks, and measure the similarity between the target text and one or more reference texts. The results are shown in Table 1. For all indicators, the higher the better.

[0084] Table 1

[0085]

[0086] In addition, ablation experiments were performed to verify the effectiveness of DGCN and FFARM in this embodiment. As shown in Table 2. First, FFARM is removed and its function is replaced by directly connecting the global-level features and the local-level features. The final result can be seen from the comparison of the second and fifth rows, and the index is reduced by 1.7%. This shows that learning the relationship between global-level features and local-level features in chest X-ray images can effectively improve the performance of the model. Secondly, the first main graph convolutional network used to construct the grid feature map and the second main graph convolutional network used to construct the category feature map were removed respectively to verify the effectiveness of modeling relationships within features at different levels. This also shows that DGCN improves the performance indicators of the report generation model.

[0087] Table 2

[0088]

[0089] In order to further verify the effectiveness of FFARM in the report generation task, two sets of experiments were designed for comparison:

[0090] M1: Directly connect the graph representation of global-level features and the graph representation of local-level features.

[0091] M2: The graph representation of global-level features and the graph representation of local-level features are linearly transformed and then connected.

[0092] The above two methods are relatively simple and standard methods for coupling features from different sources. The superiority of our method will be demonstrated from two aspects: natural language evaluation indicators and qualitative results. As shown in Table 3, the above two methods are inferior to the medical imaging report generation method of the dual graph convolutional network provided in this application in all indicators.

[0093] Table 3

[0094]

[0095] It can be seen that since medical imaging data is usually high-dimensional, highly complex and diverse, traditional methods are difficult to effectively extract and analyze key information. The technical means provided in this embodiment can process the complex structures and relationships in the image, and capture the spatial information and features in the image through the graph structure, thereby improving the accuracy and efficiency of the analysis.

[0096] In addition, traditional medical imaging reports are usually written manually by doctors, which are subjective and time-consuming. The medical imaging report generation method provided in this embodiment can realize the automatic generation and standardization of medical imaging reports, reduce human errors, improve the consistency and accuracy of reports, reduce misdiagnosis and missed diagnosis, and shorten the diagnosis time, so that doctors can handle more cases faster.

[0097] It should be noted that the step division of the above methods is only for clear description. When implemented, they can be combined into one step or some steps can be split and decomposed into multiple steps. As long as they contain the same logical relationship, they are all within the scope of protection of this patent; adding insignificant modifications to the algorithm or process or introducing insignificant designs without changing the core design of the algorithm and process are all within the scope of protection of this patent.

[0098] Example 2

[0099] See also Figure 3 As shown, this embodiment also provides a medical imaging report generation system of a dual graph convolutional network, including:

[0100] The training data collection module 10 is used to obtain a medical image picture set and a corresponding medical image report set; wherein the medical image report set serves as a label corresponding to the medical image picture set.

[0101] The data feature extraction module 20 is used to extract global-level features and local-level features from each medical image in the medical image set using a dual graph convolutional network, and to perform feature fusion based on the global-level features and the local-level features to obtain aggregated features of the medical image.

[0102] The model training module 30 is used to train the encoding and decoding model according to all the aggregated features and corresponding labels of the medical image picture set, so that the trained encoding and decoding model can output the corresponding medical image report based on the input medical image picture.

[0103] It should be noted that the medical imaging report generation system of the dual graph convolutional network provided in the above embodiment and the medical imaging report generation method of the dual graph convolutional network provided in the above embodiment 1 belong to the same concept, wherein the specific manner in which each module and unit performs the operation has been described in detail in the method embodiment and will not be repeated here. In practical applications, the medical imaging report generation method of the dual graph convolutional network provided in the above embodiment 1 can distribute the above functions to different functional modules as needed, that is, divide the internal structure of the device into different functional modules to complete all or part of the functions described above, and this is not limited here.

[0104] Example 3

[0105] See also Figure 4 As shown, an embodiment of the present application further provides an electronic device, comprising a memory 2, a processor 1, and a program stored in the memory and executable on the processor, wherein the processor implements the steps of any of the above methods when executing the program.

[0106] Among them, the memory includes at least one type of readable storage medium, and the readable storage medium includes flash memory, mobile hard disk, multimedia card, card-type memory (for example: SD or DX memory, etc.), magnetic memory, disk, optical disk, etc. In some embodiments, the memory can be an internal storage unit of the electronic device, such as a mobile hard disk of the electronic device. In other embodiments, the memory can also be an external storage device of the electronic device, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device. Further, the memory can also include both an internal storage unit of the electronic device and an external storage device. The memory can not only be used to store application software and various types of data installed in the electronic device, but also can be used to temporarily store data that has been output or is to be output.

[0107] In some embodiments, the processor may be composed of an integrated circuit, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and a combination of various control chips. The processor is the control core (Control Unit) of the electronic device, which connects the various components of the entire electronic device using various interfaces and lines, and executes various functions of the electronic device and processes data by running or executing programs or modules stored in the memory, and calling data stored in the memory.

[0108] The processor executes the operating system of the electronic device and various installed application programs. The processor executes the application programs to implement the steps in the above method embodiment.

[0109] Exemplarily, the program may be divided into one or more modules, which are stored in the memory and executed by the processor to complete the present invention. The one or more modules may be a series of program instruction segments capable of completing specific functions, which are used to describe the execution process of the program in the electronic device.

[0110] The above-mentioned integrated unit implemented in the form of software function modules can be stored in a computer-readable storage medium. The above-mentioned software function modules are stored in a storage medium and include several instructions for enabling a computer device (which can be a personal computer, a computer device, or a network device, etc.) or a processor to perform part of the functions of various embodiments of the present invention.

[0111] In summary, the present invention provides a medical imaging report generation method based on a dual graph convolutional network, which integrates grid feature maps and category feature maps into the medical imaging report generation process, and fuses different types of features to mine more useful relationship information in medical imaging images.

[0112] The present invention can be used to automatically generate diagnostic reports for medical images, such as X-rays, CT scans, MRIs, etc. By performing deep learning and analysis on medical images, it can assist doctors in quickly and accurately diagnosing diseases.

[0113] The present invention can provide quantitative and qualitative analysis based on imaging data in the medical decision-making process, helping doctors to formulate more reasonable treatment plans, especially in complex cases or when multidisciplinary consultation is required, which can speed up the decision-making process.

[0114] The present invention can be used for medical research and teaching, providing standardized image analysis cases and report templates, which is helpful for medical education and training and improving the understanding and skills of medical professionals in image diagnosis.

[0115] The above embodiments are merely illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Anyone familiar with the technology may modify or change the above embodiments without violating the spirit and scope of the present invention. Therefore, all equivalent modifications or changes made by a person of ordinary skill in the art without departing from the spirit and technical ideas disclosed by the present invention shall still be covered by the claims of the present invention.

Claims

1. A medical imaging report generation method based on a dual graph convolutional network, characterized in that: include: Obtain a medical image picture set and a corresponding medical image report set; wherein the medical image report set serves as a label corresponding to the medical image picture set; For each medical image in the medical image set, extract global-level features and local-level features from the medical image using a dual graph convolutional network, perform feature fusion based on the global-level features and the local-level features, and obtain aggregated features of the medical image; Training a codec model based on all aggregated features and corresponding labels of the medical image set, so that the trained codec model can output a corresponding medical image report based on the input medical image; Among them, the steps of extracting global-level features and local-level features from the medical image using a dual graph convolutional network, and performing feature fusion based on the global-level features and the local-level features to obtain aggregated features of the medical image include: constructing a grid feature map of the medical image using a first main graph convolutional network based on the medical image; constructing a category feature map of the medical image using a second main graph convolutional network based on the grid feature map; extracting global-level features and local-level features from the grid feature map and the category feature map respectively using an auxiliary graph convolutional network; aggregating the global-level features and the local-level features through convolutional association operations, and calculating the correlation of global-local relationship weights using a multi-head attention function to obtain aggregated features of the medical image; and the dual graph convolutional network is composed of the first main graph convolutional network, the second main graph convolutional network, and the auxiliary graph convolutional network.

2. The medical imaging report generation method of the dual graph convolutional network according to claim 1, characterized in that: According to the medical image, the step of constructing a grid feature map of the medical image using a first main image convolutional network includes: Mapping and extracting a feature map corresponding to the medical image using the first main image convolutional network, and reshaping the feature map into a feature array, wherein an element in the feature array represents a grid; For each element in the feature array, identifying the adjacent relationship between it and other elements to construct an edge set between grids; Constructing a grid feature map of the medical image according to the grids contained in the feature array and the edge sets between the grids; Among them, the first main graph convolutional network does not include a pooling layer and a fully connected layer.

3. The medical imaging report generation method of the dual graph convolutional network according to claim 1, characterized in that: The step of constructing the category feature map of the medical image using the second main graph convolutional network according to the grid feature map includes: Processing the grid feature map using the second main graph convolutional network to extract category features corresponding to the medical image; Corresponding category features are associated with preset category labels to construct edge sets between category features; Constructing a category feature graph of the medical image according to the category features and the edge sets between the category features; The second main graph convolutional network is pre-trained using category features manually defined by medical experts and category features obtained by analyzing radiology reports in a data-driven manner.

4. The medical imaging report generation method of the dual graph convolutional network according to claim 1, characterized in that: The formation formula of the auxiliary graph convolutional network is as follows: ， ， ， Among them, the pre-built graph , represents a node set, represents the node with Nth dimension D, represents the edge set, m represents the number of edges, and X represents the edges formed by all nodes Feature matrix, adjacency matrix A represents the relationship between nodes, and its elements represents the weight of the edge between node i and node j, Indicates a The diagonal matrix of , W represents the learnable weight matrix, represents the activation function, represents batch normalization, represents the graph convolution operation, represents the connection operation of the convolution association operation, Represents the feature vector of the node in layer l.

5. The medical imaging report generation method of the dual graph convolutional network according to claim 1, characterized in that: The steps of aggregating the global-level features and the local-level features by using a convolutional correlation operation, and calculating the correlation of the global-local relationship weights by using a multi-head attention function to obtain the aggregated features of the medical image include: The global-local relationship weight e is calculated according to the following formula: ， Performing a dot multiplication on the global-local relationship weight e to obtain a weighted feature representation, and using it as an aggregate feature of the medical image; in, represents the multi-head attention function, represents the connection operation of the convolution association operation, represents a node in the category feature graph, represents a node in the grid feature map, and Respectively represent and The associated first learnable weight parameters.

6. The medical imaging report generation method of the dual graph convolutional network according to claim 5, characterized in that: The weighted feature is calculated according to the following formula: : ， in, represents channel addition, represents element-wise multiplication, and Respectively represent and The associated second learnable weight parameter.

7. The medical imaging report generation method of the dual graph convolutional network according to claim 1, characterized in that: The dual graph convolutional network includes two auxiliary graph convolutional networks, corresponding to the first main graph convolutional network and the second main graph convolutional network respectively.

8. The medical imaging report generation method of the dual graph convolutional network according to claim 1, characterized in that: The encoder and decoder in the encoding and decoding model are composed of three sub-layers respectively, and each sub-layer includes a multi-head attention layer and a feed-forward layer.

9. A dual graph convolutional network medical imaging report generation system, characterized in that: include: A training data collection module, used to obtain a medical image picture set and a corresponding medical image report set; wherein the medical image report set serves as a label corresponding to the medical image picture set; A data feature extraction module, for each medical image in the medical image set, extracting global-level features and local-level features from the medical image using a dual graph convolutional network, and performing feature fusion based on the global-level features and the local-level features to obtain aggregated features of the medical image; A model training module, used for training a codec model according to all aggregated features and corresponding labels of the medical image set, so that the trained codec model can output a corresponding medical image report based on the input medical image; Among them, the steps of extracting global-level features and local-level features from the medical image using a dual graph convolutional network, and performing feature fusion based on the global-level features and the local-level features to obtain aggregated features of the medical image include: constructing a grid feature map of the medical image using a first main graph convolutional network based on the medical image; constructing a category feature map of the medical image using a second main graph convolutional network based on the grid feature map; extracting global-level features and local-level features from the grid feature map and the category feature map respectively using an auxiliary graph convolutional network; aggregating the global-level features and the local-level features through convolutional association operations, and calculating the correlation of global-local relationship weights using a multi-head attention function to obtain aggregated features of the medical image; and the dual graph convolutional network is composed of the first main graph convolutional network, the second main graph convolutional network, and the auxiliary graph convolutional network.

Images