A medical image-based disease diagnosis apparatus and method

Abstract

The application provides a disease diagnosis device and method based on medical images. The device comprises: a feature extraction module for extracting features of input medical images using a convolutional neural network; a first diagnosis module for disease diagnosis reasoning based on a Bayesian network; a second diagnosis module for disease diagnosis and attribute category reasoning based on a graph convolution network; and a diagnosis result fusion module for fusing the outputs of the first diagnosis module and the second diagnosis module to obtain a final diagnosis result. The application enables the disease intelligent diagnosis model to think like a professional doctor by reasoning the disease based on the Bayesian network and the graph convolution network, inferring the disease from the imaging characteristics of the lesion, realizing the verifiable disease intelligent diagnosis, integrating the professional knowledge of doctors, and thus improving the accuracy, verifiability and generalization of automatic diagnosis.

Description

Technical Field

[0001] This invention belongs to the field of medical imaging technology, specifically relating to a disease diagnosis device and method based on medical imaging. Background Technology

[0002] In recent computer vision research, artificial intelligence, especially deep neural networks, has achieved accuracy surpassing classical machine learning models in a range of tasks, including image recognition, object detection, and semantic segmentation. Simultaneously, deep neural networks have been applied to medical image analysis, achieving good recognition results in the automatic diagnosis of several diseases, such as automatic detection of intracranial hemorrhage areas based on CT images, automatic classification and lesion localization of chest diseases based on X-ray images, automatic diagnosis of breast cancer based on mammography, and automatic screening for diabetes based on fundus images. The superior model performance of deep learning in medical image analysis is primarily due to its powerful image feature extraction and feature learning capabilities. It can learn representative and discriminative image features of lesions from various medical images, thereby achieving high-precision disease diagnosis.

[0003] Existing AI-assisted medical imaging diagnostic systems transmit medical images to an AI computing platform, which then uses a corresponding deep learning model to automatically identify them. The results are then sent to doctors for reference, review, and verification. However, these systems only use general deep learning models to identify and classify lesions. Since deep learning is essentially a data-driven model, its learning process is close to a black box. Therefore, the verifiability of these models is poor, and their diagnostic accuracy is low. Although doctors receive the lesion identification results from the deep learning model, the identification process is invisible to them. Doctors find it difficult to fully trust the automated diagnostic results and must spend time carefully reviewing them in practice. This fails to meet doctors' requirements for the generalization of AI model reasoning processes. Specifically, verifiability means that the reasoning and decision-making processes within the intelligent disease diagnostic model not only need to be interpretable but also need to be consistent with medical knowledge and rules, ensuring the verifiability of the diagnostic results. Taking the classification of benign and malignant pulmonary nodules in medical imaging diagnosis as an example, the intelligent disease diagnosis model first needs to identify and classify the imaging attributes of pulmonary nodules like a doctor. Then, it needs to learn the causal relationship between these lesion attributes (such as lobulation, spiculation, sphericity, etc.) and the benign and malignant nature of the lesions in order to realize the restoration of the doctor's diagnostic process and make the diagnostic results verifiable. Summary of the Invention

[0004] To address the aforementioned problems in the prior art, the present invention provides a disease diagnosis device and method based on medical imaging.

[0005] To achieve the above objectives, the present invention adopts the following technical solution.

[0006] In a first aspect, the present invention provides a disease diagnosis device based on medical imaging, comprising:

[0007] The feature extraction module is used to extract features from the input medical images using a convolutional neural network, and then adapts and transforms the extracted features before inputting them into the first diagnostic module and the second diagnostic module respectively.

[0008] The first diagnostic module is used for disease diagnosis reasoning based on Bayesian networks;

[0009] The second diagnostic module is used for disease diagnosis and attribute category reasoning based on graph convolutional networks;

[0010] The diagnostic result fusion module is used to fuse the outputs of the first diagnostic module and the second diagnostic module to obtain the final diagnostic result.

[0011] Furthermore, the input medical images include CT images, MRI images, or X-ray images.

[0012] Furthermore, the feature extraction module uses ResNet or EfficientNet convolutional neural networks as the backbone for feature extraction; it employs a feature pyramid network to extend ResNet or EfficientNet to multiple scales, and uses a global average pooling operator to perform spatial scale reduction on the feature vectors of each layer of the feature pyramid, resulting in feature vectors with a width and height of 1. It is set that the feature output dimension is the same for all scales. After the features at different scales pass through the global average pooling operator, a fully connected layer is used to transform the features. At the same time, an addition operation is used to fuse the features at all scales, and another fully connected layer is used to continue to transform the fused features. Finally, the extracted image features are output.

[0013] Furthermore, the first diagnostic module establishes a probabilistic dependency model between disease categories and lesion attributes based on a Bayesian network; the Bayesian network is represented as B = <V B E B ,θ>, where <V B E B > represents a directed acyclic graph, V B E represents the set of nodes. BLet θ represent the set of edges, and let θ represent the Bayesian network parameters consisting of the conditional probability tables for each node. An edge connecting two nodes represents the probabilistic dependency between them. First, the input image features are converted into probability scores of attributes and fed into the Bayesian network as evidence for forward inference. The input evidence for each node is combined with the conditional probability tables to obtain the marginal posterior probability of each node. The marginal posterior probability P of node v0 is given by θ. B (v0) is:

[0014] P B (v0)=∫...∫ V P(v0,v1,...,v n )dv1...dv n (1)

[0015]

[0016] Where n is the number of nodes, Parents(v i ) represents node v i The parent node, when v i When there is no parent node, Parents(v i () is empty.

[0017] Furthermore, the second diagnostic module establishes a relationship model between lesion attributes and between lesion attributes and disease categories based on a graph convolutional network; the graph convolutional network is based on an undirected graph G = (V G E G ) is constructed, where V G E G Let L represent the set of nodes and L represent the set of edges, respectively. Each node and its contained feature vector correspond to a certain attribute of the lesion or disease category. If two nodes are connected, it means that the two nodes are related. For the edge between two nodes, a learnable weight is used to represent the degree of correlation between the two nodes. The graph convolutional network contains a total of L layers, and the number of nodes in each layer is the same. The feature transformation from the l-th layer to the (l+1)-th layer is implemented using residual graph convolution operation.

[0018]

[0019] Among them, F(G l W l ) represents the graph convolution operator of the l-th layer. These represent the feature vector matrices of all nodes in the l-th and (l+1)-th layers, respectively.

[0020] Furthermore, the diagnostic result fusion module employs a residual fusion algorithm, using the Bayesian network of the first diagnostic module as the main branch and the graph convolutional network of the second diagnostic module as the residual branch, with the prediction results superimposed on the output of the Bayesian network; during the fusion process, the prediction results of disease categories and lesion attributes are fused separately, as shown in equations (4) and (5):

[0021]

[0022]

[0023] in, The probability of the merged disease category. The probability of the attributes of the fused lesions. and represent the marginal posterior probabilities of the Bayesian network output corresponding to the disease category and lesion attribute, respectively. Let w represent the probabilities of disease category and lesion attribute output by the graph convolutional network, respectively. B and w' B These represent two learnable coefficients, W0 and W'0 represent the weight matrices of the two fully connected layers, σ represents the softmax function, and Concat() represents the feature stacking operator.

[0024] Furthermore, the device also includes a model optimization module for optimizing model parameters through model training, wherein the model training method includes:

[0025] Step S1: Keeping the structure and parameters of the Bayesian network unchanged, update the parameters of the convolutional neural network of the feature extraction module and the graph convolutional network of the second diagnostic module using the backpropagation algorithm.

[0026] Step S2: The structure of the Bayesian network of the first diagnostic module is updated using dynamic programming, and then the conditional probability table is updated using maximum likelihood estimation.

[0027] Steps S1 and S2 are executed repeatedly. When the number of repetitions exceeds the preset maximum number of Bayesian network updates, only step S1 is executed.

[0028] Secondly, the present invention provides a disease diagnosis method based on medical imaging, comprising the following steps:

[0029] Convolutional neural networks are used to extract features from input medical images, and the extracted features are adapted and transformed before being input into Bayesian networks and graph convolutional networks, respectively.

[0030] Disease diagnosis inference based on Bayesian networks;

[0031] Disease diagnosis and attribute category reasoning based on graph convolutional networks;

[0032] The outputs of the Bayesian network and the graph convolutional network are fused to obtain the final diagnostic result.

[0033] Furthermore, the input medical images include CT images, MRI images, or X-ray images.

[0034] Furthermore, the feature extraction module uses ResNet or EfficientNet convolutional neural networks (CNNs) as the backbone for feature extraction; it employs a feature pyramid network to extend ResNet or EfficientNet to multiple scales, and uses a global average pooling operator to perform spatial scale reduction on the feature vectors of each layer of the feature pyramid, resulting in feature vectors with a width and height of 1. It is assumed that the feature output dimensions are the same across all scales. After the features at different scales pass through the global average pooling operator, a fully connected layer is used to transform their features. Simultaneously, addition is used to fuse the features at all scales, and another fully connected layer is used to further transform the fused features. Finally, the extracted image features are output.

[0035] Furthermore, the first diagnostic module establishes a probabilistic dependency model between disease categories and lesion attributes based on a Bayesian network; the Bayesian network is represented as B = <V B E B ,θ>, where <V B E B > represents a directed acyclic graph, V B E represents the set of nodes. B Let θ represent the set of edges, and let θ represent the Bayesian network parameters consisting of the conditional probability tables for each node. An edge connecting two nodes represents the probabilistic dependency between them. First, the input image features are converted into probability scores of attributes and fed into the Bayesian network as evidence for forward inference. The input evidence for each node is combined with the conditional probability tables to obtain the marginal posterior probability of each node. The marginal posterior probability P of node v0 is given by θ. B (v0) is:

[0036] P B (v0)=∫...∫ V P(v0,v1,...,v n )dv1...dv n (1)

[0037]

[0038] Where n is the number of nodes, Parents(v i ) represents node v iThe parent node, when v i When there is no parent node, Parents(v i () is empty.

[0039] Furthermore, the graph convolutional network is represented as an undirected graph G = (V G E G ), where V G E G Let L represent the set of nodes and L represent the set of edges, respectively. Each node and its contained feature vector correspond to a certain attribute of the lesion or disease category. If two nodes are connected, it means that the two nodes are related. For the edge between two nodes, a learnable weight is used to represent the degree of correlation between the two nodes. The graph convolutional network contains a total of L layers, and the number of nodes in each layer is the same. The feature transformation from the l-th layer to the (l+1)-th layer is implemented using residual graph convolution operation.

[0040]

[0041] Among them, F(G l W l ) represents the graph convolution operator of the l-th layer. These represent the feature vector matrices of all nodes in the l-th and (l+1)-th layers, respectively.

[0042] Furthermore, the method employs a residual fusion algorithm, using a Bayesian network as the main branch and a graph convolutional network as the residual branch, with the prediction results superimposed on the output of the Bayesian network. During the fusion process, the prediction results for disease categories and lesion attributes are fused separately, as shown in equations (4) and (5), respectively:

[0043]

[0044]

[0045] in, The probability of the merged disease category. The probability of the attributes of the fused lesions. and represent the marginal posterior probabilities of the Bayesian network output corresponding to the disease category and lesion attribute, respectively. Let w represent the probabilities of disease category and lesion attribute output by the graph convolutional network, respectively. B and w' B These represent two learnable coefficients, W0 and W'0 represent the weight matrices of the two fully connected layers, σ represents the softmax function, and Concat() represents the feature stacking operator.

[0046] Furthermore, the method also includes a model optimization step, which optimizes model parameters through model training. The model training method includes:

[0047] Step S1: Keeping the structure and parameters of the Bayesian network unchanged, update the parameters of the convolutional neural network and graph convolutional network for feature extraction using the backpropagation algorithm;

[0048] Step S2: Update the structure of the Bayesian network using dynamic programming, and then update the conditional probability table using maximum likelihood estimation.

[0049] Steps S1 and S2 are executed repeatedly. When the number of repetitions exceeds the preset maximum number of Bayesian network updates, only step S1 is executed.

[0050] Compared with the prior art, the present invention has the following beneficial effects.

[0051] This invention achieves automatic disease diagnosis based on input medical images by setting up a feature extraction module, a first diagnostic module for disease diagnosis reasoning based on Bayesian networks, a second diagnostic module for disease diagnosis and attribute category reasoning based on graph convolutional networks, and a diagnostic result fusion module for fusing the outputs of the first and second diagnostic modules. By using Bayesian networks and graph convolutional networks for disease diagnosis reasoning, this invention enables the intelligent disease diagnosis model to think like a professional doctor, inferring diseases from the imaging representation of lesions, achieving verifiable intelligent disease diagnosis, integrating the professional knowledge of doctors, and thus improving the accuracy of automatic diagnosis. Attached Figure Description

[0052] Figure 1 This is a block diagram illustrating the components of a disease diagnosis device based on medical imaging, according to an embodiment of the present invention.

[0053] Figure 2 This is a flowchart of a disease diagnosis method based on medical imaging, according to an embodiment of the present invention. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of this invention clearer and more understandable, the invention will be further described below in conjunction with the accompanying drawings and specific embodiments. Obviously, the described embodiments are merely some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0055] Figure 1 This is a block diagram of a disease diagnosis device based on medical imaging according to an embodiment of the present invention. The device includes:

[0056] The feature extraction module 11 is used to extract features from the input medical image using a convolutional neural network, and then adapt and convert the extracted features before inputting them into the first diagnostic module and the second diagnostic module respectively.

[0057] The first diagnostic module 12 is used for disease diagnosis reasoning based on Bayesian networks;

[0058] The second diagnostic module 13 is used for disease diagnosis and attribute category reasoning based on graph convolutional networks;

[0059] The diagnostic result fusion module 14 is used to fuse the outputs of the first diagnostic module 12 and the second diagnostic module 13 to obtain the final diagnostic result.

[0060] This embodiment provides a disease diagnosis device based on medical imaging. The device mainly consists of a feature extraction module 11, a first diagnosis module 12, a second diagnosis module 13, and a diagnosis result fusion module 14. Figure 1 As shown, the output of feature extraction module 11 is connected to the inputs of the first diagnosis module 12 and the second diagnosis module 13, respectively, and the outputs of the first diagnosis module 12 and the second diagnosis module 13 are connected to the inputs of the diagnosis result fusion module 14, respectively. Each module will be described in detail below.

[0061] The feature extraction module 11 is mainly used to extract features from the input medical images. The feature extraction module 11 generally uses a convolutional neural network as its backbone. Through a series of convolution operations on the input medical images, it transforms the original medical images from the data space to the feature space. These extracted features are input to the first diagnostic module 12 and the second diagnostic module 13, serving as the basis for subsequent inference. Many convolutional neural networks can be used for feature extraction; this embodiment does not limit the specific convolutional neural network structure. Later embodiments will provide an implementation method for the feature extraction module 11. Of course, to improve the efficiency of feature extraction, some necessary preprocessing is required before inputting the original medical images into the feature extraction module 11, such as denoising, cropping, and normalization. After feature extraction, the extracted features are adapted and transformed, including necessary feature probability score transformation when facing Bayesian networks, and transformation of features into individual attribute features through fully connected layers when facing graph convolutional networks.

[0062] The first diagnostic module 12 is mainly used for disease diagnosis reasoning based on Bayesian networks. The first diagnostic module 12 is implemented by a Bayesian network model. Firstly, the first diagnostic module 12 converts the medical image features output by the feature extraction module 11 into probability scores of attributes, and feeds these scores as evidence into the Bayesian network. Through forward inference of the Bayesian network, the posterior probability of the disease category is finally obtained. Bayesian networks, also known as confidence networks, are an extension of the Bayesian method and are one of the most effective theoretical models in the field of uncertain knowledge representation and reasoning. Since its introduction by Pearl in 1988, it has become a research hotspot in recent years.

[0063] The second diagnostic module 13 is primarily used for disease diagnosis and attribute category reasoning based on graph convolutional networks. Module 13 is implemented using a graph convolutional network model, which models the relationships between lesion attributes based on medical image features. In the graph convolutional network modeling process, a node in the network represents an attribute of the lesion, and the edges between nodes represent the relationships between attributes. The graph convolutional network enhances the features corresponding to each node by mining the relationships between attributes, thereby aiding in the classification of lesion attributes and obtaining more accurate disease diagnostic results.

[0064] The diagnostic result fusion module 14 is mainly used to fuse the diagnostic results of the first diagnostic module 12 and the second diagnostic module 13 to obtain the final diagnostic result. The Bayesian network of the first diagnostic module 12 and the graph convolutional network of the second diagnostic module 13 are two parallel diagnostic inference branches. To better utilize the functions of these two branches, this embodiment fuses the outputs of the Bayesian network and the graph convolutional network. Since the two inference branches use different network models, their strengths can be combined to compensate for each other's weaknesses. By fusing the inference results of the two different network models, the accuracy of the diagnostic result can be significantly improved. A more specific fusion method will be given in later embodiments.

[0065] This embodiment models causal and non-causal relationships in medical image diagnosis using Bayesian networks and graph convolutional networks, respectively. This quantitatively integrates medical expertise into the intelligent disease diagnosis model, improving its verifiability and generalization. Each node in both the Bayesian and graph convolutional networks represents a high-level concept (such as lesion attributes). The model explores the relationships between diseases and these high-level concepts to achieve intelligent reasoning and diagnosis, thus simulating and recreating the process of doctors diagnosing diseases based on images, thereby enhancing the model's verifiability. Specifically, the Bayesian network establishes a causal relationship between the imaging attributes of lesions and lesion categories, enabling direct reasoning from the imaging attributes of lesions to arrive at a diagnostic result. The graph convolutional network, on the other hand, places the imaging attributes of lesions into a graph network and models the relationships between attribute features through graph convolution operations to enhance the representativeness of features, thereby completing the reasoning process from features to disease categories.

[0066] As an optional embodiment, the input medical image includes CT images, MRI images, or X-ray images.

[0067] This embodiment illustrates several image formats for inputting medical images. Medical images refer to images of internal tissues of the human body or a part of the human body obtained non-invasively for medical treatment or research. Depending on the imaging mechanism and imaging equipment, the medical images in this embodiment can be CT images, MRI images, or X-ray images.

[0068] As an optional embodiment, the feature extraction module 11 uses a ResNet or EfficientNet convolutional neural network as the backbone for feature extraction; it employs a feature pyramid network to extend ResNet or EfficientNet to multiple scales, and uses a global average pooling operator to perform spatial scale reduction on the feature vectors of each layer of the feature pyramid to obtain feature vectors with a width and height of 1. It is set that the feature output dimension is the same for all scales. After the features at different scales pass through the global average pooling operator, a fully connected layer is used to transform the features. At the same time, an addition operation is used to fuse the features at all scales, and another fully connected layer is used to continue to transform the fused features. Finally, the extracted image features are output.

[0069] This embodiment presents a technical solution for the feature extraction module 11. In this embodiment, the feature extraction module 11 uses a ResNet or EfficientNet convolutional neural network as the backbone for feature extraction. ResNet proposes the idea of ​​residual learning. Traditional convolutional neural networks or fully connected networks suffer from information loss and degradation during information transmission, and also lead to gradient vanishing or gradient exploding, making deep networks untrainable. ResNet solves this problem to some extent by directly routing input information to the output, protecting the integrity of the information. The entire network only needs to learn the difference between the input and output, simplifying the learning objective and difficulty. EfficientNet was proposed in a Google 2019 paper, "EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks ICML 2019". This paper mainly describes how to use composite coefficients to uniformly scale all dimensions of the model to achieve the highest accuracy and efficiency. Composite coefficients include w, d, and r, where w represents the kernel size, which determines the receptive field size; d represents the depth of the neural network; and r represents the resolution. To achieve multi-scale aggregation of medical image features, this embodiment employs a feature pyramid network to extend ResNet or EfficientNet to multiple scales. Simultaneously, to reduce network parameters, Global Average Pooling (GAP) is used to reduce the spatial scale of the feature vectors at each layer of the feature pyramid, resulting in feature vectors with a width and height of 1. Furthermore, to minimize information loss at different scales of the feature pyramid, the feature output dimension is set to be the same across all scales. After the features at different scales pass through GAP, a fully connected layer (FC) is used for feature transformation. Addition is then used to fuse features from all scales, and another fully connected layer is used to further transform the fused features, resulting in the final output of the feature extraction backbone network.

[0070] As an optional embodiment, the first diagnostic module 12 establishes a probabilistic dependency model between disease categories and lesion attributes based on a Bayesian network; the Bayesian network is represented as B = <V B E B ,θ>, where <V B E B . represents a directed acyclic graph, V B E represents the set of nodes. BLet θ represent the set of edges, and let θ represent the Bayesian network parameters consisting of the conditional probability tables for each node. An edge connecting two nodes represents the probabilistic dependency between them. The input image features are converted into probability scores of attributes and fed as evidence into the Bayesian network for forward inference. The input evidence for each node is combined with the conditional probability tables to obtain the marginal posterior probability of each node. The marginal posterior probability P of node v0 is given by θ. B (v0) is:

[0071] P B (v0)=∫...∫ V P(v0,v1,...,v n )dv1...dv n (1)

[0072]

[0073] Where n is the number of nodes, Parents(v i ) represents node v i The parent node, when v i When there is no parent node, Parents(v i () is empty.

[0074] This embodiment provides a technical solution for the first diagnostic module 12. The first diagnostic module 12 is implemented using a Bayesian network. A Bayesian network is a belief network, a network structure based on a directed acyclic graph to characterize the dependencies between attributes, and uses a conditional probability table (CPT) to describe the joint probability distribution. Specifically, a Bayesian network consists of two parts: structure and parameters. The structure uses a directed acyclic graph. <V B E B > indicates that, where V B E represents the set of nodes. B The set of edges represents the network; the parameters are the Bayesian network parameters, denoted by θ, which consist of the conditional probability tables for each node. If there is a connection (edge) between two nodes, it means that there is a probabilistic dependency between the two nodes.

[0075] The learning of a Bayesian network involves two steps: structure learning and parameter learning. The following embodiments will provide a technical solution for optimizing the structure and parameters of a Bayesian network through model training (learning). For the sake of generality, assume that there is only one disease, and that each node in the Bayesian network has a grade score for its corresponding lesion attribute. To connect the Bayesian network to the backbone of the feature extraction module 11, the output of the backbone needs to be dimensionally transformed. A fully connected layer is used to transform the output of the backbone to a dimension of (N+1)×C, making it easier to input into the Bayesian network. Here, N represents the number of lesion attributes, 1 represents the number of disease categories, and C represents the number of grades for the lesion attribute score. Furthermore, during forward inference of the Bayesian network, the input evidence and conditional probability table of each node are combined to obtain the marginal posterior probability of each node. The marginal posterior probability calculation formula for node v0 is shown in equations (1) and (2). Equation (2) is derived based on the local Markov property of the Bayesian network, while equation (1) is implemented using a belief propagation algorithm.

[0076] As an optional embodiment, the second diagnostic module 13 establishes a relationship model between lesion attributes and between lesion attributes and disease categories based on a graph convolutional network; the graph convolutional network is based on an undirected graph G = (V G E G ) is constructed, where V G E G Let L represent the set of nodes and L represent the set of edges, respectively. Each node and its contained feature vector correspond to a certain attribute of the lesion or disease category. If two nodes are connected, it means that the two nodes are related. For the edge between two nodes, a learnable weight is used to represent the degree of correlation between the two nodes. The graph convolutional network contains a total of L layers, and the number of nodes in each layer is the same. The feature transformation from the l-th layer to the (l+1)-th layer is implemented using residual graph convolution operation.

[0077]

[0078] Among them, F(G l W l ) represents the graph convolution operator of the l-th layer. These represent the feature vector matrices of all nodes in the l-th and (l+1)-th layers, respectively.

[0079] This embodiment provides a technical solution for the second diagnostic module 13. The second diagnostic module 13 is implemented by a graph convolutional network. As the name suggests, a graph convolutional network performs convolution operations on a graph. The convolution operation on the graph utilizes the Fourier transform on the graph and then applies the convolution theorem, thus representing the convolution operation as the product of two Fourier transforms. Specifically, the graph convolutional network is represented as an undirected graph G = (V... G EG ), where V G E G These represent the set of nodes and the set of edges, respectively. In this embodiment, each node and its contained feature vector correspond to a certain attribute (or disease category) of the lesion. If two nodes are connected, it indicates that there is a certain correlation between their corresponding lesion attributes (or disease categories). For the edge between these two nodes, a learnable weight is set to describe the connection strength between them. Specifically, the graph convolutional network in this embodiment contains a total of L layers, where the number of nodes in each layer is the same, but the weights corresponding to the edges in each layer may be different. In the construction of the graph convolution, residual graph convolution is used to realize the feature transformation from the current (layer l) to the next layer (layer l+1), and the transformation formula is shown in equation (3). At the same time, in order to combine the above graph convolutional network with the backbone network of the feature extraction module 11, it is necessary to transform the dimension of the output feature vector of the backbone network. By adding a fully connected layer between the backbone network and the graph convolutional network, the dimension of the output feature of the backbone network is transformed to the dimension of (N+1)×D, which is convenient for subsequent input into the graph convolutional network. Where N represents the number of lesion attributes, 1 represents the number of diseases, and D represents the feature vector dimension of a node in the graph convolutional network. Through the above connections, end-to-end training of the graph convolutional network and the backbone network can be achieved.

[0080] As an optional embodiment, the diagnostic result fusion module 14 adopts a residual fusion algorithm, using the Bayesian network of the first diagnostic module 12 as the main branch and the graph convolutional network of the second diagnostic module 13 as the residual branch, and superimposing the prediction results onto the output of the Bayesian network; during the fusion process, the prediction results of disease categories and the prediction results of lesion attributes are fused respectively, as shown in equations (4) and (5):

[0081]

[0082]

[0083] in, The probability of the merged disease category. The probability of the attributes of the fused lesions. and represent the marginal posterior probabilities of the Bayesian network output corresponding to the disease category and lesion attribute, respectively. Let w represent the probabilities of disease category and lesion attribute output by the graph convolutional network, respectively. B and w' B These represent two learnable coefficients, W0 and W'0 represent the weight matrices of the two fully connected layers, σ represents the softmax function, and Concat() represents the feature stacking operator.

[0084] This embodiment presents a technical solution for the diagnostic result fusion module 14. Having constructed a Bayesian network and a graph convolutional network respectively, the diagnostic result fusion module 14 needs to fuse their outputs to further improve the accuracy of intelligent disease diagnosis. Specifically, the diagnostic result fusion module 14 employs a residual fusion algorithm, using the Bayesian network as the main branch and the graph convolutional network as the residual branch, with the predicted results superimposed on the output of the Bayesian network. The Bayesian network is chosen as the main branch because it can still provide reliable prediction results even with limited training samples. In medical image analysis, insufficient training data is a frequent occurrence, and this residual fusion fully utilizes the Bayesian network's insensitivity to the number of training samples to improve the model's generalization ability. Simultaneously, using the graph convolutional network as the residual branch can improve the feature learning ability of lesion attributes. By modeling the potential relationships between lesion attributes, a higher level of knowledge representation is provided for disease diagnosis, thereby further improving the model's prediction accuracy even with sufficient training samples. During the fusion process, this embodiment fuses the prediction results of lesion attributes and the prediction results of disease categories separately. Among them, the fusion of lesion attributes adopts the residual fusion method shown in equation (4); the fusion of disease categories adopts the residual fusion method shown in equation (5).

[0085] As an optional embodiment, the apparatus further includes a model optimization module for optimizing model parameters through model training, wherein the model training method includes:

[0086] Step S1: Keeping the structure and parameters of the Bayesian network unchanged, update the parameters of the convolutional neural network of the feature extraction module 11 and the graph convolutional network of the second diagnostic module 13 using the backpropagation algorithm.

[0087] Step S2: The structure of the Bayesian network of the first diagnostic module 12 is updated using dynamic programming, and then the conditional probability table is updated using maximum likelihood estimation.

[0088] Steps S1 and S2 are executed repeatedly. When the number of repetitions exceeds the preset maximum number of Bayesian network updates, only step S1 is executed.

[0089] This embodiment presents a technical solution for model training. After model construction is completed, it needs to be trained to optimize and update model parameters. In this embodiment, the backbone network of the feature extraction module 11 and the graph convolutional network of the second diagnostic module 13 are both optimized using the stochastic gradient descent algorithm; the Bayesian network of the first diagnostic module 12 adopts a different training method. Since the forward inference process of the Bayesian network is differentiable, this differentiability only applies to the network input and parameters; the optimization process of the network structure is not differentiable. This means that the gradient can propagate backward from the output layer of the Bayesian network to its input layer. Since the parameter learning and structure learning of the Bayesian network are tightly bound together, forcibly decoupling parameter learning from structure learning and using stochastic gradient descent for optimization will not achieve good training results. To better train the model, this embodiment proposes an alternating optimization algorithm, that is, the training of the backbone network and the graph convolutional network is performed alternately with the training of the Bayesian network, and the two training methods are implemented by steps S1 and S2, respectively. Step S1 updates the parameters of the backbone network and the graph convolutional network using the backpropagation algorithm. At this stage, the gradient only propagates back through the Bayesian network and does not update its structure or parameters; that is, step S1 keeps the structure and parameters of the Bayesian network unchanged. Step S2 first updates the structure of the Bayesian network using dynamic programming, and then updates the conditional probability table using maximum likelihood estimation. Steps S1 and S2 are executed alternately to ensure the smooth progress of the model training process. Simultaneously, a maximum number of updates to the Bayesian network is set. When the number of repetitions is within this maximum, model training proceeds in the above alternating manner. Once the preset maximum number of repetitions is exceeded, the structure and parameters of the Bayesian network are fixed, and only the parameters of the backbone network and the graph convolutional network are updated; that is, only step S1 is executed.

[0090] This embodiment utilizes the belief propagation algorithm for forward inference computation of the Bayesian network. However, during the training of the graph convolutional network, it's necessary to reverse the gradient through the Bayesian network, i.e., obtain the gradient from the belief propagation algorithm. Specifically, the gradient to be calculated is the marginal posterior distribution of all nodes relative to all input evidence of the Bayesian network. Since the input and output of the Bayesian network are multi-dimensional, the gradient of the belief propagation algorithm is a Jacobian matrix. For lack of generality, it is assumed that the Bayesian network is a multi-tree structure. In this structure, it is assumed that there exists N... B Nodes and N E Each edge, or conditional distribution, has M discrete levels. Because in the structure of a multitree, N... B There are N nodes. B There are -1 edges, and each edge corresponds to two opposite message propagation directions. Therefore, the total number of variables in belief propagation is M(N). B +2NE )=M(3N B -2). For node X, its marginal distribution and propagated information at time t depend on the information it obtains from its parent and child nodes at time t-1. Therefore, an M(N) can be defined. B +2N E )×M(N B +2N E A Jacobian matrix of size J. t Let J represent the gradient of all variables at time t with respect to all variables at time t-1. Assuming the belief propagation algorithm requires T time steps to converge, then the Jacobian matrix J of the final variables relative to the initial variables is... all It is the product of the Jacobian matrices at all T time points, i.e. Therefore, the gradient of the entire belief propagation algorithm, i.e., the partial derivative of the marginal posterior distribution of all nodes with respect to all input evidence, constitutes J. all A submatrix, which can be derived from J all Extracted from.

[0091] Figure 2 This is a flowchart of a disease diagnosis method based on medical imaging according to an embodiment of the present invention, including the following steps:

[0092] Step 101: Use a convolutional neural network to extract features from the input medical image, and then adapt and transform the extracted features before inputting them into a Bayesian network and a graph convolutional network, respectively.

[0093] Step 102: Perform disease diagnosis reasoning based on Bayesian networks;

[0094] Step 103: Perform disease diagnosis and attribute category reasoning based on graph convolutional networks;

[0095] Step 104: Fuse the outputs of the Bayesian network and the graph convolutional network to obtain the final diagnostic result.

[0096] The method in this embodiment is similar to... Figure 1 The implementation principle and technical effects of the device embodiments shown are similar to those of the embodiments described above, and will not be repeated here. The same applies to the subsequent embodiments, which will not be described in detail.

[0097] As an optional embodiment, the input medical image includes CT images, MRI images, or X-ray images.

[0098] As an optional embodiment, the feature extraction module 11 uses a ResNet or EfficientNet convolutional neural network as the backbone for feature extraction; it employs a feature pyramid network to extend ResNet or EfficientNet to multiple scales, and uses a global average pooling operator to perform spatial scale reduction on the feature vectors of each layer of the feature pyramid to obtain feature vectors with a width and height of 1. It is set that the feature output dimension is the same for all scales. After the features at different scales pass through the global average pooling operator, a fully connected layer is used to transform the features. At the same time, an addition operation is used to fuse the features at all scales, and another fully connected layer is used to continue to transform the fused features. Finally, the extracted image features are output.

[0099] As an optional embodiment, the first diagnostic module 12 establishes a probabilistic dependency model between disease categories and lesion attributes based on a Bayesian network; the Bayesian network is represented as B = <V B E B ,θ>, where <V B E B > represents a directed acyclic graph, V B E represents the set of nodes. B Let θ represent the set of edges, and let θ represent the Bayesian network parameters consisting of the conditional probability tables for each node. An edge connecting two nodes represents the probabilistic dependency between them. First, the input image features are converted into probability scores of attributes and fed into the Bayesian network as evidence for forward inference. The input evidence for each node is combined with the conditional probability tables to obtain the marginal posterior probability of each node. The marginal posterior probability P of node v0 is given by θ. B (v0) is:

[0100] P B (v0)=∫...∫ V P(v0,v1,...,v n )dv1...dv n (1)

[0101]

[0102] Where n is the number of nodes, Parents(v i ) represents node v i The parent node, when v i When there is no parent node, Parents(v i () is empty.

[0103] As an optional embodiment, the graph convolutional network is represented as an undirected graph G = (V G E G ), where VG E G Let L represent the set of nodes and L represent the set of edges, respectively. Each node and its contained feature vector correspond to a certain attribute of the lesion or disease category. If two nodes are connected, it means that the two nodes are related. For the edge between two nodes, a learnable weight is used to represent the degree of correlation between the two nodes. The graph convolutional network contains a total of L layers, and the number of nodes in each layer is the same. The feature transformation from the l-th layer to the (l+1)-th layer is implemented using residual graph convolution operation.

[0104]

[0105] Among them, F(G l W l ) represents the graph convolution operator of the l-th layer. These represent the feature vector matrices of all nodes in the l-th and (l+1)-th layers, respectively.

[0106] As an optional embodiment, the method employs a residual fusion algorithm, using a Bayesian network as the main branch and a graph convolutional network as the residual branch, with the prediction results superimposed on the output of the Bayesian network; during the fusion process, the prediction results for disease categories and lesion attributes are fused separately, as shown in equations (4) and (5):

[0107]

[0108]

[0109] in, The probability of the merged disease category. The probability of the attributes of the fused lesions. and represent the marginal posterior probabilities of the Bayesian network output corresponding to the disease category and lesion attribute, respectively. Let w represent the probabilities of disease category and lesion attribute output by the graph convolutional network, respectively. B and w' B These represent two learnable coefficients, W0 and W'0 represent the weight matrices of the two fully connected layers, σ represents the softmax function, and Concat() represents the feature stacking operator.

[0110] As an optional embodiment, the method further includes a model optimization step, which optimizes model parameters through model training. The model training method includes:

[0111] Step S1: Keeping the structure and parameters of the Bayesian network unchanged, update the parameters of the convolutional neural network and graph convolutional network for feature extraction using the backpropagation algorithm;

[0112] Step S2: Update the structure of the Bayesian network using dynamic programming, and then update the conditional probability table using maximum likelihood estimation.

[0113] Steps S1 and S2 are executed repeatedly. When the number of repetitions exceeds the preset maximum number of Bayesian network updates, only step S1 is executed.

[0114] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A disease diagnostic device based on medical imaging, characterized in that, The device includes: The feature extraction module is used to extract features from the input medical images using a convolutional neural network, and then adapts and transforms the extracted features before inputting them into the first diagnostic module and the second diagnostic module respectively. The first diagnostic module is used for disease diagnosis reasoning based on Bayesian networks; The second diagnostic module is used for disease diagnosis and attribute category reasoning based on graph convolutional networks; The diagnostic result fusion module is used to fuse the outputs of the first diagnostic module and the second diagnostic module to obtain the final diagnostic result. The diagnostic result fusion module employs a residual fusion algorithm, using the Bayesian network of the first diagnostic module as the main branch and the graph convolutional network of the second diagnostic module as the residual branch, with the prediction results superimposed on the output of the Bayesian network. During the fusion process, the prediction results for disease categories and lesion attributes are fused separately.

2. The disease diagnostic device based on medical imaging according to claim 1, characterized in that, The input medical images include CT images, MRI images, or X-ray images.

3. The disease diagnostic device based on medical imaging according to claim 1, characterized in that, The feature extraction module uses ResNet or EfficientNet convolutional neural networks as the backbone for feature extraction. It employs a feature pyramid network to extend ResNet or EfficientNet to multiple scales, and uses a global average pooling operator to reduce the spatial scale of the feature vectors in each layer of the feature pyramid, resulting in feature vectors with a width and height of 1. It is set that the feature output dimension is the same for all scales. After the features at different scales pass through the global average pooling operator, a fully connected layer is used to transform the features. At the same time, an addition operation is used to fuse the features at all scales, and another fully connected layer is used to further transform the fused features. Finally, the extracted image features are output.

4. The disease diagnostic device based on medical imaging according to claim 1, characterized in that, The first diagnostic module establishes a probabilistic dependency model between disease categories and lesion attributes based on a Bayesian network; the Bayesian network is represented as B = <V B E B ,θ>, where <V B E B > represents a directed acyclic graph, V B E represents the set of nodes. B Let θ represent the set of edges, and let θ represent the Bayesian network parameters consisting of the conditional probability tables for each node. An edge connecting two nodes represents the probabilistic dependency between them. First, the input image features are converted into probability scores of attributes and fed into the Bayesian network as evidence for forward inference. The input evidence for each node is combined with the conditional probability tables to obtain the marginal posterior probability of each node. The marginal posterior probability P of node v0 is given by θ. B (v0) is: P B (v0)=∫...∫ V P(v0,v1,...,v n )dv1...dv n (1) Where n is the number of nodes, Parents(v i ) represents node v i The parent node, when v i When there is no parent node, Parents(v i () is empty.

5. The disease diagnostic device based on medical imaging according to claim 3, characterized in that, The second diagnostic module establishes a model of the relationships between lesion attributes and between lesion attributes and disease categories based on a graph convolutional network; the graph convolutional network is based on an undirected graph G = (V G E G ) is constructed, where V G E G Let L represent the set of nodes and L represent the set of edges, respectively. Each node and its contained feature vector correspond to a certain attribute of the lesion or disease category. If two nodes are connected, it means that the two nodes are related. For the edge between two nodes, a learnable weight is used to represent the degree of correlation between the two nodes. The graph convolutional network contains a total of L layers, and the number of nodes in each layer is the same. The feature transformation from the l-th layer to the (l+1)-th layer is implemented using residual graph convolution operation. Among them, F(G l W l ) represents the graph convolution operator of the l-th layer. These represent the feature vector matrices of all nodes in the l-th and (l+1)-th layers, respectively.

6. The disease diagnostic device based on medical imaging according to claim 5, characterized in that, The prediction results for disease categories and lesion attributes are fused together, as shown in equations (4) and (5), respectively: in, The probability of the merged disease category. The probability of the attributes of the fused lesions. and represent the marginal posterior probabilities of the Bayesian network output corresponding to the disease category and lesion attribute, respectively. Let w represent the probabilities of disease category and lesion attribute output by the graph convolutional network, respectively. B and w' B Let W0 and W0' represent two learnable coefficients, respectively. Let W0 and W0' represent the weight matrices of the two fully connected layers, respectively. Let σ represent the softmax function, and let Concat() represent the feature stacking operator.

7. The disease diagnostic device based on medical imaging according to claim 1, characterized in that, The device further includes a model optimization module for optimizing model parameters through model training, wherein the model training method includes: Step S1: Keeping the structure and parameters of the Bayesian network unchanged, update the parameters of the convolutional neural network of the feature extraction module and the graph convolutional network of the second diagnostic module using the backpropagation algorithm. Step S2: The structure of the Bayesian network of the first diagnostic module is updated using dynamic programming, and then the conditional probability table is updated using maximum likelihood estimation. Steps S1 and S2 are executed repeatedly. When the number of repetitions exceeds the preset maximum number of Bayesian network updates, only step S1 is executed.

8. A disease diagnosis method based on medical imaging, characterized in that, The method includes the following steps: Convolutional neural networks are used to extract features from input medical images, and the extracted features are adapted and transformed before being input into Bayesian networks and graph convolutional networks, respectively. Disease diagnosis inference based on Bayesian networks; Disease diagnosis and attribute category reasoning based on graph convolutional networks; The outputs of the Bayesian network and the graph convolutional network are fused to obtain the final diagnostic result; The method employs a residual fusion algorithm, using a Bayesian network as the main branch and a graph convolutional network as the residual branch, with the prediction results superimposed on the output of the Bayesian network. During the fusion process, the prediction results for disease categories and lesion attributes are fused separately.

9. The disease diagnosis method based on medical imaging according to claim 8, characterized in that, The prediction results for disease categories and lesion attributes are fused together, as shown in equations (4) and (5), respectively: in, The probability of the merged disease category. The probability of the attributes of the fused lesions. and represent the marginal posterior probabilities of the Bayesian network output corresponding to the disease category and lesion attribute, respectively. Let w represent the probabilities of disease category and lesion attribute output by the graph convolutional network, respectively. B and w' B Let W0 and W0' represent two learnable coefficients, respectively. Let W0 and W0' represent the weight matrices of the two fully connected layers, respectively. Let σ represent the softmax function, and let Concat() represent the feature stacking operator.

10. The disease diagnosis method based on medical imaging according to claim 8, characterized in that, The method further includes a model optimization step, which optimizes model parameters through model training. The model training method includes: Step S1: Keeping the structure and parameters of the Bayesian network unchanged, update the parameters of the convolutional neural network and graph convolutional network for feature extraction using the backpropagation algorithm; Step S2: Update the structure of the Bayesian network using dynamic programming, and then update the conditional probability table using maximum likelihood estimation. Steps S1 and S2 are executed repeatedly. When the number of repetitions exceeds the preset maximum number of Bayesian network updates, only step S1 is executed.

Images